[ceph-users] Sizing for bluestore db and wal
Hi folks, i upgraded our ceph cluster from jewel to luminous and want to migrate from filestore to bluestore. Currently we use one SSD as journal for thre 8TB Sata Drives with a journal partition size of 40GB. If my understanding of the bluestore documentation is correct, i can use a wal partition for the writeahead log (to decrease write latency, similar to filestore) and a db partition for metadata (decreasing write AND read latency/throughput). Now I have two questions: a) Do I really need an WAL partition if both wal and db are on the same SSD? b) If so, what would the ratio look like? 99% db, 1% wal? Best regards, Felix -- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Using Cephfs Snapshots in Luminous
Hi folks, is anybody using cephfs with snapshots on luminous? Cephfs snapshots are declared stable in mimic, but I'd like to know about the risks using them on luminous. Do I risk a complete cephfs failure or just some not working snapshots? It is one namespace, one fs, one data and one metadata pool. Also I'd like to know if there is a way to list all current snapshot (folders) in cephfs. Since a snaphot is created by the mkdir command, snapshot creation can be done by every user and I would like to monitor snapshotted folders and snaphshot size (if possible). Best regards Felix Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs set quota without mount
Our ceph cluster is mainly used for openstack but we also need to provide storage to linux workstations via nfs and smb for our windows clients. Even though our linux workstations could talk to cephs directly we don't want them to be in our ceph public network. Ceph public network is only connected to openstack and ceph nodes. In addition we are implementing two gateway servers to export cephfs via nfs/smb using ctdb as HA On 10/11/2018 08:42 PM, solarflow99 wrote: I am just interested to know more about your use case for NFS as opposed to just using cephfs directly, and what are you using for HA? On Thu, Oct 11, 2018 at 1:54 AM Felix Stolte <mailto:f.sto...@fz-juelich.de>> wrote: Hey folks, I use nfs-ganesha to export cephfs to nfs. nfs-ganesha can talk to cephfs via libcephfs so there is no need for mounting cephfs manually. I also like to use directory quotas from cephfs. Anyone knows a way to set quota on directories without the need to mount it first? I was thinking about an admin socket, but the mds does not seem to offer this functionality. Regards Felix Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ___ ceph-users mailing list ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Felix Stolte IT-Services Tel.: +49 2461 61-9243 Email: f.sto...@fz-juelich.de Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] cephfs set quota without mount
Hey folks, I use nfs-ganesha to export cephfs to nfs. nfs-ganesha can talk to cephfs via libcephfs so there is no need for mounting cephfs manually. I also like to use directory quotas from cephfs. Anyone knows a way to set quota on directories without the need to mount it first? I was thinking about an admin socket, but the mds does not seem to offer this functionality. Regards Felix Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] vfs_ceph ignoring quotas
That's bad news, but maybe there is a workarround. Samba offers the opportunity to define a custom df command. If I could extract the current utilization or size of a directory with a quota, i think i should be able to write a little df command. The quota is stored as an extended attribute, where can i get its utilization? Regards Felix On 10/09/2018 10:59 AM, John Spray wrote: On Tue, Oct 9, 2018 at 9:14 AM Felix Stolte wrote: Hi folks, i'm running a luminous cluster on Ubuntu 18.04 an want to share folders on cephfs with samba using the vfs_ceph. Sharing works fine, but the quotas I set on the directories is ignored and every share reports its size as the total size of the cephfs. Anyone got this working? Or is the vfs lacking quota support? Looks like the Samba VFS is always mounting CephFS at the root[1], including if you're exporting a subdir (nfs ganesha used to be this way too) -- CephFS can only do the "magic" quota-based statfs if it knows it's mounting a particular subdir. John 1. https://github.com/samba-team/samba/blob/master/source3/modules/vfs_ceph.c#L123 Regards Felix -- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Felix Stolte IT-Services Tel.: +49 2461 61-9243 Email: f.sto...@fz-juelich.de Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] vfs_ceph ignoring quotas
Hi folks, i'm running a luminous cluster on Ubuntu 18.04 an want to share folders on cephfs with samba using the vfs_ceph. Sharing works fine, but the quotas I set on the directories is ignored and every share reports its size as the total size of the cephfs. Anyone got this working? Or is the vfs lacking quota support? Regards Felix -- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to use vfs_ceph
Hi David, I am using Samba 4.6.7 (shipped with Ubuntu 17.10). I've got it working now by copying the ceph.client.admin.keyring to /etc/ceph (I'm very unhappy with that). Which Samba Version & Linux Distribution are using? Are you using quotas on subdirectories and are they applied when you export the subdirectory via samba? Regards Felix On 12/21/2017 06:19 PM, David Disseldorp wrote: Hi Felix, On Thu, 21 Dec 2017 14:16:23 +0100, Felix Stolte wrote: Hello folks, is anybody using the vfs_ceph module for exporting cephfs as samba shares? Yes, alongside Luminous. Which version of Samba are you using? Make sure it includes the fix for https://bugzilla.samba.org/show_bug.cgi?id=12911 . We are running ceph jewel with cephx enabled. Manpage of vfs_ceph only references the option ceph:config_file. How do I need to configure my share (or maybe ceph.conf)? log.smbd: '/' does not exist or permission denied when connecting to [vfs] Error was Transport endpoint is not connected I have a user ctdb with keyring file /etc/ceph/ceph.client.ctdb.keyring with permissions: caps: [mds] allow rw caps: [mon] allow r caps: [osd] allow rwx pool=cephfs_metadata,allow rwx pool=cephfs_data I can mount cephfs with cephf-fuse using the id ctdb and its keyfile. My share definition is: [vfs] comment = vfs path = / read only = No vfs objects = acl_xattr ceph ceph:user_id = ctdb ceph:config_file = /etc/ceph/ceph.conf Your configuration looks fine - can you confirm that the *mapped* Samba user is permitted access to the root of the CephFS filesystem? Have you tried using ceph-fuse as the mapped user? If you're still running into problems, feel free to raise a ticket at bugzilla.samba.org and assign it to me. Cheers, David -- Felix Stolte IT-Services Tel.: +49 2461 61-9243 Email: f.sto...@fz-juelich.de Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] How to use vfs_ceph
Hello folks, is anybody using the vfs_ceph module for exporting cephfs as samba shares? We are running ceph jewel with cephx enabled. Manpage of vfs_ceph only references the option ceph:config_file. How do I need to configure my share (or maybe ceph.conf)? log.smbd: '/' does not exist or permission denied when connecting to [vfs] Error was Transport endpoint is not connected I have a user ctdb with keyring file /etc/ceph/ceph.client.ctdb.keyring with permissions: caps: [mds] allow rw caps: [mon] allow r caps: [osd] allow rwx pool=cephfs_metadata,allow rwx pool=cephfs_data I can mount cephfs with cephf-fuse using the id ctdb and its keyfile. My share definition is: [vfs] comment = vfs path = / read only = No vfs objects = acl_xattr ceph ceph:user_id = ctdb ceph:config_file = /etc/ceph/ceph.conf Any advice is appreciated. Regards Felix -- Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir. Dr. Karl Eugen Huthmacher Geschäftsführung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
