Re: [ceph-users] Weird behaviour of cephfs with samba
Gregory Farnum greg@... writes:
You can mount subtrees with the -r option to ceph-fuse.
Yay! That did the trick to properly mount via fuse. And I can confirm that
the directory list results are now stable both locally and via samba.
Once you've started it up you should find a file like
client.admin.[0-9]*.asok in (I think?) /var/run/ceph. You can run
ceph --admin-daemon /var/run/ceph/{client_asok} status and provide
the output to see if it's doing anything useful. Or set debug client
= 20 in the config and then upload the client log file either
publicly or with ceph-post-file and I'll take a quick look to see
what's going on.
root@gru:/mnt# ceph --admin-daemon /var/run/ceph/ceph-client.admin.asok status
{
metadata: {
ceph_sha1: 5fb85614ca8f354284c713a2f9c610860720bbf3,
ceph_version: ceph version 0.94.2
(5fb85614ca8f354284c713a2f9c610860720bbf3),
entity_id: admin,
hostname: gru,
mount_point: \/mnt\/test
},
dentry_count: 0,
dentry_pinned_count: 0,
inode_count: 1,
mds_epoch: 1075,
osd_epoch: 2391,
osd_epoch_barrier: 0
}
It seems like if cephfs was only ever mounted with a non-root basepath, the
cephfs root does not contain a directory entry for that base path. At least
that's what I guess from dentry_count: 0.
Mmm, that looks like a Samba config issue which unfortunately I don't
know much about. Perhaps you need to install these modules
individually? It looks like our nightly tests are just getting the
Ceph VFS installed by default. :/
Ubuntu's samba package isn't configured with --with-libcephfs and therefore
simply doesn't ship ceph.so. I am currently trying to recompile it with that
flag. If that doesn't work, that's no biggie, because the re-export of the
ceph-fuse mounted directory behaves flawlessly now.
Thanks a ton for your help!
Joerg Henne
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
On Mon, Jul 27, 2015 at 6:25 PM, Jörg Henne henn...@gmail.com wrote:
Gregory Farnum greg@... writes:
Yeah, I think there were some directory listing bugs in that version
that Samba is probably running into. They're fixed in a newer kernel
release (I'm not sure which one exactly, sorry).
Ok, thanks, good to know!
and then detaches itself but the mountpoint stays empty no matter what.
/var/log/ceph/ceph-client.admin.log isn't enlighting as well. I've never
used a FUSE before, though, so I might be overlooking something.
Uh, that's odd. What do you mean it's empty no matter what? Is the
ceph-fuse process actually still running?
Yes, e.g.
8525 pts/0Sl 0:00 ceph-fuse -m 10.208.66.1:6789 /mnt/regtest2
But
root@gru:/mnt# ls /mnt/regtest2 | wc -l
0
With the kernel module I mount just a subpath of the cephfs space like in
/etc/fstab:
my_monhost:/regression-test /mnt/regtest ...
which ceph-fuse doesn't seem to support, but then I would expect
regression-test to simply be a sub-directory of /mnt/regtest2.
You can mount subtrees with the -r option to ceph-fuse.
Once you've started it up you should find a file like
client.admin.[0-9]*.asok in (I think?) /var/run/ceph. You can run
ceph --admin-daemon /var/run/ceph/{client_asok} status and provide
the output to see if it's doing anything useful. Or set debug client
= 20 in the config and then upload the client log file either
publicly or with ceph-post-file and I'll take a quick look to see
what's going on.
-Greg
(You should also be able to talk to Ceph directly via the Samba
daemon; the bindings are in upstream Samba although you probably need
to install one of the Ceph packages to make it work. That's the way we
test in our nightlies.)
Indeed, it seems like something is missing:
[2015/07/27 19:21:40.080572, 0] ../lib/util/modules.c:48(load_module)
Error loading module '/usr/lib/x86_64-linux-gnu/samba/vfs/ceph.so':
/usr/lib/x86_64-linux-gnu/samba/vfs/ceph.so: cannot open shared object file:
No such file or directory
Mmm, that looks like a Samba config issue which unfortunately I don't
know much about. Perhaps you need to install these modules
individually? It looks like our nightly tests are just getting the
Ceph VFS installed by default. :/
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
I use cephfs over samba vfs and have some issues. 1) If I use 1 stacked vfs (ceph scannedonly) - I have problems with file order, but solved by dirsort vfs (vfs objects = scannedonly dirsort ceph). Single ceph vfs looks good too (and I use it single for fast internal shares), but you can try add dirsort (vfs objects = dirsort ceph). 2) I use 2 my patches: https://github.com/mahatma-kaganovich/raw/tree/master/app-portage/ppatch/files/extensions/net-fs/samba/compile - to support max disk size and to secure chown. About chown: I unsure about strict follow standard system behaviour, but it works for me, without - user can chown() even to root. I put first (max disk size) patch into samba bugzilla times ago, second patch - no, I unsure about it correctness, but sure about security hole. Jörg Henne пишет: Hi all, the faq at http://ceph.com/docs/cuttlefish/faq/ mentions the possibility to run export a mounted cephfs via samba. This combination exhibits a very weird behaviour, though. We have a directory on cephfs with many small xml snippets. If I repeadtedly ls the directory on Unix, I get the same answer each and every time: root@gru:/mnt/regtest/regressiontestdata2/assets# while true; do ls|wc -l; sleep 1; done 851 851 851 ... and so on If I do the same on the directory exported and mounted via SMB under Windows the result looks like that (output generated unter cygwin, but effect is present with Windows Explorer as well): $ while true; do ls|wc -l; sleep 1; done 380 380 380 380 380 1451 362 851 851 851 851 851 851 851 851 1451 362 851 851 851 ... The problem does not seem to be related to Samba. If I copy the files to an XFS volume and export that, things look fine. Thanks Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.unibel.by/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
PS I start to use this patches with samba 4.1. IMHO some of problems may (or must) be solved not inside vfs code, but outside - in samba kernel, but I still use both in samba 4.2.3 without verification. Dzianis Kahanovich пишет: I use cephfs over samba vfs and have some issues. 1) If I use 1 stacked vfs (ceph scannedonly) - I have problems with file order, but solved by dirsort vfs (vfs objects = scannedonly dirsort ceph). Single ceph vfs looks good too (and I use it single for fast internal shares), but you can try add dirsort (vfs objects = dirsort ceph). 2) I use 2 my patches: https://github.com/mahatma-kaganovich/raw/tree/master/app-portage/ppatch/files/extensions/net-fs/samba/compile - to support max disk size and to secure chown. About chown: I unsure about strict follow standard system behaviour, but it works for me, without - user can chown() even to root. I put first (max disk size) patch into samba bugzilla times ago, second patch - no, I unsure about it correctness, but sure about security hole. Jörg Henne пишет: Hi all, the faq at http://ceph.com/docs/cuttlefish/faq/ mentions the possibility to run export a mounted cephfs via samba. This combination exhibits a very weird behaviour, though. We have a directory on cephfs with many small xml snippets. If I repeadtedly ls the directory on Unix, I get the same answer each and every time: root@gru:/mnt/regtest/regressiontestdata2/assets# while true; do ls|wc -l; sleep 1; done 851 851 851 ... and so on If I do the same on the directory exported and mounted via SMB under Windows the result looks like that (output generated unter cygwin, but effect is present with Windows Explorer as well): $ while true; do ls|wc -l; sleep 1; done 380 380 380 380 380 1451 362 851 851 851 851 851 851 851 851 1451 362 851 851 851 ... The problem does not seem to be related to Samba. If I copy the files to an XFS volume and export that, things look fine. Thanks Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.unibel.by/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
I don't have any answers but I am also seeing some strange results exporting a Ceph file system using the Samba VFS interface on Ceph version 9.0.2. If I mount a Linux client with vers=1, I see the file system the same as I see it on a ceph file system mount. If I use vers=2.0 or vers=3.0 on the mount line, some directories disappear and the ownership changes. I am still early in my testing of this issue. Eric On Mon, Jul 27, 2015 at 8:16 AM, Jörg Henne henn...@gmail.com wrote: Hi all, the faq at http://ceph.com/docs/cuttlefish/faq/ mentions the possibility to run export a mounted cephfs via samba. This combination exhibits a very weird behaviour, though. We have a directory on cephfs with many small xml snippets. If I repeadtedly ls the directory on Unix, I get the same answer each and every time: root@gru:/mnt/regtest/regressiontestdata2/assets# while true; do ls|wc -l; sleep 1; done 851 851 851 ... and so on If I do the same on the directory exported and mounted via SMB under Windows the result looks like that (output generated unter cygwin, but effect is present with Windows Explorer as well): $ while true; do ls|wc -l; sleep 1; done 380 380 380 380 380 1451 362 851 851 851 851 851 851 851 851 1451 362 851 851 851 ... The problem does not seem to be related to Samba. If I copy the files to an XFS volume and export that, things look fine. Thanks Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
What's the full stack you're using to run this with? If you're using the kernel client, try updating it or switching to the userspace (ceph-fuse, or Samba built-in) client. If using userspace, please make sure you've got the latest one. -Greg On Mon, Jul 27, 2015 at 3:16 PM, Jörg Henne henn...@gmail.com wrote: Hi all, the faq at http://ceph.com/docs/cuttlefish/faq/ mentions the possibility to run export a mounted cephfs via samba. This combination exhibits a very weird behaviour, though. We have a directory on cephfs with many small xml snippets. If I repeadtedly ls the directory on Unix, I get the same answer each and every time: root@gru:/mnt/regtest/regressiontestdata2/assets# while true; do ls|wc -l; sleep 1; done 851 851 851 ... and so on If I do the same on the directory exported and mounted via SMB under Windows the result looks like that (output generated unter cygwin, but effect is present with Windows Explorer as well): $ while true; do ls|wc -l; sleep 1; done 380 380 380 380 380 1451 362 851 851 851 851 851 851 851 851 1451 362 851 851 851 ... The problem does not seem to be related to Samba. If I copy the files to an XFS volume and export that, things look fine. Thanks Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
Gregory Farnum greg@... writes: Yeah, I think there were some directory listing bugs in that version that Samba is probably running into. They're fixed in a newer kernel release (I'm not sure which one exactly, sorry). Ok, thanks, good to know! and then detaches itself but the mountpoint stays empty no matter what. /var/log/ceph/ceph-client.admin.log isn't enlighting as well. I've never used a FUSE before, though, so I might be overlooking something. Uh, that's odd. What do you mean it's empty no matter what? Is the ceph-fuse process actually still running? Yes, e.g. 8525 pts/0Sl 0:00 ceph-fuse -m 10.208.66.1:6789 /mnt/regtest2 But root@gru:/mnt# ls /mnt/regtest2 | wc -l 0 With the kernel module I mount just a subpath of the cephfs space like in /etc/fstab: my_monhost:/regression-test /mnt/regtest ... which ceph-fuse doesn't seem to support, but then I would expect regression-test to simply be a sub-directory of /mnt/regtest2. (You should also be able to talk to Ceph directly via the Samba daemon; the bindings are in upstream Samba although you probably need to install one of the Ceph packages to make it work. That's the way we test in our nightlies.) Indeed, it seems like something is missing: [2015/07/27 19:21:40.080572, 0] ../lib/util/modules.c:48(load_module) Error loading module '/usr/lib/x86_64-linux-gnu/samba/vfs/ceph.so': /usr/lib/x86_64-linux-gnu/samba/vfs/ceph.so: cannot open shared object file: No such file or directory Unfortunately, finding out what to install will have to wait till Thursday. Thanks for your help so far! Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
On Mon, Jul 27, 2015 at 5:46 PM, Jörg Henne henn...@gmail.com wrote: Gregory Farnum greg@... writes: What's the full stack you're using to run this with? If you're using the kernel client, try updating it or switching to the userspace (ceph-fuse, or Samba built-in) client. If using userspace, please make sure you've got the latest one. -Greg The system is: root@gru:~# uname -a; cat /etc/issue; ceph --version Linux gru 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Ubuntu 14.04.2 LTS \n \l ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e) The filesystem is mounted via the kernel module: root@gru:~# modinfo ceph filename: /lib/modules/3.13.0-57-generic/kernel/fs/ceph/ceph.ko license:GPL description:Ceph filesystem for Linux author: Patience Warnick patie...@newdream.net author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net alias: fs-ceph srcversion: 659145B67321D11C1A5C076 depends:fscache,libceph intree: Y vermagic: 3.13.0-57-generic SMP mod_unload modversions signer: Magrathea: Glacier signing key sig_key:A5:5B:8D:E6:8A:60:7A:DB:FA:6D:F9:60:00:50:69:50:CD:AE:A0:41 sig_hashalgo: sha512 Yeah, I think there were some directory listing bugs in that version that Samba is probably running into. They're fixed in a newer kernel release (I'm not sure which one exactly, sorry). I updated to hammer, but the problem persisted: root@gru:/etc/puppet# ceph --version; modinfo ceph ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) filename: /lib/modules/3.13.0-58-generic/kernel/fs/ceph/ceph.ko license:GPL description:Ceph filesystem for Linux author: Patience Warnick patie...@newdream.net author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net alias: fs-ceph srcversion: 659145B67321D11C1A5C076 depends:fscache,libceph intree: Y vermagic: 3.13.0-58-generic SMP mod_unload modversions signer: Magrathea: Glacier signing key sig_key:70:A9:7E:EA:B2:62:56:39:DF:8C:99:79:47:6F:D1:51:6D:CD:65:0B sig_hashalgo: sha512 Ok, so on to ceph-fuse. Unfortunately, I can't seem to be able to get this to work at all. The process starts Ok with root@gru:/mnt# ceph-fuse -m 10.208.66.1:6789 /mnt/regtest2 ceph-fuse[8525]: starting ceph client 2015-07-27 18:43:27.790224 7f72dc0497c0 -1 init, newargv = 0x44e3d30 newargc=11 ceph-fuse[8525]: starting fuse and then detaches itself but the mountpoint stays empty no matter what. /var/log/ceph/ceph-client.admin.log isn't enlighting as well. I've never used a FUSE before, though, so I might be overlooking something. Uh, that's odd. What do you mean it's empty no matter what? Is the ceph-fuse process actually still running? (You should also be able to talk to Ceph directly via the Samba daemon; the bindings are in upstream Samba although you probably need to install one of the Ceph packages to make it work. That's the way we test in our nightlies.) -Greg Thanks Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Weird behaviour of cephfs with samba
Gregory Farnum greg@... writes: What's the full stack you're using to run this with? If you're using the kernel client, try updating it or switching to the userspace (ceph-fuse, or Samba built-in) client. If using userspace, please make sure you've got the latest one. -Greg The system is: root@gru:~# uname -a; cat /etc/issue; ceph --version Linux gru 3.13.0-57-generic #95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Ubuntu 14.04.2 LTS \n \l ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e) The filesystem is mounted via the kernel module: root@gru:~# modinfo ceph filename: /lib/modules/3.13.0-57-generic/kernel/fs/ceph/ceph.ko license:GPL description:Ceph filesystem for Linux author: Patience Warnick patie...@newdream.net author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net alias: fs-ceph srcversion: 659145B67321D11C1A5C076 depends:fscache,libceph intree: Y vermagic: 3.13.0-57-generic SMP mod_unload modversions signer: Magrathea: Glacier signing key sig_key:A5:5B:8D:E6:8A:60:7A:DB:FA:6D:F9:60:00:50:69:50:CD:AE:A0:41 sig_hashalgo: sha512 I updated to hammer, but the problem persisted: root@gru:/etc/puppet# ceph --version; modinfo ceph ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3) filename: /lib/modules/3.13.0-58-generic/kernel/fs/ceph/ceph.ko license:GPL description:Ceph filesystem for Linux author: Patience Warnick patie...@newdream.net author: Yehuda Sadeh yeh...@hq.newdream.net author: Sage Weil s...@newdream.net alias: fs-ceph srcversion: 659145B67321D11C1A5C076 depends:fscache,libceph intree: Y vermagic: 3.13.0-58-generic SMP mod_unload modversions signer: Magrathea: Glacier signing key sig_key:70:A9:7E:EA:B2:62:56:39:DF:8C:99:79:47:6F:D1:51:6D:CD:65:0B sig_hashalgo: sha512 Ok, so on to ceph-fuse. Unfortunately, I can't seem to be able to get this to work at all. The process starts Ok with root@gru:/mnt# ceph-fuse -m 10.208.66.1:6789 /mnt/regtest2 ceph-fuse[8525]: starting ceph client 2015-07-27 18:43:27.790224 7f72dc0497c0 -1 init, newargv = 0x44e3d30 newargc=11 ceph-fuse[8525]: starting fuse and then detaches itself but the mountpoint stays empty no matter what. /var/log/ceph/ceph-client.admin.log isn't enlighting as well. I've never used a FUSE before, though, so I might be overlooking something. Thanks Joerg Henne ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
