Have you tried just running “sync;sync” on the originating node? Does that achieve the same thing or not? (I guess it could/should).
Jan > On 16 Jun 2015, at 13:37, negillen negillen <negil...@gmail.com> wrote: > > Thanks again, > > even 'du' performance is terrible on node B (testing on a directory taken > from Phoronix): > > # time du -hs /storage/test9/installed-tests/pts/pgbench-1.5.1/ > 73M /storage/test9/installed-tests/pts/pgbench-1.5.1/ > real 0m21.044s > user 0m0.010s > sys 0m0.067s > > > Reading the files from node B doesn't seem to help with subsequent accesses > in this case: > > # time tar c /storage/test9/installed-tests/pts/pgbench-1.5.1/>/dev/null > real 1m47.650s > user 0m0.041s > sys 0m0.212s > > # time tar c /storage/test9/installed-tests/pts/pgbench-1.5.1/>/dev/null > real 1m45.636s > user 0m0.042s > sys 0m0.214s > > # time ls -laR /storage/test9/installed-tests/pts/pgbench-1.5.1>/dev/null > > real 1m43.180s > user 0m0.069s > sys 0m0.236s > > > Of course, once I dismount the CephFS on node A everything gets as fast as it > can be. > > Am I missing something obvious here? > Yes I could drop the Linux cache as a 'fix' but that would drop the entire > system's cache, sounds a bit extreme! :P > Unless is there a way to drop the cache only for that single dir...? > > > On Tue, Jun 16, 2015 at 12:15 PM, Gregory Farnum <g...@gregs42.com > <mailto:g...@gregs42.com>> wrote: > On Tue, Jun 16, 2015 at 12:11 PM, negillen negillen <negil...@gmail.com > <mailto:negil...@gmail.com>> wrote: > > Thank you very much for your reply! > > > > Is there anything I can do to go around that? e.g. setting access caps to be > > released after a short while? Or is there a command to manually release > > access caps (so that I could run it in cron)? > > Well, you can drop the caches. ;) > > More generally, you're running into a specific hole here. If your > clients are actually *accessing* the files then they should go into > shared mode and this will be much faster on subsequent accesses. > > > This is quite a problem because we have several applications that need to > > access a large number of files and when we set them to work on CephFS > > latency skyrockets. > > What kind of shared-file access do they have? If you have a bunch of > files being shared for read I'd expect this to be very fast. If > different clients are writing small amounts to them in round-robin > then that's unfortunately not going to work well. :( > -Greg > > > > > Thanks again and regards. > > > > On Tue, Jun 16, 2015 at 10:59 AM, Gregory Farnum <g...@gregs42.com > > <mailto:g...@gregs42.com>> wrote: > >> > >> On Mon, Jun 15, 2015 at 11:34 AM, negillen negillen <negil...@gmail.com > >> <mailto:negil...@gmail.com>> > >> wrote: > >> > Hello everyone, > >> > > >> > something very strange is driving me crazy with CephFS (kernel driver). > >> > I copy a large directory on the CephFS from one node. If I try to > >> > perform a > >> > 'time ls -alR' on that directory it gets executed in less than one > >> > second. > >> > If I try to do the same 'time ls -alR' from another node it takes > >> > several > >> > minutes. No matter how many times I repeat the command, the speed is > >> > always > >> > abysmal. The ls works fine on the node where the initial copy was > >> > executed > >> > from. This happens with any directory I have tried, no matter what kind > >> > of > >> > data is inside. > >> > > >> > After lots of experimenting I have found that in order to have fast ls > >> > speed > >> > for that dir from every node I need to flush the Linux cache on the > >> > original > >> > node: > >> > echo 3 > /proc/sys/vm/drop_caches > >> > Unmounting and remounting the CephFS on that node does the trick too. > >> > > >> > Anyone has a clue about what's happening here? Could this be a problem > >> > with > >> > the writeback fscache for the CephFS? > >> > > >> > Any help appreciated! Thanks and regards. :) > >> > >> This is a consequence of the CephFS "file capabilities" that we use to > >> do distributed locking on file states. When you copy the directory on > >> client A, it has full capabilities on the entire tree. When client B > >> tries to do a stat on each file in that tree, it doesn't have any > >> capabilities. So it sends a stat request to the MDS, which sends a cap > >> update to client A requiring it to pause updates on the file and share > >> its current state. Then the MDS tells client A it can keep going and > >> sends the stat to client B. > >> So that's: > >> B -> MDS > >> MDS -> A > >> A -> MDS > >> MDS -> B | MDS -> A > >> for every file you touch. > >> > >> I think the particular oddity you're encountering here is that CephFS > >> generally tries not to make clients drop their "exclusive" access caps > >> just to satisfy a stat. If you had client B doing something with the > >> files (like reading them) you would probably see different behavior. > >> I'm not sure if there's something effective we can do here or not > >> (it's just a bunch of heuristics when we should or should not drop > >> caps), but please file a bug on the tracker (tracker.ceph.com > >> <http://tracker.ceph.com/>) with > >> this case. :) > >> -Greg > > > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
