Re: [Nfs-ganesha-devel] Readdir results for FSAL_CEPH
So, it sounds like there's a small benefit to the caching, but not much. I'll look into that initial jump, to see if it's caused by my stop-caching code. Daniel On 01/19/2017 05:09 AM, Jiffin Tony Thottan wrote: > Hi, > > I have done similar exercise on FSAL_GLUSTER > > configuration : > > gluster volume 2x2 using 4 servers with 4GB RAM and IGB ethernet cable > b/w them > > It was export by one of the above server and mounted in another client > machine, > > created 1 1kb files > > Got a similar kind of result like FSAL_CEPH > > i) dirent caching enabled > real0m2.518s > real0m2.706s > real0m2.628s > real0m2.712s > real0m2.197s > real0m2.065s > real0m2.072s > real0m2.674s > real0m2.649s > real0m2.652s > > ii.) dirent caching disabled > real0m6.178s > real0m2.638s > real0m2.481s > real0m2.501s > real0m2.524s > real0m2.515s > real0m2.476s > real0m2.551s > real0m2.580s > real0m2.543s > > Regards, > Jiffin > > On 14/01/17 02:31, Frank Filz wrote: >> Here are the FSAL_CEPH results: >> >> With 1 files, dirent caching enabled >> real 0m1.032s >> real 0m1.315s >> real 0m1.421s >> real 0m1.429s >> real 0m1.377s >> real 0m1.344s >> real 0m0.667s >> real 0m1.400s >> real 0m1.389s >> real 0m1.728s >> With 1 files, dirent caching disabled >> real 0m15.483s >> real 0m1.457s >> real 0m1.487s >> real 0m1.511s >> real 0m1.498s >> real 0m1.765s >> real 0m1.445s >> real 0m0.930s >> real 0m0.793s >> real 0m1.178s >> >> We need to investigate why in the uncached case it takes 15 seconds for the >> initial readdir. I repeated the experiment several times cached and uncached >> and consistently got comparable results. >> >> Ceph (as was expected) saw less differentiation between cached and uncached, >> there might still be a small advantage to the cached, but it may not be >> statistically significant. >> >> Frank >> >> >> --- >> This email has been checked for viruses by Avast antivirus software. >> https://www.avast.com/antivirus >> >> >> -- >> Developer Access Program for Intel Xeon Phi Processors >> Access to Intel Xeon Phi processor-based developer platforms. >> With one year of Intel Parallel Studio XE. >> Training and support from Colfax. >> Order your platform today. http://sdm.link/xeonphi >> ___ >> Nfs-ganesha-devel mailing list >> Nfs-ganesha-devel@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > > > -- > Check out the vibrant tech community on one of the world's most > engaging tech sites, SlashDot.org! http://sdm.link/slashdot > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Readdir results for FSAL_CEPH
Hi, I have done similar exercise on FSAL_GLUSTER configuration : gluster volume 2x2 using 4 servers with 4GB RAM and IGB ethernet cable b/w them It was export by one of the above server and mounted in another client machine, created 1 1kb files Got a similar kind of result like FSAL_CEPH i) dirent caching enabled real0m2.518s real0m2.706s real0m2.628s real0m2.712s real0m2.197s real0m2.065s real0m2.072s real0m2.674s real0m2.649s real0m2.652s ii.) dirent caching disabled real0m6.178s real0m2.638s real0m2.481s real0m2.501s real0m2.524s real0m2.515s real0m2.476s real0m2.551s real0m2.580s real0m2.543s Regards, Jiffin On 14/01/17 02:31, Frank Filz wrote: > Here are the FSAL_CEPH results: > > With 1 files, dirent caching enabled > real 0m1.032s > real 0m1.315s > real 0m1.421s > real 0m1.429s > real 0m1.377s > real 0m1.344s > real 0m0.667s > real 0m1.400s > real 0m1.389s > real 0m1.728s > With 1 files, dirent caching disabled > real 0m15.483s > real 0m1.457s > real 0m1.487s > real 0m1.511s > real 0m1.498s > real 0m1.765s > real 0m1.445s > real 0m0.930s > real 0m0.793s > real 0m1.178s > > We need to investigate why in the uncached case it takes 15 seconds for the > initial readdir. I repeated the experiment several times cached and uncached > and consistently got comparable results. > > Ceph (as was expected) saw less differentiation between cached and uncached, > there might still be a small advantage to the cached, but it may not be > statistically significant. > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > -- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Readdir results for FSAL_CEPH
Here are the FSAL_CEPH results: With 1 files, dirent caching enabled real0m1.032s real0m1.315s real0m1.421s real0m1.429s real0m1.377s real0m1.344s real0m0.667s real0m1.400s real0m1.389s real0m1.728s With 1 files, dirent caching disabled real0m15.483s real0m1.457s real0m1.487s real0m1.511s real0m1.498s real0m1.765s real0m1.445s real0m0.930s real0m0.793s real0m1.178s We need to investigate why in the uncached case it takes 15 seconds for the initial readdir. I repeated the experiment several times cached and uncached and consistently got comparable results. Ceph (as was expected) saw less differentiation between cached and uncached, there might still be a small advantage to the cached, but it may not be statistically significant. Frank --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Readdir results
Frank's result based on testing of fsal vfs can't be generalized to configurations which actually have latency, so, no? Matt - Original Message - > From: "William Allen Simpson" <william.allen.simp...@gmail.com> > To: "Frank Filz" <ffilz...@mindspring.com>, > nfs-ganesha-devel@lists.sourceforge.net > Sent: Wednesday, January 11, 2017 5:19:46 AM > Subject: Re: [Nfs-ganesha-devel] Readdir results > > On 1/10/17 4:29 PM, Frank Filz wrote: > > It looks like dirent caching does buy us something, though interestingly > > the > > initial populating tends to be the quickest run... > > > If the initial populating is the quickest run, there must be some > interaction between the cache and the re-fetch that's causing thrashing. > > I make no pretense of understanding this code. But my gut feeling at this > point is to toss this caching entirely. Faster stores and faster networks > mean fetching the entries should be fairly low latency. Need to optimize > moving the data through the system, with less examining it. > > -- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309 -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Readdir results
On 1/10/17 4:29 PM, Frank Filz wrote: > It looks like dirent caching does buy us something, though interestingly the > initial populating tends to be the quickest run... > If the initial populating is the quickest run, there must be some interaction between the cache and the re-fetch that's causing thrashing. I make no pretense of understanding this code. But my gut feeling at this point is to toss this caching entirely. Faster stores and faster networks mean fetching the entries should be fairly low latency. Need to optimize moving the data through the system, with less examining it. -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
Re: [Nfs-ganesha-devel] Readdir results
On Di, 2017-01-10 at 13:29 -0800, Frank Filz wrote: > I did some testing between having dirent caching and not (Dir_Max > left > default, or Dir_Max set to 1): > > With 1 files, dirent caching enabled > real 0m1.940s > real 0m3.652s > real 0m2.409s > real 0m1.968s > real 0m2.098s > real 0m2.209s > real 0m3.792s > real 0m2.625s > real 0m2.201s > real 0m2.963s > > With 1 files, dirent caching disabled > real 0m2.423s > real 0m4.142s > real 0m3.883s > real 0m2.939s > real 0m2.283s > real 0m5.105s > real 0m3.050s > real 0m3.609s > real 0m2.574s > real 0m2.727s Do you see any differences w/ and w/o caching if the directory is being modified while you're executing ls -l (like adding / removing files or changing attributes) ? I remember having to deal with customers doing something alike with >200k files in a single directory and the execution time jumped up to 10-30 minutes. Cheers Swen > > It looks like dirent caching does buy us something, though > interestingly the > initial populating tends to be the quickest run... > > These were run with Ganesha having freshly started, and the following > script: > > mount /mnt4 > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > time ls -l /mnt4/test1/many | wc > echo 3 > /proc/sys/vm/drop_caches > umount /mnt4 > > With the results piped to grep real. > > Frank > > > --- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus > > > --- > --- > Developer Access Program for Intel Xeon Phi Processors > Access to Intel Xeon Phi processor-based developer platforms. > With one year of Intel Parallel Studio XE. > Training and support from Colfax. > Order your platform today. http://sdm.link/xeonphi > ___ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel > -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
[Nfs-ganesha-devel] Readdir results
I did some testing between having dirent caching and not (Dir_Max left default, or Dir_Max set to 1): With 1 files, dirent caching enabled real0m1.940s real0m3.652s real0m2.409s real0m1.968s real0m2.098s real0m2.209s real0m3.792s real0m2.625s real0m2.201s real0m2.963s With 1 files, dirent caching disabled real0m2.423s real0m4.142s real0m3.883s real0m2.939s real0m2.283s real0m5.105s real0m3.050s real0m3.609s real0m2.574s real0m2.727s It looks like dirent caching does buy us something, though interestingly the initial populating tends to be the quickest run... These were run with Ganesha having freshly started, and the following script: mount /mnt4 time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches time ls -l /mnt4/test1/many | wc echo 3 > /proc/sys/vm/drop_caches umount /mnt4 With the results piped to grep real. Frank --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -- Developer Access Program for Intel Xeon Phi Processors Access to Intel Xeon Phi processor-based developer platforms. With one year of Intel Parallel Studio XE. Training and support from Colfax. Order your platform today. http://sdm.link/xeonphi ___ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
