Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
On Wed, Feb 13, 2019 at 11:16 AM Manoj Pillai wrote: > > > On Wed, Feb 13, 2019 at 10:51 AM Raghavendra Gowdappa > wrote: > >> >> >> On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa >> wrote: >> >>> All, >>> >>> We've found perf xlators io-cache and read-ahead not adding any >>> performance improvement. At best read-ahead is redundant due to kernel >>> read-ahead >>> >> >> One thing we are still figuring out is whether kernel read-ahead is >> tunable. From what we've explored, it _looks_ like (may not be entirely >> correct), ra is capped at 128KB. If that's the case, I am interested in few >> things: >> * Are there any realworld applications/usecases, which would benefit from >> larger read-ahead (Manoj says block devices can do ra of 4MB)? >> > > kernel read-ahead is adaptive but influenced by the read-ahead setting on > the block device (/sys/block//queue/read_ahead_kb), which can be > tuned. For RHEL specifically, the default is 128KB (last I checked) but the > default RHEL tuned-profile, throughput-performance, bumps that up to 4MB. > It should be fairly easy to rig up a test where 4MB read-ahead on the > block device gives better performance than 128KB read-ahead. > Thanks Manoj. To add to what Manoj said and give more context here, Glusterfs being a fuse-based fs is not exposed as a block device. So, that's the first problem of where/how to tune and I've listed other problems earlier. > -- Manoj > > * Is the limit on kernel ra tunable a hard one? IOW, what does it take to >> make it to do higher ra? If its difficult, can glusterfs read-ahead provide >> the expected performance improvement for these applications that would >> benefit from aggressive ra (as glusterfs can support larger ra sizes)? >> >> I am still inclined to prefer kernel ra as I think its more intelligent >> and can identify more sequential patterns than Glusterfs read-ahead [1][2]. >> [1] https://www.kernel.org/doc/ols/2007/ols2007v2-pages-273-284.pdf >> [2] https://lwn.net/Articles/155510/ >> >> and at worst io-cache is degrading the performance for workloads that >>> doesn't involve re-read. Given that VFS already have both these >>> functionalities, I am proposing to have these two translators turned off by >>> default for native fuse mounts. >>> >>> For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have >>> these xlators on by having custom profiles. Comments? >>> >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 >>> >>> regards, >>> Raghavendra >>> >> ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
On Wed, Feb 13, 2019 at 10:51 AM Raghavendra Gowdappa wrote: > > > On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa > wrote: > >> All, >> >> We've found perf xlators io-cache and read-ahead not adding any >> performance improvement. At best read-ahead is redundant due to kernel >> read-ahead >> > > One thing we are still figuring out is whether kernel read-ahead is > tunable. From what we've explored, it _looks_ like (may not be entirely > correct), ra is capped at 128KB. If that's the case, I am interested in few > things: > * Are there any realworld applications/usecases, which would benefit from > larger read-ahead (Manoj says block devices can do ra of 4MB)? > kernel read-ahead is adaptive but influenced by the read-ahead setting on the block device (/sys/block//queue/read_ahead_kb), which can be tuned. For RHEL specifically, the default is 128KB (last I checked) but the default RHEL tuned-profile, throughput-performance, bumps that up to 4MB. It should be fairly easy to rig up a test where 4MB read-ahead on the block device gives better performance than 128KB read-ahead. -- Manoj * Is the limit on kernel ra tunable a hard one? IOW, what does it take to > make it to do higher ra? If its difficult, can glusterfs read-ahead provide > the expected performance improvement for these applications that would > benefit from aggressive ra (as glusterfs can support larger ra sizes)? > > I am still inclined to prefer kernel ra as I think its more intelligent > and can identify more sequential patterns than Glusterfs read-ahead [1][2]. > [1] https://www.kernel.org/doc/ols/2007/ols2007v2-pages-273-284.pdf > [2] https://lwn.net/Articles/155510/ > > and at worst io-cache is degrading the performance for workloads that >> doesn't involve re-read. Given that VFS already have both these >> functionalities, I am proposing to have these two translators turned off by >> default for native fuse mounts. >> >> For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have >> these xlators on by having custom profiles. Comments? >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 >> >> regards, >> Raghavendra >> > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa wrote: > All, > > We've found perf xlators io-cache and read-ahead not adding any > performance improvement. At best read-ahead is redundant due to kernel > read-ahead > One thing we are still figuring out is whether kernel read-ahead is tunable. From what we've explored, it _looks_ like (may not be entirely correct), ra is capped at 128KB. If that's the case, I am interested in few things: * Are there any realworld applications/usecases, which would benefit from larger read-ahead (Manoj says block devices can do ra of 4MB)? * Is the limit on kernel ra tunable a hard one? IOW, what does it take to make it to do higher ra? If its difficult, can glusterfs read-ahead provide the expected performance improvement for these applications that would benefit from aggressive ra (as glusterfs can support larger ra sizes)? I am still inclined to prefer kernel ra as I think its more intelligent and can identify more sequential patterns than Glusterfs read-ahead [1][2]. [1] https://www.kernel.org/doc/ols/2007/ols2007v2-pages-273-284.pdf [2] https://lwn.net/Articles/155510/ and at worst io-cache is degrading the performance for workloads that > doesn't involve re-read. Given that VFS already have both these > functionalities, I am proposing to have these two translators turned off by > default for native fuse mounts. > > For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have > these xlators on by having custom profiles. Comments? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 > > regards, > Raghavendra > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Disabling read-ahead and io-cache for native fuse mounts
On Tue, Feb 12, 2019 at 11:09 PM Darrell Budic wrote: > Is there an example of a custom profile you can share for my ovirt use > case (with gfapi enabled)? > I was speaking about a group setting like "group metadata-cache". Its just that custom options one would turn on for a class of applications or problems. Or are you just talking about the standard group settings for virt as a > custom profile? > > On Feb 12, 2019, at 7:22 AM, Raghavendra Gowdappa > wrote: > > https://review.gluster.org/22203 > > On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa > wrote: > >> All, >> >> We've found perf xlators io-cache and read-ahead not adding any >> performance improvement. At best read-ahead is redundant due to kernel >> read-ahead and at worst io-cache is degrading the performance for workloads >> that doesn't involve re-read. Given that VFS already have both these >> functionalities, I am proposing to have these two translators turned off by >> default for native fuse mounts. >> >> For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have >> these xlators on by having custom profiles. Comments? >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 >> >> regards, >> Raghavendra >> > ___ > Gluster-users mailing list > gluster-us...@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
The volume is not stopped before unmounting the bricks. I will send a fix. On Wed, 13 Feb 2019 at 10:00, Raghavendra Gowdappa wrote: > > > On Wed, Feb 13, 2019 at 9:54 AM Amar Tumballi Suryanarayan < > atumb...@redhat.com> wrote: > >> >> >> On Wed, Feb 13, 2019 at 9:51 AM Nithya Balachandran >> wrote: >> >>> I'll take a look at this today. The logs indicate the test completed in >>> under 3 minutes but something seems to be holding up the cleanup. >>> >>> >> Just a look on some successful runs show output like below: >> >> -- >> >> *17:44:49* ok 57, LINENUM:155*17:44:49* umount: /d/backends/patchy1: target >> is busy.*17:44:49* (In some cases useful info about processes that >> use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* >> umount: /d/backends/patchy2: target is busy.*17:44:49* (In some >> cases useful info about processes that use*17:44:49* the device is >> found by lsof(8) or fuser(1))*17:44:49* umount: /d/backends/patchy3: target >> is busy.*17:44:49* (In some cases useful info about processes that >> use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* >> N*17:44:49* ok >> >> -- >> >> This is just before finish, so , the cleanup is being held for sure. >> > > Yes. In my tests too, I saw these msgs. But, i thought they are not > accounted in waiting time. > > >> Regards, >> Amar >> >> On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa >>> wrote: >>> On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal wrote: > Hi, > > I have observed the test case ./tests/bugs/distribute/bug-1161311.t is > getting timed > I've seen failure of this too in some of my patches. out on build server at the time of running centos regression on one of > my patch https://review.gluster.org/22166 > > I have executed test case for i in {1..30}; do time prove -vf > ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that > is > similar to build infra, the test case is not taking time more than 3 > minutes but on build server test case is getting timed out. > > Kindly share your input if you are facing the same. > > Thanks, > Mohit Agrawal > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel >>> >>> ___ >>> Gluster-devel mailing list >>> Gluster-devel@gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-devel >> >> >> >> -- >> Amar Tumballi (amarts) >> > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
On Wed, Feb 13, 2019 at 9:54 AM Amar Tumballi Suryanarayan < atumb...@redhat.com> wrote: > > > On Wed, Feb 13, 2019 at 9:51 AM Nithya Balachandran > wrote: > >> I'll take a look at this today. The logs indicate the test completed in >> under 3 minutes but something seems to be holding up the cleanup. >> >> > Just a look on some successful runs show output like below: > > -- > > *17:44:49* ok 57, LINENUM:155*17:44:49* umount: /d/backends/patchy1: target > is busy.*17:44:49* (In some cases useful info about processes that > use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* > umount: /d/backends/patchy2: target is busy.*17:44:49* (In some cases > useful info about processes that use*17:44:49* the device is found > by lsof(8) or fuser(1))*17:44:49* umount: /d/backends/patchy3: target is > busy.*17:44:49* (In some cases useful info about processes that > use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* > N*17:44:49* ok > > -- > > This is just before finish, so , the cleanup is being held for sure. > Yes. In my tests too, I saw these msgs. But, i thought they are not accounted in waiting time. > Regards, > Amar > > On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa >> wrote: >> >>> >>> >>> On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal >>> wrote: >>> Hi, I have observed the test case ./tests/bugs/distribute/bug-1161311.t is getting timed >>> >>> I've seen failure of this too in some of my patches. >>> >>> out on build server at the time of running centos regression on one of my patch https://review.gluster.org/22166 I have executed test case for i in {1..30}; do time prove -vf ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is similar to build infra, the test case is not taking time more than 3 minutes but on build server test case is getting timed out. Kindly share your input if you are facing the same. Thanks, Mohit Agrawal ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel >>> >>> ___ >>> Gluster-devel mailing list >>> Gluster-devel@gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-devel >> >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > > > -- > Amar Tumballi (amarts) > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
On Wed, Feb 13, 2019 at 9:51 AM Nithya Balachandran wrote: > I'll take a look at this today. The logs indicate the test completed in > under 3 minutes but something seems to be holding up the cleanup. > > Just a look on some successful runs show output like below: -- *17:44:49* ok 57, LINENUM:155*17:44:49* umount: /d/backends/patchy1: target is busy.*17:44:49* (In some cases useful info about processes that use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* umount: /d/backends/patchy2: target is busy.*17:44:49* (In some cases useful info about processes that use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* umount: /d/backends/patchy3: target is busy.*17:44:49* (In some cases useful info about processes that use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49* N*17:44:49* ok -- This is just before finish, so , the cleanup is being held for sure. Regards, Amar On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa > wrote: > >> >> >> On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal >> wrote: >> >>> Hi, >>> >>> I have observed the test case ./tests/bugs/distribute/bug-1161311.t is >>> getting timed >>> >> >> I've seen failure of this too in some of my patches. >> >> out on build server at the time of running centos regression on one of my >>> patch https://review.gluster.org/22166 >>> >>> I have executed test case for i in {1..30}; do time prove -vf >>> ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is >>> similar to build infra, the test case is not taking time more than 3 >>> minutes but on build server test case is getting timed out. >>> >>> Kindly share your input if you are facing the same. >>> >>> Thanks, >>> Mohit Agrawal >>> ___ >>> Gluster-devel mailing list >>> Gluster-devel@gluster.org >>> https://lists.gluster.org/mailman/listinfo/gluster-devel >> >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel -- Amar Tumballi (amarts) ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
I'll take a look at this today. The logs indicate the test completed in under 3 minutes but something seems to be holding up the cleanup. On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa wrote: > > > On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal wrote: > >> Hi, >> >> I have observed the test case ./tests/bugs/distribute/bug-1161311.t is >> getting timed >> > > I've seen failure of this too in some of my patches. > > out on build server at the time of running centos regression on one of my >> patch https://review.gluster.org/22166 >> >> I have executed test case for i in {1..30}; do time prove -vf >> ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is >> similar to build infra, the test case is not taking time more than 3 >> minutes but on build server test case is getting timed out. >> >> Kindly share your input if you are facing the same. >> >> Thanks, >> Mohit Agrawal >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> https://lists.gluster.org/mailman/listinfo/gluster-devel > > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] [Gluster-infra]Softserve will be down
Hi, After all the RAX builders are moved to AWS. We are planning to migrate softserve application to AWS completely. As a part of this migration, we're bringing down softserve for a few days. So softserve will not be able to lend any more machines until we are ready with the migrated code. Thanks, Deepshikha ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal wrote: > Hi, > > I have observed the test case ./tests/bugs/distribute/bug-1161311.t is > getting timed > I've seen failure of this too in some of my patches. out on build server at the time of running centos regression on one of my > patch https://review.gluster.org/22166 > > I have executed test case for i in {1..30}; do time prove -vf > ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is > similar to build infra, the test case is not taking time more than 3 > minutes but on build server test case is getting timed out. > > Kindly share your input if you are facing the same. > > Thanks, > Mohit Agrawal > ___ > Gluster-devel mailing list > Gluster-devel@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
Hi, I have observed the test case ./tests/bugs/distribute/bug-1161311.t is getting timed out on build server at the time of running centos regression on one of my patch https://review.gluster.org/22166 I have executed test case for i in {1..30}; do time prove -vf ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is similar to build infra, the test case is not taking time more than 3 minutes but on build server test case is getting timed out. Kindly share your input if you are facing the same. Thanks, Mohit Agrawal ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
https://review.gluster.org/22203 On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa wrote: > All, > > We've found perf xlators io-cache and read-ahead not adding any > performance improvement. At best read-ahead is redundant due to kernel > read-ahead and at worst io-cache is degrading the performance for workloads > that doesn't involve re-read. Given that VFS already have both these > functionalities, I am proposing to have these two translators turned off by > default for native fuse mounts. > > For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have > these xlators on by having custom profiles. Comments? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 > > regards, > Raghavendra > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
All, We've found perf xlators io-cache and read-ahead not adding any performance improvement. At best read-ahead is redundant due to kernel read-ahead and at worst io-cache is degrading the performance for workloads that doesn't involve re-read. Given that VFS already have both these functionalities, I am proposing to have these two translators turned off by default for native fuse mounts. For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have these xlators on by having custom profiles. Comments? [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 regards, Raghavendra ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel