Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
https://review.gluster.org/22203 On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa wrote: > All, > > We've found perf xlators io-cache and read-ahead not adding any > performance improvement. At best read-ahead is redundant due to kernel > read-ahead and at worst io-cache is degrading the performance for workloads > that doesn't involve re-read. Given that VFS already have both these > functionalities, I am proposing to have these two translators turned off by > default for native fuse mounts. > > For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have > these xlators on by having custom profiles. Comments? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 > > regards, > Raghavendra > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
Hi,
I have observed the test case ./tests/bugs/distribute/bug-1161311.t is
getting timed
out on build server at the time of running centos regression on one of my
patch https://review.gluster.org/22166
I have executed test case for i in {1..30}; do time prove -vf
./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is
similar to build infra, the test case is not taking time more than 3
minutes but on build server test case is getting timed out.
Kindly share your input if you are facing the same.
Thanks,
Mohit Agrawal
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal wrote:
> Hi,
>
> I have observed the test case ./tests/bugs/distribute/bug-1161311.t is
> getting timed
>
I've seen failure of this too in some of my patches.
out on build server at the time of running centos regression on one of my
> patch https://review.gluster.org/22166
>
> I have executed test case for i in {1..30}; do time prove -vf
> ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is
> similar to build infra, the test case is not taking time more than 3
> minutes but on build server test case is getting timed out.
>
> Kindly share your input if you are facing the same.
>
> Thanks,
> Mohit Agrawal
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] [Gluster-infra]Softserve will be down
Hi, After all the RAX builders are moved to AWS. We are planning to migrate softserve application to AWS completely. As a part of this migration, we're bringing down softserve for a few days. So softserve will not be able to lend any more machines until we are ready with the migrated code. Thanks, Deepshikha ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
On Wed, Feb 13, 2019 at 9:51 AM Nithya Balachandran
wrote:
> I'll take a look at this today. The logs indicate the test completed in
> under 3 minutes but something seems to be holding up the cleanup.
>
>
Just a look on some successful runs show output like below:
--
*17:44:49* ok 57, LINENUM:155*17:44:49* umount: /d/backends/patchy1:
target is busy.*17:44:49* (In some cases useful info about
processes that use*17:44:49* the device is found by lsof(8)
or fuser(1))*17:44:49* umount: /d/backends/patchy2: target is
busy.*17:44:49* (In some cases useful info about processes
that use*17:44:49* the device is found by lsof(8) or
fuser(1))*17:44:49* umount: /d/backends/patchy3: target is
busy.*17:44:49* (In some cases useful info about processes
that use*17:44:49* the device is found by lsof(8) or
fuser(1))*17:44:49* N*17:44:49* ok
--
This is just before finish, so , the cleanup is being held for sure.
Regards,
Amar
On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa
> wrote:
>
>>
>>
>> On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal
>> wrote:
>>
>>> Hi,
>>>
>>> I have observed the test case ./tests/bugs/distribute/bug-1161311.t is
>>> getting timed
>>>
>>
>> I've seen failure of this too in some of my patches.
>>
>> out on build server at the time of running centos regression on one of my
>>> patch https://review.gluster.org/22166
>>>
>>> I have executed test case for i in {1..30}; do time prove -vf
>>> ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is
>>> similar to build infra, the test case is not taking time more than 3
>>> minutes but on build server test case is getting timed out.
>>>
>>> Kindly share your input if you are facing the same.
>>>
>>> Thanks,
>>> Mohit Agrawal
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
--
Amar Tumballi (amarts)
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
The volume is not stopped before unmounting the bricks. I will send a fix.
On Wed, 13 Feb 2019 at 10:00, Raghavendra Gowdappa
wrote:
>
>
> On Wed, Feb 13, 2019 at 9:54 AM Amar Tumballi Suryanarayan <
> atumb...@redhat.com> wrote:
>
>>
>>
>> On Wed, Feb 13, 2019 at 9:51 AM Nithya Balachandran
>> wrote:
>>
>>> I'll take a look at this today. The logs indicate the test completed in
>>> under 3 minutes but something seems to be holding up the cleanup.
>>>
>>>
>> Just a look on some successful runs show output like below:
>>
>> --
>>
>> *17:44:49* ok 57, LINENUM:155*17:44:49* umount: /d/backends/patchy1: target
>> is busy.*17:44:49* (In some cases useful info about processes that
>> use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49*
>> umount: /d/backends/patchy2: target is busy.*17:44:49* (In some
>> cases useful info about processes that use*17:44:49* the device is
>> found by lsof(8) or fuser(1))*17:44:49* umount: /d/backends/patchy3: target
>> is busy.*17:44:49* (In some cases useful info about processes that
>> use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49*
>> N*17:44:49* ok
>>
>> --
>>
>> This is just before finish, so , the cleanup is being held for sure.
>>
>
> Yes. In my tests too, I saw these msgs. But, i thought they are not
> accounted in waiting time.
>
>
>> Regards,
>> Amar
>>
>> On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa
>>> wrote:
>>>
On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal
wrote:
> Hi,
>
> I have observed the test case ./tests/bugs/distribute/bug-1161311.t is
> getting timed
>
I've seen failure of this too in some of my patches.
out on build server at the time of running centos regression on one of
> my patch https://review.gluster.org/22166
>
> I have executed test case for i in {1..30}; do time prove -vf
> ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that
> is
> similar to build infra, the test case is not taking time more than 3
> minutes but on build server test case is getting timed out.
>
> Kindly share your input if you are facing the same.
>
> Thanks,
> Mohit Agrawal
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>>
>> --
>> Amar Tumballi (amarts)
>>
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
On Wed, Feb 13, 2019 at 10:51 AM Raghavendra Gowdappa wrote: > > > On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa > wrote: > >> All, >> >> We've found perf xlators io-cache and read-ahead not adding any >> performance improvement. At best read-ahead is redundant due to kernel >> read-ahead >> > > One thing we are still figuring out is whether kernel read-ahead is > tunable. From what we've explored, it _looks_ like (may not be entirely > correct), ra is capped at 128KB. If that's the case, I am interested in few > things: > * Are there any realworld applications/usecases, which would benefit from > larger read-ahead (Manoj says block devices can do ra of 4MB)? > kernel read-ahead is adaptive but influenced by the read-ahead setting on the block device (/sys/block//queue/read_ahead_kb), which can be tuned. For RHEL specifically, the default is 128KB (last I checked) but the default RHEL tuned-profile, throughput-performance, bumps that up to 4MB. It should be fairly easy to rig up a test where 4MB read-ahead on the block device gives better performance than 128KB read-ahead. -- Manoj * Is the limit on kernel ra tunable a hard one? IOW, what does it take to > make it to do higher ra? If its difficult, can glusterfs read-ahead provide > the expected performance improvement for these applications that would > benefit from aggressive ra (as glusterfs can support larger ra sizes)? > > I am still inclined to prefer kernel ra as I think its more intelligent > and can identify more sequential patterns than Glusterfs read-ahead [1][2]. > [1] https://www.kernel.org/doc/ols/2007/ols2007v2-pages-273-284.pdf > [2] https://lwn.net/Articles/155510/ > > and at worst io-cache is degrading the performance for workloads that >> doesn't involve re-read. Given that VFS already have both these >> functionalities, I am proposing to have these two translators turned off by >> default for native fuse mounts. >> >> For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have >> these xlators on by having custom profiles. Comments? >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 >> >> regards, >> Raghavendra >> > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
On Wed, Feb 13, 2019 at 11:16 AM Manoj Pillai wrote: > > > On Wed, Feb 13, 2019 at 10:51 AM Raghavendra Gowdappa > wrote: > >> >> >> On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa >> wrote: >> >>> All, >>> >>> We've found perf xlators io-cache and read-ahead not adding any >>> performance improvement. At best read-ahead is redundant due to kernel >>> read-ahead >>> >> >> One thing we are still figuring out is whether kernel read-ahead is >> tunable. From what we've explored, it _looks_ like (may not be entirely >> correct), ra is capped at 128KB. If that's the case, I am interested in few >> things: >> * Are there any realworld applications/usecases, which would benefit from >> larger read-ahead (Manoj says block devices can do ra of 4MB)? >> > > kernel read-ahead is adaptive but influenced by the read-ahead setting on > the block device (/sys/block//queue/read_ahead_kb), which can be > tuned. For RHEL specifically, the default is 128KB (last I checked) but the > default RHEL tuned-profile, throughput-performance, bumps that up to 4MB. > It should be fairly easy to rig up a test where 4MB read-ahead on the > block device gives better performance than 128KB read-ahead. > Thanks Manoj. To add to what Manoj said and give more context here, Glusterfs being a fuse-based fs is not exposed as a block device. So, that's the first problem of where/how to tune and I've listed other problems earlier. > -- Manoj > > * Is the limit on kernel ra tunable a hard one? IOW, what does it take to >> make it to do higher ra? If its difficult, can glusterfs read-ahead provide >> the expected performance improvement for these applications that would >> benefit from aggressive ra (as glusterfs can support larger ra sizes)? >> >> I am still inclined to prefer kernel ra as I think its more intelligent >> and can identify more sequential patterns than Glusterfs read-ahead [1][2]. >> [1] https://www.kernel.org/doc/ols/2007/ols2007v2-pages-273-284.pdf >> [2] https://lwn.net/Articles/155510/ >> >> and at worst io-cache is degrading the performance for workloads that >>> doesn't involve re-read. Given that VFS already have both these >>> functionalities, I am proposing to have these two translators turned off by >>> default for native fuse mounts. >>> >>> For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have >>> these xlators on by having custom profiles. Comments? >>> >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 >>> >>> regards, >>> Raghavendra >>> >> ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
On Wed, Feb 13, 2019 at 9:54 AM Amar Tumballi Suryanarayan <
atumb...@redhat.com> wrote:
>
>
> On Wed, Feb 13, 2019 at 9:51 AM Nithya Balachandran
> wrote:
>
>> I'll take a look at this today. The logs indicate the test completed in
>> under 3 minutes but something seems to be holding up the cleanup.
>>
>>
> Just a look on some successful runs show output like below:
>
> --
>
> *17:44:49* ok 57, LINENUM:155*17:44:49* umount: /d/backends/patchy1: target
> is busy.*17:44:49* (In some cases useful info about processes that
> use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49*
> umount: /d/backends/patchy2: target is busy.*17:44:49* (In some cases
> useful info about processes that use*17:44:49* the device is found
> by lsof(8) or fuser(1))*17:44:49* umount: /d/backends/patchy3: target is
> busy.*17:44:49* (In some cases useful info about processes that
> use*17:44:49* the device is found by lsof(8) or fuser(1))*17:44:49*
> N*17:44:49* ok
>
> --
>
> This is just before finish, so , the cleanup is being held for sure.
>
Yes. In my tests too, I saw these msgs. But, i thought they are not
accounted in waiting time.
> Regards,
> Amar
>
> On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa
>> wrote:
>>
>>>
>>>
>>> On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal
>>> wrote:
>>>
Hi,
I have observed the test case ./tests/bugs/distribute/bug-1161311.t is
getting timed
>>>
>>> I've seen failure of this too in some of my patches.
>>>
>>> out on build server at the time of running centos regression on one of
my patch https://review.gluster.org/22166
I have executed test case for i in {1..30}; do time prove -vf
./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is
similar to build infra, the test case is not taking time more than 3
minutes but on build server test case is getting timed out.
Kindly share your input if you are facing the same.
Thanks,
Mohit Agrawal
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>> ___
>>> Gluster-devel mailing list
>>> Gluster-devel@gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
> Amar Tumballi (amarts)
>
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Failing test case ./tests/bugs/distribute/bug-1161311.t
I'll take a look at this today. The logs indicate the test completed in
under 3 minutes but something seems to be holding up the cleanup.
On Tue, 12 Feb 2019 at 19:30, Raghavendra Gowdappa
wrote:
>
>
> On Tue, Feb 12, 2019 at 7:16 PM Mohit Agrawal wrote:
>
>> Hi,
>>
>> I have observed the test case ./tests/bugs/distribute/bug-1161311.t is
>> getting timed
>>
>
> I've seen failure of this too in some of my patches.
>
> out on build server at the time of running centos regression on one of my
>> patch https://review.gluster.org/22166
>>
>> I have executed test case for i in {1..30}; do time prove -vf
>> ./tests/bugs/distribute/bug-1161311.t; done 30 times on softserv vm that is
>> similar to build infra, the test case is not taking time more than 3
>> minutes but on build server test case is getting timed out.
>>
>> Kindly share your input if you are facing the same.
>>
>> Thanks,
>> Mohit Agrawal
>> ___
>> Gluster-devel mailing list
>> Gluster-devel@gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
> ___
> Gluster-devel mailing list
> Gluster-devel@gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
___
Gluster-devel mailing list
Gluster-devel@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] [Gluster-users] Disabling read-ahead and io-cache for native fuse mounts
On Tue, Feb 12, 2019 at 11:09 PM Darrell Budic wrote: > Is there an example of a custom profile you can share for my ovirt use > case (with gfapi enabled)? > I was speaking about a group setting like "group metadata-cache". Its just that custom options one would turn on for a class of applications or problems. Or are you just talking about the standard group settings for virt as a > custom profile? > > On Feb 12, 2019, at 7:22 AM, Raghavendra Gowdappa > wrote: > > https://review.gluster.org/22203 > > On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa > wrote: > >> All, >> >> We've found perf xlators io-cache and read-ahead not adding any >> performance improvement. At best read-ahead is redundant due to kernel >> read-ahead and at worst io-cache is degrading the performance for workloads >> that doesn't involve re-read. Given that VFS already have both these >> functionalities, I am proposing to have these two translators turned off by >> default for native fuse mounts. >> >> For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have >> these xlators on by having custom profiles. Comments? >> >> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 >> >> regards, >> Raghavendra >> > ___ > Gluster-users mailing list > gluster-us...@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
On Tue, Feb 12, 2019 at 5:38 PM Raghavendra Gowdappa wrote: > All, > > We've found perf xlators io-cache and read-ahead not adding any > performance improvement. At best read-ahead is redundant due to kernel > read-ahead > One thing we are still figuring out is whether kernel read-ahead is tunable. From what we've explored, it _looks_ like (may not be entirely correct), ra is capped at 128KB. If that's the case, I am interested in few things: * Are there any realworld applications/usecases, which would benefit from larger read-ahead (Manoj says block devices can do ra of 4MB)? * Is the limit on kernel ra tunable a hard one? IOW, what does it take to make it to do higher ra? If its difficult, can glusterfs read-ahead provide the expected performance improvement for these applications that would benefit from aggressive ra (as glusterfs can support larger ra sizes)? I am still inclined to prefer kernel ra as I think its more intelligent and can identify more sequential patterns than Glusterfs read-ahead [1][2]. [1] https://www.kernel.org/doc/ols/2007/ols2007v2-pages-273-284.pdf [2] https://lwn.net/Articles/155510/ and at worst io-cache is degrading the performance for workloads that > doesn't involve re-read. Given that VFS already have both these > functionalities, I am proposing to have these two translators turned off by > default for native fuse mounts. > > For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have > these xlators on by having custom profiles. Comments? > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 > > regards, > Raghavendra > ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Disabling read-ahead and io-cache for native fuse mounts
All, We've found perf xlators io-cache and read-ahead not adding any performance improvement. At best read-ahead is redundant due to kernel read-ahead and at worst io-cache is degrading the performance for workloads that doesn't involve re-read. Given that VFS already have both these functionalities, I am proposing to have these two translators turned off by default for native fuse mounts. For non-native fuse mounts like gfapi (NFS-ganesha/samba) we can have these xlators on by having custom profiles. Comments? [1] https://bugzilla.redhat.com/show_bug.cgi?id=1665029 regards, Raghavendra ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
