On Fri, Jun 29, 2018 at 8:24 PM, Pat Haley <[email protected]> wrote: > > Hi Raghavendra, > > Our technician was able to try the manual setting today. He found that > our upper limit for performance.md-cache-timeout was 60 not 600, so he > used that value, along with the network.inode-lru-limit=50000. > > The result was another small (~1%) increase in speed. Does this suggest > some addition tests/changes we could try? >
Can you set gluster option diagnostics.client-log-level to TRACE and run sequential read tests again (with md-cache-timeout value of 60)? #gluster volume set <volname> diagnostics.client-log-level TRACE Also are you sure that open-behind was turned off? Can you give the output of, # gluster volume info <volname> > Thanks > > Pat > > > > > On 06/25/2018 09:39 PM, Raghavendra Gowdappa wrote: > > > > On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley <[email protected]> wrote: > >> >> Hi Raghavendra, >> >> Setting the performance.write-behind off had a small improvement on the >> write speed (~3%), >> >> We were unable to turn on "group metadata-cache". When we try get errors >> like >> >> # gluster volume set data-volume group metadata-cache >> '/var/lib/glusterd/groups/metadata-cache' file format not valid. >> >> Was metadata-cache available for gluster 3.7.11? We ask because the >> release notes for 3.11 mentions “Feature for metadata-caching/small file >> performance is production ready.” (https://gluster.readthedocs.i >> o/en/latest/release-notes/3.11.0/). >> >> Do any of these results suggest anything? If not, what further tests >> would be useful? >> > > Group metadata-cache is just a bunch of options one sets on a volume. So, > You can set them manually using gluster cli. Following are the options and > their values: > > performance.md-cache-timeout=600 > network.inode-lru-limit=50000 > > > >> Thanks >> >> Pat >> >> >> >> >> On 06/22/2018 07:51 AM, Raghavendra Gowdappa wrote: >> >> >> >> On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley <[email protected]> wrote: >> >>> >>> Hi Raghavendra, >>> >>> Thanks for the suggestions. Our technician will be in on Monday. We'll >>> test then and let you know the results. >>> >>> One question I have, is the "group metadata-cache" option supposed to >>> directly impact the performance or is it to help collect data? If the >>> latter, where will the data be located? >>> >> >> It impacts performance. >> >> >>> Thanks again. >>> >>> Pat >>> >>> >>> >>> On 06/21/2018 01:01 AM, Raghavendra Gowdappa wrote: >>> >>> >>> >>> On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa < >>> [email protected]> wrote: >>> >>>> For the case of writes to glusterfs mount, >>>> >>>> I saw in earlier conversations that there are too many lookups, but >>>> small number of writes. Since writes cached in write-behind would >>>> invalidate metadata cache, lookups won't be absorbed by md-cache. I am >>>> wondering what would results look like if we turn off >>>> performance.write-behind. >>>> >>>> @Pat, >>>> >>>> Can you set, >>>> >>>> # gluster volume set <volname> performance.write-behind off >>>> >>> >>> Please turn on "group metadata-cache" for write tests too. >>> >>> >>>> and redo the tests writing to glusterfs mount? Let us know about the >>>> results you see. >>>> >>>> regards, >>>> Raghavendra >>>> >>>> On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa < >>>> [email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa < >>>>> [email protected]> wrote: >>>>> >>>>>> For the case of reading from Glusterfs mount, read-ahead should help. >>>>>> However, we've known issues with read-ahead[1][2]. To work around these, >>>>>> can you try with, >>>>>> >>>>>> 1. Turn off performance.open-behind >>>>>> #gluster volume set <volname> performance.open-behind off >>>>>> >>>>>> 2. enable group meta metadata-cache >>>>>> # gluster volume set <volname> group metadata-cache >>>>>> >>>>> >>>>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508 >>>>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489 >>>>> >>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley <[email protected]> wrote: >>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We were recently revisiting our problems with the slowness of >>>>>>> gluster writes (http://lists.gluster.org/pipe >>>>>>> rmail/gluster-users/2017-April/030529.html). Specifically we were >>>>>>> testing the suggestions in a recent post ( >>>>>>> http://lists.gluster.org/pipermail/gluster-users/2018-March >>>>>>> /033699.html). The first two suggestions (specifying a >>>>>>> negative-timeout in the mount settings or adding >>>>>>> rpc-auth-allow-insecure to >>>>>>> glusterd.vol) did not improve our performance, while setting >>>>>>> "disperse.eager-lock off" provided a tiny (5%) speed-up. >>>>>>> >>>>>>> Some of the various tests we have tried earlier can be seen in the >>>>>>> links below. Do any of the above observations suggest what we could try >>>>>>> next to either improve the speed or debug the issue? Thanks >>>>>>> >>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-June/0 >>>>>>> 31565.html >>>>>>> http://lists.gluster.org/pipermail/gluster-users/2017-May/03 >>>>>>> 0937.html >>>>>>> >>>>>>> Pat >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>>>>>> Pat Haley Email: [email protected] >>>>>>> Center for Ocean Engineering Phone: (617) 253-6824 >>>>>>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>>>>>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>>>>>> 77 Massachusetts Avenue >>>>>>> Cambridge, MA 02139-4301 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Gluster-users mailing list >>>>>>> [email protected] >>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> -- >>> >>> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >>> Pat Haley Email: [email protected] >>> Center for Ocean Engineering Phone: (617) 253-6824 >>> Dept. of Mechanical Engineering Fax: (617) 253-8125 >>> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >>> 77 Massachusetts Avenue >>> Cambridge, MA 02139-4301 >>> >>> >>> _______________________________________________ >>> Gluster-users mailing list >>> [email protected] >>> http://lists.gluster.org/mailman/listinfo/gluster-users >>> >> >> >> -- >> >> -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- >> Pat Haley Email: [email protected] >> Center for Ocean Engineering Phone: (617) 253-6824 >> Dept. of Mechanical Engineering Fax: (617) 253-8125 >> MIT, Room 5-213 http://web.mit.edu/phaley/www/ >> 77 Massachusetts Avenue >> Cambridge, MA 02139-4301 >> >> > > -- > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Pat Haley Email: [email protected] > Center for Ocean Engineering Phone: (617) 253-6824 > Dept. of Mechanical Engineering Fax: (617) 253-8125 > MIT, Room 5-213 http://web.mit.edu/phaley/www/ > 77 Massachusetts Avenue > Cambridge, MA 02139-4301 > >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
