Hi Vijay Did you manage to look into the gluster profile logs ?
Thanks Kashif On Mon, Jun 12, 2017 at 11:40 AM, mohammad kashif <[email protected]> wrote: > Hi Vijay > > I have enabled client profiling and used this script > https://github.com/bengland2/gluster-profile-analysis/blob/ > master/gvp-client.sh to extract data. I am attaching output files. I > don't have any reference data to compare with my output. Hopefully you can > make some sense out of it. > > On Sat, Jun 10, 2017 at 10:47 AM, Vijay Bellur <[email protected]> wrote: > >> Would it be possible for you to turn on client profiling and then run du? >> Instructions for turning on client profiling can be found at [1]. Providing >> the client profile information can help us figure out where the latency >> could be stemming from. >> >> Regards, >> Vijay >> >> [1] https://gluster.readthedocs.io/en/latest/Administrator% >> 20Guide/Performance%20Testing/#client-side-profiling >> >> On Fri, Jun 9, 2017 at 7:22 PM, mohammad kashif <[email protected]> >> wrote: >> >>> Hi Vijay >>> >>> Thanks for your quick response. I am using gluster 3.8.11 on Centos 7 >>> servers >>> glusterfs-3.8.11-1.el7.x86_64 >>> >>> clients are centos 6 but I tested with a centos 7 client as well and >>> results didn't change >>> >>> gluster volume info Volume Name: atlasglust >>> Type: Distribute >>> Volume ID: fbf0ebb8-deab-4388-9d8a-f722618a624b >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 5 >>> Transport-type: tcp >>> Bricks: >>> Brick1: pplxgluster01.x.y.z:/glusteratlas/brick001/gv0 >>> Brick2: pplxgluster02..x.y.z:/glusteratlas/brick002/gv0 >>> Brick3: pplxgluster03.x.y.z:/glusteratlas/brick003/gv0 >>> Brick4: pplxgluster04.x.y.z:/glusteratlas/brick004/gv0 >>> Brick5: pplxgluster05.x.y.z:/glusteratlas/brick005/gv0 >>> Options Reconfigured: >>> nfs.disable: on >>> performance.readdir-ahead: on >>> transport.address-family: inet >>> auth.allow: x.y.z >>> >>> I am not using directory quota. >>> >>> Please let me know if you require some more info >>> >>> Thanks >>> >>> Kashif >>> >>> >>> >>> On Fri, Jun 9, 2017 at 2:34 PM, Vijay Bellur <[email protected]> wrote: >>> >>>> Can you please provide more details about your volume configuration and >>>> the version of gluster that you are using? >>>> >>>> Regards, >>>> Vijay >>>> >>>> On Fri, Jun 9, 2017 at 5:35 PM, mohammad kashif <[email protected]> >>>> wrote: >>>> >>>>> Hi >>>>> >>>>> I have just moved our 400 TB HPC storage from lustre to gluster. It is >>>>> part of a research institute and users have very small files to big files >>>>> ( few KB to 20GB) . Our setup consists of 5 servers, each with 96TB RAID 6 >>>>> disks. All servers are connected through 10G ethernet but not all clients. >>>>> Gluster volumes are distributed without any replication. There are >>>>> approximately 80 million files in file system. >>>>> I am mounting using glusterfs on clients. >>>>> >>>>> I have copied everything from lustre to gluster but old file system >>>>> exist so I can compare. >>>>> >>>>> The problem, I am facing is extremely slow du on even a small >>>>> directory. Also the time taken is substantially different each time. >>>>> I tried du from same client on a particular directory twice and got >>>>> these results. >>>>> >>>>> time du -sh /data/aa/bb/cc >>>>> 3.7G /data/aa/bb/cc >>>>> real 7m29.243s >>>>> user 0m1.448s >>>>> sys 0m7.067s >>>>> >>>>> time du -sh /data/aa/bb/cc >>>>> 3.7G /data/aa/bb/cc >>>>> real 16m43.735s >>>>> user 0m1.097s >>>>> sys 0m5.802s >>>>> >>>>> 16m and 7m is too long for a 3.7 G directory. I must mention that the >>>>> directory contains huge number of files (208736) >>>>> >>>>> but running du on same directory on old data gives this result >>>>> >>>>> time du -sh /olddata/aa/bb/cc >>>>> 4.0G /olddata/aa/bb/cc >>>>> real 3m1.255s >>>>> user 0m0.755s >>>>> sys 0m38.099s >>>>> >>>>> much better if I run same command again >>>>> >>>>> time du -sh /olddata/aa/bb/cc >>>>> 4.0G /olddata/aa/bb/cc >>>>> real 0m8.309s >>>>> user 0m0.313s >>>>> sys 0m7.755s >>>>> >>>>> Is there anything I can do to improve this performance? I would also >>>>> like hear from some one who is running same kind of setup. >>>>> >>>>> Thanks >>>>> >>>>> Kashif >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> [email protected] >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>> >> >
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
