Hi, OK, thanks. I'd also asked for gluster version you're running. Could you share that information as well?
-Krutika On Thu, Mar 7, 2019 at 9:38 PM Drew Rash <drew.r...@gmail.com> wrote: > Here is the output for our ssd gluster which exhibits the same issue as > the hdd glusters. > However, I can replicate the issue on an 8TB WD Gold disk NFS mounted as > well ( removed the gluster part ) Which is the reason I'm on the oVirt > site. I can start a file copy that writes at max speed, then after a gb or > 2 it drops down to 3-10 MBps maxing at 13.3 ish overall. > Testing outside of oVirt using dd doesn't have the same behavior. Outside > ovirt (directly on the ovirtnode to the gluster or 8tb nfs mount results is > max drive speeds consistently for large file copies) > > I enabled writeback (as someone suggested) on the virtio-scsi windows disk > and one of our windows 10 installs speed up. Still suffers from sustained > write issue which causes the whole box to cripple. Opening chrome for > example cripples the box or sql server management studio also. > > Volume Name: gv1 > Type: Replicate > Volume ID: 7340a436-d971-4d69-84f9-12a23cd76ec8 > Status: Started > Snapshot Count: 0 > Number of Bricks: 1 x (2 + 1) = 3 > Transport-type: tcp > Bricks: > Brick1: 10.30.30.121:/gluster_bricks/gv1/brick > Brick2: 10.30.30.122:/gluster_bricks/gv1/brick > Brick3: 10.30.30.123:/gluster_bricks/gv1/brick (arbiter) > Options Reconfigured: > network.ping-timeout: 30 > performance.strict-o-direct: on > storage.owner-gid: 36 > storage.owner-uid: 36 > user.cifs: off > cluster.shd-wait-qlength: 10000 > cluster.shd-max-threads: 8 > cluster.locking-scheme: granular > cluster.data-self-heal-algorithm: full > cluster.server-quorum-type: server > cluster.quorum-type: auto > cluster.eager-lock: enable > network.remote-dio: off > performance.low-prio-threads: 32 > performance.io-cache: off > performance.read-ahead: off > performance.quick-read: off > features.shard: off > cluster.granular-entry-heal: enable > transport.address-family: inet > nfs.disable: on > performance.client-io-threads: off > > > On Thu, Mar 7, 2019 at 1:00 AM Krutika Dhananjay <kdhan...@redhat.com> > wrote: > >> So from the profile, it appears the XATTROPs and FINODELKs are way higher >> than the number of WRITEs: >> >> <excerpt> >> ... >> ... >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop >> --------- ----------- ----------- ----------- ------------ >> ---- >> 0.43 384.83 us 51.00 us 65375.00 us 13632 >> FXATTROP >> 7.54 13535.70 us 225.00 us 210298.00 us 6816 >> WRITE >> 45.99 28508.86 us 7.00 us 2591280.00 us 19751 >> FINODELK >> >> ... >> ... >> </excerpt> >> >> We'd noticed something similar in our internal tests and found >> inefficiencies in gluster's eager-lock implementation. This was fixed at >> https://review.gluster.org/c/glusterfs/+/19503. >> I need the two things I asked for in the prev mail to confirm if you're >> hitting the same issue. >> >> -Krutika >> >> On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <kdhan...@redhat.com> >> wrote: >> >>> Hi, >>> >>> Could you share the following pieces of information to begin with - >>> >>> 1. output of `gluster volume info $AFFECTED_VOLUME_NAME` >>> 2. glusterfs version you're running >>> >>> -Krutika >>> >>> >>> On Sat, Mar 2, 2019 at 3:38 AM Drew R <drew.r...@gmail.com> wrote: >>> >>>> Saw some people asking for profile info. So I had started a migration >>>> from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep >>>> gluster and it's been running a while for a 100GB file thin provisioned >>>> with like 28GB actually used. Here is the profile info. I started the >>>> profiler like 5 minutes ago. The migration had been running for like >>>> 30minutes: >>>> >>>> gluster volume profile gv2 info >>>> Brick: 10.30.30.122:/gluster_bricks/gv2/brick >>>> --------------------------------------------- >>>> Cumulative Stats: >>>> Block Size: 256b+ 512b+ >>>> 1024b+ >>>> No. of Reads: 1189 8 >>>> 12 >>>> No. of Writes: 4 3245 >>>> 883 >>>> >>>> Block Size: 2048b+ 4096b+ >>>> 8192b+ >>>> No. of Reads: 10 20 >>>> 2 >>>> No. of Writes: 1087 312228 >>>> 124080 >>>> >>>> Block Size: 16384b+ 32768b+ >>>> 65536b+ >>>> No. of Reads: 0 1 >>>> 52 >>>> No. of Writes: 5188 3617 >>>> 5532 >>>> >>>> Block Size: 131072b+ >>>> No. of Reads: 70191 >>>> No. of Writes: 634192 >>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>>> Fop >>>> --------- ----------- ----------- ----------- ------------ >>>> ---- >>>> 0.00 0.00 us 0.00 us 0.00 us 2 >>>> FORGET >>>> 0.00 0.00 us 0.00 us 0.00 us 202 >>>> RELEASE >>>> 0.00 0.00 us 0.00 us 0.00 us 1297 >>>> RELEASEDIR >>>> 0.00 14.50 us 9.00 us 20.00 us 4 >>>> READDIR >>>> 0.00 38.00 us 9.00 us 120.00 us 7 >>>> GETXATTR >>>> 0.00 66.00 us 34.00 us 128.00 us 6 >>>> OPEN >>>> 0.00 137.25 us 52.00 us 195.00 us 4 >>>> SETATTR >>>> 0.00 23.19 us 11.00 us 46.00 us 26 >>>> INODELK >>>> 0.00 41.58 us 18.00 us 79.00 us 24 >>>> OPENDIR >>>> 0.00 166.70 us 15.00 us 775.00 us 27 >>>> READDIRP >>>> 0.01 135.29 us 12.00 us 11695.00 us 221 >>>> STATFS >>>> 0.01 176.54 us 22.00 us 22944.00 us 364 >>>> FSTAT >>>> 0.02 626.21 us 13.00 us 17308.00 us 168 >>>> STAT >>>> 0.09 834.84 us 9.00 us 34337.00 us 607 >>>> LOOKUP >>>> 0.73 146.18 us 6.00 us 52255.00 us 29329 >>>> FINODELK >>>> 1.03 298.20 us 42.00 us 43711.00 us 20204 >>>> FXATTROP >>>> 15.38 8903.40 us 213.00 us 213832.00 us 10102 >>>> WRITE >>>> 39.14 26796.37 us 222.00 us 122696.00 us 8538 >>>> READ >>>> 43.59 39536.79 us 259.00 us 183630.00 us 6446 >>>> FSYNC >>>> >>>> Duration: 15078 seconds >>>> Data Read: 9207377205 bytes >>>> Data Written: 86214017762 bytes >>>> >>>> Interval 2 Stats: >>>> Block Size: 256b+ 512b+ >>>> 1024b+ >>>> No. of Reads: 17 0 >>>> 0 >>>> No. of Writes: 0 43 >>>> 7 >>>> >>>> Block Size: 2048b+ 4096b+ >>>> 8192b+ >>>> No. of Reads: 0 7 >>>> 0 >>>> No. of Writes: 16 1881 >>>> 1010 >>>> >>>> Block Size: 16384b+ 32768b+ >>>> 65536b+ >>>> No. of Reads: 0 0 >>>> 6 >>>> No. of Writes: 305 586 >>>> 2359 >>>> >>>> Block Size: 131072b+ >>>> No. of Reads: 7162 >>>> No. of Writes: 610 >>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>>> Fop >>>> --------- ----------- ----------- ----------- ------------ >>>> ---- >>>> 0.00 0.00 us 0.00 us 0.00 us 6 >>>> RELEASE >>>> 0.00 0.00 us 0.00 us 0.00 us 20 >>>> RELEASEDIR >>>> 0.00 14.50 us 9.00 us 20.00 us 4 >>>> READDIR >>>> 0.00 38.00 us 9.00 us 120.00 us 7 >>>> GETXATTR >>>> 0.00 66.00 us 34.00 us 128.00 us 6 >>>> OPEN >>>> 0.00 137.25 us 52.00 us 195.00 us 4 >>>> SETATTR >>>> 0.00 23.19 us 11.00 us 46.00 us 26 >>>> INODELK >>>> 0.00 40.05 us 18.00 us 79.00 us 20 >>>> OPENDIR >>>> 0.00 180.33 us 16.00 us 775.00 us 21 >>>> READDIRP >>>> 0.01 181.77 us 12.00 us 11695.00 us 149 >>>> STATFS >>>> 0.01 511.23 us 15.00 us 16954.00 us 111 >>>> STAT >>>> 0.01 210.22 us 22.00 us 22944.00 us 291 >>>> FSTAT >>>> 0.09 979.73 us 9.00 us 23539.00 us 404 >>>> LOOKUP >>>> 0.81 179.58 us 6.00 us 52255.00 us 19786 >>>> FINODELK >>>> 1.07 342.71 us 42.00 us 43711.00 us 13633 >>>> FXATTROP >>>> 14.18 9087.30 us 213.00 us 213832.00 us 6817 >>>> WRITE >>>> 40.00 40227.38 us 259.00 us 157934.00 us 4343 >>>> FSYNC >>>> 43.81 26601.57 us 222.00 us 118756.00 us 7192 >>>> READ >>>> >>>> Duration: 177 seconds >>>> Data Read: 939534945 bytes >>>> Data Written: 363763200 bytes >>>> >>>> Brick: 10.30.30.121:/gluster_bricks/gv2/brick >>>> --------------------------------------------- >>>> Cumulative Stats: >>>> Block Size: 8b+ 256b+ >>>> 512b+ >>>> No. of Reads: 0 142626 >>>> 329092 >>>> No. of Writes: 12 98 >>>> 123230 >>>> >>>> Block Size: 1024b+ 2048b+ >>>> 4096b+ >>>> No. of Reads: 21175 18223 >>>> 3345043 >>>> No. of Writes: 58646 69147 >>>> 6092143 >>>> >>>> Block Size: 8192b+ 16384b+ >>>> 32768b+ >>>> No. of Reads: 1090937 216037 >>>> 748595 >>>> No. of Writes: 2030403 243915 >>>> 240942 >>>> >>>> Block Size: 65536b+ 131072b+ >>>> No. of Reads: 1524667 4828662 >>>> No. of Writes: 605125 1889892 >>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>>> Fop >>>> --------- ----------- ----------- ----------- ------------ >>>> ---- >>>> 0.00 0.00 us 0.00 us 0.00 us 133 >>>> FORGET >>>> 0.00 0.00 us 0.00 us 0.00 us 4821 >>>> RELEASE >>>> 0.00 0.00 us 0.00 us 0.00 us 42742 >>>> RELEASEDIR >>>> 0.00 28.50 us 12.00 us 61.00 us 4 >>>> READDIR >>>> 0.00 71.75 us 59.00 us 88.00 us 4 >>>> SETATTR >>>> 0.00 44.57 us 12.00 us 94.00 us 7 >>>> GETXATTR >>>> 0.00 62.17 us 42.00 us 89.00 us 6 >>>> OPEN >>>> 0.00 35.92 us 1.00 us 76.00 us 24 >>>> OPENDIR >>>> 0.00 325.54 us 22.00 us 1117.00 us 13 >>>> READDIRP >>>> 0.00 482.13 us 19.00 us 12473.00 us 46 >>>> STAT >>>> 0.00 217.65 us 15.00 us 32841.00 us 221 >>>> STATFS >>>> 0.01 497.52 us 11.00 us 29179.00 us 539 >>>> LOOKUP >>>> 0.02 685.44 us 16.00 us 61131.00 us 615 >>>> FSTAT >>>> 0.04 31691.92 us 17.00 us 349648.00 us 26 >>>> INODELK >>>> 0.45 451.95 us 51.00 us 65375.00 us 20206 >>>> FXATTROP >>>> 6.90 13906.03 us 225.00 us 210298.00 us 10103 >>>> WRITE >>>> 18.47 58375.96 us 389.00 us 218211.00 us 6446 >>>> FSYNC >>>> 27.45 27279.84 us 184.00 us 241444.00 us 20503 >>>> READ >>>> 46.66 32462.82 us 7.00 us 2750519.00 us 29283 >>>> FINODELK >>>> >>>> Duration: 475152 seconds >>>> Data Read: 799630169437 bytes >>>> Data Written: 362708560796 bytes >>>> >>>> Interval 2 Stats: >>>> Block Size: 256b+ 512b+ >>>> 1024b+ >>>> No. of Reads: 53 0 >>>> 0 >>>> No. of Writes: 0 43 >>>> 7 >>>> >>>> Block Size: 2048b+ 4096b+ >>>> 8192b+ >>>> No. of Reads: 0 891 >>>> 616 >>>> No. of Writes: 16 1881 >>>> 1010 >>>> >>>> Block Size: 16384b+ 32768b+ >>>> 65536b+ >>>> No. of Reads: 288 750 >>>> 52 >>>> No. of Writes: 304 586 >>>> 2359 >>>> >>>> Block Size: 131072b+ >>>> No. of Reads: 9445 >>>> No. of Writes: 610 >>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>>> Fop >>>> --------- ----------- ----------- ----------- ------------ >>>> ---- >>>> 0.00 0.00 us 0.00 us 0.00 us 6 >>>> RELEASE >>>> 0.00 0.00 us 0.00 us 0.00 us 20 >>>> RELEASEDIR >>>> 0.00 28.50 us 12.00 us 61.00 us 4 >>>> READDIR >>>> 0.00 71.75 us 59.00 us 88.00 us 4 >>>> SETATTR >>>> 0.00 44.57 us 12.00 us 94.00 us 7 >>>> GETXATTR >>>> 0.00 62.17 us 42.00 us 89.00 us 6 >>>> OPEN >>>> 0.00 34.50 us 1.00 us 76.00 us 20 >>>> OPENDIR >>>> 0.00 55.16 us 19.00 us 128.00 us 25 >>>> STAT >>>> 0.00 363.18 us 25.00 us 1117.00 us 11 >>>> READDIRP >>>> 0.00 71.61 us 15.00 us 4323.00 us 149 >>>> STATFS >>>> 0.01 251.40 us 11.00 us 15344.00 us 336 >>>> LOOKUP >>>> 0.02 594.11 us 16.00 us 61131.00 us 450 >>>> FSTAT >>>> 0.07 31691.92 us 17.00 us 349648.00 us 26 >>>> INODELK >>>> 0.43 384.83 us 51.00 us 65375.00 us 13632 >>>> FXATTROP >>>> 7.54 13535.70 us 225.00 us 210298.00 us 6816 >>>> WRITE >>>> 20.03 56497.17 us 394.00 us 218211.00 us 4340 >>>> FSYNC >>>> 25.91 26229.86 us 184.00 us 241444.00 us 12095 >>>> READ >>>> 45.99 28508.86 us 7.00 us 2591280.00 us 19751 >>>> FINODELK >>>> >>>> Duration: 177 seconds >>>> Data Read: 1298631621 bytes >>>> Data Written: 363744256 bytes >>>> >>>> Brick: 10.30.30.123:/gluster_bricks/gv2/brick >>>> --------------------------------------------- >>>> Cumulative Stats: >>>> Block Size: 1b+ >>>> No. of Reads: 0 >>>> No. of Writes: 18126558 >>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>>> Fop >>>> --------- ----------- ----------- ----------- ------------ >>>> ---- >>>> 0.00 0.00 us 0.00 us 0.00 us 383 >>>> FORGET >>>> 0.00 0.00 us 0.00 us 0.00 us 9602 >>>> RELEASE >>>> 0.00 0.00 us 0.00 us 0.00 us 70644 >>>> RELEASEDIR >>>> 0.00 20.75 us 13.00 us 29.00 us 4 >>>> READDIR >>>> 0.00 87.00 us 72.00 us 129.00 us 4 >>>> SETATTR >>>> 0.00 71.00 us 46.00 us 93.00 us 6 >>>> OPEN >>>> 0.00 66.00 us 9.00 us 114.00 us 7 >>>> GETXATTR >>>> 0.00 50.58 us 16.00 us 90.00 us 26 >>>> INODELK >>>> 0.00 56.38 us 1.00 us 111.00 us 24 >>>> OPENDIR >>>> 0.02 66.23 us 28.00 us 146.00 us 113 >>>> FSTAT >>>> 0.17 143.88 us 13.00 us 444.00 us 539 >>>> LOOKUP >>>> 1.17 51.64 us 10.00 us 166.00 us 10103 >>>> WRITE >>>> 2.89 44.06 us 7.00 us 2177.00 us 29349 >>>> FINODELK >>>> 5.55 122.83 us 54.00 us 9281.00 us 20206 >>>> FXATTROP >>>> 90.19 6249.64 us 237.00 us 59996.00 us 6448 >>>> FSYNC >>>> >>>> Duration: 772792 seconds >>>> Data Read: 0 bytes >>>> Data Written: 18126558 bytes >>>> >>>> Interval 2 Stats: >>>> Block Size: 1b+ >>>> No. of Reads: 0 >>>> No. of Writes: 6816 >>>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>>> Fop >>>> --------- ----------- ----------- ----------- ------------ >>>> ---- >>>> 0.00 0.00 us 0.00 us 0.00 us 6 >>>> RELEASE >>>> 0.00 0.00 us 0.00 us 0.00 us 20 >>>> RELEASEDIR >>>> 0.00 20.75 us 13.00 us 29.00 us 4 >>>> READDIR >>>> 0.00 87.00 us 72.00 us 129.00 us 4 >>>> SETATTR >>>> 0.00 71.00 us 46.00 us 93.00 us 6 >>>> OPEN >>>> 0.00 66.00 us 9.00 us 114.00 us 7 >>>> GETXATTR >>>> 0.00 58.10 us 1.00 us 111.00 us 20 >>>> OPENDIR >>>> 0.00 50.58 us 16.00 us 90.00 us 26 >>>> INODELK >>>> 0.03 66.23 us 28.00 us 146.00 us 113 >>>> FSTAT >>>> 0.16 145.84 us 13.00 us 346.00 us 336 >>>> LOOKUP >>>> 1.18 51.98 us 11.00 us 157.00 us 6816 >>>> WRITE >>>> 2.91 44.05 us 7.00 us 2177.00 us 19798 >>>> FINODELK >>>> 5.59 122.70 us 54.00 us 9281.00 us 13632 >>>> FXATTROP >>>> 90.11 6210.61 us 238.00 us 54711.00 us 4342 >>>> FSYNC >>>> >>>> Duration: 177 seconds >>>> Data Read: 0 bytes >>>> Data Written: 6816 bytes >>>> _______________________________________________ >>>> Users mailing list -- users@ovirt.org >>>> To unsubscribe send an email to users-le...@ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/XHAUIMGRCVEDCLS7F6WALUWJVQ4IXAWU/ >>>> >>>
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZNGJXLFKNPAPHY4Q3N3S4E4KCBG5CJPO/
