So from the profile, it appears the XATTROPs and FINODELKs are way higher
than the number of WRITEs:
<excerpt>
...
...
%-latency Avg-latency Min-Latency Max-Latency No. of calls
Fop
--------- ----------- ----------- ----------- ------------
----
0.43 384.83 us 51.00 us 65375.00 us 13632
FXATTROP
7.54 13535.70 us 225.00 us 210298.00 us 6816
WRITE
45.99 28508.86 us 7.00 us 2591280.00 us 19751
FINODELK
...
...
</excerpt>
We'd noticed something similar in our internal tests and found
inefficiencies in gluster's eager-lock implementation. This was fixed at
https://review.gluster.org/c/glusterfs/+/19503.
I need the two things I asked for in the prev mail to confirm if you're
hitting the same issue.
-Krutika
On Thu, Mar 7, 2019 at 12:24 PM Krutika Dhananjay <[email protected]>
wrote:
> Hi,
>
> Could you share the following pieces of information to begin with -
>
> 1. output of `gluster volume info $AFFECTED_VOLUME_NAME`
> 2. glusterfs version you're running
>
> -Krutika
>
>
> On Sat, Mar 2, 2019 at 3:38 AM Drew R <[email protected]> wrote:
>
>> Saw some people asking for profile info. So I had started a migration
>> from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep
>> gluster and it's been running a while for a 100GB file thin provisioned
>> with like 28GB actually used. Here is the profile info. I started the
>> profiler like 5 minutes ago. The migration had been running for like
>> 30minutes:
>>
>> gluster volume profile gv2 info
>> Brick: 10.30.30.122:/gluster_bricks/gv2/brick
>> ---------------------------------------------
>> Cumulative Stats:
>> Block Size: 256b+ 512b+
>> 1024b+
>> No. of Reads: 1189 8
>> 12
>> No. of Writes: 4 3245
>> 883
>>
>> Block Size: 2048b+ 4096b+
>> 8192b+
>> No. of Reads: 10 20
>> 2
>> No. of Writes: 1087 312228
>> 124080
>>
>> Block Size: 16384b+ 32768b+
>> 65536b+
>> No. of Reads: 0 1
>> 52
>> No. of Writes: 5188 3617
>> 5532
>>
>> Block Size: 131072b+
>> No. of Reads: 70191
>> No. of Writes: 634192
>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>> Fop
>> --------- ----------- ----------- ----------- ------------
>> ----
>> 0.00 0.00 us 0.00 us 0.00 us 2
>> FORGET
>> 0.00 0.00 us 0.00 us 0.00 us 202
>> RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us 1297
>> RELEASEDIR
>> 0.00 14.50 us 9.00 us 20.00 us 4
>> READDIR
>> 0.00 38.00 us 9.00 us 120.00 us 7
>> GETXATTR
>> 0.00 66.00 us 34.00 us 128.00 us 6
>> OPEN
>> 0.00 137.25 us 52.00 us 195.00 us 4
>> SETATTR
>> 0.00 23.19 us 11.00 us 46.00 us 26
>> INODELK
>> 0.00 41.58 us 18.00 us 79.00 us 24
>> OPENDIR
>> 0.00 166.70 us 15.00 us 775.00 us 27
>> READDIRP
>> 0.01 135.29 us 12.00 us 11695.00 us 221
>> STATFS
>> 0.01 176.54 us 22.00 us 22944.00 us 364
>> FSTAT
>> 0.02 626.21 us 13.00 us 17308.00 us 168
>> STAT
>> 0.09 834.84 us 9.00 us 34337.00 us 607
>> LOOKUP
>> 0.73 146.18 us 6.00 us 52255.00 us 29329
>> FINODELK
>> 1.03 298.20 us 42.00 us 43711.00 us 20204
>> FXATTROP
>> 15.38 8903.40 us 213.00 us 213832.00 us 10102
>> WRITE
>> 39.14 26796.37 us 222.00 us 122696.00 us 8538
>> READ
>> 43.59 39536.79 us 259.00 us 183630.00 us 6446
>> FSYNC
>>
>> Duration: 15078 seconds
>> Data Read: 9207377205 bytes
>> Data Written: 86214017762 bytes
>>
>> Interval 2 Stats:
>> Block Size: 256b+ 512b+
>> 1024b+
>> No. of Reads: 17 0
>> 0
>> No. of Writes: 0 43
>> 7
>>
>> Block Size: 2048b+ 4096b+
>> 8192b+
>> No. of Reads: 0 7
>> 0
>> No. of Writes: 16 1881
>> 1010
>>
>> Block Size: 16384b+ 32768b+
>> 65536b+
>> No. of Reads: 0 0
>> 6
>> No. of Writes: 305 586
>> 2359
>>
>> Block Size: 131072b+
>> No. of Reads: 7162
>> No. of Writes: 610
>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>> Fop
>> --------- ----------- ----------- ----------- ------------
>> ----
>> 0.00 0.00 us 0.00 us 0.00 us 6
>> RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us 20
>> RELEASEDIR
>> 0.00 14.50 us 9.00 us 20.00 us 4
>> READDIR
>> 0.00 38.00 us 9.00 us 120.00 us 7
>> GETXATTR
>> 0.00 66.00 us 34.00 us 128.00 us 6
>> OPEN
>> 0.00 137.25 us 52.00 us 195.00 us 4
>> SETATTR
>> 0.00 23.19 us 11.00 us 46.00 us 26
>> INODELK
>> 0.00 40.05 us 18.00 us 79.00 us 20
>> OPENDIR
>> 0.00 180.33 us 16.00 us 775.00 us 21
>> READDIRP
>> 0.01 181.77 us 12.00 us 11695.00 us 149
>> STATFS
>> 0.01 511.23 us 15.00 us 16954.00 us 111
>> STAT
>> 0.01 210.22 us 22.00 us 22944.00 us 291
>> FSTAT
>> 0.09 979.73 us 9.00 us 23539.00 us 404
>> LOOKUP
>> 0.81 179.58 us 6.00 us 52255.00 us 19786
>> FINODELK
>> 1.07 342.71 us 42.00 us 43711.00 us 13633
>> FXATTROP
>> 14.18 9087.30 us 213.00 us 213832.00 us 6817
>> WRITE
>> 40.00 40227.38 us 259.00 us 157934.00 us 4343
>> FSYNC
>> 43.81 26601.57 us 222.00 us 118756.00 us 7192
>> READ
>>
>> Duration: 177 seconds
>> Data Read: 939534945 bytes
>> Data Written: 363763200 bytes
>>
>> Brick: 10.30.30.121:/gluster_bricks/gv2/brick
>> ---------------------------------------------
>> Cumulative Stats:
>> Block Size: 8b+ 256b+
>> 512b+
>> No. of Reads: 0 142626
>> 329092
>> No. of Writes: 12 98
>> 123230
>>
>> Block Size: 1024b+ 2048b+
>> 4096b+
>> No. of Reads: 21175 18223
>> 3345043
>> No. of Writes: 58646 69147
>> 6092143
>>
>> Block Size: 8192b+ 16384b+
>> 32768b+
>> No. of Reads: 1090937 216037
>> 748595
>> No. of Writes: 2030403 243915
>> 240942
>>
>> Block Size: 65536b+ 131072b+
>> No. of Reads: 1524667 4828662
>> No. of Writes: 605125 1889892
>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>> Fop
>> --------- ----------- ----------- ----------- ------------
>> ----
>> 0.00 0.00 us 0.00 us 0.00 us 133
>> FORGET
>> 0.00 0.00 us 0.00 us 0.00 us 4821
>> RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us 42742
>> RELEASEDIR
>> 0.00 28.50 us 12.00 us 61.00 us 4
>> READDIR
>> 0.00 71.75 us 59.00 us 88.00 us 4
>> SETATTR
>> 0.00 44.57 us 12.00 us 94.00 us 7
>> GETXATTR
>> 0.00 62.17 us 42.00 us 89.00 us 6
>> OPEN
>> 0.00 35.92 us 1.00 us 76.00 us 24
>> OPENDIR
>> 0.00 325.54 us 22.00 us 1117.00 us 13
>> READDIRP
>> 0.00 482.13 us 19.00 us 12473.00 us 46
>> STAT
>> 0.00 217.65 us 15.00 us 32841.00 us 221
>> STATFS
>> 0.01 497.52 us 11.00 us 29179.00 us 539
>> LOOKUP
>> 0.02 685.44 us 16.00 us 61131.00 us 615
>> FSTAT
>> 0.04 31691.92 us 17.00 us 349648.00 us 26
>> INODELK
>> 0.45 451.95 us 51.00 us 65375.00 us 20206
>> FXATTROP
>> 6.90 13906.03 us 225.00 us 210298.00 us 10103
>> WRITE
>> 18.47 58375.96 us 389.00 us 218211.00 us 6446
>> FSYNC
>> 27.45 27279.84 us 184.00 us 241444.00 us 20503
>> READ
>> 46.66 32462.82 us 7.00 us 2750519.00 us 29283
>> FINODELK
>>
>> Duration: 475152 seconds
>> Data Read: 799630169437 bytes
>> Data Written: 362708560796 bytes
>>
>> Interval 2 Stats:
>> Block Size: 256b+ 512b+
>> 1024b+
>> No. of Reads: 53 0
>> 0
>> No. of Writes: 0 43
>> 7
>>
>> Block Size: 2048b+ 4096b+
>> 8192b+
>> No. of Reads: 0 891
>> 616
>> No. of Writes: 16 1881
>> 1010
>>
>> Block Size: 16384b+ 32768b+
>> 65536b+
>> No. of Reads: 288 750
>> 52
>> No. of Writes: 304 586
>> 2359
>>
>> Block Size: 131072b+
>> No. of Reads: 9445
>> No. of Writes: 610
>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>> Fop
>> --------- ----------- ----------- ----------- ------------
>> ----
>> 0.00 0.00 us 0.00 us 0.00 us 6
>> RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us 20
>> RELEASEDIR
>> 0.00 28.50 us 12.00 us 61.00 us 4
>> READDIR
>> 0.00 71.75 us 59.00 us 88.00 us 4
>> SETATTR
>> 0.00 44.57 us 12.00 us 94.00 us 7
>> GETXATTR
>> 0.00 62.17 us 42.00 us 89.00 us 6
>> OPEN
>> 0.00 34.50 us 1.00 us 76.00 us 20
>> OPENDIR
>> 0.00 55.16 us 19.00 us 128.00 us 25
>> STAT
>> 0.00 363.18 us 25.00 us 1117.00 us 11
>> READDIRP
>> 0.00 71.61 us 15.00 us 4323.00 us 149
>> STATFS
>> 0.01 251.40 us 11.00 us 15344.00 us 336
>> LOOKUP
>> 0.02 594.11 us 16.00 us 61131.00 us 450
>> FSTAT
>> 0.07 31691.92 us 17.00 us 349648.00 us 26
>> INODELK
>> 0.43 384.83 us 51.00 us 65375.00 us 13632
>> FXATTROP
>> 7.54 13535.70 us 225.00 us 210298.00 us 6816
>> WRITE
>> 20.03 56497.17 us 394.00 us 218211.00 us 4340
>> FSYNC
>> 25.91 26229.86 us 184.00 us 241444.00 us 12095
>> READ
>> 45.99 28508.86 us 7.00 us 2591280.00 us 19751
>> FINODELK
>>
>> Duration: 177 seconds
>> Data Read: 1298631621 bytes
>> Data Written: 363744256 bytes
>>
>> Brick: 10.30.30.123:/gluster_bricks/gv2/brick
>> ---------------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> No. of Reads: 0
>> No. of Writes: 18126558
>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>> Fop
>> --------- ----------- ----------- ----------- ------------
>> ----
>> 0.00 0.00 us 0.00 us 0.00 us 383
>> FORGET
>> 0.00 0.00 us 0.00 us 0.00 us 9602
>> RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us 70644
>> RELEASEDIR
>> 0.00 20.75 us 13.00 us 29.00 us 4
>> READDIR
>> 0.00 87.00 us 72.00 us 129.00 us 4
>> SETATTR
>> 0.00 71.00 us 46.00 us 93.00 us 6
>> OPEN
>> 0.00 66.00 us 9.00 us 114.00 us 7
>> GETXATTR
>> 0.00 50.58 us 16.00 us 90.00 us 26
>> INODELK
>> 0.00 56.38 us 1.00 us 111.00 us 24
>> OPENDIR
>> 0.02 66.23 us 28.00 us 146.00 us 113
>> FSTAT
>> 0.17 143.88 us 13.00 us 444.00 us 539
>> LOOKUP
>> 1.17 51.64 us 10.00 us 166.00 us 10103
>> WRITE
>> 2.89 44.06 us 7.00 us 2177.00 us 29349
>> FINODELK
>> 5.55 122.83 us 54.00 us 9281.00 us 20206
>> FXATTROP
>> 90.19 6249.64 us 237.00 us 59996.00 us 6448
>> FSYNC
>>
>> Duration: 772792 seconds
>> Data Read: 0 bytes
>> Data Written: 18126558 bytes
>>
>> Interval 2 Stats:
>> Block Size: 1b+
>> No. of Reads: 0
>> No. of Writes: 6816
>> %-latency Avg-latency Min-Latency Max-Latency No. of calls
>> Fop
>> --------- ----------- ----------- ----------- ------------
>> ----
>> 0.00 0.00 us 0.00 us 0.00 us 6
>> RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us 20
>> RELEASEDIR
>> 0.00 20.75 us 13.00 us 29.00 us 4
>> READDIR
>> 0.00 87.00 us 72.00 us 129.00 us 4
>> SETATTR
>> 0.00 71.00 us 46.00 us 93.00 us 6
>> OPEN
>> 0.00 66.00 us 9.00 us 114.00 us 7
>> GETXATTR
>> 0.00 58.10 us 1.00 us 111.00 us 20
>> OPENDIR
>> 0.00 50.58 us 16.00 us 90.00 us 26
>> INODELK
>> 0.03 66.23 us 28.00 us 146.00 us 113
>> FSTAT
>> 0.16 145.84 us 13.00 us 346.00 us 336
>> LOOKUP
>> 1.18 51.98 us 11.00 us 157.00 us 6816
>> WRITE
>> 2.91 44.05 us 7.00 us 2177.00 us 19798
>> FINODELK
>> 5.59 122.70 us 54.00 us 9281.00 us 13632
>> FXATTROP
>> 90.11 6210.61 us 238.00 us 54711.00 us 4342
>> FSYNC
>>
>> Duration: 177 seconds
>> Data Read: 0 bytes
>> Data Written: 6816 bytes
>> _______________________________________________
>> Users mailing list -- [email protected]
>> To unsubscribe send an email to [email protected]
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/[email protected]/message/XHAUIMGRCVEDCLS7F6WALUWJVQ4IXAWU/
>>
>
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/CDUJZB4WB3F6VU5ZQLKL4CZPN4YU3TTW/
