Hi
My tests on 3.5.1qa4 are doing well, but performance seems weak. Here is gluster
volume profile information at the end of this message.
If I understand correctly, I have INODELK and LOOKUP operations that take more
than one second, which seems huge. However, the average is still reasonable,
therefore I wonder if it is really significant. How can I investigate this
further?
It may be related: on bricks that have slow LOOKUP, glustershd.log gets filled
with entries like this:
[2013-12-24 05:10:18.562322] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk]
0-gfs351-client-2: remote operation failed: Stale NFS file handle. Path:
<gfid:fe59ae94-f6a4-4b28-a279-4b5b8d75c936>
(fe59ae94-f6a4-4b28-a279-4b5b8d75c936). Key: glusterfs.gfid2path
[2013-12-24 05:10:18.564198] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk]
0-gfs351-client-2: remote operation failed: Stale NFS file handle. Path:
<gfid:0b70f051-df96-4a53-94a8-bc2c340a77af>
(0b70f051-df96-4a53-94a8-bc2c340a77af). Key: glusterfs.gfid2path
[2013-12-24 05:10:18.566070] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk]
0-gfs351-client-2: remote operation failed: Stale NFS file handle. Path:
<gfid:f822879c-29d4-425a-ad8b-023bb5e51f6c>
(f822879c-29d4-425a-ad8b-023bb5e51f6c). Key: glusterfs.gfid2path
And the brick log loops on the same kind of errors:
[2013-12-24 05:15:21.477211] I [server-rpc-fops.c:776:server_getxattr_cbk]
0-gfs351-server: 18770957: GETXATTR (null)
(d93d337b-3d61-4a80-9516-3ba379ed2ab7) (glusterfs.gfid2path) ==> (Stale NFS file
handle)
[2013-12-24 05:15:21.477663] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 18770958: LOOKUP (null) (0438b8f0-2268-4f51-b014-13b70cdfd555)
==> (Stale NFS file handle)
[2013-12-24 05:15:21.478267] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 9014209: LOOKUP (null) (5379c125-b0dd-4636-8b7f-880247a6368f)
==> (Stale NFS file handle)
[2013-12-24 05:15:21.479118] I [server-rpc-fops.c:776:server_getxattr_cbk]
0-gfs351-server: 18770959: GETXATTR (null)
(0438b8f0-2268-4f51-b014-13b70cdfd555) (glusterfs.gfid2path) ==> (Stale NFS file
handle)
[2013-12-24 05:15:21.479546] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 18770960: LOOKUP (null) (bf230ee9-6942-4eb0-a7b4-c9707ce31c19)
==> (Stale NFS file handle)
[2013-12-24 05:15:21.480171] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 9014210: LOOKUP (null) (9dcb2827-21c4-4dab-b55e-a0102efe4456)
==> (Stale NFS file handle)
What does it means?
gluster volume profile information:
Brick: silo:/export/wd2a
------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 7 26
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 0 0
No. of Writes: 517 226 396
Block Size: 64b+ 128b+ 256b+
No. of Reads: 166 391 292
No. of Writes: 523 861 1486
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 137 172 161
No. of Writes: 7074 8974 11592
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1303 1331 1380
No. of Writes: 6597 4380 3152
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1198 12077 2574709
No. of Writes: 3846 619 940
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 28171 FORGET
0.00 0.00 us 0.00 us 0.00 us 67177 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.00 30.00 us 30.00 us 30.00 us 2 STAT
0.00 69.00 us 69.00 us 69.00 us 1 FSTAT
0.00 111.00 us 111.00 us 111.00 us 1 GETXATTR
0.00 43.00 us 40.00 us 45.00 us 3 READLINK
0.01 38.75 us 23.00 us 68.00 us 4 ENTRYLK
0.01 47.50 us 33.00 us 76.00 us 4 STATFS
0.01 65.25 us 36.00 us 98.00 us 4 FTRUNCATE
0.03 43.56 us 23.00 us 97.00 us 16 FINODELK
0.06 413.25 us 335.00 us 493.00 us 4 FSYNC
0.06 432.50 us 39.00 us 1589.00 us 4 WRITE
0.47 40.28 us 21.00 us 322.00 us 314 INODELK
0.62 8402.00 us 1877.00 us 14927.00 us 2 CREATE
0.73 148.71 us 28.00 us 381.00 us 132 READ
3.84 6478.19 us 2798.00 us 23951.00 us 16 FXATTROP
34.45 221.39 us 79.00 us 296105.00 us 4195 LOOKUP
59.70 10253.01 us 59.00 us 392512.00 us 157 SETATTR
Duration: 402686 seconds
Data Read: 338773217672 bytes
Data Written: 502853550 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 7 26
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 0 0
No. of Writes: 517 226 396
Block Size: 64b+ 128b+ 256b+
No. of Reads: 166 391 292
No. of Writes: 523 861 1486
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 137 172 161
No. of Writes: 7074 8974 11592
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1303 1331 1380
No. of Writes: 6597 4380 3152
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1198 12077 2574709
No. of Writes: 3846 619 940
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 28171 FORGET
0.00 0.00 us 0.00 us 0.00 us 67177 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.00 30.00 us 30.00 us 30.00 us 2 STAT
0.00 69.00 us 69.00 us 69.00 us 1 FSTAT
0.00 111.00 us 111.00 us 111.00 us 1 GETXATTR
0.00 43.00 us 40.00 us 45.00 us 3 READLINK
0.01 38.75 us 23.00 us 68.00 us 4 ENTRYLK
0.01 47.50 us 33.00 us 76.00 us 4 STATFS
0.01 65.25 us 36.00 us 98.00 us 4 FTRUNCATE
0.03 43.56 us 23.00 us 97.00 us 16 FINODELK
0.06 413.25 us 335.00 us 493.00 us 4 FSYNC
0.06 432.50 us 39.00 us 1589.00 us 4 WRITE
0.47 40.28 us 21.00 us 322.00 us 314 INODELK
0.62 8402.00 us 1877.00 us 14927.00 us 2 CREATE
0.73 148.71 us 28.00 us 381.00 us 132 READ
3.84 6478.19 us 2798.00 us 23951.00 us 16 FXATTROP
34.45 221.39 us 79.00 us 296105.00 us 4195 LOOKUP
59.70 10253.01 us 59.00 us 392512.00 us 157 SETATTR
Duration: 402686 seconds
Data Read: 338773217672 bytes
Data Written: 502853550 bytes
Brick: debacle:/export/wd1a
---------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 1 0 0
No. of Writes: 3 12 29
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 3 3
No. of Writes: 549 388 543
Block Size: 64b+ 128b+ 256b+
No. of Reads: 188 478 270
No. of Writes: 686 1264 2242
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 141 237 158
No. of Writes: 8500 11940 17533
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1357 1331 3030
No. of Writes: 6858 4259 3119
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1652 5581 44543
No. of Writes: 3610 629 1132
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 28094 FORGET
0.00 0.00 us 0.00 us 0.00 us 61813 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.00 32.00 us 32.00 us 32.00 us 1 GETXATTR
0.00 22.50 us 21.00 us 24.00 us 2 STAT
0.00 39.00 us 37.00 us 41.00 us 2 WRITE
0.00 20.75 us 17.00 us 24.00 us 4 ENTRYLK
0.00 17.75 us 13.00 us 21.00 us 8 FINODELK
0.00 73.00 us 41.00 us 105.00 us 2 FTRUNCATE
0.00 54.75 us 20.00 us 155.00 us 4 STATFS
0.00 169.50 us 164.00 us 175.00 us 2 FSYNC
0.00 50.38 us 44.00 us 58.00 us 8 FSTAT
0.01 101.32 us 91.00 us 123.00 us 44 READDIR
0.06 3362.38 us 1069.00 us 17592.00 us 8 FXATTROP
0.19 328.27 us 45.00 us 62554.00 us 249 SETATTR
0.67 140898.50 us 96307.00 us 185490.00 us 2 CREATE
8.15 6867.40 us 11.00 us 1675137.00 us 499 INODELK
90.91 6765.62 us 57.00 us 3022228.00 us 5649 LOOKUP
Duration: 402683 seconds
Data Read: 6536179250 bytes
Data Written: 545243668 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 1 0 0
No. of Writes: 3 12 29
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 3 3
No. of Writes: 549 388 543
Block Size: 64b+ 128b+ 256b+
No. of Reads: 188 478 270
No. of Writes: 686 1264 2242
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 141 237 158
No. of Writes: 8500 11940 17533
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1357 1331 3030
No. of Writes: 6858 4259 3119
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1652 5581 44543
No. of Writes: 3610 629 1132
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 28094 FORGET
0.00 0.00 us 0.00 us 0.00 us 61813 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.00 32.00 us 32.00 us 32.00 us 1 GETXATTR
0.00 22.50 us 21.00 us 24.00 us 2 STAT
0.00 39.00 us 37.00 us 41.00 us 2 WRITE
0.00 20.75 us 17.00 us 24.00 us 4 ENTRYLK
0.00 17.75 us 13.00 us 21.00 us 8 FINODELK
0.00 73.00 us 41.00 us 105.00 us 2 FTRUNCATE
0.00 54.75 us 20.00 us 155.00 us 4 STATFS
0.00 169.50 us 164.00 us 175.00 us 2 FSYNC
0.00 50.38 us 44.00 us 58.00 us 8 FSTAT
0.01 101.32 us 91.00 us 123.00 us 44 READDIR
0.06 3362.38 us 1069.00 us 17592.00 us 8 FXATTROP
0.19 328.27 us 45.00 us 62554.00 us 249 SETATTR
0.67 140898.50 us 96307.00 us 185490.00 us 2 CREATE
8.15 6867.40 us 11.00 us 1675137.00 us 499 INODELK
90.91 6765.62 us 57.00 us 3022228.00 us 5649 LOOKUP
Duration: 402683 seconds
Data Read: 6536179250 bytes
Data Written: 545243668 bytes
Brick: hangar:/export/wd1a
--------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 7 26
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 2 0
No. of Writes: 517 226 396
Block Size: 64b+ 128b+ 256b+
No. of Reads: 205 432 335
No. of Writes: 523 861 1486
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 120 124 175
No. of Writes: 7074 8974 11592
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1376 5933 1325
No. of Writes: 6597 4380 3152
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1116 6665 2201096
No. of Writes: 3846 619 940
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 28311 FORGET
0.00 0.00 us 0.00 us 0.00 us 67175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.01 47.00 us 47.00 us 47.00 us 1 GETXATTR
0.02 38.25 us 25.00 us 51.00 us 4 ENTRYLK
0.02 47.25 us 34.00 us 76.00 us 4 STATFS
0.03 58.00 us 46.00 us 75.00 us 4 WRITE
0.04 74.50 us 40.00 us 115.00 us 4 FTRUNCATE
0.06 31.56 us 20.00 us 61.00 us 16 FINODELK
0.22 435.00 us 368.00 us 516.00 us 4 FSYNC
0.32 39.58 us 32.00 us 50.00 us 65 READLINK
1.46 37.34 us 20.00 us 197.00 us 314 INODELK
4.23 17009.00 us 12995.00 us 21023.00 us 2 CREATE
7.76 3901.38 us 2761.00 us 10117.00 us 16 FXATTROP
8.62 441.71 us 53.00 us 44487.00 us 157 SETATTR
77.23 149.34 us 82.00 us 10172.00 us 4162 LOOKUP
Duration: 402683 seconds
Data Read: 289280402340 bytes
Data Written: 502853550 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 7 26
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 2 0
No. of Writes: 517 226 396
Block Size: 64b+ 128b+ 256b+
No. of Reads: 205 432 335
No. of Writes: 523 861 1486
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 120 124 175
No. of Writes: 7074 8974 11592
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1376 5933 1325
No. of Writes: 6597 4380 3152
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1116 6665 2201096
No. of Writes: 3846 619 940
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 28311 FORGET
0.00 0.00 us 0.00 us 0.00 us 67175 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.01 47.00 us 47.00 us 47.00 us 1 GETXATTR
0.02 38.25 us 25.00 us 51.00 us 4 ENTRYLK
0.02 47.25 us 34.00 us 76.00 us 4 STATFS
0.03 58.00 us 46.00 us 75.00 us 4 WRITE
0.04 74.50 us 40.00 us 115.00 us 4 FTRUNCATE
0.06 31.56 us 20.00 us 61.00 us 16 FINODELK
0.22 435.00 us 368.00 us 516.00 us 4 FSYNC
0.32 39.58 us 32.00 us 50.00 us 65 READLINK
1.46 37.34 us 20.00 us 197.00 us 314 INODELK
4.23 17009.00 us 12995.00 us 21023.00 us 2 CREATE
7.76 3901.38 us 2761.00 us 10117.00 us 16 FXATTROP
8.62 441.71 us 53.00 us 44487.00 us 157 SETATTR
77.23 149.34 us 82.00 us 10172.00 us 4162 LOOKUP
Duration: 402683 seconds
Data Read: 289280402340 bytes
Data Written: 502853550 bytes
Brick: hangar:/export/wd3a
--------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 12 29
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 4 2
No. of Writes: 549 388 543
Block Size: 64b+ 128b+ 256b+
No. of Reads: 89 383 302
No. of Writes: 686 1264 2241
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 179 170 146
No. of Writes: 8499 11938 17532
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1367 1366 1816
No. of Writes: 6858 4258 3119
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1962 13661 92348
No. of Writes: 3610 629 1132
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 27647 FORGET
0.00 0.00 us 0.00 us 0.00 us 61843 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.00 34.25 us 21.00 us 50.00 us 4 ENTRYLK
0.00 71.50 us 68.00 us 75.00 us 2 WRITE
0.00 72.50 us 72.00 us 73.00 us 2 FTRUNCATE
0.00 68.50 us 39.00 us 141.00 us 4 STATFS
0.00 186.50 us 88.00 us 285.00 us 2 GETXATTR
0.00 64.25 us 26.00 us 224.00 us 8 FINODELK
0.00 288.50 us 266.00 us 311.00 us 2 FSYNC
0.01 168.28 us 115.00 us 249.00 us 32 READDIR
0.08 4721.12 us 2756.00 us 9548.00 us 8 FXATTROP
0.12 28292.50 us 12698.00 us 43887.00 us 2 CREATE
1.10 1024.51 us 23.00 us 208491.00 us 500 INODELK
3.59 6658.77 us 59.00 us 333387.00 us 250 SETATTR
95.09 7812.38 us 92.00 us 2591816.00 us 5651 LOOKUP
Duration: 402683 seconds
Data Read: 13735588693 bytes
Data Written: 545224769 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 3 12 29
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 4 2
No. of Writes: 549 388 543
Block Size: 64b+ 128b+ 256b+
No. of Reads: 89 383 302
No. of Writes: 686 1264 2241
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 179 170 146
No. of Writes: 8499 11938 17532
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 1367 1366 1816
No. of Writes: 6858 4258 3119
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1962 13661 92348
No. of Writes: 3610 629 1132
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 27647 FORGET
0.00 0.00 us 0.00 us 0.00 us 61843 RELEASE
0.00 0.00 us 0.00 us 0.00 us 36654 RELEASEDIR
0.00 34.25 us 21.00 us 50.00 us 4 ENTRYLK
0.00 71.50 us 68.00 us 75.00 us 2 WRITE
0.00 72.50 us 72.00 us 73.00 us 2 FTRUNCATE
0.00 68.50 us 39.00 us 141.00 us 4 STATFS
0.00 186.50 us 88.00 us 285.00 us 2 GETXATTR
0.00 64.25 us 26.00 us 224.00 us 8 FINODELK
0.00 288.50 us 266.00 us 311.00 us 2 FSYNC
0.01 168.28 us 115.00 us 249.00 us 32 READDIR
0.08 4721.12 us 2756.00 us 9548.00 us 8 FXATTROP
0.12 28292.50 us 12698.00 us 43887.00 us 2 CREATE
1.10 1024.51 us 23.00 us 208491.00 us 500 INODELK
3.59 6658.77 us 59.00 us 333387.00 us 250 SETATTR
95.09 7812.38 us 92.00 us 2591816.00 us 5651 LOOKUP
Duration: 402683 seconds
Data Read: 13735588693 bytes
Data Written: 545224769 bytes
--
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
m...@netbsd.org
_______________________________________________
Gluster-devel mailing list
Gluster-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/gluster-devel
