Hi, Could you share the following pieces of information to begin with -
1. output of `gluster volume info $AFFECTED_VOLUME_NAME` 2. glusterfs version you're running -Krutika On Sat, Mar 2, 2019 at 3:38 AM Drew R <[email protected]> wrote: > Saw some people asking for profile info. So I had started a migration > from a 6TB WDGold 2+1arb replicated gluster to a 1TB samsung ssd 2+1 rep > gluster and it's been running a while for a 100GB file thin provisioned > with like 28GB actually used. Here is the profile info. I started the > profiler like 5 minutes ago. The migration had been running for like > 30minutes: > > gluster volume profile gv2 info > Brick: 10.30.30.122:/gluster_bricks/gv2/brick > --------------------------------------------- > Cumulative Stats: > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 1189 8 > 12 > No. of Writes: 4 3245 > 883 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 10 20 > 2 > No. of Writes: 1087 312228 > 124080 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 0 1 > 52 > No. of Writes: 5188 3617 > 5532 > > Block Size: 131072b+ > No. of Reads: 70191 > No. of Writes: 634192 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 2 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 202 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 1297 > RELEASEDIR > 0.00 14.50 us 9.00 us 20.00 us 4 > READDIR > 0.00 38.00 us 9.00 us 120.00 us 7 > GETXATTR > 0.00 66.00 us 34.00 us 128.00 us 6 > OPEN > 0.00 137.25 us 52.00 us 195.00 us 4 > SETATTR > 0.00 23.19 us 11.00 us 46.00 us 26 > INODELK > 0.00 41.58 us 18.00 us 79.00 us 24 > OPENDIR > 0.00 166.70 us 15.00 us 775.00 us 27 > READDIRP > 0.01 135.29 us 12.00 us 11695.00 us 221 > STATFS > 0.01 176.54 us 22.00 us 22944.00 us 364 > FSTAT > 0.02 626.21 us 13.00 us 17308.00 us 168 > STAT > 0.09 834.84 us 9.00 us 34337.00 us 607 > LOOKUP > 0.73 146.18 us 6.00 us 52255.00 us 29329 > FINODELK > 1.03 298.20 us 42.00 us 43711.00 us 20204 > FXATTROP > 15.38 8903.40 us 213.00 us 213832.00 us 10102 > WRITE > 39.14 26796.37 us 222.00 us 122696.00 us 8538 > READ > 43.59 39536.79 us 259.00 us 183630.00 us 6446 > FSYNC > > Duration: 15078 seconds > Data Read: 9207377205 bytes > Data Written: 86214017762 bytes > > Interval 2 Stats: > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 17 0 > 0 > No. of Writes: 0 43 > 7 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 0 7 > 0 > No. of Writes: 16 1881 > 1010 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 0 0 > 6 > No. of Writes: 305 586 > 2359 > > Block Size: 131072b+ > No. of Reads: 7162 > No. of Writes: 610 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 6 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 20 > RELEASEDIR > 0.00 14.50 us 9.00 us 20.00 us 4 > READDIR > 0.00 38.00 us 9.00 us 120.00 us 7 > GETXATTR > 0.00 66.00 us 34.00 us 128.00 us 6 > OPEN > 0.00 137.25 us 52.00 us 195.00 us 4 > SETATTR > 0.00 23.19 us 11.00 us 46.00 us 26 > INODELK > 0.00 40.05 us 18.00 us 79.00 us 20 > OPENDIR > 0.00 180.33 us 16.00 us 775.00 us 21 > READDIRP > 0.01 181.77 us 12.00 us 11695.00 us 149 > STATFS > 0.01 511.23 us 15.00 us 16954.00 us 111 > STAT > 0.01 210.22 us 22.00 us 22944.00 us 291 > FSTAT > 0.09 979.73 us 9.00 us 23539.00 us 404 > LOOKUP > 0.81 179.58 us 6.00 us 52255.00 us 19786 > FINODELK > 1.07 342.71 us 42.00 us 43711.00 us 13633 > FXATTROP > 14.18 9087.30 us 213.00 us 213832.00 us 6817 > WRITE > 40.00 40227.38 us 259.00 us 157934.00 us 4343 > FSYNC > 43.81 26601.57 us 222.00 us 118756.00 us 7192 > READ > > Duration: 177 seconds > Data Read: 939534945 bytes > Data Written: 363763200 bytes > > Brick: 10.30.30.121:/gluster_bricks/gv2/brick > --------------------------------------------- > Cumulative Stats: > Block Size: 8b+ 256b+ > 512b+ > No. of Reads: 0 142626 > 329092 > No. of Writes: 12 98 > 123230 > > Block Size: 1024b+ 2048b+ > 4096b+ > No. of Reads: 21175 18223 > 3345043 > No. of Writes: 58646 69147 > 6092143 > > Block Size: 8192b+ 16384b+ > 32768b+ > No. of Reads: 1090937 216037 > 748595 > No. of Writes: 2030403 243915 > 240942 > > Block Size: 65536b+ 131072b+ > No. of Reads: 1524667 4828662 > No. of Writes: 605125 1889892 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 133 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 4821 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 42742 > RELEASEDIR > 0.00 28.50 us 12.00 us 61.00 us 4 > READDIR > 0.00 71.75 us 59.00 us 88.00 us 4 > SETATTR > 0.00 44.57 us 12.00 us 94.00 us 7 > GETXATTR > 0.00 62.17 us 42.00 us 89.00 us 6 > OPEN > 0.00 35.92 us 1.00 us 76.00 us 24 > OPENDIR > 0.00 325.54 us 22.00 us 1117.00 us 13 > READDIRP > 0.00 482.13 us 19.00 us 12473.00 us 46 > STAT > 0.00 217.65 us 15.00 us 32841.00 us 221 > STATFS > 0.01 497.52 us 11.00 us 29179.00 us 539 > LOOKUP > 0.02 685.44 us 16.00 us 61131.00 us 615 > FSTAT > 0.04 31691.92 us 17.00 us 349648.00 us 26 > INODELK > 0.45 451.95 us 51.00 us 65375.00 us 20206 > FXATTROP > 6.90 13906.03 us 225.00 us 210298.00 us 10103 > WRITE > 18.47 58375.96 us 389.00 us 218211.00 us 6446 > FSYNC > 27.45 27279.84 us 184.00 us 241444.00 us 20503 > READ > 46.66 32462.82 us 7.00 us 2750519.00 us 29283 > FINODELK > > Duration: 475152 seconds > Data Read: 799630169437 bytes > Data Written: 362708560796 bytes > > Interval 2 Stats: > Block Size: 256b+ 512b+ > 1024b+ > No. of Reads: 53 0 > 0 > No. of Writes: 0 43 > 7 > > Block Size: 2048b+ 4096b+ > 8192b+ > No. of Reads: 0 891 > 616 > No. of Writes: 16 1881 > 1010 > > Block Size: 16384b+ 32768b+ > 65536b+ > No. of Reads: 288 750 > 52 > No. of Writes: 304 586 > 2359 > > Block Size: 131072b+ > No. of Reads: 9445 > No. of Writes: 610 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 6 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 20 > RELEASEDIR > 0.00 28.50 us 12.00 us 61.00 us 4 > READDIR > 0.00 71.75 us 59.00 us 88.00 us 4 > SETATTR > 0.00 44.57 us 12.00 us 94.00 us 7 > GETXATTR > 0.00 62.17 us 42.00 us 89.00 us 6 > OPEN > 0.00 34.50 us 1.00 us 76.00 us 20 > OPENDIR > 0.00 55.16 us 19.00 us 128.00 us 25 > STAT > 0.00 363.18 us 25.00 us 1117.00 us 11 > READDIRP > 0.00 71.61 us 15.00 us 4323.00 us 149 > STATFS > 0.01 251.40 us 11.00 us 15344.00 us 336 > LOOKUP > 0.02 594.11 us 16.00 us 61131.00 us 450 > FSTAT > 0.07 31691.92 us 17.00 us 349648.00 us 26 > INODELK > 0.43 384.83 us 51.00 us 65375.00 us 13632 > FXATTROP > 7.54 13535.70 us 225.00 us 210298.00 us 6816 > WRITE > 20.03 56497.17 us 394.00 us 218211.00 us 4340 > FSYNC > 25.91 26229.86 us 184.00 us 241444.00 us 12095 > READ > 45.99 28508.86 us 7.00 us 2591280.00 us 19751 > FINODELK > > Duration: 177 seconds > Data Read: 1298631621 bytes > Data Written: 363744256 bytes > > Brick: 10.30.30.123:/gluster_bricks/gv2/brick > --------------------------------------------- > Cumulative Stats: > Block Size: 1b+ > No. of Reads: 0 > No. of Writes: 18126558 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 383 > FORGET > 0.00 0.00 us 0.00 us 0.00 us 9602 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 70644 > RELEASEDIR > 0.00 20.75 us 13.00 us 29.00 us 4 > READDIR > 0.00 87.00 us 72.00 us 129.00 us 4 > SETATTR > 0.00 71.00 us 46.00 us 93.00 us 6 > OPEN > 0.00 66.00 us 9.00 us 114.00 us 7 > GETXATTR > 0.00 50.58 us 16.00 us 90.00 us 26 > INODELK > 0.00 56.38 us 1.00 us 111.00 us 24 > OPENDIR > 0.02 66.23 us 28.00 us 146.00 us 113 > FSTAT > 0.17 143.88 us 13.00 us 444.00 us 539 > LOOKUP > 1.17 51.64 us 10.00 us 166.00 us 10103 > WRITE > 2.89 44.06 us 7.00 us 2177.00 us 29349 > FINODELK > 5.55 122.83 us 54.00 us 9281.00 us 20206 > FXATTROP > 90.19 6249.64 us 237.00 us 59996.00 us 6448 > FSYNC > > Duration: 772792 seconds > Data Read: 0 bytes > Data Written: 18126558 bytes > > Interval 2 Stats: > Block Size: 1b+ > No. of Reads: 0 > No. of Writes: 6816 > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop > --------- ----------- ----------- ----------- ------------ > ---- > 0.00 0.00 us 0.00 us 0.00 us 6 > RELEASE > 0.00 0.00 us 0.00 us 0.00 us 20 > RELEASEDIR > 0.00 20.75 us 13.00 us 29.00 us 4 > READDIR > 0.00 87.00 us 72.00 us 129.00 us 4 > SETATTR > 0.00 71.00 us 46.00 us 93.00 us 6 > OPEN > 0.00 66.00 us 9.00 us 114.00 us 7 > GETXATTR > 0.00 58.10 us 1.00 us 111.00 us 20 > OPENDIR > 0.00 50.58 us 16.00 us 90.00 us 26 > INODELK > 0.03 66.23 us 28.00 us 146.00 us 113 > FSTAT > 0.16 145.84 us 13.00 us 346.00 us 336 > LOOKUP > 1.18 51.98 us 11.00 us 157.00 us 6816 > WRITE > 2.91 44.05 us 7.00 us 2177.00 us 19798 > FINODELK > 5.59 122.70 us 54.00 us 9281.00 us 13632 > FXATTROP > 90.11 6210.61 us 238.00 us 54711.00 us 4342 > FSYNC > > Duration: 177 seconds > Data Read: 0 bytes > Data Written: 6816 bytes > _______________________________________________ > Users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/[email protected]/message/XHAUIMGRCVEDCLS7F6WALUWJVQ4IXAWU/ >
_______________________________________________ Users mailing list -- [email protected] To unsubscribe send an email to [email protected] Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/[email protected]/message/AMJ7LXV7SCSMJAQ46QLWNBPZ3TGZMKHS/
