Hello,
I want to estimate how much longer a running mmrestripefs -R –metadata-only –qos maintenance job will take to finish. We did switch from ‘-m 1’ to ‘-m 2 ’ and now run mmrestripefs to get the second copy of all metadata. I know that the % values in the output are useless, but now it looks like the number of inodes processed is of no use, too? The filesystem has about 1e9 allocated inodes, but the mmrestripefs reports up to 7e9 inodes processed? The number of inodes reported increases heavily since it passed the number of allocated inodes??? 4.41 % complete on Fri May 28 09:22:05 2021 (7219160122 inodes with total 8985067 MB data processed) Should I consider the ‘MB data processed’ instead – will the job finish when all used metadata disk space is processed? I see little iops on the metadata disks and wonder if something is stuck, I’m not sure if it makes sense to wait any longer. But without an estimate on how much longer the job will take I hesitate to restart. A restart will need to scan all metadata again … Metadata is on SSD or NVMe, disks iops never were limiting. I turned qos off on the filesystem, previously I restricted the iops. But I don’t see any increase in iops. Thank you. I know this is an old issue, but still I would strongly welcome any advice or comment. Cheers, Heiner I’m aware of RFE ID:150409 mmrestripefs % complete metric is near useless. I’m not sure if it covers the metadata-only case, too. Inode Information ----------------- Total number of used inodes in all Inode spaces: 735005692 Total number of free inodes in all Inode spaces: 335462660 Total number of allocated inodes in all Inode spaces: 1070468352 Total of Maximum number of inodes in all Inode spaces: 1223782912 Mmrestripefs output: START Mon May 24 20:27:25 CEST 2021 Scanning file system metadata, phase 1 ... 100 % complete on Mon May 24 21:49:27 2021 Scan completed successfully. Scanning file system metadata, phase 2 ... 100 % complete on Mon May 24 21:49:28 2021 Scanning file system metadata for data storage pool 90 % complete on Mon May 24 21:49:34 2021 100 % complete on Mon May 24 21:49:35 2021 Scanning file system metadata for Capacity storage pool 49 % complete on Mon May 24 21:49:41 2021 100 % complete on Mon May 24 21:49:46 2021 Scan completed successfully. Scanning file system metadata, phase 3 ... Scan completed successfully. Scanning file system metadata, phase 4 ... 100 % complete on Mon May 24 21:49:48 2021 Scan completed successfully. Scanning file system metadata, phase 5 ... 100 % complete on Mon May 24 21:49:52 2021 Scan completed successfully. Scanning user file metadata ... 0.01 % complete on Mon May 24 21:50:13 2021 ( 613108 inodes with total 76693 MB data processed) 0.01 % complete on Mon May 24 21:50:33 2021 ( 1090048 inodes with total 80495 MB data processed) 0.01 % complete on Mon May 24 21:50:53 2021 ( 1591808 inodes with total 84576 MB data processed) … 3.97 % complete on Thu May 27 22:01:02 2021 (1048352497 inodes with total 8480467 MB data processed) 3.99 % complete on Thu May 27 22:30:18 2021 (1050254855 inodes with total 8495969 MB data processed) 4.01 % complete on Thu May 27 22:59:39 2021 (1052304294 inodes with total 8512683 MB data processed) 4.03 % complete on Thu May 27 23:29:11 2021 (1055390220 inodes with total 8537615 MB data processed) 4.05 % complete on Thu May 27 23:58:58 2021 (1059333989 inodes with total 8568871 MB data processed) 4.07 % complete on Fri May 28 00:28:48 2021 (1064728403 inodes with total 8611605 MB data processed) 4.09 % complete on Fri May 28 00:58:50 2021 (1067749260 inodes with total 8636120 MB data processed). <<< approximately number of allocated inodes 4.11 % complete on Fri May 28 01:29:00 2021 (1488665433 inodes with total 8661588 MB data processed) <<< whats going?? 4.13 % complete on Fri May 28 01:59:23 2021 (1851124480 inodes with total 8682324 MB data processed) 4.15 % complete on Fri May 28 02:29:55 2021 (1885948840 inodes with total 8700082 MB data processed) 4.17 % complete on Fri May 28 03:00:38 2021 (2604503808 inodes with total 8724069 MB data processed) 4.19 % complete on Fri May 28 03:31:38 2021 (2877196260 inodes with total 8746504 MB data processed) 4.21 % complete on Fri May 28 04:02:38 2021 (2933166080 inodes with total 8762555 MB data processed) 4.23 % complete on Fri May 28 04:33:48 2021 (2956295936 inodes with total 8782298 MB data processed) 4.25 % complete on Fri May 28 05:05:09 2021 (3628799151 inodes with total 8802452 MB data processed) 4.27 % complete on Fri May 28 05:36:40 2021 (3970093965 inodes with total 8823885 MB data processed) 4.29 % complete on Fri May 28 06:08:20 2021 (4012553472 inodes with total 8841407 MB data processed) 4.31 % complete on Fri May 28 06:40:11 2021 (4029545087 inodes with total 8858676 MB data processed) 4.33 % complete on Fri May 28 07:12:11 2021 (6080613874 inodes with total 8889395 MB data processed) 4.35 % complete on Fri May 28 07:44:21 2021 (6146937531 inodes with total 8907253 MB data processed) 4.37 % complete on Fri May 28 08:16:45 2021 (6167408718 inodes with total 8925236 MB data processed) 4.39 % complete on Fri May 28 08:49:18 2021 (7151592448 inodes with total 8968126 MB data processed) 4.41 % complete on Fri May 28 09:22:05 2021 (7219160122 inodes with total 8985067 MB data processed) # mmdf fsxxxx -m --block-size M disk disk size failure holds holds free in MB free in MB name in MB group metadata data in full blocks in fragments --------------- ------------- -------- -------- ----- -------------------- ------------------- Disks in storage pool: system (Maximum disk size allowed is 32.20 TB) fsxxxx_rg07b_1 2861295 3 yes no 1684452 ( 59%) 35886 ( 1%). << 1’140’957MB used – old nsd fsxxxx_rg07a_1 2861295 3 yes no 1685002 ( 59%) 35883 ( 1% fsxxxx_rg03b_1 2861295 3 yes no 1681515 ( 59%) 35627 ( 1%) fsxxxx_rg03a_1 2861295 3 yes no 1680859 ( 59%) 35651 ( 1%) RG003LG004VS015 2046239 12 yes no 904369 ( 44%) 114 ( 0%) << 1’141’756MB used – newly added nsd RG003LG003VS015 2046239 12 yes no 904291 ( 44%) 118 ( 0%) RG003LG002VS015 2046239 12 yes no 903632 ( 44%) 114 ( 0%) RG003LG001VS015 2046239 12 yes no 903247 ( 44%) 115 ( 0%) ------------- -------------------- ------------------- (pool total) 19630136 10347367 ( 53%) 143503 ( 1%) We run spectrum scale 5.0.5.4/5.0.5.5 on Power. Filesystem: -f 32768 Minimum fragment (subblock) size in bytes -i 4096 Inode size in bytes -I 32768 Indirect block size in bytes -m 2 Default number of metadata replicas -M 2 Maximum number of metadata replicas -r 1 Default number of data replicas -R 2 Maximum number of data replicas -j scatter Block allocation type -V 23.00 (5.0.5.0) Current file system version 15.01 (4.2.0.0) Original file system version -L 33554432 Logfile size --subblocks-per-full-block 32 Number of subblocks per full block
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
