On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss <[email protected]<mailto:[email protected]>> wrote:
Hi, I'm trying to remove 4 OSTs. # lfs osts OBDS: 0: cluster-OST0000_UUID ACTIVE 1: cluster-OST0001_UUID ACTIVE 2: cluster-OST0002_UUID ACTIVE 3: cluster-OST0003_UUID ACTIVE . . . I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs. # time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster real 936m8.528s user 13m48.298s sys 210m1.245s But still: 2624 inods are in use and 14.5G total size. # lfs df -i | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003 cluster-OST0000_UUID 4293438576 644 4293437932 1% /cluster[OST:0] cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1] cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2] cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3] # lfs df -h | grep -e OST0000 -e OST0001 -e OST0002 -e OST0003 cluster-OST0000_UUID 29.2T 3.8G 27.6T 1% /cluster[OST:0] cluster-OST0001_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:1] cluster-OST0002_UUID 29.2T 3.3G 27.6T 1% /cluster[OST:2] cluster-OST0003_UUID 29.2T 3.7G 27.6T 1% /cluster[OST:3] I tried to check the file-system for errors: # umount /lustre/ost01 # e2fsck -fy /dev/mapper/ost01 and # lctl lfsck_start --device cluster-OST0001 # lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub . . . status: completed I tried to mount OST as ldiskfs and there are several files in /O/0/d*/ # umount /lustre/ost01 # mount -t ldiskfs /dev/mapper/ost01 /mnt/ # ls -Rhl /mnt/O/0/d*/ . . . /mnt/O/0/d11/: -rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003 /mnt/O/0/d12/: -rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028 -rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484 . . . Is it expected behavior? Is it save to delere OST even with those files? You can run the debugfs "stat" command to print the "fid" xattr and it will print the MDT parent FID for use with "lfs fid2path" on the client to see if there are any files related to these objects. You could also run "ll_decode_filter_fid" to do the same thing on the mounted ldiskfs filesystem. It is likely that there are a few stray objects from deleted files, but hard to say for sure. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud
_______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
