Re: [lustre-discuss] OST still has inodes and size after deleting all files
Thank you. I checked all those files with "ll_decode_filter_fid" and "lfs fid2path": On OSS servers: # umount /lustre/ost01 # mount -t ldiskfs /dev/mapper/ost01 /mnt/ # find /mnt/O/0/d*/ -type f /mnt/O/0/d11/450605003 /mnt/O/0/d12/450605484 . . . # ll_decode_filter_fid /mnt/O/0/d11/450605003 /mnt/O/0/d11/450605003: parent=[0x200019425:0x733f:0x0] stripe=0 stripe_size=1048576 stripe_count=1 layout_version=0 range=0 # umount /mnt # mount -t lustre /dev/mapper/ost01 /lustre/ost01 On client: # lfs fid2path /cluster [0x200019425:0x733f:0x0] lfs fid2path: cannot find /cluster [0x200019425:0x733f:0x0]: No such file or directory I got a reply "No such file or directory" for all those unknown files. So I can assume those are "stray objects from deleted files". Best regards, Pavlo Khmel > On 20 Jan 2024, at 04:45, Andreas Dilger wrote: > > > >> On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss >> mailto:lustre-discuss@lists.lustre.org>> >> wrote: >> >> Hi, >> >> I'm trying to remove 4 OSTs. >> >> # lfs osts >> OBDS: >> 0: cluster-OST_UUID ACTIVE >> 1: cluster-OST0001_UUID ACTIVE >> 2: cluster-OST0002_UUID ACTIVE >> 3: cluster-OST0003_UUID ACTIVE >> . . . >> >> I moved all files to other OSTs. "lfs find" cannot find any files on these 4 >> OSTs. >> >> # time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster >> >> real 936m8.528s >> user 13m48.298s >> sys 210m1.245s >> >> But still: 2624 inods are in use and 14.5G total size. >> >> # lfs df -i | grep -e OST -e OST0001 -e OST0002 -e OST0003 >> cluster-OST_UUID 4293438576 644 4293437932 1% /cluster[OST:0] >> cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1] >> cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2] >> cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3] >> >> # lfs df -h | grep -e OST -e OST0001 -e OST0002 -e OST0003 >> cluster-OST_UUID 29.2T3.8G 27.6T 1% /cluster[OST:0] >> cluster-OST0001_UUID 29.2T3.7G 27.6T 1% /cluster[OST:1] >> cluster-OST0002_UUID 29.2T3.3G 27.6T 1% /cluster[OST:2] >> cluster-OST0003_UUID 29.2T3.7G 27.6T 1% /cluster[OST:3] >> >> I tried to check the file-system for errors: >> >> # umount /lustre/ost01 >> # e2fsck -fy /dev/mapper/ost01 >> >> and >> >> # lctl lfsck_start --device cluster-OST0001 >> # lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub >> . . . >> status: completed >> >> I tried to mount OST as ldiskfs and there are several files in /O/0/d*/ >> >> # umount /lustre/ost01 >> # mount -t ldiskfs /dev/mapper/ost01 /mnt/ >> # ls -Rhl /mnt/O/0/d*/ >> . . . >> /mnt/O/0/d11/: >> -rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003 >> /mnt/O/0/d12/: >> -rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028 >> -rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484 >> . . . >> >> Is it expected behavior? Is it save to delere OST even with those files? > > > You can run the debugfs "stat" command to print the "fid" xattr and it will > print the MDT > parent FID for use with "lfs fid2path" on the client to see if there are any > files related > to these objects. You could also run "ll_decode_filter_fid" to do the same > thing on the > mounted ldiskfs filesystem. > > It is likely that there are a few stray objects from deleted files, but hard > to say for sure. > > Cheers, Andreas > -- > Andreas Dilger > Lustre Principal Architect > Whamcloud > > > > > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OST still has inodes and size after deleting all files
On Jan 19, 2024, at 13:48, Pavlo Khmel via lustre-discuss mailto:lustre-discuss@lists.lustre.org>> wrote: Hi, I'm trying to remove 4 OSTs. # lfs osts OBDS: 0: cluster-OST_UUID ACTIVE 1: cluster-OST0001_UUID ACTIVE 2: cluster-OST0002_UUID ACTIVE 3: cluster-OST0003_UUID ACTIVE . . . I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs. # time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster real 936m8.528s user 13m48.298s sys 210m1.245s But still: 2624 inods are in use and 14.5G total size. # lfs df -i | grep -e OST -e OST0001 -e OST0002 -e OST0003 cluster-OST_UUID 4293438576 644 4293437932 1% /cluster[OST:0] cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1] cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2] cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3] # lfs df -h | grep -e OST -e OST0001 -e OST0002 -e OST0003 cluster-OST_UUID 29.2T3.8G 27.6T 1% /cluster[OST:0] cluster-OST0001_UUID 29.2T3.7G 27.6T 1% /cluster[OST:1] cluster-OST0002_UUID 29.2T3.3G 27.6T 1% /cluster[OST:2] cluster-OST0003_UUID 29.2T3.7G 27.6T 1% /cluster[OST:3] I tried to check the file-system for errors: # umount /lustre/ost01 # e2fsck -fy /dev/mapper/ost01 and # lctl lfsck_start --device cluster-OST0001 # lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub . . . status: completed I tried to mount OST as ldiskfs and there are several files in /O/0/d*/ # umount /lustre/ost01 # mount -t ldiskfs /dev/mapper/ost01 /mnt/ # ls -Rhl /mnt/O/0/d*/ . . . /mnt/O/0/d11/: -rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003 /mnt/O/0/d12/: -rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028 -rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484 . . . Is it expected behavior? Is it save to delere OST even with those files? You can run the debugfs "stat" command to print the "fid" xattr and it will print the MDT parent FID for use with "lfs fid2path" on the client to see if there are any files related to these objects. You could also run "ll_decode_filter_fid" to do the same thing on the mounted ldiskfs filesystem. It is likely that there are a few stray objects from deleted files, but hard to say for sure. Cheers, Andreas -- Andreas Dilger Lustre Principal Architect Whamcloud ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] OST still has inodes and size after deleting all files
Hi, I'm trying to remove 4 OSTs. # lfs osts OBDS: 0: cluster-OST_UUID ACTIVE 1: cluster-OST0001_UUID ACTIVE 2: cluster-OST0002_UUID ACTIVE 3: cluster-OST0003_UUID ACTIVE . . . I moved all files to other OSTs. "lfs find" cannot find any files on these 4 OSTs. # time lfs find --ost 0 --ost 1 --ost 2 --ost 3 /cluster real936m8.528s user13m48.298s sys 210m1.245s But still: 2624 inods are in use and 14.5G total size. # lfs df -i | grep -e OST -e OST0001 -e OST0002 -e OST0003 cluster-OST_UUID 4293438576 644 4293437932 1% /cluster[OST:0] cluster-OST0001_UUID 4293438576 640 4293437936 1% /cluster[OST:1] cluster-OST0002_UUID 4293438576 671 4293437905 1% /cluster[OST:2] cluster-OST0003_UUID 4293438576 669 4293437907 1% /cluster[OST:3] # lfs df -h | grep -e OST -e OST0001 -e OST0002 -e OST0003 cluster-OST_UUID 29.2T3.8G 27.6T 1% /cluster[OST:0] cluster-OST0001_UUID 29.2T3.7G 27.6T 1% /cluster[OST:1] cluster-OST0002_UUID 29.2T3.3G 27.6T 1% /cluster[OST:2] cluster-OST0003_UUID 29.2T3.7G 27.6T 1% /cluster[OST:3] I tried to check the file-system for errors: # umount /lustre/ost01 # e2fsck -fy /dev/mapper/ost01 and # lctl lfsck_start --device cluster-OST0001 # lctl get_param -n osd-ldiskfs.cluster-OST0001.oi_scrub . . . status: completed I tried to mount OST as ldiskfs and there are several files in /O/0/d*/ # umount /lustre/ost01 # mount -t ldiskfs /dev/mapper/ost01 /mnt/ # ls -Rhl /mnt/O/0/d*/ . . . /mnt/O/0/d11/: -rw-rw-rw- 1 user1 group1 603K Nov 8 21:37 450605003 /mnt/O/0/d12/: -rw-rw-rw- 1 user1 group1 110K Jun 16 2023 450322028 -rw-rw-rw- 1 user1 group1 21M Nov 8 22:17 450605484 . . . Is it expected behavior? Is it save to delere OST even with those files? Best regards, Pavlo Khmel ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
