[Lustre-discuss] Lost folders after changing MDS

[Alastair Ferguson]

OK, so our old MDS had hardware issues so I configured a new MGS / MDS on a VM (this is a backup lustre filesystem and I wanted to separate the MGS / MDS from OSS of the previous), and then did this: For example: mount -t ldiskfs /dev/old /mnt/ost_old mount -t ldiskfs /dev/new /mnt/ost_new rsync -aSv /mnt/ost_old/ /mnt/ost_new # note trailing slash on ost_old/ If you are unable to connect both sets of disk to the same computer, use rsync to copy over the network using rsh (or ssh with -e ssh): rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new Use the same procedure for the MDS, with one additional step: cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \<copy all MDS files as above>; cd /mnt/mds_new; setfattr \--restore=/tmp/mdsea Run on all nodes (clients and OSS's & MDS / MGS) to change MGS (where mgs is the name of the server and /dev/sdb1 is the name of the unmounted oat) tunefs.lustre --mgsnode=mgs@tcp0 /dev/sdb1 tunefs.lustre --erase-param --mgsnode=192.168.16.3@tcp --writeconf /dev/sdc1 After copying with rsync, I had to cd /srv/mdt; rm CATALOGS OBJECTS/* on the new MDT partition. I also upgraded from 1.8.8 to 2. I managed to mount the Lustre filesystem and if I do lfs df -h, I get: NB> I deactivated those two OSTs below. [root@mgs data]# lfs df -h UUID                       bytes        Used   Available Use% Mounted on AC3-MDT0000_UUID           37.5G      499.5M       34.5G   1% /data[MDT:0] AC3-OST0000_UUID           16.4T        2.2T       13.3T  14% /data[OST:0] AC3-OST0001_UUID           16.4T        1.8T       13.7T  12% /data[OST:1] AC3-OST0002_UUID            6.4T        6.0T       49.2G  99% /data[OST:2] AC3-OST0003_UUID            6.4T        6.1T      912.9M 100% /data[OST:3] AC3-OST0004_UUID            4.3T        4.1T       17.2G 100% /data[OST:4] AC3-OST0005_UUID            1.9T        1.8T       29.0M 100% /data[OST:5] AC3-OST0006_UUID            1.9T        1.5T      282.1G  85% /data[OST:6] AC3-OST0007_UUID            1.9T        1.8T      434.3M 100% /data[OST:7] AC3-OST0008_UUID            1.9T        1.8T       12.9G  99% /data[OST:8] OST0009             : Resource temporarily unavailable AC3-OST000a_UUID            2.0T        1.9T      971.7M 100% /data[OST:10] AC3-OST000b_UUID            2.0T        1.9T      537.4M 100% /data[OST:11] AC3-OST000c_UUID            2.0T        1.9T        1.4G 100% /data[OST:12] AC3-OST000d_UUID            2.0T        1.6T      234.5G  88% /data[OST:13] AC3-OST000e_UUID            2.0T        1.9T        3.1G 100% /data[OST:14] OST000f             : Resource temporarily unavailable AC3-OST0010_UUID            2.0T        1.9T        3.6G 100% /data[OST:16] AC3-OST0011_UUID            2.0T        1.6T      248.3G  87% /data[OST:17] AC3-OST0012_UUID            2.0T        1.9T        3.0G 100% /data[OST:18] AC3-OST0013_UUID            2.0T        1.9T        1.4G 100% /data[OST:19] AC3-OST0014_UUID            2.0T        1.9T        2.8G 100% /data[OST:20] AC3-OST0015_UUID            2.0T        1.9T        2.7G 100% /data[OST:21] AC3-OST0016_UUID            2.0T        1.4T      473.3G  75% /data[OST:22] AC3-OST0017_UUID            1.2T      917.9G      230.1G  80% /data[OST:23] AC3-OST0018_UUID            2.0T        1.3T      616.7G  68% /data[OST:24] AC3-OST0019_UUID            2.0T        1.2T      671.9G  65% /data[OST:25] AC3-OST001a_UUID            2.0T        1.1T      769.4G  59% /data[OST:26] AC3-OST001b_UUID            2.0T        1.2T      642.2G  66% /data[OST:27] AC3-OST001c_UUID            2.0T        1.2T      633.0G  67% /data[OST:28] filesystem summary:        92.1T       55.6T       31.9T  64% /data However, there are hardly any files or folders there - most of them have disappeared, but they are clearly on the OSTs (according to the above command). Is there any way I can reindex the OST'S / MDS to recognise the files and folders properly? When I do lctl dl I get: [root@mgs data]# lctl dl   1 UP mgc MGC192.168.16.3@tcp 285232ea-c153-187d-1033-47c23b432296 5   2 UP lov AC3-MDT0000-mdtlov AC3-MDT0000-mdtlov_UUID 4   3 UP mdt AC3-MDT0000 AC3-MDT0000_UUID 5   4 UP mds mdd_obd-AC3-MDT0000 mdd_obd_uuid-AC3-MDT0000 3   5 UP osc AC3-OST0002-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5   6 UP osc AC3-OST0003-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5   7 UP osc AC3-OST0004-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5   8 UP osc AC3-OST000a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5   9 UP osc AC3-OST000b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  10 UP osc AC3-OST000c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  11 UP osc AC3-OST000d-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  12 UP osc AC3-OST000e-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  13 UP osc AC3-OST0011-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  14 UP osc AC3-OST0013-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  15 UP osc AC3-OST0012-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  16 UP osc AC3-OST0014-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  17 UP osc AC3-OST0015-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  18 UP osc AC3-OST0016-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  19 UP osc AC3-OST0017-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  20 UP osc AC3-OST0001-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  21 UP osc AC3-OST0000-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  22 UP osc AC3-OST0010-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  23 UP osc AC3-OST0005-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  24 UP osc AC3-OST0006-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  25 UP osc AC3-OST0008-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  26 UP osc AC3-OST0007-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  27 UP osc AC3-OST0018-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  28 UP osc AC3-OST0019-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  29 UP osc AC3-OST001a-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  30 UP osc AC3-OST001b-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  31 UP osc AC3-OST001c-osc-MDT0000 AC3-MDT0000-mdtlov_UUID 5  32 UP lov AC3-clilov-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4  33 UP lmv AC3-clilmv-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 4  34 UP mdc AC3-MDT0000-mdc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  35 UP osc AC3-OST0002-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  36 UP osc AC3-OST0003-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  37 UP osc AC3-OST0004-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  38 UP osc AC3-OST000a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  39 UP osc AC3-OST000b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  40 UP osc AC3-OST000c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  41 UP osc AC3-OST000d-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  42 UP osc AC3-OST000e-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  43 UP osc AC3-OST0011-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  44 UP osc AC3-OST0013-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  45 UP osc AC3-OST0012-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  46 UP osc AC3-OST0014-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  47 UP osc AC3-OST0015-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  48 UP osc AC3-OST0016-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  49 UP osc AC3-OST0017-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  50 UP osc AC3-OST0001-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  51 UP osc AC3-OST0000-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  52 UP osc AC3-OST0010-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  53 UP osc AC3-OST0005-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  54 UP osc AC3-OST0006-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  55 UP osc AC3-OST0008-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  56 UP osc AC3-OST0007-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  57 UP osc AC3-OST0018-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  58 UP osc AC3-OST0019-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  59 UP osc AC3-OST001a-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  60 UP osc AC3-OST001b-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5  61 UP osc AC3-OST001c-osc-ffff810120a12800 ea98bf43-336d-bdc6-1e8c-54738d5b8774 5 SO it all looks fine and /var/log/messages and dmesg seem to look OK. Thanks in advance! Alastair Ferguson IT Manager Capital Markets CRC Limited (CMCRC) Telephone: +61 2 8088 4222 Mobile: +61 424 235 159 Fax: +61 2 8088 4201 www.cmcrc.com  Capital Markets CRC Ltd - Confidential Communication The information contained in this e-mail is confidential.  It is intended for the addressee only.  If you receive this e-mail by mistake please promptly inform us by reply e-mail and then delete the e-mail and destroy any printed copy. You must not disclose or use in any way the information in the e-mail. There is no warranty that this e-mail is error or virus free. It may be a private communication, and if so, does not represent the views of the CMCRC and its associates.  If it is a private communication, care should be taken in opening it to ensure that undue offence is not given.

_______________________________________________
Lustre-discuss mailing list
Lustre-discuss@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss