Dear all, We have an odd problem on a lustre client in that a couple of OST elements have become "inactive".
However, lctl dl shows all devices are in "UP" state. The "UP" status is also echoed on the mds as well as on another client - please see below. I have tried rebooting this client, but with no joy. I have also searched around the discussion list, and can find a few others had experienced similar problems before. But unfortunatley none of them has reported back with a good recovery procedure, other than a complete reformatting of the lustre storage. So can I trouble you experts to give me some clue as to how to recover from this? Many thanks. Regards, Peter PS. Client: SLES 11 with 2.6.32.29 kernel, Lustre 1.8.5 =================================================================================== Client Capuchin: capuchin:~ # lfs df -h UUID bytes Used Available Use% Mounted on ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0] ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0] ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1] ceda3-OST0002_UUID : inactive device ceda3-OST0003_UUID : inactive device ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4] ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5] ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6] filesystem summary: 35.8T 34.0T 32.5G 94% /disks/ceda3 capuchin:~ # lctl dl 0 UP mgc MGC130.246.191.64@tcp be6c0da2-275f-c59d-fa3a-c792134a0d9d 5 1 UP lov ceda3-clilov-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 4 2 UP mdc ceda3-MDT0000-mdc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 3 UP osc ceda3-OST0000-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 4 UP osc ceda3-OST0001-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 5 UP osc ceda3-OST0002-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 6 UP osc ceda3-OST0003-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 7 UP osc ceda3-OST0004-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 8 UP osc ceda3-OST0005-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 9 UP osc ceda3-OST0006-osc-ffff880c11d23c00 afc735ad-0668-0f3f-0e86-cd9b26d45f40 5 capuchin:~ # Client2 cmip-dn1: [root@aspre script]# ssh cmip-dn1 lfs df -h UUID bytes Used Available Use% Mounted on ceda3-MDT0000_UUID 350.0G 689.2M 329.3G 0% /disks/ceda3[MDT:0] ceda3-OST0000_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:0] ceda3-OST0001_UUID 7.2T 6.8T 6.5G 94% /disks/ceda3[OST:1] ceda3-OST0002_UUID 7.2T 451.6M 6.8T 0% /disks/ceda3[OST:2] ceda3-OST0003_UUID 7.2T 451.8M 6.8T 0% /disks/ceda3[OST:3] ceda3-OST0004_UUID 7.2T 6.8T 6.6G 94% /disks/ceda3[OST:4] ceda3-OST0005_UUID 7.2T 6.8T 6.7G 94% /disks/ceda3[OST:5] ceda3-OST0006_UUID 7.2T 6.8T 6.2G 94% /disks/ceda3[OST:6] filesystem summary: 50.1T 34.0T 13.6T 67% /disks/ceda3 Mds: [root@mds02 ~]# lctl dl 0 UP mgc MGC130.246.191.64@tcp 8aa29420-11f1-8ca3-a361-ce5135a09be2 5 1 UP mdt MDS MDS_uuid 3 2 UP lov ceda3-mdtlov ceda3-mdtlov_UUID 4 3 UP mds ceda3-MDT0000 ceda3-MDT0000_UUID 9 4 UP osc ceda3-OST0000-osc ceda3-mdtlov_UUID 5 5 UP osc ceda3-OST0001-osc ceda3-mdtlov_UUID 5 6 UP osc ceda3-OST0002-osc ceda3-mdtlov_UUID 5 7 UP osc ceda3-OST0003-osc ceda3-mdtlov_UUID 5 8 UP osc ceda3-OST0004-osc ceda3-mdtlov_UUID 5 9 UP osc ceda3-OST0005-osc ceda3-mdtlov_UUID 5 10 UP osc ceda3-OST0006-osc ceda3-mdtlov_UUID 5 11 UP lov ceda4-mdtlov ceda4-mdtlov_UUID 4 12 UP mds ceda4-MDT0000 ceda4-MDT0000_UUID 9 13 UP osc ceda4-OST0000-osc ceda4-mdtlov_UUID 5 14 UP osc ceda4-OST0001-osc ceda4-mdtlov_UUID 5 15 UP osc ceda4-OST0002-osc ceda4-mdtlov_UUID 5 16 UP osc ceda4-OST0003-osc ceda4-mdtlov_UUID 5 17 UP osc ceda4-OST0004-osc ceda4-mdtlov_UUID 5 18 UP osc ceda4-OST0005-osc ceda4-mdtlov_UUID 5 [root@mds02 ~]# -- Scanned by iCritical. _______________________________________________ Lustre-discuss mailing list [email protected] http://lists.lustre.org/mailman/listinfo/lustre-discuss
