Hi Guido,
thanks for the tip, that was successful, with the exact same commands,
tune2fs -O ^quota /dev/mdt (took about ~3min)
tunefs.lustre --quota /dev/mdt (took about ~30min with ~200M used inodes)
A subsequent
lctl lfsck_start -M nyx-MDT0000
ran for less than 45 min and seems to have cleaned up the mdt mess.
Cheers,
Thomas
On 07/12/2016 11:18 PM, Guido Laubender wrote:
English version (I'm sorry for my previous mail in German - but should have
been a personal mail to Thomas only :( ):
We were recently able to fix wrong Lustre inode quotas by disabling and
re-enabling quota support on the MDT by 'tune2fs -O ^quota /dev/mdt' and
'tunefs.lustre --quota /dev/mdt'.
Maybe it helps here as well.
On Tue, 12 Jul 2016, Guido Laubender wrote:
Bei uns waren vor kurzem die Inode-Quoten nicht korrekt; durch Deaktivieren und
anschließendes Aktivieren der Quoten-Unterstützung (mittels 'tune2fs
-O ^quota' und anschließendem 'tunefs.lustre --quota') auf dem MDT konnten wir
es wieder reparieren.
Vielleicht hilft das bei Euch auch...
On Tue, 12 Jul 2016, Thomas Roth wrote:
Hi all,
we are running Lustre 2.5.3 on our servers. OSTs are on ZFS, MDS is on ldiskfs.
After a MDS crash and e2fsck 1.42.9.wc1 on the partition, the MDS mounts but
causes high-frequency log entries
Jul 12 16:06:38 lxmds12 kernel: VFS: find_free_dqentry(): Data block full but
it shouldn't.
Jul 12 16:06:38 lxmds12 kernel: VFS: Error -5 occurred while creating quota.
interspersed with
Jul 12 16:06:38 lxmds12 kernel: LustreError:
13159:0:(qsd_handler.c:1155:qsd_op_adjust()) nyx-MDT0000: fail to locate lqe
for id:6763, type:0
Jul 12 16:06:38 lxmds12 kernel: LustreError:
13159:0:(qsd_handler.c:1155:qsd_op_adjust()) Skipped 4973 previous similar
messages
or
Jul 12 15:59:26 lxmds12 kernel: LustreError:
13414:0:(qsd_entry.c:211:qsd_refresh_usage()) $$$ failed to read disk usage,
rc:-3 qsd:nyx-MDT0000
qtype:usr id:7408 enforced:0 granted:0 pending:0 waiting:0 req:0 usage:0
qunit:0 qtune:0 edquot:0
Jul 12 15:59:26 lxmds12 kernel: LustreError:
13414:0:(qsd_entry.c:211:qsd_refresh_usage()) Skipped 5166 previous similar
messages
According to our experience from the last few days, this will eventually bring
all Lustre operations to a halt.
Both the web and the e2fsck-messages ([QUOTA WARNING] Usage inconsistent for ID
7989:actual (278528000, 738675) != expected (222507008, 531071))
hint towards quota issues.
Therefore, we have 'switched off' quota by "lctl conf_param
fsname.quota.ost|mdt=u|g|ug|none", restarted, umounted and 'switched on' quota again,
restarted, unmounted.
-> The VSF-Errors still appear.
Is there anything else we could do?
Mount the MDT as ldiskfs and do nasty things on the disk?
Is there any command that recalculates / rewrites the quota files on the MDT?
(As long as Lustre is still accessible, 'lfs quota' gives results for both
users and groups, but at least the file count is entirely wrong (all of
my own Lustre files amount to exactle 0 files).
And the update of the usage numbers does not work either - I managed to copy a
1GB-file and still had the same kbytes used...)
Regards,
Thomas
--
--------------------------------------------------------------------
Thomas Roth
Department: Informationstechnologie
Location: SB3 1.250
Phone: +49-6159-71 1453 Fax: +49-6159-71 2986
GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt
www.gsi.de
Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528
Geschäftsführung: Ursula Weyrich
Professor Dr. Karlheinz Langanke
Jörg Blaurock
Vorsitzende des Aufsichtsrates: St Dr. Georg Schütte
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
--
--------------------------------------------------------------------
Thomas Roth
Department: HPC
Location: SB3 1.262
Phone: +49-6159-71 1453 Fax: +49-6159-71 2986
GSI Helmholtzzentrum für Schwerionenforschung GmbH
Planckstraße 1
64291 Darmstadt
www.gsi.de
Gesellschaft mit beschränkter Haftung
Sitz der Gesellschaft: Darmstadt
Handelsregister: Amtsgericht Darmstadt, HRB 1528
Geschäftsführung: Professor Dr. Karlheinz Langanke
Ursula Weyrich
Jörg Blaurock
Vorsitzender des Aufsichtsrates: St Dr. Georg Schütte
Stellvertreter: Ministerialdirigent Dr. Rolf Bernhardt
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
