On Apr 16, 2019, at 13:53, Fernando Pérez <[email protected]> wrote: > >> You can try... >> unmount the filesystem and the run a force e2fsck on all OSTs and MDT. >> >> e2fsck -f -p /dev/ost... >> >> Regargs. > > Thank you Mahmoud. > > The problem is that e2fsck in the MDT runs very very slowly…10 inodes per > second for more than 100 million inodes. > >>> Are there a lot of inodes moved to lost+found by the fsck, which contribute >>> to the occupied quota now? > > Thank you Martin. > > I can’t finish the e2fsck in the MDT. > >> Correct, LFSCK will repair the OST object ownership, but does not *directly* >> repair quota >> files. That said, if an object is owned by the wrong owner for some reason, >> changing the >> ownership _should_ transfer the quota usage properly to the new owner, so in >> that regard >> the quota usage should be indirectly repaired by an LFSCK run. > > > Thank you Andreas. > > I can confirm that lfsck layout does not repair the wrong quotes. > > I’m running lfsck namespace in the MDT to try other way.
Note that "namespace" checking only verifes/repairs the MDT consistency. Running "layout" checking is what verifies the MDT-OST consistency. That said, it looks like you are running "layout" checking based on the output below. >> Note the "tune2fs -O ^quota" to repair the quota accounting is a bit of a >> "cargo cult" behaviour. >> >> Running "e2fsck -fp" is the proper way to repair the quota files, since it >> not only recalculates the quota accounting using the same code as "tune2fs >> -O quota" does, but it also ensures that the files themselves are valid in >> the first place. They should take about the same time, except in the case >> your filesystem is corrupted, in which case you'd want e2fsck to repair the >> filesystem anyway. > > I ran this in the lfsck MDT and lfsck OSTs. After do that the quotes for all > users were more corrupted than before (for example for a user with 27 TB the > quota reports 900 MB). > > Then I ran the e2fsck in the lfsck OSTs. I tried to do the same in the lfsck > MDT but it ran very very slow….and I canceled it. > > I suppose that the correct way to repair quota is to run the e2fsck in the > MDT, but the problem is the system downtime. > > I copy the lfsck stats at the end of this message: > >> lctl get_param -n mdd.lustre-MDT0000.lfsck_layout >> name: lfsck_layout >> magic: 0xb17371b9 >> version: 2 >> status: partial >> flags: >> param: >> last_completed_time: 1555393553 >> time_since_last_completed: 50098 seconds >> latest_start_time: 1555339300 >> time_since_latest_start: 104351 seconds >> last_checkpoint_time: 1555393553 >> time_since_last_checkpoint: 50098 seconds >> latest_start_position: 77 >> last_checkpoint_position: 102983584 >> first_failure_position: 21342296 >> success_count: 1 >> repaired_dangling: 2470024 >> repaired_unmatched_pair: 7487028 >> repaired_multiple_referenced: 0 >> repaired_orphan: 0 >> repaired_inconsistent_owner: 18950288 This is a lot of repairs that would affect quota. >> repaired_others: 0 >> skipped: 0 >> failed_phase1: 1622485 >> failed_phase2: 0 >> checked_phase1: 119083557 >> checked_phase2: 0 >> run_time_phase1: 31777 seconds >> run_time_phase2: 22423 seconds >> average_speed_phase1: 3747 items/sec >> average_speed_phase2: 0 objs/sec >> real-time_speed_phase1: N/A >> real-time_speed_phase2: N/A >> current_position: N/A > > >> lctl lfsck_query -t namespace -M lustre-MDT0000 >> namespace_mdts_init: 0 >> namespace_mdts_scanning-phase1: 0 >> namespace_mdts_scanning-phase2: 1 >> namespace_mdts_completed: 0 >> namespace_mdts_failed: 0 >> namespace_mdts_stopped: 0 >> namespace_mdts_paused: 0 >> namespace_mdts_crashed: 0 >> namespace_mdts_partial: 0 >> namespace_mdts_co-failed: 0 >> namespace_mdts_co-stopped: 0 >> namespace_mdts_co-paused: 0 >> namespace_mdts_unknown: 0 >> namespace_osts_init: 0 >> namespace_osts_scanning-phase1: 0 >> namespace_osts_scanning-phase2: 0 >> namespace_osts_completed: 0 >> namespace_osts_failed: 0 >> namespace_osts_stopped: 0 >> namespace_osts_paused: 0 >> namespace_osts_crashed: 0 >> namespace_osts_partial: 0 >> namespace_osts_co-failed: 0 >> namespace_osts_co-stopped: 0 >> namespace_osts_co-paused: 0 >> namespace_osts_unknown: 0 >> namespace_repaired: 4550034 > > > > ============================================= > Fernando Pérez > Institut de Ciències del Mar (CMIMA-CSIC) > Departament Oceanografía Física i Tecnològica > Passeig Marítim de la Barceloneta,37-49 > 08003 Barcelona > Phone: (+34) 93 230 96 35 > ============================================= > >> El 16 abr 2019, a las 20:43, Martin Hecht <[email protected]> escribió: >> >> Are there a lot of inodes moved to lost+found by the fsck, which contribute >> to the occupied quota now? >> >> ----- Ursprüngliche Mail ----- >> Von: Fernando Pérez <[email protected]> >> An: [email protected] >> Gesendet: Tue, 16 Apr 2019 16:24:13 +0200 (CEST) >> Betreff: Re: [lustre-discuss] lfsck repair quota >> >> Thank you Rick. >> >> I followed these steps for the ldiskfs OSTs and MDT, but the quotes for all >> users is more corrupted than before. >> >> I tried to run e2fsck in ldiskfs OSTs MDT, but the problem was the MDT >> e2fsck ran very slow ( 10 inodes per second for more than 100 million >> inodes). >> >> According to the lustre wiki I though that the lfsck could repair corrupted >> quotes: >> >> http://wiki.lustre.org/Lustre_Quota_Troubleshooting >> >> Regards. >> >> ============================================ >> Fernando Pérez >> Institut de Ciències del Mar (CSIC) >> Departament Oceanografía Física i Tecnològica >> Passeig Marítim de la Barceloneta,37-49 >> 08003 Barcelona >> Phone: (+34) 93 230 96 35 >> ============================================ >> >>> El 16 abr 2019, a las 15:34, Mohr Jr, Richard Frank (Rick Mohr) >>> <[email protected]> escribió: >>> >>> >>>> On Apr 15, 2019, at 10:54 AM, Fernando Perez <[email protected]> wrote: >>>> >>>> Could anyone confirm me that the correct way to repair wrong quotes in a >>>> ldiskfs mdt is lctl lfsck_start -t layout -A? >>> >>> As far as I know, lfsck doesn’t repair quota info. It only fixes internal >>> consistency within Lustre. >>> >>> Whenever I have had to repair quotas, I just follow the procedure you did >>> (unmount everything, run “tune2fs -O ^quota <dev>”, run “tune2fs -O quota >>> <dev>”, and then remount). But all my systems used ldiskfs, so I don’t >>> know if the ZFS OSTs introduce any sort of complication. (Actually, I am >>> not even sure if/how you can regenerate quota info for ZFS.) >>> >>> -- >>> Rick Mohr >>> Senior HPC System Administrator >>> National Institute for Computational Sciences >>> http://www.nics.tennessee.edu >>> >> > Cheers, Andreas --- Andreas Dilger CTO Whamcloud _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
