Re: "Run fsck manually"..?
On Wed 03 Feb 2021 at 15:56:58 (-0500), Greg Wooledge wrote: > On Wed, Feb 03, 2021 at 02:06:04PM -0600, David Wright wrote: > > >From foo/main/init where foo is > > $ unmkinitramfs /boot/initrd.img foo/ > > Interesting I used locate --regex to find where this is coming > from, and it seems there's a copy in /usr/share/initramfs-tools/init > which I'm guessing is what gets installed as init inside the initrd. > > /usr/share/initramfs-tools/init is from the initramfs-tools-core > package which appears to come from Debian itself (according to > the copyright file), having been forked from Ubuntu's version. > > A google search for "linux fsck.mode=force" led me indirectly to the > systemd-fsck man page. This one tells me that fsck.mode= is an option > supported by systemd-fsck with the values auto, force or skip. Debian's > /usr/share/initramfs-tools/init script appears to be using a superset > of these values, including your forcefsck alias. > > Putting these clues all together is tricky for me because nothing is > really spelled out clearly. Near as I can manage to guess, Debian's > initrd init script parses the kernel command line and looks for these > fsck-ish options, applies them during the initrd stage, and then... > then, I don't know what happens. > > There is still no clear end-user-comprehensible documentation that says > how to make fsck happen on the next boot. The first ~43 lines of grub.cfg juggles with the contexts of grubenv. foo/main/init pulls in foo/main/scripts/{functions,local}. The former contains _checkfs_once which translates the forcefsck words etc into options for fsck. The latter mounts local filesystems, calling checkfs. > > > What is the "normal mechanism"? Where is it documented? > > > > https://www.gnu.org/software/grub/manual/grub/grub.html > > (I told you guys how my email is being mangled) > > > I'm guessing the comment is a signature. §15.2 outlines the grubenv > > method. As Stefan mentioned, grubenv is designed to remain static, > > and grub-editenv carefully alters its contents. Configuring Grub > > uses /etc/grub.d/00_header and the variables in /etc/default/grub to > > write the code at the top of /boot/grub/grub.cfg for juggling the > > boot selection. > > > > You start all this off by editing /etc/default/grub: GRUB_DEFAULT=0 → > > GRUB_DEFAULT=savedand running grub-set-default 0. Then: > > > > $ head -c 72 /boot/grub/grubenv ; echo > > # GRUB Environment Block > > saved_entry=0 > > # > > $ > > > > # grub-reboot 'fsck>fsck' > > At this point I give up. I don't know where this "fsck>fsck" argument > comes from or how you use it. It's definitely not in the grub-reboot > man page. That's my implementation, my choice. My grub.cfg postprocessing script turns: submenu 'Advanced options for Debian GNU/Linux' $menuentry_id_option 'gnulinux-advanced-f3bf6eef-3c26-4070-b180-fd1914377253' { menuentry 'Debian GNU/Linux, with Linux 4.19.0-14-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-14-amd64-advanced-f3bf6eef-3c26-4070-b180-fd1914377253' { […] linux /boot/vmlinuz-4.19.0-14-amd64 root=UUID=f3bf6eef-3c26-4070-b180-fd1914377253 ro systemd.show_status=true quiet into: submenu 'Advanced options' $menuentry_id_option 'fsck' { menuentry 'FSCK Debian 4.19.0-14-amd64 FSCK' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'fsck' { […] linux /boot/vmlinuz-4.19.0-14-amd64 root=LABEL=toto04 ro systemd.show_status=true quiet forcefsck The 'FSCK … FSCK' labelling is for visibility when using Grub interactively; the two 'fsck' labels match the invocation grub-reboot 'fsck>fsck' when using the grubenv mechanism. 1024 characters in grubenv is ample to hold two or three of the tediously long default labels, but I choose to use shorter, more meaningful ones. BTW there was some dispute over whether the > operator worked in handling these nested menuentries. I've seen no actual evidence presented, having regularly used them in grub-reboot, and tested them with grub-set-default. No follow-ups to https://lists.debian.org/debian-user/2020/12/msg00641.html Cheers, David.
Re: "Run fsck manually"..?
On Wed, Feb 03, 2021 at 02:06:04PM -0600, David Wright wrote: > >From foo/main/init where foo is > $ unmkinitramfs /boot/initrd.img foo/ Interesting I used locate --regex to find where this is coming from, and it seems there's a copy in /usr/share/initramfs-tools/init which I'm guessing is what gets installed as init inside the initrd. /usr/share/initramfs-tools/init is from the initramfs-tools-core package which appears to come from Debian itself (according to the copyright file), having been forked from Ubuntu's version. A google search for "linux fsck.mode=force" led me indirectly to the systemd-fsck man page. This one tells me that fsck.mode= is an option supported by systemd-fsck with the values auto, force or skip. Debian's /usr/share/initramfs-tools/init script appears to be using a superset of these values, including your forcefsck alias. Putting these clues all together is tricky for me because nothing is really spelled out clearly. Near as I can manage to guess, Debian's initrd init script parses the kernel command line and looks for these fsck-ish options, applies them during the initrd stage, and then... then, I don't know what happens. There is still no clear end-user-comprehensible documentation that says how to make fsck happen on the next boot. > > What is the "normal mechanism"? Where is it documented? > > https://www.gnu.org/software/grub/manual/grub/grub.html (I told you guys how my email is being mangled) > I'm guessing the comment is a signature. §15.2 outlines the grubenv > method. As Stefan mentioned, grubenv is designed to remain static, > and grub-editenv carefully alters its contents. Configuring Grub > uses /etc/grub.d/00_header and the variables in /etc/default/grub to > write the code at the top of /boot/grub/grub.cfg for juggling the > boot selection. > > You start all this off by editing /etc/default/grub: GRUB_DEFAULT=0 → > GRUB_DEFAULT=savedand running grub-set-default 0. Then: > > $ head -c 72 /boot/grub/grubenv ; echo > # GRUB Environment Block > saved_entry=0 > # > $ > > # grub-reboot 'fsck>fsck' At this point I give up. I don't know where this "fsck>fsck" argument comes from or how you use it. It's definitely not in the grub-reboot man page.
Re: "Run fsck manually"..?
On Wed 03 Feb 2021 at 11:24:38 (-0500), Greg Wooledge wrote: > On Wed, Feb 03, 2021 at 10:02:42AM -0600, David Wright wrote: > > T'other way, I think, but no matter. For the record, there are > > two different "strengths" for fsck: > > > > fastboot|fsck.mode=skip) > > fastboot=y > > ;; > > forcefsck|fsck.mode=force) > > forcefsck=y > > ;; > > fsckfix|fsck.repair=yes) > > fsckfix=y > > ;; > > fsck.repair=no) > > fsckfix=n > > ;; > > Where is this coming from? >From foo/main/init where foo is $ unmkinitramfs /boot/initrd.img foo/ > > If you have a separate /boot that is not on a filesystem needing > > fscking, then you can use the normal mechanism supported by grub¹. > > > ¹ It writes to the file /boot/grub/grubenv, hence the caution. > > What is the "normal mechanism"? Where is it documented? https://www.gnu.org/software/grub/manual/grub/grub.html > There are no man pages for "grub" or for "grubenv" on my system. > There is a /boot/grub/grubenv file but it consists of one comment line, > ending in a newline, and then enough # characters to make the file size > exactly 1024 bytes. Incomprehensible. I'm guessing the comment is a signature. §15.2 outlines the grubenv method. As Stefan mentioned, grubenv is designed to remain static, and grub-editenv carefully alters its contents. Configuring Grub uses /etc/grub.d/00_header and the variables in /etc/default/grub to write the code at the top of /boot/grub/grub.cfg for juggling the boot selection. You start all this off by editing /etc/default/grub: GRUB_DEFAULT=0 → GRUB_DEFAULT=savedand running grub-set-default 0. Then: $ head -c 72 /boot/grub/grubenv ; echo # GRUB Environment Block saved_entry=0 # $ # grub-reboot 'fsck>fsck' # $ head -c 72 /boot/grub/grubenv ; echo # GRUB Environment Block saved_entry=0 next_entry=fsck>fsck $ I'm doing the next bit now to restore normality, but this would also be done automatically if you boot with a next_entry set, so that the altered boot is a one-off. # grub-set-default 0 # $ head -c 72 /boot/grub/grubenv ; echo # GRUB Environment Block saved_entry=0 # $ Cheers, David.
Re: "Run fsck manually"..?
> There are no man pages for "grub" or for "grubenv" on my system. > There is a /boot/grub/grubenv file but it consists of one comment line, > ending in a newline, and then enough # characters to make the file size > exactly 1024 bytes. Incomprehensible. This is just a place holder to make sure the file can be modified without having to allocate any new space in the filesystem. A single write of a single sector is all it takes. Stefan
Re: "Run fsck manually"..?
On Wed, Feb 03, 2021 at 10:02:42AM -0600, David Wright wrote: > T'other way, I think, but no matter. For the record, there are > two different "strengths" for fsck: > > fastboot|fsck.mode=skip) > fastboot=y > ;; > forcefsck|fsck.mode=force) > forcefsck=y > ;; > fsckfix|fsck.repair=yes) > fsckfix=y > ;; > fsck.repair=no) > fsckfix=n > ;; Where is this coming from? > If you have a separate /boot that is not on a filesystem needing > fscking, then you can use the normal mechanism supported by grub¹. > ¹ It writes to the file /boot/grub/grubenv, hence the caution. What is the "normal mechanism"? Where is it documented? There are no man pages for "grub" or for "grubenv" on my system. There is a /boot/grub/grubenv file but it consists of one comment line, ending in a newline, and then enough # characters to make the file size exactly 1024 bytes. Incomprehensible.
Re: "Run fsck manually"..?
On Wed 03 Feb 2021 at 07:32:25 (-0500), Greg Wooledge wrote: > On Tue, Feb 02, 2021 at 10:01:30PM -0600, David Wright wrote: > > > # touch /forcefsck > > > > I think this is somewhat out of date, is it not. > > It is, yes. That no longer works in systemd. > > > I force fsck by adding forcefsck in grub, ie press e in > > the grub menu, move the cursor to the end of the linux line, > > type forcefsck and press Ctrl-X or F10 to boot. For example, > > > > linux /boot/vmlinuz-4.19.0-14-amd64 root=LABEL=toto04 ro > > systemd.show_status=true quiet forcefsck > > The instructions from the bot in #debian say to add fsck.mode=force > to the kernel parameters. Perhaps your variant is some sort of alias. T'other way, I think, but no matter. For the record, there are two different "strengths" for fsck: fastboot|fsck.mode=skip) fastboot=y ;; forcefsck|fsck.mode=force) forcefsck=y ;; fsckfix|fsck.repair=yes) fsckfix=y ;; fsck.repair=no) fsckfix=n ;; > I am not aware of any one-shot variant that can be used *before* > rebooting, without being physically present at the console. Sadly. > The closest approximation would be to permanently edit the grub menu, > reboot, and then restore your backup copy of the grub menu. If you have a separate /boot that is not on a filesystem needing fscking, then you can use the normal mechanism supported by grub¹. On my systems, I would run a script containing grub-reboot 'fsck>fsck' followed by either reboot or /sbin/shutdown -P now (to leave checking until the next day). Yes, it involves either editing the Grub menu, or learning how to change it the official way through the scripts in /etc/grub.d/. Being lazy, I just postprocess grub.cfg whenever it's upgraded. My script² takes advantage of the fact that the first menuentry and the first ("Advanced") submenu's menuentry are identical, so all it has to do is change the labelling of the submenu and menuentry (hence 'fsck>fsck' above) and add forcefsck to the kernel line. I suppose you could do it through a /etc/grub.d/40_custom entry, keeping it up to date by using the symlinks in /, or by manually maintaining corresponding symlinks in /boot if it's separate. I've never bothered with automating a fsckfix option—if things are that bad, I want to be present when fsck runs. ¹ It writes to the file /boot/grub/grubenv, hence the caution. ² The principal function of my script is to convert all the UUIDs into LABELs by looking up /run/udev/data/. Cheers, David.
Re: "Run fsck manually"..?
On Tue, Feb 02, 2021 at 10:01:30PM -0600, David Wright wrote: > > # touch /forcefsck > > I think this is somewhat out of date, is it not. It is, yes. That no longer works in systemd. > I force fsck by adding forcefsck in grub, ie press e in > the grub menu, move the cursor to the end of the linux line, > type forcefsck and press Ctrl-X or F10 to boot. For example, > > linux /boot/vmlinuz-4.19.0-14-amd64 root=LABEL=toto04 ro > systemd.show_status=true quiet forcefsck The instructions from the bot in #debian say to add fsck.mode=force to the kernel parameters. Perhaps your variant is some sort of alias. I am not aware of any one-shot variant that can be used *before* rebooting, without being physically present at the console. Sadly. The closest approximation would be to permanently edit the grub menu, reboot, and then restore your backup copy of the grub menu.
Re: "Run fsck manually"..?
On 03/02/2021 04:01, David Wright wrote: > On Wed 03 Feb 2021 at 01:41:54 (+), Andy Smith wrote: >> On Tue, Feb 02, 2021 at 07:13:16PM -0500, hobie of RMN wrote: >>> My brother's Debian system suddenly says on attempt to boot, "/dev/sda1: >>> UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were part of >>> a corrupted orphan linked list found." >>> >>> He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck >>> identifying it's version, and nothing else. >> There can be issues trying to run fsck on a mounted filesystem. What >> happens if you do: >> >> # touch /forcefsck > I think this is somewhat out of date, is it not. > > I force fsck by adding forcefsck in grub, ie press e in > the grub menu, move the cursor to the end of the linux line, > type forcefsck and press Ctrl-X or F10 to boot. For example, > > linux /boot/vmlinuz-4.19.0-14-amd64 root=LABEL=toto04 ro > systemd.show_status=true quiet forcefsck Good point. I seem to recall there being some discussion a few years ago around the wisdom of having to write to a file system to fsck it. That is, if you know the file system is broken and want to fsck it but you don't have a way to interactively run fsck yourself, do you _really_ want to be modifying the file system? It's entirely possible that, by writing /forcefsck to the file system, you destroy some other valuable bit of data that fsck might have been able to save. OpenPGP_signature Description: OpenPGP digital signature
[SOLVED} Re: "Run fsck manually"..?
> On Wed, Feb 03, 2021 at 01:41:54AM +, Andy Smith wrote: >> On Tue, Feb 02, 2021 at 07:13:16PM -0500, hobie of RMN wrote: >> > He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck >> > identifying it's version, and nothing else. >> >> There can be issues trying to run fsck on a mounted filesystem. What >> happens if you do: >> >> # touch /forcefsck > > Oh, sorry, I missed your mention of (initramfs) prompt. So your > filesystem is too damaged to allow boot to complete and you won't be > able to do that "touch /forcefsck" thing. > > If fsck is just printing its version it may think it doesn't need to > be run. You can force it to do a check/repair with "-f", so: > > (initramfs) fsck.ext4 -vf /dev/sda1 > > If it find things that it wants to fix it will ask yuo and you'll > have to press 'y' each time. If you're certain that you always want > to answer 'y' then you can ctrl-c that and try again with -y: > > (initramfs) fsck.ext4 -yvf /dev/sda1 > > If you want to see what it would do without it actually doing it you > can use -n instead of -y. > > Cheers, > Andy Thanks, Andy and everyone. :) From the (initramfs) prompt, fsck -y /dev/sda1 did the job. :) My brother finally realized he'd entered an extra character originally, causing fsck to fail on his original attempt - he had entered "./dev/sda1" instead of "/dev/sda1" - so removing that '.' was part of solving this. Like so many these days, he spends mos or all of his time in the GUI rather than at the command line. --hobie
Re: "Run fsck manually"..?
On Wed 03 Feb 2021 at 01:41:54 (+), Andy Smith wrote: > > On Tue, Feb 02, 2021 at 07:13:16PM -0500, hobie of RMN wrote: > > My brother's Debian system suddenly says on attempt to boot, "/dev/sda1: > > UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were part of > > a corrupted orphan linked list found." > > > > He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck > > identifying it's version, and nothing else. > > There can be issues trying to run fsck on a mounted filesystem. What > happens if you do: > > # touch /forcefsck I think this is somewhat out of date, is it not. I force fsck by adding forcefsck in grub, ie press e in the grub menu, move the cursor to the end of the linux line, type forcefsck and press Ctrl-X or F10 to boot. For example, linux /boot/vmlinuz-4.19.0-14-amd64 root=LABEL=toto04 ro systemd.show_status=true quiet forcefsck > That will force the system to do a fsck on boot, before the > filesystem is mounted for use. If that doesn't help I think you will > indeed have to try this from a live or rescue environment. The > Debian install media can boot into a rescue mode for tasks like > this. Cheers, David.
Re: "Run fsck manually"..?
On Wed, Feb 03, 2021 at 01:41:54AM +, Andy Smith wrote: > On Tue, Feb 02, 2021 at 07:13:16PM -0500, hobie of RMN wrote: > > He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck > > identifying it's version, and nothing else. > > There can be issues trying to run fsck on a mounted filesystem. What > happens if you do: > > # touch /forcefsck Oh, sorry, I missed your mention of (initramfs) prompt. So your filesystem is too damaged to allow boot to complete and you won't be able to do that "touch /forcefsck" thing. If fsck is just printing its version it may think it doesn't need to be run. You can force it to do a check/repair with "-f", so: (initramfs) fsck.ext4 -vf /dev/sda1 If it find things that it wants to fix it will ask yuo and you'll have to press 'y' each time. If you're certain that you always want to answer 'y' then you can ctrl-c that and try again with -y: (initramfs) fsck.ext4 -yvf /dev/sda1 If you want to see what it would do without it actually doing it you can use -n instead of -y. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: "Run fsck manually"..?
> You might have to boot from a recovery CD image, such as a Debian live > install image, or GParted Live. You can't actually run fsck on a drive > while said drive is mounted. Thank, Jeremy. But - is /dev/sda1 mounted at this point? Isn't it being indicated to us that it can't be successfully mounted? (Thinking of the Busybox appearance and (intramfs) reference.) > On Tue, 2 Feb 2021 at 19:24, Stefan Monnier > wrote: > >> >> My brother's Debian system suddenly says on attempt to boot, >> "/dev/sda1: >> >> UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were >> part >> of >> >> a corrupted orphan linked list found." >> >> >> >> He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck >> >> identifying it's version, and nothing else. Tha appears to take >> place >> >> from (initramfs) and Busybox. An attempt to reboot just starts the >> >> problem all over again. >> >> >> >> We'd be grateful for help with this. Thanks. >> >> >> > hello, >> > >> > fsck -fy /dev/sda1 is probably what you want >> >> Then again, after the "UNEXPECTED INCONSISTENCY", the `-f` flag to >> `fsck` shouldn't be needed. This is weird. >> >> >> Stefan >> >> >
Re: "Run fsck manually"..?
Hi, On Tue, Feb 02, 2021 at 07:13:16PM -0500, hobie of RMN wrote: > My brother's Debian system suddenly says on attempt to boot, "/dev/sda1: > UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were part of > a corrupted orphan linked list found." > > He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck > identifying it's version, and nothing else. There can be issues trying to run fsck on a mounted filesystem. What happens if you do: # touch /forcefsck # reboot ? That will force the system to do a fsck on boot, before the filesystem is mounted for use. If that doesn't help I think you will indeed have to try this from a live or rescue environment. The Debian install media can boot into a rescue mode for tasks like this. The only time I've had something like this was when I created an ext4 filesystem in Debian buster and then used it as a root filesystem for CentOS 7. The ext code in buster used a new filesystem feature that isn't present in the ext4 driver in CentOS 7, so when CentOS 7 tried to mount its root filesystem it said there were "inconsistencies" every time. Yet doing a fsck in CentOS or trying the /forcefsck strategy I mentioned above just said the filesystem was fine, and the filesystem seemed fine in everyday use. This was because the fsck in CentOS also could not understand the new filesystem feature. In the end I had to fsck it from Debian buster and then remove the feature with tune2fs. CentOS 7 was happy with it then. I am not saying this is what has happened to you. I'm just giving an example of one weird set of circumstances that can lead to something like this. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: "Run fsck manually"..?
You might have to boot from a recovery CD image, such as a Debian live install image, or GParted Live. You can't actually run fsck on a drive while said drive is mounted. On Tue, 2 Feb 2021 at 19:24, Stefan Monnier wrote: > >> My brother's Debian system suddenly says on attempt to boot, "/dev/sda1: > >> UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were part > of > >> a corrupted orphan linked list found." > >> > >> He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck > >> identifying it's version, and nothing else. Tha appears to take place > >> from (initramfs) and Busybox. An attempt to reboot just starts the > >> problem all over again. > >> > >> We'd be grateful for help with this. Thanks. > >> > > hello, > > > > fsck -fy /dev/sda1 is probably what you want > > Then again, after the "UNEXPECTED INCONSISTENCY", the `-f` flag to > `fsck` shouldn't be needed. This is weird. > > > Stefan > >
Re: "Run fsck manually"..?
>> My brother's Debian system suddenly says on attempt to boot, "/dev/sda1: >> UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were part of >> a corrupted orphan linked list found." >> >> He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck >> identifying it's version, and nothing else. Tha appears to take place >> from (initramfs) and Busybox. An attempt to reboot just starts the >> problem all over again. >> >> We'd be grateful for help with this. Thanks. >> > hello, > > fsck -fy /dev/sda1 is probably what you want Then again, after the "UNEXPECTED INCONSISTENCY", the `-f` flag to `fsck` shouldn't be needed. This is weird. Stefan
Re: "Run fsck manually"..?
On 2/2/21 19:13, hobie of RMN wrote: My brother's Debian system suddenly says on attempt to boot, "/dev/sda1: UNEXPECTED INCONSISTENCY:Runfsck manually", and, "inodes that were part of a corrupted orphan linked list found." He enters "fsck" or "fsck /dev/sda1", and in a short while gets fsck identifying it's version, and nothing else. Tha appears to take place from (initramfs) and Busybox. An attempt to reboot just starts the problem all over again. We'd be grateful for help with this. Thanks. hello, fsck -fy /dev/sda1 is probably what you want