[Solved(?)] Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

After 3 months of going Little Red Hen on this, I think I figured it
out. My system had live-config & live-boot (with their dependencies,
live-boot-initramfs-tools, user-setup, live-tools and
live-config-systemd) installed. An aptitude purge command on these
packages got my system booting normally again.

Despite 
http://debian-live.alioth.debian.org/live-manual/stable/manual/html/live-manual.en.html#139
saying that installing live-config & live-boot 'will do no harm and is
useful for reference purposes', it clearly bugged up booting.

I can't say that I'll miss having to hit E at the grub boot loader,
add single or systemd.debug-shell to the boot parameters, F10 and type
`sudo systemctl isolate default` at my first shell EVERY SINGLE TIME I
boot my computer. I only hope that others benefit from my suffering
and the irreplaceable hours of my life sucked into this time vampire
so that the cycle of despair can shorten if ever so slightly.

Off to whine to the package managers unless there are any insights to
explain how these packages can coexist on my system.

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

I'm still stuck with this problem after wasting another 5 hours on it
last night. I'm still getting the time out error message but, without
any useful debugging information, don't know where to start
troubleshooting. Further, it appears the systemd maintainers have
given up responding to bug reports since they haven't responded to bug
#758808 in over 5 months.

Since it will be at least a year before the next Debian release comes
out - when I usually wipe and reload my machine - it's actually worth
my while to put a bounty of a couple hundred bucks for the person who
can help me solve this problem.

Who's interested?

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

As my previous messages have gone entirely ignored, I'm hoping that
someone can help me with this problem, which I hope is simpler. My
boot times out trying to start a number of devices. One of them, for
example, is dev-mapper-LVGx2dtmp.device. Another is
dev-disk-byx2duuid-.device. I can `systemctl show ` these
devices but I can't disable them. Can someone tell me what they're
there for, what created them and how I can, for debugging purposes,
disable them from being called?

On 22 July 2016 at 00:09, Borden Rhodes  wrote:
> During the 90 second wait before my boot times out, I dropped into a
> debug-shell and ran `systemctl list-units`. This was the output:
> http://paste.debian.net/783995/ (good for 3 days). Of curiosity is
> that the other dev-mapper-LVG... devices show that they're inactive
> except for dev-mapper-LVG\x2droot.device, which makes sense since this
> is given a pass of 1 in /etc/fstab. What sorts of operations would be
> running on dev-mapper-LVG\x2droot.device so I can isolate which one
> might be hanging up? Also note in the dump that -.mount is both active
> and successfully mounted, which may explain why, even after the
> dev-mapper-LVG... processes time out that the debug-shell still shows
> a properly-mounted filesystem.
>
> I suspected it might be fsck but disabling it both from fstab (pass=0)
> and the kernel command line (fsck.mode=skip) didn't work. What else
> can I check?
>
> On 19 July 2016 at 02:54, Borden Rhodes  wrote:
>> Thank you for your message, Michael, and please forgive the delay in 
>> responding.
>>
>> I tried booting with the 4.5 kernel after 4.6 failed to boot. It
>> seems, by then, that the damage had been done as I got identical
>> symptoms on both boots. I agree with you that the cryptsetup/LVM is to
>> blame (although I'd blame LVM more).
>>
>> The hypothesis to test multi-user.wants came from being able to boot
>> into single user mode without incident and isolate default.target once
>> I'm in single user mode. I can also isolate default.target from the
>> early debug shell.
>>
>> I tried to follow your advice. It seems that my box could accurately
>> identify the partitions from `ls`-ing through the /dev directory and
>> seeing everything set up correctly. fstab and crypttab also seem to be
>> intact during the hangup.
>>
>> I ran `udevadm info` on everything I could find in /sys/class/block/
>> the settings you told me to check are as follows:
>> ./dm-0 (mapper/sda5_crypt): SYSTEMD_READY=1; TAGS=:systemd:
>> ./dm-1 (mapper/LVG-root): TAGS=:systemd: (SYSTEMD_READY is not present)
>> ./dm-2 (mapper/LVG-var): TAGS=:systemd: (SYSTEMD_READY is not present)
>> ./dm-3 (mapper/LVG-tmp): TAGS=:systemd: (SYSTEMD_READY is not present)
>> ./dm-4 (mapper/LVG-home): TAGS=:systemd: (SYSTEMD_READY is not present)
>> ./sda: TAGS=:systemd: (SYSTEMD_READY is not present)
>> ./sda1 (/boot): TAGS=:systemd: (SYSTEMD_READY is not present)
>> ./sda5 (crypttab/LVM partition): TAGS=:systemd: (SYSTEMD_READY is not 
>> present)
>>
>> I hope that's legible. I can pastebin the full output for each of
>> those commands if it helps.
>>
>> For kicks and giggles, I ran `sudo lvmconfig --type diff` which yielded
>> devices {
>> cache_dir="/run/lvm"
>> }
>> I'm grasping at straws so I don't know if this is relevant or not.
>>
>> With thanks,
>>
>> By-the-by, since it's been a while since I've been able to tackle
>> this, here's the rest of the e-mail thread for context:
>> https://lists.debian.org/debian-user/2016/06/msg00670.html

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

During the 90 second wait before my boot times out, I dropped into a
debug-shell and ran `systemctl list-units`. This was the output:
http://paste.debian.net/783995/ (good for 3 days). Of curiosity is
that the other dev-mapper-LVG... devices show that they're inactive
except for dev-mapper-LVG\x2droot.device, which makes sense since this
is given a pass of 1 in /etc/fstab. What sorts of operations would be
running on dev-mapper-LVG\x2droot.device so I can isolate which one
might be hanging up? Also note in the dump that -.mount is both active
and successfully mounted, which may explain why, even after the
dev-mapper-LVG... processes time out that the debug-shell still shows
a properly-mounted filesystem.

I suspected it might be fsck but disabling it both from fstab (pass=0)
and the kernel command line (fsck.mode=skip) didn't work. What else
can I check?

On 19 July 2016 at 02:54, Borden Rhodes  wrote:
> Thank you for your message, Michael, and please forgive the delay in 
> responding.
>
> I tried booting with the 4.5 kernel after 4.6 failed to boot. It
> seems, by then, that the damage had been done as I got identical
> symptoms on both boots. I agree with you that the cryptsetup/LVM is to
> blame (although I'd blame LVM more).
>
> The hypothesis to test multi-user.wants came from being able to boot
> into single user mode without incident and isolate default.target once
> I'm in single user mode. I can also isolate default.target from the
> early debug shell.
>
> I tried to follow your advice. It seems that my box could accurately
> identify the partitions from `ls`-ing through the /dev directory and
> seeing everything set up correctly. fstab and crypttab also seem to be
> intact during the hangup.
>
> I ran `udevadm info` on everything I could find in /sys/class/block/
> the settings you told me to check are as follows:
> ./dm-0 (mapper/sda5_crypt): SYSTEMD_READY=1; TAGS=:systemd:
> ./dm-1 (mapper/LVG-root): TAGS=:systemd: (SYSTEMD_READY is not present)
> ./dm-2 (mapper/LVG-var): TAGS=:systemd: (SYSTEMD_READY is not present)
> ./dm-3 (mapper/LVG-tmp): TAGS=:systemd: (SYSTEMD_READY is not present)
> ./dm-4 (mapper/LVG-home): TAGS=:systemd: (SYSTEMD_READY is not present)
> ./sda: TAGS=:systemd: (SYSTEMD_READY is not present)
> ./sda1 (/boot): TAGS=:systemd: (SYSTEMD_READY is not present)
> ./sda5 (crypttab/LVM partition): TAGS=:systemd: (SYSTEMD_READY is not present)
>
> I hope that's legible. I can pastebin the full output for each of
> those commands if it helps.
>
> For kicks and giggles, I ran `sudo lvmconfig --type diff` which yielded
> devices {
> cache_dir="/run/lvm"
> }
> I'm grasping at straws so I don't know if this is relevant or not.
>
> With thanks,
>
> By-the-by, since it's been a while since I've been able to tackle
> this, here's the rest of the e-mail thread for context:
> https://lists.debian.org/debian-user/2016/06/msg00670.html

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

Thank you for your message, Michael, and please forgive the delay in responding.

I tried booting with the 4.5 kernel after 4.6 failed to boot. It
seems, by then, that the damage had been done as I got identical
symptoms on both boots. I agree with you that the cryptsetup/LVM is to
blame (although I'd blame LVM more).

The hypothesis to test multi-user.wants came from being able to boot
into single user mode without incident and isolate default.target once
I'm in single user mode. I can also isolate default.target from the
early debug shell.

I tried to follow your advice. It seems that my box could accurately
identify the partitions from `ls`-ing through the /dev directory and
seeing everything set up correctly. fstab and crypttab also seem to be
intact during the hangup.

I ran `udevadm info` on everything I could find in /sys/class/block/
the settings you told me to check are as follows:
./dm-0 (mapper/sda5_crypt): SYSTEMD_READY=1; TAGS=:systemd:
./dm-1 (mapper/LVG-root): TAGS=:systemd: (SYSTEMD_READY is not present)
./dm-2 (mapper/LVG-var): TAGS=:systemd: (SYSTEMD_READY is not present)
./dm-3 (mapper/LVG-tmp): TAGS=:systemd: (SYSTEMD_READY is not present)
./dm-4 (mapper/LVG-home): TAGS=:systemd: (SYSTEMD_READY is not present)
./sda: TAGS=:systemd: (SYSTEMD_READY is not present)
./sda1 (/boot): TAGS=:systemd: (SYSTEMD_READY is not present)
./sda5 (crypttab/LVM partition): TAGS=:systemd: (SYSTEMD_READY is not present)

I hope that's legible. I can pastebin the full output for each of
those commands if it helps.

For kicks and giggles, I ran `sudo lvmconfig --type diff` which yielded
devices {
cache_dir="/run/lvm"
}
I'm grasping at straws so I don't know if this is relevant or not.

With thanks,

By-the-by, since it's been a while since I've been able to tackle
this, here's the rest of the e-mail thread for context:
https://lists.debian.org/debian-user/2016/06/msg00670.html

Re: boot times out after dist-upgrade on Stretch

[Michael Biebl]

Am 15.06.2016 um 09:58 schrieb Borden Rhodes:
> linux-image-4.6.0-1-amd64:amd64 (4.6.1-1, automatic)

...

> Could I get direction on how to troubleshoot this?

In your list of upgraded packages, only the kernel package looks
suspiciuos. I suspect your system worked correctly before the upgrade?
Does downgrading the kernel work?

For further debugging there is
https://freedesktop.org/wiki/Software/systemd/Debugging/#index1h1

Starting the early debug shell let's you examine the system

I don't expect that services in multi-user.target are causing the issue,
but it's rather a cryptsetup/lvm issue. Most likely the devices are not
properly marked as available, so systemd times out waiting for them.

In your early debug shell, check if the devices listed in /etc/crypttab
and /etc/fstab are available.
Also check the udev db via
udevadm info /sys/class/block/<...>
for your devices. Look for TAGS=:systemd: and the SYSTEMD_READY property.



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

Thank you, Mark, and I apologise for not acknowledging your message
sooner. I moved all of the symlinks in
/etc/systemd/system/multi-user.target.wants/ to my home directory and
restarted. The same symptoms appeared.

I don't suppose there's some utility that can walk through a target
unit and display all of the dependant units in a nice, pretty format
to save me from having to do it myself (and making lots of mistakes in
the process)?

With thanks,

On 16 June 2016 at 23:51, Mark Fletcher <mark2...@gmail.com> wrote:
>
> On Fri, 17 Jun 2016 at 03:12, Borden Rhodes <j...@bordenrhodes.com> wrote:
>>
>> Thank you for getting back to me, Sven,
>>
>> I normally run apt-get update; apt-get dist-upgrade immediately after
>> my computer boots. According to the messages log, I turned the
>> computer on about 5 minutes before running that command and the last
>> log entry was about 3.5 hours later at 22:59. I hadn't fiddled with
>> any other settings during that boot. Unfortunately, without /var
>> loading on the dead boots, I can't get any log information except when
>> I successfully boot into the recovery console.
>>
>> I should have mentioned that I also tried booting from the 4.5 kernel
>> and got the exact same symptoms. I also tried running update-grub in
>> case it had made a mistake whilst installing the 4.6 kernel.
>>
>> Notwithstanding better ideas, I'm thinking of doing the Windows Safe
>> Mode troubleshooting method where I work out the systemd differences
>> between the default and recovery targets and gradually add services
>> until I find the one that breaks the system. I'm inferring that, since
>> recovery mode works but normal mode doesn't, then one of the
>> targets/services in normal mode is to blame for the lock up. I don't
>> suppose I could trouble the list for a resource on how to do that?
>>
>> > Date: Wed, 15 Jun 2016 19:35:34 +0200
>> > From: Sven Joachim <svenj...@gmx.de>
>> > To: debian-user@lists.debian.org
>> > Subject: Re: boot times out after dist-upgrade on Stretch
>> > Message-ID: <8760tapd7d@turtle.gmx.de>
>> > Content-Type: text/plain
>> >
>> > On 2016-06-15 07:58 +, Borden Rhodes wrote:
>> >
>> >> I ran apt dist-upgrade on Stretch (with a few Sid packages) which made
>> >> the following changes:
>> >>
>> >> Start-Date: 2016-06-14  19:42:39
>> >> Commandline: apt-get dist-upgrade
>> >> Requested-By: me (1000)
>> >> Install: libdw1:amd64 (0.163-5.1, automatic),
>> >> linux-image-4.6.0-1-amd64:amd64 (4.6.1-1, automatic)
>> >> Upgrade: wwwconfig-common:amd64 (0.2.2, 0.3.0), libcomerr2:amd64
>> >> (1.43-3, 1.43.1-1), libcomerr2:i386 (1.43-3, 1.43.1-1), libcups2:amd64
>> >> (2.1.3-5, 2.1.3-6), fuse2fs:amd64 (1.43-3, 1.43.1-1), e2fsprogs:amd64
>> >> (1.43-3, 1.43.1-1), boinc-client:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1),
>> >> libbabeltrace1:amd64 (1.3.2-1, 1.4.0-1), cups-server-common:amd64
>> >> (2.1.3-5, 2.1.3-6), e2fslibs:amd64 (1.43-3, 1.43.1-1),
>> >> cups-common:amd64 (2.1.3-5, 2.1.3-6), libspice-server1:amd64
>> >> (0.12.6-4, 0.12.6-4.1), boinc-manager:amd64 (7.6.32+dfsg-2,
>> >> 7.6.33+dfsg-1), libss2:amd64 (1.43-3, 1.43.1-1), live-config-doc:amd64
>> >> (5.20151121, 5.20160608), libdatetime-timezone-perl:amd64
>> >> (1:1.98-1+2016d, 1:2.00-1+2016d), cups-ppdc:amd64 (2.1.3-5, 2.1.3-6),
>> >> libcupsmime1:amd64 (2.1.3-5, 2.1.3-6), python-paramiko:amd64
>> >> (1.16.0-1, 2.0.0-1), linux-image-amd64:amd64 (4.5+73, 4.6+74),
>> >> libboinc7:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1), libcupsppdc1:amd64
>> >> (2.1.3-5, 2.1.3-6), libbabeltrace-ctf1:amd64 (1.3.2-1, 1.4.0-1),
>> >> live-config:amd64 (5.20151121, 5.20160608), cups-bsd:amd64 (2.1.3-5,
>> >> 2.1.3-6), cups-core-drivers:amd64 (2.1.3-5, 2.1.3-6),
>> >> cups-daemon:amd64 (2.1.3-5, 2.1.3-6), libcupsimage2:amd64 (2.1.3-5,
>> >> 2.1.3-6), cups:amd64 (2.1.3-5, 2.1.3-6), boinc:amd64 (7.6.32+dfsg-2,
>> >> 7.6.33+dfsg-1), libcupscgi1:amd64 (2.1.3-5, 2.1.3-6),
>> >> cups-client:amd64 (2.1.3-5, 2.1.3-6), live-config-systemd:amd64
>> >> (5.20151121, 5.20160608), libjpeg62-turbo:amd64 (1:1.4.2-2,
>> >> 1:1.5.0-1), libjpeg62-turbo:i386 (1:1.4.2-2, 1:1.5.0-1), xterm:amd64
>> >> (324-2, 325-1)
>> >> End-Date: 2016-06-14  19:46:44
>> >
>> > The only package related to the boot process seems to be
>> > linux-image-4.6.0-1-amd64.  However, there could be others which were
&g

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

First, a huge shout out to /usr/share/doc/systemd/README.Debian.gz for
telling me that appending systemd.debug-shell to the linux line in
grub would let me run a shell even whilst the main boot couldn't drop
into an emergency shell. I think this should be writ in large friendly
letters at the start of any boot-related troubleshooting guide from
now on.

With that, I could dump journalctl and systemctl list-jobs to files.
In perusing these files, it confirms that boot chokes at 4
dev-disk-by\x2duuid-<...>.device, dev-mapper-sda5_crypt.device &
dev-mapper-LVG\x2d<home|var|tmp>.device, but curiously not
dev-mapper-LVG\x2droot. root, of course, has a fstab pass value of 1
whereas the others have a pass value of 2. Is this a clue? The
list-jobs dumps also show some 94 other units waiting in line to run.

A little search-fu later, it seems I may have caught this bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=758808. However,
other searches on other distros suggest everything from hdparm needing
to be removed (didn't work), to incorrect fstab entries (did you guys
see any error in them?), to lunar cycles, to a butterfly flapping its
wings etc.

Anyhow, notwithstanding any better ideas, I think I'm going to
redirect this conversation to bug 758808.

> Date: Thu, 16 Jun 2016 13:56:48 -0400
> From: Borden Rhodes <j...@bordenrhodes.com>
> To: debian-user@lists.debian.org
> Subject: Re: boot times out after dist-upgrade on Stretch
> Message-ID: 
> <CABeHYofhP+9tVT9BM=a3txx5hwufu3rztpqcnb99mbmhsog...@mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Thank you for getting back to me, Sven,
>
> I normally run apt-get update; apt-get dist-upgrade immediately after
> my computer boots. According to the messages log, I turned the
> computer on about 5 minutes before running that command and the last
> log entry was about 3.5 hours later at 22:59. I hadn't fiddled with
> any other settings during that boot. Unfortunately, without /var
> loading on the dead boots, I can't get any log information except when
> I successfully boot into the recovery console.
>
> I should have mentioned that I also tried booting from the 4.5 kernel
> and got the exact same symptoms. I also tried running update-grub in
> case it had made a mistake whilst installing the 4.6 kernel.
>
> Notwithstanding better ideas, I'm thinking of doing the Windows Safe
> Mode troubleshooting method where I work out the systemd differences
> between the default and recovery targets and gradually add services
> until I find the one that breaks the system. I'm inferring that, since
> recovery mode works but normal mode doesn't, then one of the
> targets/services in normal mode is to blame for the lock up. I don't
> suppose I could trouble the list for a resource on how to do that?

Re: boot times out after dist-upgrade on Stretch

[Mark Fletcher]

On Fri, 17 Jun 2016 at 03:12, Borden Rhodes <j...@bordenrhodes.com> wrote:

> Thank you for getting back to me, Sven,
>
> I normally run apt-get update; apt-get dist-upgrade immediately after
> my computer boots. According to the messages log, I turned the
> computer on about 5 minutes before running that command and the last
> log entry was about 3.5 hours later at 22:59. I hadn't fiddled with
> any other settings during that boot. Unfortunately, without /var
> loading on the dead boots, I can't get any log information except when
> I successfully boot into the recovery console.
>
> I should have mentioned that I also tried booting from the 4.5 kernel
> and got the exact same symptoms. I also tried running update-grub in
> case it had made a mistake whilst installing the 4.6 kernel.
>
> Notwithstanding better ideas, I'm thinking of doing the Windows Safe
> Mode troubleshooting method where I work out the systemd differences
> between the default and recovery targets and gradually add services
> until I find the one that breaks the system. I'm inferring that, since
> recovery mode works but normal mode doesn't, then one of the
> targets/services in normal mode is to blame for the lock up. I don't
> suppose I could trouble the list for a resource on how to do that?
>
> > Date: Wed, 15 Jun 2016 19:35:34 +0200
> > From: Sven Joachim <svenj...@gmx.de>
> > To: debian-user@lists.debian.org
> > Subject: Re: boot times out after dist-upgrade on Stretch
> > Message-ID: <8760tapd7d@turtle.gmx.de>
> > Content-Type: text/plain
> >
> > On 2016-06-15 07:58 +, Borden Rhodes wrote:
> >
> >> I ran apt dist-upgrade on Stretch (with a few Sid packages) which made
> >> the following changes:
> >>
> >> Start-Date: 2016-06-14  19:42:39
> >> Commandline: apt-get dist-upgrade
> >> Requested-By: me (1000)
> >> Install: libdw1:amd64 (0.163-5.1, automatic),
> >> linux-image-4.6.0-1-amd64:amd64 (4.6.1-1, automatic)
> >> Upgrade: wwwconfig-common:amd64 (0.2.2, 0.3.0), libcomerr2:amd64
> >> (1.43-3, 1.43.1-1), libcomerr2:i386 (1.43-3, 1.43.1-1), libcups2:amd64
> >> (2.1.3-5, 2.1.3-6), fuse2fs:amd64 (1.43-3, 1.43.1-1), e2fsprogs:amd64
> >> (1.43-3, 1.43.1-1), boinc-client:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1),
> >> libbabeltrace1:amd64 (1.3.2-1, 1.4.0-1), cups-server-common:amd64
> >> (2.1.3-5, 2.1.3-6), e2fslibs:amd64 (1.43-3, 1.43.1-1),
> >> cups-common:amd64 (2.1.3-5, 2.1.3-6), libspice-server1:amd64
> >> (0.12.6-4, 0.12.6-4.1), boinc-manager:amd64 (7.6.32+dfsg-2,
> >> 7.6.33+dfsg-1), libss2:amd64 (1.43-3, 1.43.1-1), live-config-doc:amd64
> >> (5.20151121, 5.20160608), libdatetime-timezone-perl:amd64
> >> (1:1.98-1+2016d, 1:2.00-1+2016d), cups-ppdc:amd64 (2.1.3-5, 2.1.3-6),
> >> libcupsmime1:amd64 (2.1.3-5, 2.1.3-6), python-paramiko:amd64
> >> (1.16.0-1, 2.0.0-1), linux-image-amd64:amd64 (4.5+73, 4.6+74),
> >> libboinc7:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1), libcupsppdc1:amd64
> >> (2.1.3-5, 2.1.3-6), libbabeltrace-ctf1:amd64 (1.3.2-1, 1.4.0-1),
> >> live-config:amd64 (5.20151121, 5.20160608), cups-bsd:amd64 (2.1.3-5,
> >> 2.1.3-6), cups-core-drivers:amd64 (2.1.3-5, 2.1.3-6),
> >> cups-daemon:amd64 (2.1.3-5, 2.1.3-6), libcupsimage2:amd64 (2.1.3-5,
> >> 2.1.3-6), cups:amd64 (2.1.3-5, 2.1.3-6), boinc:amd64 (7.6.32+dfsg-2,
> >> 7.6.33+dfsg-1), libcupscgi1:amd64 (2.1.3-5, 2.1.3-6),
> >> cups-client:amd64 (2.1.3-5, 2.1.3-6), live-config-systemd:amd64
> >> (5.20151121, 5.20160608), libjpeg62-turbo:amd64 (1:1.4.2-2,
> >> 1:1.5.0-1), libjpeg62-turbo:i386 (1:1.4.2-2, 1:1.5.0-1), xterm:amd64
> >> (324-2, 325-1)
> >> End-Date: 2016-06-14  19:46:44
> >
> > The only package related to the boot process seems to be
> > linux-image-4.6.0-1-amd64.  However, there could be others which were
> > upgraded earlier.  When did you last boot before this upgrade?
> >
> >> The system worked normally until I rebooted a few hours later. After
> >> entering my encryption password (more on that later), boot up stalls
> >> with a message saying that "A start job is running for" and then
> >> switches between sda5_crypt.device, x2dhome.device, x2dvar.device,
> >> x2dtmp.device, .device and
> >> .device.
> >>
> >> After 90 seconds, the start up jobs timeout and the boot tries to
> >> start an emergency shell. However, the prompt never appears, responds
> >> to ^C or ^D as some suggest it might. However, CTRL+ALT+DEL works, so
> >> I know the system isn't completely locked up.
> >>
&g

Re: boot times out after dist-upgrade on Stretch

[Borden Rhodes]

Thank you for getting back to me, Sven,

I normally run apt-get update; apt-get dist-upgrade immediately after
my computer boots. According to the messages log, I turned the
computer on about 5 minutes before running that command and the last
log entry was about 3.5 hours later at 22:59. I hadn't fiddled with
any other settings during that boot. Unfortunately, without /var
loading on the dead boots, I can't get any log information except when
I successfully boot into the recovery console.

I should have mentioned that I also tried booting from the 4.5 kernel
and got the exact same symptoms. I also tried running update-grub in
case it had made a mistake whilst installing the 4.6 kernel.

Notwithstanding better ideas, I'm thinking of doing the Windows Safe
Mode troubleshooting method where I work out the systemd differences
between the default and recovery targets and gradually add services
until I find the one that breaks the system. I'm inferring that, since
recovery mode works but normal mode doesn't, then one of the
targets/services in normal mode is to blame for the lock up. I don't
suppose I could trouble the list for a resource on how to do that?

> Date: Wed, 15 Jun 2016 19:35:34 +0200
> From: Sven Joachim <svenj...@gmx.de>
> To: debian-user@lists.debian.org
> Subject: Re: boot times out after dist-upgrade on Stretch
> Message-ID: <8760tapd7d@turtle.gmx.de>
> Content-Type: text/plain
>
> On 2016-06-15 07:58 +, Borden Rhodes wrote:
>
>> I ran apt dist-upgrade on Stretch (with a few Sid packages) which made
>> the following changes:
>>
>> Start-Date: 2016-06-14  19:42:39
>> Commandline: apt-get dist-upgrade
>> Requested-By: me (1000)
>> Install: libdw1:amd64 (0.163-5.1, automatic),
>> linux-image-4.6.0-1-amd64:amd64 (4.6.1-1, automatic)
>> Upgrade: wwwconfig-common:amd64 (0.2.2, 0.3.0), libcomerr2:amd64
>> (1.43-3, 1.43.1-1), libcomerr2:i386 (1.43-3, 1.43.1-1), libcups2:amd64
>> (2.1.3-5, 2.1.3-6), fuse2fs:amd64 (1.43-3, 1.43.1-1), e2fsprogs:amd64
>> (1.43-3, 1.43.1-1), boinc-client:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1),
>> libbabeltrace1:amd64 (1.3.2-1, 1.4.0-1), cups-server-common:amd64
>> (2.1.3-5, 2.1.3-6), e2fslibs:amd64 (1.43-3, 1.43.1-1),
>> cups-common:amd64 (2.1.3-5, 2.1.3-6), libspice-server1:amd64
>> (0.12.6-4, 0.12.6-4.1), boinc-manager:amd64 (7.6.32+dfsg-2,
>> 7.6.33+dfsg-1), libss2:amd64 (1.43-3, 1.43.1-1), live-config-doc:amd64
>> (5.20151121, 5.20160608), libdatetime-timezone-perl:amd64
>> (1:1.98-1+2016d, 1:2.00-1+2016d), cups-ppdc:amd64 (2.1.3-5, 2.1.3-6),
>> libcupsmime1:amd64 (2.1.3-5, 2.1.3-6), python-paramiko:amd64
>> (1.16.0-1, 2.0.0-1), linux-image-amd64:amd64 (4.5+73, 4.6+74),
>> libboinc7:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1), libcupsppdc1:amd64
>> (2.1.3-5, 2.1.3-6), libbabeltrace-ctf1:amd64 (1.3.2-1, 1.4.0-1),
>> live-config:amd64 (5.20151121, 5.20160608), cups-bsd:amd64 (2.1.3-5,
>> 2.1.3-6), cups-core-drivers:amd64 (2.1.3-5, 2.1.3-6),
>> cups-daemon:amd64 (2.1.3-5, 2.1.3-6), libcupsimage2:amd64 (2.1.3-5,
>> 2.1.3-6), cups:amd64 (2.1.3-5, 2.1.3-6), boinc:amd64 (7.6.32+dfsg-2,
>> 7.6.33+dfsg-1), libcupscgi1:amd64 (2.1.3-5, 2.1.3-6),
>> cups-client:amd64 (2.1.3-5, 2.1.3-6), live-config-systemd:amd64
>> (5.20151121, 5.20160608), libjpeg62-turbo:amd64 (1:1.4.2-2,
>> 1:1.5.0-1), libjpeg62-turbo:i386 (1:1.4.2-2, 1:1.5.0-1), xterm:amd64
>> (324-2, 325-1)
>> End-Date: 2016-06-14  19:46:44
>
> The only package related to the boot process seems to be
> linux-image-4.6.0-1-amd64.  However, there could be others which were
> upgraded earlier.  When did you last boot before this upgrade?
>
>> The system worked normally until I rebooted a few hours later. After
>> entering my encryption password (more on that later), boot up stalls
>> with a message saying that "A start job is running for" and then
>> switches between sda5_crypt.device, x2dhome.device, x2dvar.device,
>> x2dtmp.device, .device and
>> .device.
>>
>> After 90 seconds, the start up jobs timeout and the boot tries to
>> start an emergency shell. However, the prompt never appears, responds
>> to ^C or ^D as some suggest it might. However, CTRL+ALT+DEL works, so
>> I know the system isn't completely locked up.
>>
>> The only error messages I can read after that, as earlier ones would
>> get truncated, are that systemd-tmpfiles.setup.service,
>> binfmt-support.service and networking.service all failed to start.
>
> Those probably fail because /tmp and /var could not be mounted.
>
>> I can, however, boot into single user recovery without the stall,
>> timetout or any error messages.
>>
>> I think it's relevant to note that my hard drive has

Re: boot times out after dist-upgrade on Stretch

[Sven Joachim]

On 2016-06-15 07:58 +, Borden Rhodes wrote:

> I ran apt dist-upgrade on Stretch (with a few Sid packages) which made
> the following changes:
>
> Start-Date: 2016-06-14  19:42:39
> Commandline: apt-get dist-upgrade
> Requested-By: me (1000)
> Install: libdw1:amd64 (0.163-5.1, automatic),
> linux-image-4.6.0-1-amd64:amd64 (4.6.1-1, automatic)
> Upgrade: wwwconfig-common:amd64 (0.2.2, 0.3.0), libcomerr2:amd64
> (1.43-3, 1.43.1-1), libcomerr2:i386 (1.43-3, 1.43.1-1), libcups2:amd64
> (2.1.3-5, 2.1.3-6), fuse2fs:amd64 (1.43-3, 1.43.1-1), e2fsprogs:amd64
> (1.43-3, 1.43.1-1), boinc-client:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1),
> libbabeltrace1:amd64 (1.3.2-1, 1.4.0-1), cups-server-common:amd64
> (2.1.3-5, 2.1.3-6), e2fslibs:amd64 (1.43-3, 1.43.1-1),
> cups-common:amd64 (2.1.3-5, 2.1.3-6), libspice-server1:amd64
> (0.12.6-4, 0.12.6-4.1), boinc-manager:amd64 (7.6.32+dfsg-2,
> 7.6.33+dfsg-1), libss2:amd64 (1.43-3, 1.43.1-1), live-config-doc:amd64
> (5.20151121, 5.20160608), libdatetime-timezone-perl:amd64
> (1:1.98-1+2016d, 1:2.00-1+2016d), cups-ppdc:amd64 (2.1.3-5, 2.1.3-6),
> libcupsmime1:amd64 (2.1.3-5, 2.1.3-6), python-paramiko:amd64
> (1.16.0-1, 2.0.0-1), linux-image-amd64:amd64 (4.5+73, 4.6+74),
> libboinc7:amd64 (7.6.32+dfsg-2, 7.6.33+dfsg-1), libcupsppdc1:amd64
> (2.1.3-5, 2.1.3-6), libbabeltrace-ctf1:amd64 (1.3.2-1, 1.4.0-1),
> live-config:amd64 (5.20151121, 5.20160608), cups-bsd:amd64 (2.1.3-5,
> 2.1.3-6), cups-core-drivers:amd64 (2.1.3-5, 2.1.3-6),
> cups-daemon:amd64 (2.1.3-5, 2.1.3-6), libcupsimage2:amd64 (2.1.3-5,
> 2.1.3-6), cups:amd64 (2.1.3-5, 2.1.3-6), boinc:amd64 (7.6.32+dfsg-2,
> 7.6.33+dfsg-1), libcupscgi1:amd64 (2.1.3-5, 2.1.3-6),
> cups-client:amd64 (2.1.3-5, 2.1.3-6), live-config-systemd:amd64
> (5.20151121, 5.20160608), libjpeg62-turbo:amd64 (1:1.4.2-2,
> 1:1.5.0-1), libjpeg62-turbo:i386 (1:1.4.2-2, 1:1.5.0-1), xterm:amd64
> (324-2, 325-1)
> End-Date: 2016-06-14  19:46:44

The only package related to the boot process seems to be
linux-image-4.6.0-1-amd64.  However, there could be others which were
upgraded earlier.  When did you last boot before this upgrade?

> The system worked normally until I rebooted a few hours later. After
> entering my encryption password (more on that later), boot up stalls
> with a message saying that "A start job is running for" and then
> switches between sda5_crypt.device, x2dhome.device, x2dvar.device,
> x2dtmp.device, .device and
> .device.
>
> After 90 seconds, the start up jobs timeout and the boot tries to
> start an emergency shell. However, the prompt never appears, responds
> to ^C or ^D as some suggest it might. However, CTRL+ALT+DEL works, so
> I know the system isn't completely locked up.
>
> The only error messages I can read after that, as earlier ones would
> get truncated, are that systemd-tmpfiles.setup.service,
> binfmt-support.service and networking.service all failed to start.

Those probably fail because /tmp and /var could not be mounted.

> I can, however, boot into single user recovery without the stall,
> timetout or any error messages.
>
> I think it's relevant to note that my hard drive has a msdos partition
> table (and a legacy BIOS), a LVM partition containing dm-crypt'd
> partitions, each of which is formatted with a btrfs file system. Put
> another way, here's my fstab:
> #
> /dev/mapper/LVG-root /   btrfs   defaults0   1
> # /boot was on /dev/sda1 during installation
> UUID= /boot   btrfs   defaults0   2
> /dev/mapper/LVG-home /home   btrfs   defaults0   2
> /dev/mapper/LVG-tmp /tmpbtrfs   defaults0   2
> /dev/mapper/LVG-var /varbtrfs   defaults0   2
> /dev/sr0/media/cdrom0   udf,iso9660 user,noauto 0   0
>
> What hasn't worked:
> - One site suggested that systemd requires acl. I added acl to all of
> the options in fstab without success.

That's red herring, acl is only needed to tune the permissions for the
journal.

> - Another user on Arch had very similar symptoms to mine:
> https://bbs.archlinux.org/viewtopic.php?id=210008 . However, my system
> doesn't have mkinitcpio, so I can't try the solution that worked for
> him. However, I have initramfs, so maybe adapting his solution would
> work. I'd need guidance as to how so that I don't waste hours
> experimenting with config files.

I guess lvm already works in the initramfs, otherwise your root
filesystem could not be mounted.

> Could I get direction on how to troubleshoot this?

Does the problem show up when you boot with the previous kernel
(probably 4.5)?

Cheers,
   Sven