Bug#873613: systemd gets confused at shutdown time

[Michael Biebl]

On Fri, 1 Dec 2017 10:39:33 +0100 Harald Dunkel
 wrote:
> Hi Felipe,
> 
> sorry for the delay.
> 
> On 9/5/17 10:09 PM, Felipe Sateler wrote:
> > 
> > 
> > On Mon, Sep 4, 2017 at 3:26 AM, Harald Dunkel  > > wrote:
> > 
> > If there is already a patch 6588, then would you mind to include
> > it for Stretch?
> > 
> > 
> > Before this is considered, we need confirmation it fixes the issue. Could 
> > you try that patch?
> > 
> 
> I applied 9cbc4547702aac28466c497f720038b9e2dc510c to 232-25+deb9u1:
> No luck. It still gets stuck at shutdown time for about 3 minutes.
> 
> Attached you can find 2 screenshots I managed to take. The list-open.service
> was in place. Please note the "nfs-data not responding" due to the lost
> network connection. "nfs-data" is a CNAME for the NFServer for both
> /home and /data.
> 
> Maybe its an option to not shutdown the network at all?

That's up to the network config system.

NetworkManager e.g. does not shut down ethernet connections when being
stoppend.

-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature
___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Bug#873613: systemd gets confused at shutdown time

[Felipe Sateler]

On Mon, Sep 4, 2017 at 3:26 AM, Harald Dunkel 
wrote:

> If there is already a patch 6588, then would you mind to include
> it for Stretch?
>

Before this is considered, we need confirmation it fixes the issue. Could
you try that patch?


> How does this patch affect the big problem (important services
> being shut down too early)?
>


Which important service is being shutdown too early? portmap is stopped
after nfs mounts are stopped (even though one failed to unmount).


-- 

Saludos,
Felipe Sateler
___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Bug#873613: systemd gets confused at shutdown time

[Harald Dunkel]

If there is already a patch 6588, then would you mind to include 
it for Stretch?

How does this patch affect the big problem (important services 
being shut down too early)?


Regards
Harri

___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Bug#873613: systemd gets confused at shutdown time

[Felipe Sateler]

On Fri, Sep 1, 2017 at 7:27 AM, Harald Dunkel  wrote:
> On Wed, 30 Aug 2017 10:17:55 -0300
> Felipe Sateler  wrote:
>
>> On Wed, Aug 30, 2017 at 6:53 AM, Harald Dunkel 
>> wrote:
>>
>> > On Tue, 29 Aug 2017 11:17:03 -0300
>> > Felipe Sateler  wrote:
>> > >
>> > > Please attach the full configuration for your mount points.
>> > >
>> >
>> > /proc/mounts is attached (as it is now).
>> >
>>
>> Looks like your nfs mounts are missing the _netdev option. Does the problem
>> persist if you add that option? Is this generated from fstab or mount units?
>>
>
> Its in /etc/fstab. Is the "nfs" or "nfs4" in /proc/mounts somehow
> ambiguous?
>
> Anyhow, the (tiny) problem is that systemd ignored the NFS mount point
> completely. The local mount points were released, even though they
> don't have the _netdev attribute set, either.
>
> Even if systemd would have managed to unmount the NFS mount points,
> the bigger problem is still that systemd stops basic services like
> portmap very early at shutdown time.

AFAICS, the umount is attempted at the correct place, but it fails:

Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target Remote File 
Systems.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Unmounting /home...
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Unmounting /data...
Aug 29 15:08:40 dpcl082.example.com systemd[1]: home.mount: Mount process 
exited, code=exited status=16
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Failed unmounting /home.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Unmounted /data.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target Network is 
Online.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped Network Manager Wait 
Online.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target Remote File 
Systems (Pre).
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopped target NFS client 
services.
Aug 29 15:08:40 dpcl082.example.com systemd[1]: Stopping RPC security service 
for NFS client and server...


It appears something is keeping /home still open at the time of the
unmount. Maybe some service has an undeclared dependency on /home?
Finding out who is keeping it open can be done with a unit like the
following:

=== list-open.service
[Unit]
Description=List Open files in /home
RequiresMountsFor=/home
Before=remote-fs.target

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=-/usr/bin/lsof +f -- /home

[Install]
WantedBy=multi-user.target
===


Anyway, I think this general problem is helped by this patch:

https://github.com/systemd/systemd/pull/6588

At the time of systemd-shutdown, because the network is disconnected,
any operations on the nfs mount will fail.

>
>>
>> >
>>  [...]
>> > >
>> > > Looks like systemd shut down your network before it unnmounted remote
>> > > filesystems.
>> > >
>> >
>> > Maybe it should have tried to run "umount -f" or to kill user processes
>> > keeping the mount point busy? Anyway, if you look at the log file you
>> > will notice that portmap was stopped *very* early at shutdown time, even
>> > though /home was still mounted via NFS.
>> >
>> > >
>> > > Could you attach full logs? Attaching the info generated by reportbug
>> > > would be useful too.
>> > >
>> >
>> > journalctl.log is attached (in ASCII). Unfortunately journald stopped
>> > logging. The long delay at shutdown time doesn't show, but I took a
>> > photo.
>> >
>>
>> The full info that reportbug would have generated is important. Please
>> attach it.
>>
>
> Attached.
>
> On Wed, 30 Aug 2017 11:53:38 +0200 I had sent some attachments to this
> bug (the output of journalctl and a screen snapshot). What happened to
> these?
>

I received them, and made a cursory look only. The information from
reportbug is very complete, and avoids a lot of back-and-forth with
questions. For this reason I don't look into much detail without the
full info available. There is a reason we tell reportbug to include
it, and it is that it saves everybodys time.

-- 

Saludos,
Felipe Sateler

___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Bug#873613: systemd gets confused at shutdown time

[Felipe Sateler]

On Wed, Aug 30, 2017 at 6:53 AM, Harald Dunkel 
wrote:

> On Tue, 29 Aug 2017 11:17:03 -0300
> Felipe Sateler  wrote:
> >
> > Please attach the full configuration for your mount points.
> >
>
> /proc/mounts is attached (as it is now).
>

Looks like your nfs mounts are missing the _netdev option. Does the problem
persist if you add that option? Is this generated from fstab or mount units?


>
> > > Basic problem seems to be indicated by the message
> > >
> > > nfs: server nfs-home not responding, timed out
> > >
> >
> > Looks like systemd shut down your network before it unnmounted remote
> > filesystems.
> >
>
> Maybe it should have tried to run "umount -f" or to kill user processes
> keeping the mount point busy? Anyway, if you look at the log file you
> will notice that portmap was stopped *very* early at shutdown time, even
> though /home was still mounted via NFS.
>
> >
> > Could you attach full logs? Attaching the info generated by reportbug
> > would be useful too.
> >
>
> journalctl.log is attached (in ASCII). Unfortunately journald stopped
> logging. The long delay at shutdown time doesn't show, but I took a
> photo.
>

The full info that reportbug would have generated is important. Please
attach it.

-- 

Saludos,
Felipe Sateler
___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Bug#873613: systemd gets confused at shutdown time

[Felipe Sateler]

Control: tags -1 moreinfo

On Tue, Aug 29, 2017 at 10:35 AM, Harald Dunkel  wrote:
> Package: systemd
> Version: 232-25+deb9u1
>
> At shutdown time systemd just shows a message saying
>
> watchdog: watchdog0: watchdog did not stop!
>
> for 5 minutes :-(.
>
> If I boot without quiet, then systemd reveals that it gets
> confused by service dependencies, nfs mount points and
> probably some virtual block devices (mdadm, lvm2, crypt,
> etc).

Please attach the full configuration for your mount points.

> Basic problem seems to be indicated by the message
>
> nfs: server nfs-home not responding, timed out
>

Looks like systemd shut down your network before it unnmounted remote
filesystems.

> pretty late at shutdown time. 90 secs before it tried to
> unmount the local mount points. Apparently it ignored NFS
> completely. No wonder that it got stuck.
>
> Would you mind to check and fix for stretch?

Could you attach full logs? Attaching the info generated by reportbug
would be useful too.


-- 

Saludos,
Felipe Sateler

___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Processed: Re: Bug#873613: systemd gets confused at shutdown time

[Debian Bug Tracking System]

Processing control commands:

> tags -1 moreinfo
Bug #873613 [systemd] systemd gets confused at shutdown time
Added tag(s) moreinfo.

-- 
873613: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=873613
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

Bug#873613: systemd gets confused at shutdown time

[Harald Dunkel]

Package: systemd
Version: 232-25+deb9u1

At shutdown time systemd just shows a message saying

watchdog: watchdog0: watchdog did not stop!

for 5 minutes :-(.

If I boot without quiet, then systemd reveals that it gets 
confused by service dependencies, nfs mount points and 
probably some virtual block devices (mdadm, lvm2, crypt, 
etc). Basic problem seems to be indicated by the message

nfs: server nfs-home not responding, timed out

pretty late at shutdown time. 90 secs before it tried to 
unmount the local mount points. Apparently it ignored NFS
completely. No wonder that it got stuck.

Would you mind to check and fix for stretch? 


Thanx in advance
Harri

___
Pkg-systemd-maintainers mailing list
Pkg-systemd-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers