Re: [systemd-devel] hanging reboot

2017-03-30 Thread Sergei Franco 
Hi!

Thank you very much!

It just shows that I was never patient enough to actually test it long
enough to hit 30min timeout (after about 15 min I generally would reset the
system).
The JobTimeoutSec is a good candidate to be reduced to something like 5 min
for our use case.


Best regards.


Sergei.

On 31 March 2017 at 05:52, Lennart Poettering 
wrote:

> On Thu, 09.03.17 13:00, Sergei Franco (sergei.fra...@gmail.com) wrote:
>
> > Hello!
> >
> > Thank you very much for pointing out the "After" vs "Before", as in for
> > ExecStop to run Before filesytem is unmounted it must be configured as
> > After :).
> > This fixes the Ubuntu unattended upgrades unit that prevents from
> hanging.
> >
> > What I would like to address for future problems like this is how to
> have a
> > global timeout where systemd gives up and reboots anyway (regardless the
> > state of the units)? Even if it is a dirty reboot.
>
> We have that in place. It's configured to 30min however, in order to
> be friendly to super-slow databases.
>
> See "systemctl cat reboot.target":
>
> …
> JobTimeoutSec=30min
> JobTimeoutAction=reboot-force
> …
>
> Lennart
>
> --
> Lennart Poettering, Red Hat
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] hanging reboot

2017-03-30 Thread Lennart Poettering 
On Thu, 09.03.17 13:00, Sergei Franco (sergei.fra...@gmail.com) wrote:

> Hello!
> 
> Thank you very much for pointing out the "After" vs "Before", as in for
> ExecStop to run Before filesytem is unmounted it must be configured as
> After :).
> This fixes the Ubuntu unattended upgrades unit that prevents from hanging.
> 
> What I would like to address for future problems like this is how to have a
> global timeout where systemd gives up and reboots anyway (regardless the
> state of the units)? Even if it is a dirty reboot.

We have that in place. It's configured to 30min however, in order to
be friendly to super-slow databases.

See "systemctl cat reboot.target":

…
JobTimeoutSec=30min
JobTimeoutAction=reboot-force
…

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] hanging reboot

2017-03-08 Thread Sergei Franco 
Hello!

Thank you very much for pointing out the "After" vs "Before", as in for
ExecStop to run Before filesytem is unmounted it must be configured as
After :).
This fixes the Ubuntu unattended upgrades unit that prevents from hanging.

What I would like to address for future problems like this is how to have a
global timeout where systemd gives up and reboots anyway (regardless the
state of the units)? Even if it is a dirty reboot.

Thank you very much!


Sergei.

On 8 March 2017 at 18:04, Andrei Borzenkov  wrote:

> 08.03.2017 03:05, Sergei Franco пишет:
> > Hello,
> >
> > I am revisiting the issue (since it has been completely ignored by
> systemd
> > people).
>
> Which issue? "hanging reboot" is too generic title, it can be anything.
>
> > The actual attempt at fixing this issue did not succeed.
> >
> > The official ubuntu fix does not resolve the hang.
>
> Oh, you mean Ubuntu specific problem? Do you have evidences that it is
> caused by upstream bug?
>
> > The problem is that the unattended-upgrades script relies on /var/run
> being
> > mounted, if the /var/ is a separate filesystem it gets unmounted and thus
> > hanging the script (where it waits for the lock to be available).
> >
>
> How is it systemd problem?
>
> >
> > Here is the official ubuntu unattended-upgrade-shutdown unit:
> >
> > [Unit]
> > Description=Unattended Upgrades Shutdown
> > DefaultDependencies=no
> > Before=shutdown.target reboot.target halt.target network.target
> > local-fs.target
>
> Well, you order if before local-fs.target, which means it is stopped
> after local-fs.target. You get exactly what you ask for. If this is
> wrong, fix unit definition, but we do not really know what is
> appropriate for this service.
>
> > Documentation=man:unattended-upgrade(8)
> >
> > [Service]
> > Type=oneshot
> > RemainAfterExit=yes
> > ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown
> --debug
> > TimeoutStopSec=900
> >
> > [Install]
> > WantedBy=shutdown.target
> >
> > It appears no one who is involved in this fix understands systemd (which
> is
> > very worrying!).
> >
> > How does one would fix this unit so it is ran before the file systems get
> > unmounted?
> >
>
> Order it
>
> After=local-fs.target
>
> >
> > Also, how does one configure systemd to have a global timeout to
> guarantee
> > a reboot? Hanging on reboot is OK for a laptop but is completely
> > unacceptable for a server.
> >
>
> I usually encountered hanging reboot due to inaccessible NFS server and
> in my experience system did reset after watchdog expired.
>
> >
> > Best regards.
> >
> >
> > Sergei.
> >
> >
> >
> > On 2 March 2017 at 10:21, Sergei Franco  wrote:
> >
> >> Hello,
> >>
> >> I submitted similar question but it got stuck with moderators due to
> >> screen shot size.
> >>
> >> The actual bug report (on ubuntu side) is here:
> >> https://bugs.launchpad.net/ubuntu/+source/unattended-
> upgrades/+bug/1661611
> >>
> >> My main question is actually how to configure systemd to have global
> >> timeout on reboot, so no future services will hang it?
> >>
> >> The reboots must happen regardless if systemd can start/stop services. I
> >> am even happy to wait 15 min as long as reboots do happen. Otherwise if
> >> reboots are not guaranteed it is an epic failure on design of the
> system.
> >>
> >> Thanks a lot!
> >>
> >>
> >> Sergei.
> >>
> >> On 2 March 2017 at 04:42, Hajo Locke  wrote:
> >>
> >>> Hello list,
> >>>
> >>> sometimes i have problems rebooting some machine. i think in that cases
> >>> shutting down some services fails and machine stays somewhere between
> life
> >>> and death.
> >>> Unfortunately my ssh window closes at first and no reconnect is
> possible,
> >>> it only tells "Connection refused".
> >>> If this happens, then i have to do a call to someone who works in
> >>> datacenter and resets my machine by hand.
> >>> I would like to keep sshd alive as long as possible to reconnect and
> fix
> >>> this by hand.
> >>> How can i achive this?
> >>> System is Ubuntu 16.04 with systemd 229-4ubuntu16
> >>> I googled some similiar questions and tried but without success.
> >>> What could i do?
> >>>
> >>> Thanks,
> >>> Hajo
> >>> ___
> >>> systemd-devel mailing list
> >>> systemd-devel@lists.freedesktop.org
> >>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >>>
> >>
> >>
> >
> >
> >
> > ___
> > systemd-devel mailing list
> > systemd-devel@lists.freedesktop.org
> > https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >
>
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org

Re: [systemd-devel] hanging reboot

2017-03-07 Thread Andrei Borzenkov 
08.03.2017 03:05, Sergei Franco пишет:
> Hello,
> 
> I am revisiting the issue (since it has been completely ignored by systemd
> people).

Which issue? "hanging reboot" is too generic title, it can be anything.

> The actual attempt at fixing this issue did not succeed.
> 
> The official ubuntu fix does not resolve the hang.

Oh, you mean Ubuntu specific problem? Do you have evidences that it is
caused by upstream bug?

> The problem is that the unattended-upgrades script relies on /var/run being
> mounted, if the /var/ is a separate filesystem it gets unmounted and thus
> hanging the script (where it waits for the lock to be available).
> 

How is it systemd problem?

> 
> Here is the official ubuntu unattended-upgrade-shutdown unit:
> 
> [Unit]
> Description=Unattended Upgrades Shutdown
> DefaultDependencies=no
> Before=shutdown.target reboot.target halt.target network.target
> local-fs.target

Well, you order if before local-fs.target, which means it is stopped
after local-fs.target. You get exactly what you ask for. If this is
wrong, fix unit definition, but we do not really know what is
appropriate for this service.

> Documentation=man:unattended-upgrade(8)
> 
> [Service]
> Type=oneshot
> RemainAfterExit=yes
> ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
> TimeoutStopSec=900
> 
> [Install]
> WantedBy=shutdown.target
> 
> It appears no one who is involved in this fix understands systemd (which is
> very worrying!).
> 
> How does one would fix this unit so it is ran before the file systems get
> unmounted?
> 

Order it

After=local-fs.target

> 
> Also, how does one configure systemd to have a global timeout to guarantee
> a reboot? Hanging on reboot is OK for a laptop but is completely
> unacceptable for a server.
> 

I usually encountered hanging reboot due to inaccessible NFS server and
in my experience system did reset after watchdog expired.

> 
> Best regards.
> 
> 
> Sergei.
> 
> 
> 
> On 2 March 2017 at 10:21, Sergei Franco  wrote:
> 
>> Hello,
>>
>> I submitted similar question but it got stuck with moderators due to
>> screen shot size.
>>
>> The actual bug report (on ubuntu side) is here:
>> https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611
>>
>> My main question is actually how to configure systemd to have global
>> timeout on reboot, so no future services will hang it?
>>
>> The reboots must happen regardless if systemd can start/stop services. I
>> am even happy to wait 15 min as long as reboots do happen. Otherwise if
>> reboots are not guaranteed it is an epic failure on design of the system.
>>
>> Thanks a lot!
>>
>>
>> Sergei.
>>
>> On 2 March 2017 at 04:42, Hajo Locke  wrote:
>>
>>> Hello list,
>>>
>>> sometimes i have problems rebooting some machine. i think in that cases
>>> shutting down some services fails and machine stays somewhere between life
>>> and death.
>>> Unfortunately my ssh window closes at first and no reconnect is possible,
>>> it only tells "Connection refused".
>>> If this happens, then i have to do a call to someone who works in
>>> datacenter and resets my machine by hand.
>>> I would like to keep sshd alive as long as possible to reconnect and fix
>>> this by hand.
>>> How can i achive this?
>>> System is Ubuntu 16.04 with systemd 229-4ubuntu16
>>> I googled some similiar questions and tried but without success.
>>> What could i do?
>>>
>>> Thanks,
>>> Hajo
>>> ___
>>> systemd-devel mailing list
>>> systemd-devel@lists.freedesktop.org
>>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>>>
>>
>>
> 
> 
> 
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
> 

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] hanging reboot

2017-03-07 Thread Uoti Urpala 
On Wed, 2017-03-08 at 13:05 +1300, Sergei Franco wrote:
> 
> The official ubuntu fix does not resolve the hang.
> The problem is that the unattended-upgrades script relies on /var/run
> being mounted, if the /var/ is a separate filesystem it gets
> unmounted and thus hanging the script (where it waits for the lock to
> be available).
> 
> 
> Here is the official ubuntu unattended-upgrade-shutdown unit:

> How does one would fix this unit so it is ran before the file systems
> get unmounted?

To answer your specific question, you could declare a dependency on
/var being mounted. However, if your above comment about it relying
only on /var/run being mounted is accurate, there is a better solution.
/var/run is just a legacy compatibility symlink to /run. Fix the script
to use the real /run path directly instead of /var/run, and there will
be no dependency on /var.

OTOH if that script actually installs packages then the installation
probably requires /var (and possibly other filesystems?) to be mounted,
and you should add RequiresMountsFor and other possible dependencies
required for package installation to work.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] hanging reboot

2017-03-07 Thread Sergei Franco 
Hello,

I am revisiting the issue (since it has been completely ignored by systemd
people).
The actual attempt at fixing this issue did not succeed.

The official ubuntu fix does not resolve the hang.
The problem is that the unattended-upgrades script relies on /var/run being
mounted, if the /var/ is a separate filesystem it gets unmounted and thus
hanging the script (where it waits for the lock to be available).


Here is the official ubuntu unattended-upgrade-shutdown unit:

[Unit]
Description=Unattended Upgrades Shutdown
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target network.target
local-fs.target
Documentation=man:unattended-upgrade(8)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStop=/usr/share/unattended-upgrades/unattended-upgrade-shutdown --debug
TimeoutStopSec=900

[Install]
WantedBy=shutdown.target

It appears no one who is involved in this fix understands systemd (which is
very worrying!).

How does one would fix this unit so it is ran before the file systems get
unmounted?


Also, how does one configure systemd to have a global timeout to guarantee
a reboot? Hanging on reboot is OK for a laptop but is completely
unacceptable for a server.


Best regards.


Sergei.



On 2 March 2017 at 10:21, Sergei Franco  wrote:

> Hello,
>
> I submitted similar question but it got stuck with moderators due to
> screen shot size.
>
> The actual bug report (on ubuntu side) is here:
> https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611
>
> My main question is actually how to configure systemd to have global
> timeout on reboot, so no future services will hang it?
>
> The reboots must happen regardless if systemd can start/stop services. I
> am even happy to wait 15 min as long as reboots do happen. Otherwise if
> reboots are not guaranteed it is an epic failure on design of the system.
>
> Thanks a lot!
>
>
> Sergei.
>
> On 2 March 2017 at 04:42, Hajo Locke  wrote:
>
>> Hello list,
>>
>> sometimes i have problems rebooting some machine. i think in that cases
>> shutting down some services fails and machine stays somewhere between life
>> and death.
>> Unfortunately my ssh window closes at first and no reconnect is possible,
>> it only tells "Connection refused".
>> If this happens, then i have to do a call to someone who works in
>> datacenter and resets my machine by hand.
>> I would like to keep sshd alive as long as possible to reconnect and fix
>> this by hand.
>> How can i achive this?
>> System is Ubuntu 16.04 with systemd 229-4ubuntu16
>> I googled some similiar questions and tried but without success.
>> What could i do?
>>
>> Thanks,
>> Hajo
>> ___
>> systemd-devel mailing list
>> systemd-devel@lists.freedesktop.org
>> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>>
>
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] hanging reboot

2017-03-01 Thread Sergei Franco 
Hello,

I submitted similar question but it got stuck with moderators due to screen
shot size.

The actual bug report (on ubuntu side) is here:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1661611

My main question is actually how to configure systemd to have global
timeout on reboot, so no future services will hang it?

The reboots must happen regardless if systemd can start/stop services. I am
even happy to wait 15 min as long as reboots do happen. Otherwise if
reboots are not guaranteed it is an epic failure on design of the system.

Thanks a lot!


Sergei.

On 2 March 2017 at 04:42, Hajo Locke  wrote:

> Hello list,
>
> sometimes i have problems rebooting some machine. i think in that cases
> shutting down some services fails and machine stays somewhere between life
> and death.
> Unfortunately my ssh window closes at first and no reconnect is possible,
> it only tells "Connection refused".
> If this happens, then i have to do a call to someone who works in
> datacenter and resets my machine by hand.
> I would like to keep sshd alive as long as possible to reconnect and fix
> this by hand.
> How can i achive this?
> System is Ubuntu 16.04 with systemd 229-4ubuntu16
> I googled some similiar questions and tried but without success.
> What could i do?
>
> Thanks,
> Hajo
> ___
> systemd-devel mailing list
> systemd-devel@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/systemd-devel
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel