Re: [systemd-devel] hanging reboot

[Sergei Franco]

Hi!

Thank you very much!

It just shows that I was never patient enough to actually test it long
enough to hit 30min timeout (after about 15 min I generally would reset the
system).
The JobTimeoutSec is a good candidate to be reduced to something like 5 min
for our use case.


Best regards.


Sergei.

On 31 March 2017 at 05:52, Lennart Poettering 
wrote:

> On Thu, 09.03.17 13:00, Sergei Franco (sergei.fra...@gmail.com) wrote:
>
> > Hello!
> >
> > Thank you very much for pointing out the "After" vs "Before", as in for
> > ExecStop to run Before filesytem is unmounted it must be configured as
> > After :).
> > This fixes the Ubuntu unattended upgrades unit that prevents from
> hanging.
> >
> > What I would like to address for future problems like this is how to
> have a
> > global timeout where systemd gives up and reboots anyway (regardless the
> > state of the units)? Even if it is a dirty reboot.
>
> We have that in place. It's configured to 30min however, in order to
> be friendly to super-slow databases.
>
> See "systemctl cat reboot.target":
>
> …
> JobTimeoutSec=30min
> JobTimeoutAction=reboot-force
> …
>
> Lennart
>
> --
> Lennart Poettering, Red Hat
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] udev - ignore/hide persistent storage

[lejeczek]

sorry guys to bother you, but
 I'll see myself going slowly mad next week, for I've been 
reading and tryingand trying..


and I fail to tell udev to ignore a device and not to create 
symlinks.

I need someone to 100% confirm this should work in v.219.
The way Ithink(or hope) it should work islike:

ENV{ID_SERIAL}=="c50033d225e1", ENV{UDISKS_IGNORE}="1"

although I've tried number of different options in my .rules 
file.
I think that with sas or wwin address it would go to the 
lowest level of all those identification layers, thus most 
persistent.
But all the udev test I ran I saw: ACTION=add and symlinks 
would be created by 60-persistent-storage.rules mostly. 
(centos 7.3)


Is it possible to have udev ignore scsi/sas disks with the 
use of sas(or similarly low) addresses?


many thank,
L

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Temporarily stopping a service while oneshot is running

[Kai Krakow]

Am Thu, 30 Mar 2017 16:13:25 +0200
schrieb Lennart Poettering :

> On Tue, 21.03.17 07:47, Ian Pilcher (arequip...@gmail.com) wrote:
> 
> > I have a oneshot service (run from a timer) that updates the TLS
> > certificates in my mod_nss database.  Because NSS doesn't support
> > concurrent access to the database, I need to temporarily shut down
> > Apache while the certificate update service is running.
> > 
> > Currently, I'm using the following entries in my .service file to
> > accomplish this:
> > 
> >   [Unit]
> >   Description=Update TLS certificates in mod_nss database
> >   # Restart Apache, even if this service fails for some reason
> >   OnFailure=httpd.service
> > 
> >   [Service]
> >   Type=oneshot
> >   # Shut down Apache to avoid concurrent access to the mod_nss
> > database ExecStartPre=/usr/bin/systemctl stop httpd.service
> >   ExecStart=/usr/local/bin/update-nss-certs
> >   ExecStartPost=/usr/bin/systemctl start httpd.service
> > 
> > Is this the best way to do this?  (I can't escape the feeling that
> > there ought to be a more idiomatic way of accomplishing this.)  
> 
> Yes, this appears to be the best, and simplest way to do this to me.

Isn't there a chance that this introduces races when triggered while a
bigger transaction is being executed by systemd? It always feels wrong
to trigger actions that may affect the transaction that is currently
being executed...

-- 
Regards,
Kai

Replies to list-only preferred.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] systemd debug out of memory

[Lennart Poettering]

On Sun, 05.03.17 15:59, Pascal Kolijn (p.kol...@vu.nl) wrote:

> Peace,
> 
> On 28/02/2017 16:00, Lennart Poettering wrote:
> > On Tue, 28.02.17 13:26, Pascal kolijn (p.kol...@vu.nl) wrote:
> > 
> >> Hi List,
> >>
> >> I've subscribed to this list to ask for help in debugging a problem we
> >> seem to have with the socket activated telnetd on a rhel7 system.
> >>
> >> A default install of telnetd collects data from some small boxes
> >> deployed in the field. It works for a long time and then suddenly:
> >>
> >> Feb 26 17:46:53 bibr systemd: Created slice user-6006.slice.
> >> Feb 26 17:46:53 bibr systemd: Starting user-6006.slice.
> >> Feb 26 17:46:53 bibr systemd: Started Session 2223341 of user .
> >> Feb 26 17:46:53 bibr systemd-logind: New session 2223341 of user .
> >> Feb 26 17:46:53 bibr systemd: Starting Session 2223341 of user .
> >> Feb 26 17:46:53 bibr systemd: Started Telnet Server (:28830).
> >> Feb 26 17:46:53 pbibr001 systemd: Starting Telnet Server (:28830)...
> >> Feb 26 17:46:57 bibr systemd: Failed to fork: Cannot allocate memory
> > 
> > Hmm, Linux fork() returns ENOMEM if the maximum number of tasks on the
> > system is hit (yes this is a bit misleading, but that's how it is).
> > That max number of tasks is limited for example by the max number of
> > assignable pids as configured in /proc/sys/kernel/pid_max? Maybe you
> > hit that limit? Maybe something is leaking pids on your system? not
> > reaping zombies properly?
> 
> As far as I can determine running out of pids is not the issue, as I can
> see pids being reused in a day, which will not say that some may still
> go missing over time, but how do I determine if that is the case...?

Well, the "Failed to fork" log message is generated when "fork()"
returns ENOMEM, i.e. this is generated by a syscall, i.e. it's an
allocation of some object in the kernel that fails, not of userspace...

> What I do see is that the rss of the systemd process is slowly growing
> over time in the production environment. I've not been able (yet) to
> reproduce the situation in a test environment, which is a pity. I think
> I can simulate the telnet connects more accurately after I speak with
> the developer of the said boxes, and see if I can create a reproducible
> situation.

Well, if it grows without bounds then this would be a leak. But if it
grows only a bit during runtime that's not a problem: in-process
memory management generally doesn't return all memory back to the
kernel immediately, but keeps an allocation cache. In fact, our
hashtable implementation in systemd (which is heavily used) implements
an extra allocation cache on top. Memory in that cache is not actually
leaked, even though it is not returned to the OS: it is simply kept
around for next time we need a similar object.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] centos7/systemd/mulitpathd

[jsl6uy js16uy]

Will do sir
apologies was looking if people had experienced anything like this with
systemd and/or proper places to insert a pause/slow down in a unit file to
let devices get populated. These volumes mounted reliable under sysint and
centos 6u5.
Basically was looking to see if I have to update my thinking/or gotchas for
leveraging under a systemd based boot up
thanks very much

On Thu, Mar 30, 2017 at 5:37 AM, Lennart Poettering 
wrote:

> On Mon, 13.03.17 16:25, jsl6uy js16uy (js1...@gmail.com) wrote:
>
> > Hello all, hope all is well
> > Is there a dependency or delay that needs to be added to have volumes
> > attached via HBA and managed by multipathd/devmapper and lvm reliably
> mount
> > on boot?
>
> Please contact the multipath/DM folks about this, the systemd
> community is not the best place to look for storage advice.
>
> Sorry,
>
> Lennart
>
> --
> Lennart Poettering, Red Hat
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] hanging reboot

[Lennart Poettering]

On Thu, 09.03.17 13:00, Sergei Franco (sergei.fra...@gmail.com) wrote:

> Hello!
> 
> Thank you very much for pointing out the "After" vs "Before", as in for
> ExecStop to run Before filesytem is unmounted it must be configured as
> After :).
> This fixes the Ubuntu unattended upgrades unit that prevents from hanging.
> 
> What I would like to address for future problems like this is how to have a
> global timeout where systemd gives up and reboots anyway (regardless the
> state of the units)? Even if it is a dirty reboot.

We have that in place. It's configured to 30min however, in order to
be friendly to super-slow databases.

See "systemctl cat reboot.target":

…
JobTimeoutSec=30min
JobTimeoutAction=reboot-force
…

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] unable to create bridge name "default"

[Lennart Poettering]

On Sat, 11.03.17 08:41, William Tu (u9012...@gmail.com) wrote:

> Hi,
> I'm not sure if this is caused by systemd-udev, apology if it's not
> the right place. When doing
> # brctl addbr default
> add bridge failed: Invalid argument

If brctl fails with EINVAL, then this suggests that the kernel refuses
this already, and udev is not involved. Please ping the
bridging/networking community.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] MAC address of host0 interface for --network-bridge

[Lennart Poettering]

On Thu, 16.03.17 00:27, Bhasker C V (bhas...@unixindia.com) wrote:

> Hi all,
> 
>  I am using --network-bridge option to setup my container systemd-nspawn.
>  I however dont see any option in man systemd-nspawn to specify a MAC
> Address for the host0 interface. Is there a way to do this ?
>  I see that when using --network-bridge option, the host0 interface
> MAC is fairly constant but does this get auto-generated  ? if yes, is
> there a chance for this MAC to change ?

You can change the MAC address via the ip tool during runtime.

That said, we derive the MAC by default as hash from the host machine ID
and the container name, so that it is effectively random, but stable,
and different on each instance of the container and host.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Renaming interfaces with systemd.link

[Lennart Poettering]

On Tue, 14.03.17 19:18, Chris Trobridge (christrobri...@hotmail.com) wrote:

> 
> On Fedora 25 I can rename a tap interface with a .link file in
> /etc/systemd/network but I cannot rename any Ethernet interfaces.

It should work for all interfaces.

My educated guess is that your .link file was ignored due to the
existance of /usr/lib/systemd/network/99-default.link. Make sure to
prefix your .link file with some number < 99 to ensure it takes
precedence.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

[John Florian]

On Thu, 2017-03-30 at 16:04 +0200, Lennart Poettering wrote:
> On Thu, 30.03.17 09:53, John Florian (j...@doubledog.org) wrote:
> 
> > On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote:
> > > On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote:
> > > 
> > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at
> > > > job
> > > > sshd.service/start conflicted_by=no
> > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at
> > > > job
> > > > sshd.service/stop conflicted_by=yes
> > > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing
> > > > conflicting
> > > > jobs sshd.service/start,sshd.service/stop by deleting job
> > > > sshd.service/start
> > > 
> > > So this suggests that you have both a job that pulls in
> > > sshd.service
> > > and one that conflicts with it in your initial transaction. Try
> > > figuring out what that is...
> > > 
> > > (hint: use "systemctl show sshd.service" for that, and look for
> > > Conflicts= and ConlictedBy=)
> > > 
> > 
> > That seems like a reasonable theory, but given the following I
> > can't
> > make any sense of it.
> > 
> > # journalctl -b | grep -i conflict
> > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/start conflicted_by=no
> > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/stop conflicted_by=yes
> > Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing
> > conflicting
> > jobs sshd.service/start,sshd.service/stop by deleting job
> > sshd.service/start
> > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking
> > at
> > job sshd-keygen.target/start conflicted_by=no
> > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking
> > at
> > job sshd-keygen.target/stop conflicted_by=no
> > Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing
> > conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop
> > by
> > deleting job sshd-keygen.target/stop
> > 
> > # systemctl show sshd.service | grep -i conflict
> > Conflicts=shutdown.target
> > ConflictedBy=sshd.socket
> 
> Note that you can use sshd on Fedora in socket-activated mode or
> non-socket-activated mode. In socket-activated mode, sshd.socket is
> started at boot, and for each connection one sshd@.service instance
> is
> created. sshd.service otoh is only used in the non-socket-activated
> mode, and in that case is the singleton service processing all
> connections...

Yup, I was aware of that of neat feature...

> My educated guess is that you somehow ended up pull in both
> sshd.service and sshd.socket, but you cannot really do that, you have
> to make a choice between them, as there's only one port 22, and it's
> either systemd that listens on it or sshd.

Bingo!  That was just what I needed to put my train of thought on a
different set of tracks.  systemctl status sshd.socket showed that it
was enabled and the vendor preset was also enabled.  Wait...
what   My package that contains the sshd-persist-keys.service and
sshd-restore-keys.service units also provides a file with unit presets
and in there I had "enabled sshd.*".  That was fine back in F21, but no
so with F25 so I replaced that with a "enable sshd.service" and
"disabled sshd.socket" and all is well now.   It's funny, the problem
seems so obvious now yet only an hour ago I was totally blind to the
problem.

Thank you so much for your time and wisdom.___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] socket / service dependency

[Lennart Poettering]

On Mon, 20.03.17 12:38, Ken Bass (kb...@kenbass.com) wrote:

> I thought I followed the instructions (using Centos 7 - I believe
> systemd219) ... I am running a local news server (innd.service).
> 
> I needed to add an additional inetd style daemon for nnrpd to support SSL.
> See files below. Using the BindTo and After seems proper since I never want
> nnrpd running unless innd is already started/running. If innd is stopped, i
> want to be sure nnrpd is stopped. Initially this worked fine. But if I stop
> the innd service, and then start it again, the nnrpd.socket is not started.
> How do I handle this?
> 
> I created a drop in file for innd with a Before/Wants of nnrpd. This
> sometimes works, but normally I get 'Job innd.service/start deleted to break
> ordering cycle starting with nnrpd.socket/start'. What is the proper way to
> do this.

Please provide the full log excerpt around this message, it shows you
exactly what the ordering dep cycle is.

> /etc/systemd/system/innd.d/innd.conf

This path doesn't look right?

> /etc/systemd/system/nnrpd@service

Neither does this one.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Temporarily stopping a service while oneshot is running

[Lennart Poettering]

On Tue, 21.03.17 07:47, Ian Pilcher (arequip...@gmail.com) wrote:

> I have a oneshot service (run from a timer) that updates the TLS
> certificates in my mod_nss database.  Because NSS doesn't support
> concurrent access to the database, I need to temporarily shut down
> Apache while the certificate update service is running.
> 
> Currently, I'm using the following entries in my .service file to
> accomplish this:
> 
>   [Unit]
>   Description=Update TLS certificates in mod_nss database
>   # Restart Apache, even if this service fails for some reason
>   OnFailure=httpd.service
> 
>   [Service]
>   Type=oneshot
>   # Shut down Apache to avoid concurrent access to the mod_nss database
>   ExecStartPre=/usr/bin/systemctl stop httpd.service
>   ExecStart=/usr/local/bin/update-nss-certs
>   ExecStartPost=/usr/bin/systemctl start httpd.service
> 
> Is this the best way to do this?  (I can't escape the feeling that there
> ought to be a more idiomatic way of accomplishing this.)

Yes, this appears to be the best, and simplest way to do this to me.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Temporarily stopping a service while oneshot is running

[Lennart Poettering]

On Tue, 21.03.17 20:31, Kai Krakow (hurikha...@gmail.com) wrote:

> Am Tue, 21 Mar 2017 07:47:59 -0500
> schrieb Ian Pilcher :
> 
> > I have a oneshot service (run from a timer) that updates the TLS
> > certificates in my mod_nss database.  Because NSS doesn't support
> > concurrent access to the database, I need to temporarily shut down
> > Apache while the certificate update service is running.
> > 
> > Currently, I'm using the following entries in my .service file to
> > accomplish this:
> > 
> >[Unit]
> >Description=Update TLS certificates in mod_nss database
> ># Restart Apache, even if this service fails for some reason
> >OnFailure=httpd.service
> > 
> >[Service]
> >Type=oneshot
> ># Shut down Apache to avoid concurrent access to the mod_nss
> > database ExecStartPre=/usr/bin/systemctl stop httpd.service
> >ExecStart=/usr/local/bin/update-nss-certs
> >ExecStartPost=/usr/bin/systemctl start httpd.service
> > 
> > Is this the best way to do this?  (I can't escape the feeling that
> > there ought to be a more idiomatic way of accomplishing this.)
> 
> Would "Conflicts=" help here?

No it would not.

Conflicts= would mean the specified service is stopped when the
oneshot service is started, but it will never be started again.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] sd-bus connections & authentication timeout

[Lennart Poettering]

On Mon, 20.03.17 16:37, Stanislav Angelovič (angelovi...@gmail.com) wrote:

> Hi,
> 
> We use sd-bus for DBus IPC in our own applications. Some applications of
> ours cause the dbus daemon to issue "Connection has not authenticated soon
> enough, closing it" message, leading to the
> org.freedesktop.DBus.Error.Timeout error at the peer side when the
> connection is used for an IPC call.
> 
> The situation is the following:
> 1. We have a DBus client application.
> 2. A connection is created using sd_bus_open_system() on application
> startup.
> 3. The connection is kept open and unused (i.e. no other sd-bus functions
> invoked on it).
> 4. After 30 seconds, we get the above-mentioned authentication timeout
> message from the dbus daemon, and any attempt to issue calls against a
> service via this connection fails with immediate timeout error as mentioned
> above. However, if the call is made before the 30 seconds authentication
> timeout, the dbus daemon prints nothing and the call succeeds, just like
> any subsequent call, even if issued after 1 hour from the first one.
> 
> Is that correct behavior? We would like to open dbus connections at
> application startup, and use it for communication at any time later,
> depending on user actions. Is there something we are missing?

So, as you already found out, if you explicitly flush the connection
after creating it you can make sure the timeout is not hit.

But do note that creating a connection and not processing it for
indefinite time is problematic: other peers might send you messages,
and if you don't read them in time and your input buffer hence runs
over for too long, then dbus-daemon will assume you#re stuck and
disconnect you.

Hence: if your app runs for a longer time, then either process the
connection from time to time, or disconnect it whenever you don't need
it anymore.

Or to say this differently: if you rely on keeping bg connection
objects around you can do this safely only as long as you do this for
"short-lived" operations.

> And a side suggestion: If we want to close a connection that has not been
> used for any IPC call at all, after some struggling we have realized that
> sd_bus_unref() is not enough, as there is a lingering DBus.Hello reply
> which keeps the reference count on that connection, leading to lingering
> connections. sd_bus_flush_close_unref() is needed instead. That behavior
> could be mentioned in the documentation, e.g. at
> http://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html in the
> reference client implementation, to save others from pain.

Yeah, this has come up before, and we really need better docs for
this, indeed.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] sd-bus connections & authentication timeout

[Lennart Poettering]

On Mon, 20.03.17 19:40, Stanislav Angelovič (angelovi...@gmail.com) wrote:

> Hi Jan,
> 
> thanks for quick response. Instead of sd_bus_process(), we could perhaps
> use sd_bus_flush() after creating the connection, as that one actually
> processes the requests until the connection changes state to 'running'. I
> tried this and it helped.
> 
> Regarding polling, I understand, but I fear this way we could get race
> condition on the connection instance, since we'd have a thread doing the
> polling and processing in a loop on the connection, and some other thread
> (say handling the UI, or an external system event, or incoming DBus call
> from another connection or whatever else) issuing a DBus call on the same
> connection. I think the connection instance is not thread-safe, as once I
> experienced such data race related problems. Maybe there is another way of
> doing that? One that comes to my mind is having just one single thread
> handling the connection and communication through it, but then it would
> have to poll on other fd's as well, to be able to communicate with other
> threads, for example to get DBus call request from the UI thread and
> perform it on the connection. This involves asynchronous programming
> paradigm for quite a simple thing, IMHO. What do you think?

sd-bus is not thread-safe, you'd need to lock around this yourself. It
is written in a threads-aware style however, which permits you doing
it and it keeps very little static state, and the one it keeps is
either managed per-thread implicitly or protected by mutexes.

My recommendation would be to either run a proper event loop in your
threads (for example sd-event, or whatever you like), or only keep the
connection around for as long as you need, and close it right-away.

Async programming (i.e. going for a proper event loop) of course gets
you the best performance.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

[Lennart Poettering]

On Thu, 30.03.17 09:53, John Florian (j...@doubledog.org) wrote:

> On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote:
> > On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote:
> > 
> > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> > > sshd.service/start conflicted_by=no
> > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> > > sshd.service/stop conflicted_by=yes
> > > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing
> > > conflicting
> > > jobs sshd.service/start,sshd.service/stop by deleting job
> > > sshd.service/start
> > 
> > So this suggests that you have both a job that pulls in sshd.service
> > and one that conflicts with it in your initial transaction. Try
> > figuring out what that is...
> > 
> > (hint: use "systemctl show sshd.service" for that, and look for
> > Conflicts= and ConlictedBy=)
> > 
> 
> That seems like a reasonable theory, but given the following I can't
> make any sense of it.
> 
> # journalctl -b | grep -i conflict
> Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
> sshd.service/start conflicted_by=no
> Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
> sshd.service/stop conflicted_by=yes
> Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing conflicting
> jobs sshd.service/start,sshd.service/stop by deleting job
> sshd.service/start
> Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at
> job sshd-keygen.target/start conflicted_by=no
> Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at
> job sshd-keygen.target/stop conflicted_by=no
> Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing
> conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by
> deleting job sshd-keygen.target/stop
> 
> # systemctl show sshd.service | grep -i conflict
> Conflicts=shutdown.target
> ConflictedBy=sshd.socket

Note that you can use sshd on Fedora in socket-activated mode or
non-socket-activated mode. In socket-activated mode, sshd.socket is
started at boot, and for each connection one sshd@.service instance is
created. sshd.service otoh is only used in the non-socket-activated
mode, and in that case is the singleton service processing all
connections...

My educated guess is that you somehow ended up pull in both
sshd.service and sshd.socket, but you cannot really do that, you have
to make a choice between them, as there's only one port 22, and it's
either systemd that listens on it or sshd.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

[John Florian]

On Thu, 2017-03-30 at 15:32 +0200, Lennart Poettering wrote:
> On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote:
> 
> > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/start conflicted_by=no
> > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> > sshd.service/stop conflicted_by=yes
> > Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing
> > conflicting
> > jobs sshd.service/start,sshd.service/stop by deleting job
> > sshd.service/start
> 
> So this suggests that you have both a job that pulls in sshd.service
> and one that conflicts with it in your initial transaction. Try
> figuring out what that is...
> 
> (hint: use "systemctl show sshd.service" for that, and look for
> Conflicts= and ConlictedBy=)
> 

That seems like a reasonable theory, but given the following I can't
make any sense of it.

# journalctl -b | grep -i conflict
Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
sshd.service/start conflicted_by=no
Mar 30 09:35:57 localhost systemd[1]: sshd.service: Looking at job
sshd.service/stop conflicted_by=yes
Mar 30 09:35:57 localhost systemd[1]: sshd.service: Fixing conflicting
jobs sshd.service/start,sshd.service/stop by deleting job
sshd.service/start
Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/start conflicted_by=no
Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Looking at
job sshd-keygen.target/stop conflicted_by=no
Mar 30 09:35:57 localhost systemd[1]: sshd-keygen.target: Fixing
conflicting jobs sshd-keygen.target/start,sshd-keygen.target/stop by
deleting job sshd-keygen.target/stop

# systemctl show sshd.service | grep -i conflict
Conflicts=shutdown.target
ConflictedBy=sshd.socket


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] How do I easily resolve conflicting jobs in custom units?

[Lennart Poettering]

On Wed, 22.03.17 16:47, John Florian (j...@doubledog.org) wrote:

> Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> sshd.service/start conflicted_by=no
> Mar 22 16:11:42 localhost systemd[1]: sshd.service: Looking at job
> sshd.service/stop conflicted_by=yes
> Mar 22 16:11:42 localhost systemd[1]: sshd.service: Fixing conflicting
> jobs sshd.service/start,sshd.service/stop by deleting job
> sshd.service/start

So this suggests that you have both a job that pulls in sshd.service
and one that conflicts with it in your initial transaction. Try
figuring out what that is...

(hint: use "systemctl show sshd.service" for that, and look for
Conflicts= and ConlictedBy=)

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] systemd-journald: prioritize system log over user log

[Lennart Poettering]

On Thu, 23.03.17 19:13, Yunchih Chen (yunc...@csie.ntu.edu.tw) wrote:

> 
> 
> On 03/23/2017 06:29 PM, Michael Chapman wrote:
> > On Thu, 23 Mar 2017, Yunchih Chen wrote:
> > > Hello:
> > > 
> > >At my organization, there can be hundreds of user logins in a
> > > public workstation each day, among more than one thousand of users.
> > > Since each user login produces a 8MiB sparse log file in
> > > /var/log/journal/x/, significant disk space is consumed. From
> > > the administrator point of view, it is desirable to prioritize
> > > system log over user log.  For example, when total log size exceeds
> > > SystemMaxUse, user log is vacuumed / rotated before system log.  Can
> > > we achieve this goal under current implementation?
> > 
> > I don't think journald has support for that itself currently.
> > 
> > Have you considered using SplitMode=none in journald.conf? If you do
> > that the per-UID journal files will not be created. Instead, all
> > messages will go into the system journal.
> > 
> > Of course, using SplitMode=none does mean unprivileged users probably
> > won't be able to view their own logs, but it does mean you won't get
> > this per-UID overhead.
> > 
> Thanks for your reply.  But we actually intend to let users view their own
> logs and we don't like to pollute system log with user log.  Maybe consider
> my original proposal as a feature request?

Please file feature requests on github, we'll consider them then.

https://github.com/systemd/systemd/issues/new

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Michael Chapman]

On Thu, 30 Mar 2017, Lennart Poettering wrote:
[...]

I am sorry, but XFS is really broken here. All init systems since time
began kinda did the same thing when shutting down:

a) try to unmount all fs that can be unmounted
b) for the remaining ones, try to remount ro (the root fs usually qualifies)
c) sync()
d) reboot()

That's how sysvinit does it, how Upstart does it, and systemd does it
the same way. (Well, if the initrd supports it we go one step further
though, and optionally pivot back to the initrd which can then unmount
the root file system, too. That's a systemd innovation however, and
only supported on initrd systems where the initrd supports it)

If the XFS devs think that the sync() before reboot() can be partially
ignored, then I am sorry for them, but that makes XFS pretty much
incompatible with every init system in existence.

Or to say this differently: if they expect us to invoke some magic
per-filesystem ioctl() before reboot(), then that's nonsense. No init
system calls that, and I am strongly against such hacks. They should
just fix their APIs.

Moreover, the man page of sync() is pretty clear on this:

 "sync() causes all pending modifications to file system
 metadata and cached file data to be written to the
 underlying filesystems."

It explicitly mentions metadata. Any way you turn it, the XFS folks
are just confused if they really claim sync() doesn't have to sync
metadata. History says differently, and so does the man page
documentation.


I am not a filesystem developer (IANAFD?), but I'm pretty sure they're 
going to say "the metadata _is_ synced, it's in the journal". And it's 
hard to argue that. After all, the filesystem will be perfectly valid the 
next time it is mounted, after the journal has been replayed, and it will 
contain all data written prior to the sync call. It did exactly what the 
manpage says it does.


The problem here seems to be that GRUB is an incomplete XFS 
implementation, one which doesn't know about XFS journalling. It may be a 
good argument XFS shouldn't be used for /boot... but the issue can really 
arise with just about any other journalled filesystems, like Ext3/4.


As Mantas Mikulėnas points out, the FIFREEZE ioctl is supported wherever 
systemd is, and it's not just XFS-specific. I think it'd be smartest just 
to use it because it's there, it's cheap, and it can't make things worse.


--
Michael___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] centos7/systemd/mulitpathd

[Lennart Poettering]

On Mon, 13.03.17 16:25, jsl6uy js16uy (js1...@gmail.com) wrote:

> Hello all, hope all is well
> Is there a dependency or delay that needs to be added to have volumes
> attached via HBA and managed by multipathd/devmapper and lvm reliably mount
> on boot?

Please contact the multipath/DM folks about this, the systemd
community is not the best place to look for storage advice.

Sorry,

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Lennart Poettering]

On Wed, 22.03.17 11:05, Chris Murphy (li...@colorremedies.com) wrote:

> > Result code of "remount ro" is not evaluated or logged. systemd does
> >
> > (void) mount(NULL, m->path, NULL, MS_REMOUNT|MS_RDONLY, options);
> >
> > where "options" are those from /proc/self/mountinfo sans ro|rw.
> >
> > Probably it should log it at least with debug level.
> 
> So I've asked over on the XFS about this, and they suggest all of this
> is expected behavior under the circumstances. The sync only means data
> is committed to disk with an appropriate journal entry, it doesn't
> mean fs metadata is up to date, and it's the fs metadata that GRUB is
> depending on, but isn't up to date yet. So the suggestion is that if
> remount-ro fails, to use freeze/unfreeze and then reboot. The

I am sorry, but XFS is really broken here. All init systems since time
began kinda did the same thing when shutting down:

a) try to unmount all fs that can be unmounted
b) for the remaining ones, try to remount ro (the root fs usually qualifies)
c) sync()
d) reboot()

That's how sysvinit does it, how Upstart does it, and systemd does it
the same way. (Well, if the initrd supports it we go one step further
though, and optionally pivot back to the initrd which can then unmount
the root file system, too. That's a systemd innovation however, and
only supported on initrd systems where the initrd supports it)

If the XFS devs think that the sync() before reboot() can be partially
ignored, then I am sorry for them, but that makes XFS pretty much
incompatible with every init system in existence.

Or to say this differently: if they expect us to invoke some magic
per-filesystem ioctl() before reboot(), then that's nonsense. No init
system calls that, and I am strongly against such hacks. They should
just fix their APIs.

Moreover, the man page of sync() is pretty clear on this:

  "sync() causes all pending modifications to file system
  metadata and cached file data to be written to the
  underlying filesystems."

It explicitly mentions metadata. Any way you turn it, the XFS folks
are just confused if they really claim sync() doesn't have to sync
metadata. History says differently, and so does the man page
documentation.

> If it's useful I'll file an issue with systemd on github to get a
> freeze/unfreeze inserted. remount-ro isn't always successful, and
> clearly it's not ok to reboot anyway if remount-ro fails.

I don't think we'd merge such a patch. The XFS folks should implement
documented behaviour and that'll not just fix things with systemd, but
with any init system.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Lennart Poettering]

On Mon, 27.03.17 22:27, Mantas Mikulėnas (graw...@gmail.com) wrote:

> On Mon, Mar 27, 2017 at 10:20 PM, Chris Murphy 
> wrote:
> 
> > Ok so the dirty file system problem always happens with all pk offline
> > updates on Fedora using either ext4 or XFS with any layout; and it's
> > easy to reproduce.
> >
> > 1. Clean install any version of Fedora, defaults.
> > 2. Once Gnome Software gives notification of updates, Restart & Install
> > 3. System reboots, updates are applied, system reboots again.
> > 4. Now check the journal filtering for 'fsck' and you'll see it
> > replayed the journals; if using XFS check the filter for "XFS" and
> > you'll see the kernel did journal replace at mount time.
> >
> > Basically systemd is rebooting even though the remoun-ro fails
> > multiple times, due to plymouth running off root fs and being exempt
> > from being killed, and then reboots anyway, leaving the file system
> > dirty. So it seems like a flaw to me to allow an indefinite exemption
> > from killing a process that's holding a volume rw, preventing
> > remount-ro before reboot.
> >
> > So there's a risk that in other configurations this makes either ext4
> > and XFS systems unbootable following an offline update.
> 
> So on the one hand it's probably a Plymouth bug or misconfiguration (it
> shouldn't mark itself exempt unless it runs off an in-memory initramfs).

Correct. Plymouth shouldn't mark it itself this way, unless it runs
from the initrd. The documentation says this very explicitly:

   Again: if your code is being run from the root file system, then
   this logic suggested above is NOT for you. Sorry. Talk to us, we
   can probably help you to find a different solution to your problem.

See
https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/.

That said, a file system remaining mounting during shutdown is ugly
but shouldn't result in dataloss, as we do call sync() before
reboot(), and so does any other init system (see other mail).

Hence, there are two bugs here:

a) an ugliness in plymouth (or the way it is used by fedora's package
   update logic), resulting in something that is mostly a
   cosmetic problem

b) XFS is simply broken, if we call sync() it should sync metadata,
   that happens to be triggered by a).

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Mantas Mikulėnas]

On Thu, Mar 30, 2017 at 1:24 PM, Lennart Poettering 
wrote:

> Or to say this differently: if they expect us to invoke some magic
> per-filesystem ioctl() before reboot(), then that's nonsense. No init
> system calls that, and I am strongly against such hacks. They should
> just fix their APIs.
>

On the other hand, no other init system generally supports exclusions from
the killing spree...

As for freezing, that feature seems to have been made generic in 2.6.28
(FIFREEZE/FITHAW), although I couldn't find much documentation on it. Looks
mainly meant for snapshots and backups – not for regular reboots.

-- 
Mantas Mikulėnas 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Lennart Poettering]

On Tue, 28.03.17 11:31, Chris Murphy (li...@colorremedies.com) wrote:

> OK but it's obviously possible for a developer to run a process from
> root fs, and mark it kill exempt. That's the problem under discussion,
> the developer is doing the wrong thing, and it's allowed. And it's
> been going on for a very long time (at least 5 releases of Fedora)

We expect that people who use this functionality are careful with it,
and we made sure to document this all very explicitly:

https://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons/

We even say very clearly what the correct way is to detect whether we
are running from the initrd or the host system.

But anyway, I'd claim that the main culprit is XFS here.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] systemd reboot problem

[Lennart Poettering]

On Tue, 07.03.17 03:15, lin webber (webber...@outlook.com) wrote:

> I use “system("reboot")” in my C program and system does not shutdown.I think 
> that systemd is waiting for some services to shutdown which can't shutdown 
> immediately.
> But I don't want to wait for those services to shutdown.How can I shutdown my 
> system immediately in my C program.which API should I call?
> Thank you for reply.

There are several ways to reboot your system:

1) use the raw reboot() syscall. In this case file systems might end
   up being dirty, and some more complex daemons might not like it
   either.

2) Use "systemctl reboot -ff", which is pretty much the same as #1,
   but accessible from the shell.

3) Use "systemctl reboot -f", which is a slightly friendlier version
   than the above. In this mode, systemd won't bother with stopping
   services correctly (instead it will just SIGTERM them all and
   SIGKILL what is left then). However it will still place all file
   systems in a clean state before issuing reboot().

4) Use "systemctl reboot", which is the friendliest version and
   correctly shuts down all services, i.e. is equivalent to plain
   "reboot" the way you already tried.


Unless you have a completely stateless system with all file systems
either read-only or formatted on each boot #3 is the vastly better
option than #1/#2.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel