Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Andrei Borzenkov]

22.03.2017 00:10, Chris Murphy пишет:
> OK so I had the idea to uninstall plymouth, since that's estensibly
> what's holding up the remount read-only. But it's not true.
> 
> Sending SIGTERM to remaining processes...
> Sending SIGKILL to remaining processes...
> Unmounting file systems.
> Remounting '/tmp' read-only with options 'seclabel'.
> Unmounting /tmp.
> Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
> All filesystems unmounted.

Could you show your /proc/self/mountinfo before starting shutdown (or
ideally just before systemd goes into uount all)? This suggests that "/"
appears there three times there.

Result code of "remount ro" is not evaluated or logged. systemd does

(void) mount(NULL, m->path, NULL, MS_REMOUNT|MS_RDONLY, options);

where "options" are those from /proc/self/mountinfo sans ro|rw.

Probably it should log it at least with debug level.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Chris Murphy]

OK so I had the idea to uninstall plymouth, since that's estensibly
what's holding up the remount read-only. But it's not true.

Sending SIGTERM to remaining processes...
Sending SIGKILL to remaining processes...
Unmounting file systems.
Remounting '/tmp' read-only with options 'seclabel'.
Unmounting /tmp.
Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
Remounting '/' read-only with options 'seclabel,attr2,inode64,noquota'.
All filesystems unmounted.
Deactivating swaps.
All swaps deactivated.
Detaching loop devices.
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
All loop devices detached.
Detaching DM devices.
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
All DM devices detached.
Spawned /usr/lib/systemd/system-shutdown/mdadm.shutdown as 7058.
/usr/lib/systemd/system-shutdown/mdadm.shutdown succeeded.
system-shutdown succeeded.
Failed to read reboot parameter file: No such file or directory
Rebooting.
[   47.288419] Unregister pv shared memory for cpu 0
[   47.289140] Unregister pv shared memory for cpu 1
[   47.290013] sd 1:0:0:0: [sda] Synchronizing SCSI cache
[   47.315486] reboot: Restarting system
[   47.316036] reboot: machine restart


There are still three attempts to remount read-only. Why? Separately
checking the file system following this reboot, the fs is clean, not
dirty. So one of those remounts must have worked this time. And the
file system is bootable.

There really isn't enough debugging within system to isolate
everything that's going on here.



Chris Murphy
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Chris Murphy]

On Tue, Mar 21, 2017 at 12:04 AM, Chris Murphy  wrote:

>
> c. Only XFS is left in a dirty state following the reboot. Ext4 and Btrfs
> are OK.

This is incorrect. This problem affects ext4 as well, it's just that
on ext4, while the fs is left in a dirty state, the modified grub.cfg
is still readable and boot is possible. But boot after pk offline
update, always includes journal replay.

Basically these reboots are leaving file systems dirty. I can't tell
from the available information if it's a systemd bug, or a kernel bug.
The file system remount to read-only is failing, and a umount isn't
attempted. And I guess between systemd and the kernel, they're
deciding to reboot anyway, resulting in this problem.


-- 
Chris Murphy
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Temporarily stopping a service while oneshot is running

[Kai Krakow]

Am Tue, 21 Mar 2017 07:47:59 -0500
schrieb Ian Pilcher :

> I have a oneshot service (run from a timer) that updates the TLS
> certificates in my mod_nss database.  Because NSS doesn't support
> concurrent access to the database, I need to temporarily shut down
> Apache while the certificate update service is running.
> 
> Currently, I'm using the following entries in my .service file to
> accomplish this:
> 
>[Unit]
>Description=Update TLS certificates in mod_nss database
># Restart Apache, even if this service fails for some reason
>OnFailure=httpd.service
> 
>[Service]
>Type=oneshot
># Shut down Apache to avoid concurrent access to the mod_nss
> database ExecStartPre=/usr/bin/systemctl stop httpd.service
>ExecStart=/usr/local/bin/update-nss-certs
>ExecStartPost=/usr/bin/systemctl start httpd.service
> 
> Is this the best way to do this?  (I can't escape the feeling that
> there ought to be a more idiomatic way of accomplishing this.)

Would "Conflicts=" help here?

Or you simply do not use this as a service but better define a drop-in
for httpd.service:

# systemctl edit httpd.service

[Service]
ExecStartPre=-/usr/local/bin/update-nss-certs


Now, upon starting (or restarting) the httpd.service, the certs are
updated. You can now program the timer to restart httpd. The minus in
front makes failing to do so non-fatal to the service startup.

-- 
Regards,
Kai

Replies to list-only preferred.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] sd-bus connections & authentication timeout

[Jan Alexander Steffens]

On Mon, Mar 20, 2017 at 7:40 PM Stanislav Angelovič 
wrote:

Hi Jan,

thanks for quick response. Instead of sd_bus_process(), we could perhaps
use sd_bus_flush() after creating the connection, as that one actually
processes the requests until the connection changes state to 'running'. I
tried this and it helped.

Regarding polling, I understand, but I fear this way we could get race
condition on the connection instance, since we'd have a thread doing the
polling and processing in a loop on the connection, and some other thread
(say handling the UI, or an external system event, or incoming DBus call
from another connection or whatever else) issuing a DBus call on the same
connection. I think the connection instance is not thread-safe, as once I
experienced such data race related problems. Maybe there is another way of
doing that? One that comes to my mind is having just one single thread
handling the connection and communication through it, but then it would
have to poll on other fd's as well, to be able to communicate with other
threads, for example to get DBus call request from the UI thread and
perform it on the connection. This involves asynchronous programming
paradigm for quite a simple thing, IMHO. What do you think?

Thank you,

Stanislav.


Yes, sd-bus was designed with asynchronous programming in mind. It prefers
the sd-event event loop, but you can also integrate it into another event
loop. The blocking calls all loop internally.

Your programming language and/or framework might have a preferred DBus
implementation already. For example,  if you have a GLib/GTK event loop, I
would recommend using GIO (GDBus) instead of sd-bus. Similarly, Qt has
QDBus.
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] centos7/systemd/mulitpathd

[jsl6uy js16uy]

Apologies, pinging again, to see if anyone has had issues with
systemd+mutlipathd running on CentOS7u2 and the creation of devices and
setting up /dev/disk/by-uuid id links during boot up.

if not, no worries and have great days, will check in elsewhere

thanks and regards

On Mon, Mar 13, 2017 at 4:25 PM, jsl6uy js16uy  wrote:

> Hello all, hope all is well
> Is there a dependency or delay that needs to be added to have volumes
> attached via HBA and managed by multipathd/devmapper and lvm reliably mount
> on boot?
>
> More detail
>
> We have some volumes connected via an hba in a Dell R630 running centos 7u2
> When booting up, the volumes do not mount. Causing the server to go into
> emergency mode, upon login we could see that under /dev/disk/by-uuid/ sym
> links were not created. At this point we could run from the emergency
> maintenance shell, partprobe or restart systemd-udev-trigger.service. This
> would fully create devices under /dev/ and fully populate the links in
> /dev/disk/by-uuid. Then we run mount -a. and exit the emergency shell and
> server completes boot
> multipath is installed and enabled
>
> Server is in prod so I can't do further reboot testing till the weekend.
> How we have it boot now is treating the volumes as x-systemd.automount with
> noauto/nofail options. After an initial "ls" and/or "cd" the volumes are
> available and this prevents halts during reboots
>
>
> workaround:
> UUID=094f2d03-87a1-4c0f-9777-41d11d7e6a7d   /data03 xfs
> noauto,nofail,x-systemd.automount,nobarrier,swalloc,
> inode64,noatime,nodiratime,logbufs=8,allocsize=16m 0 2
> Do I need a _netdev? Apologies not sure with systemd at the helm
>
>
> thanks for any/all help
>
>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Temporarily stopping a service while oneshot is running

[Ian Pilcher]

On 03/21/2017 08:09 AM, Mantas Mikulėnas wrote:

Didn't NSS switch to sqlite for precisely that reason?


Yes they did.  Unfortunately, this is a FreeIPA server, which still uses
the legacy Berkeley DB format.

--

Ian Pilcher arequip...@gmail.com
 "I grew up before Mark Zuckerberg invented friendship" 


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] Temporarily stopping a service while oneshot is running

[Mantas Mikulėnas]

On Tue, Mar 21, 2017 at 2:47 PM, Ian Pilcher  wrote:

> I have a oneshot service (run from a timer) that updates the TLS
> certificates in my mod_nss database.  Because NSS doesn't support
> concurrent access to the database, I need to temporarily shut down
> Apache while the certificate update service is running.


Didn't NSS switch to sqlite for precisely that reason?

-- 
Mantas Mikulėnas 
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] Temporarily stopping a service while oneshot is running

[Ian Pilcher]

I have a oneshot service (run from a timer) that updates the TLS
certificates in my mod_nss database.  Because NSS doesn't support
concurrent access to the database, I need to temporarily shut down
Apache while the certificate update service is running.

Currently, I'm using the following entries in my .service file to
accomplish this:

  [Unit]
  Description=Update TLS certificates in mod_nss database
  # Restart Apache, even if this service fails for some reason
  OnFailure=httpd.service

  [Service]
  Type=oneshot
  # Shut down Apache to avoid concurrent access to the mod_nss database
  ExecStartPre=/usr/bin/systemctl stop httpd.service
  ExecStart=/usr/local/bin/update-nss-certs
  ExecStartPost=/usr/bin/systemctl start httpd.service

Is this the best way to do this?  (I can't escape the feeling that there
ought to be a more idiomatic way of accomplishing this.)

--

Ian Pilcher arequip...@gmail.com
 "I grew up before Mark Zuckerberg invented friendship" 


___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] more verbose debug info than systemd.log_level=debug?

[Chris Murphy]

Thanks for the reply.

On Mon, Mar 20, 2017, 11:05 PM Mantas Mikulėnas  wrote:

> First thought: Even without the exit code or anything, it's going to be
> -EBUSY like 99.999% of the time. Not much else can fail during umount.
>
> And ”Filesystem is busy" would perfectly fit the earlier error message
> which you overlooked:
>
> "Process 304 (plymouthd) has been marked to be excluded from killing.
> It is running from the root file system, and thus likely to block
> re-mounting of the root file system to read-only."
>
> So you have a process holding / open (Plymouth is the boot splash screen
> app) and the kernel doesn't allow it to be umounted due to that.
>


a. Seems flawed to have something that can block remount to read only.
Either a flaw of Plymouth directly, or running it from root fs rather than
the initramfs.

b. This message occurs, as well as the three remount ro messages,
regardless of filesystem (volume format).

c. Only XFS is left in a dirty state following the reboot. Ext4 and Btrfs
are OK.

So I'm still left with why XFS is affected, and XFS devs want to know the
exit code.

At reboot/shutdown time, exactly what does systemd issue to the kernel to
do this?


Chris Murphy

>
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel