[systemd-devel] Antw: [EXT] Re: Journald: Re-use /var/log/journal hash entry after system upgrade

[Ulrich Windl]

>>> Mantas Mikulenas  schrieb am 01.03.2022 um 16:52 in
Nachricht
:
> On Tue, Mar 1, 2022 at 4:39 PM eric.zalu...@vertiv.com <
> eric.zalu...@vertiv.com> wrote:
> 
>>  [ System Environment ] 
>> On an embedded x86-64 Linux system, I’m running systemd v241. I have
>> Systemd-Journald logging set to persistent with SystemMaxUse and
>> RuntimeMaxUse both set to 512MB.
>>
>> The Linux system mount loops /var/log on start-up from a var-log.ext4
>> file. The /var/log mount is given a fixed size of the disk (976MB).
Systemd
>> creates a journal entry directory given a hash name in
>> /var/log/journal/.
>>
>>
>>
>>  [ System Information ] 
>>
>> Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
>> GNU/Linux
>>
>>
>>  [ Problem ] 
>>
>> When the system firmware is upgraded, Systemd creates a new journal entry
>> directory with a different hash name and no longer recognizes the previous
>> hash directory.
>>
>> The old logs from the previous journal entry can no longer be managed. The
>> old logs are never rotated and cannot be manually rotated using the
>> journaldctl cli. The disk usage calculator by Journald does not account
for
>> the previous journal entry meaning if there are two previous entries in
>> /var/log that consume 900MB of space; then the new journal entry only has
>> 76MB of space to work with. Eventually, disk space will be full. Journald
>> cannot automatically flush, vacuum, clean, rotate previous logs because it
>> does not recognize the previous journal entries.
>>
>> There must be a systemd journald check that occurs where it determines
>> these other entries are not for it to manage.
>>
> 
> This is not a 'hash' – it's the machine ID
> , which
> is directly read from /etc/machine-id. Normally the machine ID is randomly
> generated during *first *boot (it's just a random UUID), and it is supposed
> to be persistent afterwards.
> 
> It sounds like your /etc doesn't have that file and is on a read-only
> rootfs, so systemd generates a new machine-id in tmpfs every boot. The

Dumb question: Is that fact logged in the journal?

> device probably has *some* persistent storage though, so try to find a way
> to make /etc/machine-id persist as well (see the machine-id(5)
>  manual
> page for a few possibilities).
> 
> The intent of using machine-id subdirectories is to allow containers'
> journal directories to be symlinked on the host, or remote system journals
> to be collected on a single system – there's actually the journalctl -m
> option to make it look at "foreign" journals, so e.g. journalctl -m -f
> could be used to watch logs of all containers.

Still I think some funcion like "remove-orphans" may seem useful (just to
prevent the user to make things worse by removing the wrong files)

> 
> (The machine ID is also used as the base for systemd-networkd's DHCPv4
> Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that,
> though, I can't think of any its uses that would be visible externally –
> aside from desktop-specific things like pulseaudio – so in some cases it
> might be "good enough" to pre-define a fixed ID in the template image as a
> last resort.)


Regards,
Ulrich

> 
> -- 
> Mantas Mikulėnas

Re: [systemd-devel] From dbus notification, how to know service entered failed state and will not start without admin action

[Michal Koutný]

Hello.

On Sat, Feb 19, 2022 at 09:28:18AM +0530, Prashantkumar dhotre 
 wrote:
> I see that in OnFailure behaviour is changed and these units in OnFailure
> gets triggerd when service is failed-and-will-not-restart-automatically.
> https://lists.freedesktop.org/archives/systemd-devel/2018-June/040879.html
> In such case, does systemd also send dbus signal  whenever it triggers
> OnFailure ?

I don't think so. (Unless you can infer it from the new unit starting
job that is announced on DBus.)

Possibly you may add your cleanup via OnFailure= drop-in?

Regards,
Michal

Re: [systemd-devel] Restarting "onshot" services

[Michal Koutný]

Hello.

On Thu, Feb 24, 2022 at 03:35:13PM +0100, Ulrich Windl 
 wrote:
> Is that intentional?

It was, see

https://github.com/systemd/systemd/commit/10e72727ee

(perhaps some other commits too, this was the first that popped up on
me).

> (systemd 228 of SLES12 SP5)

The commmit above is from v244.

Regards,
Michal

Re: [systemd-devel] Journald: Re-use /var/log/journal hash entry after system upgrade

[Mantas Mikulėnas]

On Tue, Mar 1, 2022 at 4:39 PM eric.zalu...@vertiv.com <
eric.zalu...@vertiv.com> wrote:

>  [ System Environment ] 
> On an embedded x86-64 Linux system, I’m running systemd v241. I have
> Systemd-Journald logging set to persistent with SystemMaxUse and
> RuntimeMaxUse both set to 512MB.
>
> The Linux system mount loops /var/log on start-up from a var-log.ext4
> file. The /var/log mount is given a fixed size of the disk (976MB). Systemd
> creates a journal entry directory given a hash name in
> /var/log/journal/.
>
>
>
>  [ System Information ] 
>
> Linux 4.19.0-6-2-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64
> GNU/Linux
>
>
>  [ Problem ] 
>
> When the system firmware is upgraded, Systemd creates a new journal entry
> directory with a different hash name and no longer recognizes the previous
> hash directory.
>
> The old logs from the previous journal entry can no longer be managed. The
> old logs are never rotated and cannot be manually rotated using the
> journaldctl cli. The disk usage calculator by Journald does not account for
> the previous journal entry meaning if there are two previous entries in
> /var/log that consume 900MB of space; then the new journal entry only has
> 76MB of space to work with. Eventually, disk space will be full. Journald
> cannot automatically flush, vacuum, clean, rotate previous logs because it
> does not recognize the previous journal entries.
>
> There must be a systemd journald check that occurs where it determines
> these other entries are not for it to manage.
>

This is not a 'hash' – it's the machine ID
, which
is directly read from /etc/machine-id. Normally the machine ID is randomly
generated during *first *boot (it's just a random UUID), and it is supposed
to be persistent afterwards.

It sounds like your /etc doesn't have that file and is on a read-only
rootfs, so systemd generates a new machine-id in tmpfs every boot. The
device probably has *some* persistent storage though, so try to find a way
to make /etc/machine-id persist as well (see the machine-id(5)
 manual
page for a few possibilities).

The intent of using machine-id subdirectories is to allow containers'
journal directories to be symlinked on the host, or remote system journals
to be collected on a single system – there's actually the journalctl -m
option to make it look at "foreign" journals, so e.g. journalctl -m -f
could be used to watch logs of all containers.

(The machine ID is also used as the base for systemd-networkd's DHCPv4
Client ID, DHCPv6 DUID, and IPv6 address generation. But other than that,
though, I can't think of any its uses that would be visible externally –
aside from desktop-specific things like pulseaudio – so in some cases it
might be "good enough" to pre-define a fixed ID in the template image as a
last resort.)

-- 
Mantas Mikulėnas

Re: [systemd-devel] Syntax check for a new service?

[Tom Browder]

On Tue, Mar 1, 2022 at 08:06 Mantas Mikulėnas  wrote:
...

> That actually makes it relatively privileged – it has full access to
> *your* files and processes... So unless accessing your data is the
> service's explicit purpose (e.g.
>
...

The app serves html from its base (working) directory and below.  It relies
on an http server module that prevents any access to data outside  that
directory.  So, until I have to share the directory with another user, I
think I will chance it.

Thanks.

-Tom

>

Re: [systemd-devel] Syntax check for a new service?

[Mantas Mikulėnas]

On Tue, Mar 1, 2022 at 2:32 PM Tom Browder  wrote:

> On Sat, Feb 26, 2022 at 12:06 Mantas Mikulėnas  wrote:
> ...
>
>> Use the Environment= option to set environment variables.
>>
>
> Thanks, Mantas. That works great.
>
> One more question: I'm thinking of making the service have myself as the
> user (non-privileged). Is that a bad practice since I'm also the root user
> (and currently the only user)?
>

That actually makes it relatively privileged – it has full access to *your*
files and processes... So unless accessing your data is the service's
explicit purpose (e.g. Transmission or MPD or Xvnc), then it should have
its own account, or at least have something like an AppArmor policy loaded.

[1] Obligatory https://xkcd.com/1200/

-- 
Mantas Mikulėnas

Re: [systemd-devel] Syntax check for a new service?

[Tom Browder]

On Sat, Feb 26, 2022 at 12:06 Mantas Mikulėnas  wrote:
...

> Use the Environment= option to set environment variables.
>

Thanks, Mantas. That works great.

One more question: I'm thinking of making the service have myself as the
user (non-privileged). Is that a bad practice since I'm also the root user
(and currently the only user)?

Best regards,

-Tom