Re: Update early logger logging path after remounting root as rw

[Samuel Holland]

On 9/5/20 3:26 PM, Rio Liu wrote:
>>
>> Well the catch-all logger is supposed to be just that: a catch-all
>> logger. That means that ideally, every service should have its own
>> dedicated logger, typically writing to /var/log/something, and the
>> catch-all logger is only used for exceptional stuff such as error
>> messages from the supervision tree, so it doesn't matter that its logs
>> are stored under /run.
>>
> So the best practice is that every (long run) services should have a
> "producer-for" and an extra "service-log" service? Sounds like a lot of
> repeating works... perhaps I can use some symblink to do that.

Yes, or you can preprocess your service directories. For example, the script I
use looks for a "logdir" file containing the log directory path, and uses it to
generate the pipeline and logger service:

https://github.com/smaeul/rc/blob/113e9623/update#L48

>> It's difficult to do. It's possible in theory: you could have a oneshot
>> that modifies /run/service/s6-svscan-log/run, replacing the
>> "/run/uncaught-logs" string with the new location you want, then
>> copies /run/uncaught-logs into that new location and restarting the
>> s6-svscan-log service.
>>
> 
> Yeah I don't think I'll go that route. Maybe it'll be easier to somehow
> mount root as rw before the logger starts (if it's actually easier), or
> just boot with rw root if I really need those logs to be saved.

There are a couple of other options I can think of:

1) Bring up a second service reading from the same
/run/service/s6-svscan-log/fifo pipe, and once the new logger is ready, bring
down the old s6-svscan-log service (from a oneshot depending on the new logger).
During the transition, log lines could go to either service, but nothing would
be lost.

2) Keep logging to /run/uncaught-logs, but also copy the logs to /var in a way
that doesn't interfere with the s6-svscan-log service. This could be running
`tail -f` on /run/uncaught-logs/current, or periodically `rsync`ing the whole
directory to /var.

Re: Update early logger logging path after remounting root as rw

[Casper Ti. Vector]

On Sat, Sep 05, 2020 at 11:46:44AM -0400, Rio Liu wrote:
> I'm configuring my linux to use s6-rc. It works fairly well so far. One
> thing I want to improve though, is that the early logger logs to
> /run/uncaught-logs. It's nice to have a log file during early boot and it
> helped my debugging easier. But it would be cool to able to change the
> location to a permanent location like /var/log after the root has been
> remounted as read-write.
> 
> Is it possible to update the log path of early/catch-all s6-log process to
> a new location, and perhaps copying the early logs there as well? Or if
> not, is it possible to spawn a new s6-log process that acts as a catch-all
> logger?

Your requirements of saving logs from /run/uncaught-logs and (mentioned
later) batch-producing `producer-for'/`consumer-for' files can be
automated using slew: .  (Other
encapsulations of s6/s6-rc also exist, but I am the author of slew ;)

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2022.09.20)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: Update early logger logging path after remounting root as rw

[Rio Liu]

>
> Well the catch-all logger is supposed to be just that: a catch-all
> logger. That means that ideally, every service should have its own
> dedicated logger, typically writing to /var/log/something, and the
> catch-all logger is only used for exceptional stuff such as error
> messages from the supervision tree, so it doesn't matter that its logs
> are stored under /run.
>
So the best practice is that every (long run) services should have a
"producer-for" and an extra "service-log" service? Sounds like a lot of
repeating works... perhaps I can use some symblink to do that.

It's difficult to do. It's possible in theory: you could have a oneshot
> that modifies /run/service/s6-svscan-log/run, replacing the
> "/run/uncaught-logs" string with the new location you want, then
> copies /run/uncaught-logs into that new location and restarting the
> s6-svscan-log service.
>

Yeah I don't think I'll go that route. Maybe it'll be easier to somehow
mount root as rw before the logger starts (if it's actually easier), or
just boot with rw root if I really need those logs to be saved.

For longruns, unless you want to also restart everything that depends
> on the service, you can just bypass the s6-rc layer and directly tell s6
> to restart your process: s6-svc -r /run/service/$sv
>

Thanks! I forgot that s6-rc actually runs on top of s6-supervice :)

Re: Update early logger logging path after remounting root as rw

[Laurent Bercot]

I'm configuring my linux to use s6-rc. It works fairly well so far. One
thing I want to improve though, is that the early logger logs to
/run/uncaught-logs. It's nice to have a log file during early boot and it
helped my debugging easier. But it would be cool to able to change the
location to a permanent location like /var/log after the root has been
remounted as read-write.


 Well the catch-all logger is supposed to be just that: a catch-all
logger. That means that ideally, every service should have its own
dedicated logger, typically writing to /var/log/something, and the
catch-all logger is only used for exceptional stuff such as error
messages from the supervision tree, so it doesn't matter that its logs
are stored under /run.



Is it possible to update the log path of early/catch-all s6-log process to
a new location, and perhaps copying the early logs there as well? Or if
not, is it possible to spawn a new s6-log process that acts as a catch-all
logger?


 It's difficult to do. It's possible in theory: you could have a oneshot
that modifies /run/service/s6-svscan-log/run, replacing the
"/run/uncaught-logs" string with the new location you want, then
copies /run/uncaught-logs into that new location and restarting the
s6-svscan-log service. But in practice you would add complexity to the
log infrastructure, and you need to pay close attention to all the
failure cases because you don't want to have a broken catch-all logger
for any reason at all. I don't think the benefits are worth the
additional effort; but feel free to disagree and write such a "log
mover" service.



BTW, is there a command to restart service managed by s6-rc? I've been
using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there
might be something simpler.


 For oneshots, no - but you rarely want to "restart" oneshots.
 For longruns, unless you want to also restart everything that depends
on the service, you can just bypass the s6-rc layer and directly tell s6
to restart your process: s6-svc -r /run/service/$sv

 Hope this helps,

--
 Laurent

Update early logger logging path after remounting root as rw

[Rio Liu]

Hello people

I'm configuring my linux to use s6-rc. It works fairly well so far. One
thing I want to improve though, is that the early logger logs to
/run/uncaught-logs. It's nice to have a log file during early boot and it
helped my debugging easier. But it would be cool to able to change the
location to a permanent location like /var/log after the root has been
remounted as read-write.

Is it possible to update the log path of early/catch-all s6-log process to
a new location, and perhaps copying the early logs there as well? Or if
not, is it possible to spawn a new s6-log process that acts as a catch-all
logger?

Thanks

BTW, is there a command to restart service managed by s6-rc? I've been
using "s6-rc -v2 -d change sv && s6-rc -v2 -u change sv" but I feel there
might be something simpler.