Re: Query on s6-log and s6-supervise

2021-06-08 Thread Arjun D R
Dewayne,
Thanks for the details. We already have such an implementation (multiple
producers with one consumer) but still our s6-log instances are high. Many
of our services require direct logger services. We can reduce the direct
logger services by creating a funnel and using regex to separate the logs
but that indeed is a risky and complicated process. I am just interested to
confirm the memory usage of s6-log and s6-supervise processes.

Thanks,
Arjun

On Wed, Jun 9, 2021 at 9:11 AM Dewayne Geraghty <
dewa...@heuristicsystems.com.au> wrote:

> Apologies, I'd implied that we have multiple s6-supervise processes
> running and their children pipe to one file which is read by one s6-log
> file.
>
> You can achieve this outcome by using s6-rc's, where one consumer can
> receive multiple inputs from producers.
>
> There is a special (but not unique) case where a program, such as apache
> which will have explicit log files (defined in apache's config file) to
> record web-page accesses and error logs, on a per server basis.  Because
> all the supervised apache instances can write to one error logfile, I
> instructed apache to write to a pipe.  Multiple supervised apache
> instances using the one pipe (aka funnel), which was read by one s6-log.
>  This way reducing the number of (s6-log) processes.  I could do the
> same with the access logs and use the regex function of s6-log, but I
> tend to simplicity.
>


Re: Query on s6-log and s6-supervise

2021-06-08 Thread Dewayne Geraghty
Apologies, I'd implied that we have multiple s6-supervise processes
running and their children pipe to one file which is read by one s6-log
file.

You can achieve this outcome by using s6-rc's, where one consumer can
receive multiple inputs from producers.

There is a special (but not unique) case where a program, such as apache
which will have explicit log files (defined in apache's config file) to
record web-page accesses and error logs, on a per server basis.  Because
all the supervised apache instances can write to one error logfile, I
instructed apache to write to a pipe.  Multiple supervised apache
instances using the one pipe (aka funnel), which was read by one s6-log.
 This way reducing the number of (s6-log) processes.  I could do the
same with the access logs and use the regex function of s6-log, but I
tend to simplicity.


Re: Query on s6-log and s6-supervise

2021-06-08 Thread Arjun D R
Thanks Laurent for the brief detail. That really helps.

I have checked the Private_Dirty memory in "smaps" of a s6-supervise
process and I don't see any consuming above 8kB. Just posting it here
for reference.

grep Private_Dirty /proc/991/smaps
Private_Dirty: 0 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 8 kB
Private_Dirty: 0 kB
Private_Dirty: 0 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 0 kB
Private_Dirty: 0 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 0 kB
Private_Dirty: 0 kB
Private_Dirty: 8 kB
Private_Dirty: 8 kB
Private_Dirty: 8 kB
Private_Dirty: 0 kB
Private_Dirty: 0 kB
Private_Dirty: 8 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 0 kB
Private_Dirty: 8 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 4 kB
Private_Dirty: 0 kB
Private_Dirty: 0 kB

cat /proc/991/smaps
0001-00014000 r-xp  07:00 174/bin/s6-supervise

00023000-00024000 r--p 3000 07:00 174/bin/s6-supervise

00024000-00025000 rw-p 4000 07:00 174/bin/s6-supervise

00025000-00046000 rw-p  00:00 0  [heap]

b6e1c000-b6e2d000 r-xp  07:00 3652   /lib/libpthread-2.31.so

b6e2d000-b6e3c000 ---p 00011000 07:00 3652   /lib/libpthread-2.31.so

b6e3c000-b6e3d000 r--p 0001 07:00 3652   /lib/libpthread-2.31.so

b6e3d000-b6e3e000 rw-p 00011000 07:00 3652   /lib/libpthread-2.31.so

b6e3e000-b6e4 rw-p  00:00 0

b6e4-b6e45000 r-xp  07:00 3656   /lib/librt-2.31.so

b6e45000-b6e54000 ---p 5000 07:00 3656   /lib/librt-2.31.so

b6e54000-b6e55000 r--p 4000 07:00 3656   /lib/librt-2.31.so

b6e55000-b6e56000 rw-p 5000 07:00 3656   /lib/librt-2.31.so

b6e56000-b6f19000 r-xp  07:00 3613   /lib/libc-2.31.so

b6f19000-b6f28000 ---p 000c3000 07:00 3613   /lib/libc-2.31.so

b6f28000-b6f2a000 r--p 000c2000 07:00 3613   /lib/libc-2.31.so

b6f2a000-b6f2c000 rw-p 000c4000 07:00 3613   /lib/libc-2.31.so

b6f2c000-b6f2e000 rw-p  00:00 0

b6f2e000-b6f4d000 r-xp  07:00 3665   /lib/libskarnet.so.2.9.2.1

b6f4d000-b6f5c000 ---p 0001f000 07:00 3665   /lib/libskarnet.so.2.9.2.1

b6f5c000-b6f5e000 r--p 0001e000 07:00 3665   /lib/libskarnet.so.2.9.2.1

b6f5e000-b6f5f000 rw-p 0002 07:00 3665   /lib/libskarnet.so.2.9.2.1

b6f5f000-b6f6b000 rw-p  00:00 0

b6f6b000-b6f81000 r-xp  07:00 3605   /lib/ld-2.31.so

b6f87000-b6f89000 rw-p  00:00 0

b6f91000-b6f92000 r--p 00016000 07:00 3605   /lib/ld-2.31.so

b6f92000-b6f93000 rw-p 00017000 07:00 3605   /lib/ld-2.31.so

beaf8000-beb19000 rw-p  00:00 0  [stack]
Size:132 kB
Rss:   4 kB
Pss:   4 kB
Shared_Clean:  0 kB
Shared_Dirty:  0 kB
Private_Clean: 0 kB
Private_Dirty: 4 kB
Referenced:4 kB
Anonymous: 4 kB
AnonHugePages: 0 kB
Swap:  0 kB
KernelPageSize:4 kB
MMUPageSize:   4 kB
Locked:0 kB
VmFlags: rd wr mr mw me gd ac
becd5000-becd6000 r-xp  00:00 0  [sigpage]

-1000 r-xp  00:00 0  [vectors]

Sorry I am not able to post the whole data considering the mail size.

On my Linux system,
ps -axw -o pid,vsz,rss,time,comm | grep s6
1   1732  1128 00:00:06 s6-svscan
  900   1736   452 00:00:00 s6-supervise
  901   1736   480 00:00:00 s6-supervise
  902   1736   444 00:00:00 s6-supervise
  903   1736   444 00:00:00 s6-supervise
  907   1744   496 00:00:00 s6-log
.

And I don't think ps_mem is lying , I just compared it with smem as well.
Clear data on ps_mem:

 Private  +   Shared  =  RAM used   Program

  4.8 MiB + 786.0 KiB =   5.5 MiB   s6-log (46)
 12.2 MiB +   2.1 MiB =  14.3 MiB   s6-supervise (129)

smem:

  PID User Command Swap  USS  PSS
 RSS
1020 root s6-supervise wpa_supplicant0   96   98
 996
 2001 root s6-log -F wpa_supplicant.lo0  104  106
1128

Same(almost) amount of PSS/RSS are used by other s6-supervise and s6-log
processes.

I have tried with flag "--enable-allstatic" and unfortunately I don't see
any improvement. If you were mentioning about shared memory, then yes we
are good here. It is using 2.1 MiB for 129 instances, but the private
memory is around 12.2 MiB. I am not sure whether this is the normal value
or not.

If possible, can you please share us a reference smap and ps_mem data on
s6-supervise. That would really help.

Dewayne, even though we pipe it to a file, we will be having a
s6-supervisor for the log service. Maybe I didn't understand it well. Sorry
about it. Ple

Re: Query on s6-log and s6-supervise

2021-06-08 Thread Dewayne Geraghty
Thanks Laurent, that's really interesting.  By comparison, my FBSD
system uses:

# ps -axw -o pid,vsz,rss,time,comm | grep s6
   virt KB  resident cpu total
38724   10904   1600 0:00.02 s6-log
41848   10788   1552 0:00.03 s6-log
42138   10848   1576 0:00.01 s6-log
4   10888   1596 0:00.02 s6-log
45878   10784   1516 0:00.00 s6-svscan
54453   10792   1544 0:00.00 s6-supervise
... lots ...
67937   10792   1540 0:00.00 s6-supervise
76442   10724   1484 0:00.01 s6-ipcserverd
76455   11364   1600 0:00.01 s6-fdholderd
84229   10896712 0:00.01 s6-log

Processes pull-in both ld-elf and libc.so, from procstat -v
start   end path
0x1021000   0x122a000   /usr/local/bin/s6-supervise
0x801229000 0x80124f000 /libexec/ld-elf.so.1
0x801272000 0x80144c000 /lib/libc.so.7

Yes - libc is ... large.

Arjun, if you want to reduce the number of s6-log processes perhaps
consider piping them to a file which s6-log reads from.  For example we
maintain various web servers, the accesses are unique and of interest to
customers, but they don't (really) care about the errors so we aggregate
this with one s6-log. Works very well  :)


Suspend s6-log writing to log file and resume it back after sometime

2021-06-08 Thread Thiyagarajan, Deva
Hello,



I am running s6 on an embedded platform and use s6-log for logging processes’ 
output to hard disk.


I have a requirement where I need to pause logging to hard disk when the device 
is put to standby/sleep mode (basically to reduce hard disk usage and spin it 
down) and once the device is back UP from sleep, I want the logging to resume 
since the point it paused.

Basically during the sleep, s6-log should just hold the logs in memory and when 
device is UP it should start dumping the logs as usual.



Any pointers on how I can go about this? Is there any hack or tricks that could 
be done in s6-log to achieve this?



Thanks in advance


--

Deva


Re: Suspend s6-log writing to log file and resume it back after sometime

2021-06-08 Thread Laurent Bercot

Any pointers on how I can go about this? Is there any hack or tricks that could 
be done in s6-log to achieve this?


 Sorry, but no, nothing comes to mind - s6-log was not designed for 
this.


 I don't think expecting services to keep running while not logging to
disk, whether or not in standby/sleep mode, is reasonable: if logs keep
coming up, memory fills up. What do you do if the machine doesn't wake
up before the memory is full? The logger will die and you will lose all
your carefully accumulated logs.

 Ideally, you would have dynamic verbosity in the service, and switch
it to zero when going into standby/sleep mode, so it would stop
producing logs, so you'd never wake up the disk. Of course, unless it's
the wakeup event listener, the concept of still having a service running
when in standby mode is weird: it defeats the very purpose of standby
mode, which is saving energy. The best way to not have your disk spin is
to have nothing to make it spin in the first place. :P

 s6-svc -p all your services when entering standby mode! (Except the
wakeup event listener.) :D

 Sorry for being unhelpful, and good luck,

--
 Laurent



Re: Query on s6-log and s6-supervise

2021-06-08 Thread Laurent Bercot

   1. Why do we need to have separate supervisors for producer and consumer
   long run services? Is it possible to have one supervisor for both producer
   and consumer, because anyhow the consumer service need not to run when the
   producer is down.  I can understand that s6 supervisor is meant to monitor
   only one service, but why not monitor a couple of services when it is
   logically valid if I am not wrong.


 Hi Arjun,

 The logic of the supervisor is already complex enough when it has
to monitor one process. It would be quadratically as complex if it
had to monitor two. In all likeliness, the first impact of such a
change would be more bugs, because the logic would be a lot more
difficult to understand and maintain.

 The amount of memory used by the s6 logic itself would not change
(or would *increase* somewhat) if the code was organized in a
different way in order to reduce the amount of processes, and you
would see an overall decrease in code quality.

 Worsening the design to offset operational costs is not a good
trade-off - it is not "logically valid", as you put it. I would not
do it even if the high amount of memory consumed by your processes
was due to s6 itself.

 But it is not the case: your operational costs are due to something
else. See below.




   2. Is it possible to have a single supervisor for a bundle of services?
   Like, one supervisor for a bundle (consisting of few services)?


 Again, there would be no engineering benefit to that. You would likely
see operational benefits, yes, but s6 is the wrong place to try and get
those benefits, because it is not the cause of your operational costs.



   3. Generally how many instances of s6-supervise can run? We are running
   into a problem where we have 129 instances of s6-supervise that leads to
   higher memory consumption. We are migrating from systemd to s6 init system
   considering the light weight, but we have a lot of s6-log and s6-supervise
   instances that results in higher memory usage compared to systemd.  Is it
   fine to have this many number of s6-supervise instances?ps_mem data -
  5.5 MiB   s6-log (46) ,  14.3 MiB   s6-supervise (129)


 It is normally totally fine to have this many number of s6-supervise
instances (and of s6-log instances), and it is the intended usage.
The skarnet.org server only has 256 MB of RAM, and currently sports 93
instances of s6-supervise (and 44 instances of s6-log) without any
trouble. It could triple that amount without breaking a sweat.

 The real problem here is that your instances appear to use so much
memory: *that* is not normal.
Every s6-supervise process should use at most 4 pages (16k) of private
dirty memory, so for 129 processes I would expect the memory usage to
be around 2.1 MB. Your reported total shows 7 times as much, which
sounds totally out of bounds to me, and even accounting for normal
operational overhead, a factor of 7 is *completely bonkers*.

 There are two possible explanations here:
 - Either ps_mem is not accurately tallying the memory used by a given
set of processes;
 - Or you are using a libc with an incredible amount of overhead, and
your libc (and in particular, I suspect, dynamic linking management in
your libc) is the culprit for the insane amount of memory that the
s6-supervise processes seem to be eating.

 The easiest way to understand what's going on is to find a
s6-supervise process's pid, and to perform
# cat /proc/$pid/smaps_rollup

 That will tell you what's going on for the chosen s6-supervise process
(they're all similar, so the number for the other s6-supervise processes
won't be far off). In particular, look at the Private_Dirty line: that
is the "real" amount of uncompressible memory used by that process.
 It should be around 16k, tops. Anything over that is overhead from
your libc.
 If the value is not too much over 16k, then ps_mem is simply lying to
you and there is nothing to worry about, except that you should use
another tool to tally memory usage.
 But if the value is much higher, then it is time to diagnose deeper:

# cat /proc/$pid/smaps

 That will show you all the mappings performed by your libc, and
the amount of memory that each of these mappings uses. Again, the
most important lines are the Private_Dirty ones - these are the
values that add up for every s6-supervise instance.

 My hunch is that you will see *a lot* of mappings, each using
4k or 8k, or even in some cases 12k, of Private_Dirty memory.
If it is the case, unfortunately there is nothing I can do about it,
because that overhead is entirely caused by your libc.

 However, there is something *you* can do about it:

 - If "ldd /bin/s6-supervise" gives you a line mentioning libs6.so
or libskarnet.so, try recompiling s6 with --enable-allstatic. This
will link against the static version of libs6 and libskarnet, which
will alleviate the costs of dynamic linking. (The price is that the
*text* of s6-supervise will be a little bigger, but it doesn't ma

Query on s6-log and s6-supervise

2021-06-08 Thread Arjun D R
Hi Team,



I would like to hear from you for a few queries. Please help.



   1. Why do we need to have separate supervisors for producer and consumer
   long run services? Is it possible to have one supervisor for both producer
   and consumer, because anyhow the consumer service need not to run when the
   producer is down.  I can understand that s6 supervisor is meant to monitor
   only one service, but why not monitor a couple of services when it is
   logically valid if I am not wrong.
   2. Is it possible to have a single supervisor for a bundle of services?
   Like, one supervisor for a bundle (consisting of few services)?
   3. Generally how many instances of s6-supervise can run? We are running
   into a problem where we have 129 instances of s6-supervise that leads to
   higher memory consumption. We are migrating from systemd to s6 init system
   considering the light weight, but we have a lot of s6-log and s6-supervise
   instances that results in higher memory usage compared to systemd.  Is it
   fine to have this many number of s6-supervise instances?ps_mem data -
  5.5 MiB   s6-log (46) ,  14.3 MiB   s6-supervise (129)



Thanks,
Arjun