Re: Query on s6-log and s6-supervise
Dewayne, Thanks for the details. We already have such an implementation (multiple producers with one consumer) but still our s6-log instances are high. Many of our services require direct logger services. We can reduce the direct logger services by creating a funnel and using regex to separate the logs but that indeed is a risky and complicated process. I am just interested to confirm the memory usage of s6-log and s6-supervise processes. Thanks, Arjun On Wed, Jun 9, 2021 at 9:11 AM Dewayne Geraghty < dewa...@heuristicsystems.com.au> wrote: > Apologies, I'd implied that we have multiple s6-supervise processes > running and their children pipe to one file which is read by one s6-log > file. > > You can achieve this outcome by using s6-rc's, where one consumer can > receive multiple inputs from producers. > > There is a special (but not unique) case where a program, such as apache > which will have explicit log files (defined in apache's config file) to > record web-page accesses and error logs, on a per server basis. Because > all the supervised apache instances can write to one error logfile, I > instructed apache to write to a pipe. Multiple supervised apache > instances using the one pipe (aka funnel), which was read by one s6-log. > This way reducing the number of (s6-log) processes. I could do the > same with the access logs and use the regex function of s6-log, but I > tend to simplicity. >
Re: Query on s6-log and s6-supervise
Apologies, I'd implied that we have multiple s6-supervise processes running and their children pipe to one file which is read by one s6-log file. You can achieve this outcome by using s6-rc's, where one consumer can receive multiple inputs from producers. There is a special (but not unique) case where a program, such as apache which will have explicit log files (defined in apache's config file) to record web-page accesses and error logs, on a per server basis. Because all the supervised apache instances can write to one error logfile, I instructed apache to write to a pipe. Multiple supervised apache instances using the one pipe (aka funnel), which was read by one s6-log. This way reducing the number of (s6-log) processes. I could do the same with the access logs and use the regex function of s6-log, but I tend to simplicity.
Re: Query on s6-log and s6-supervise
Thanks Laurent for the brief detail. That really helps. I have checked the Private_Dirty memory in "smaps" of a s6-supervise process and I don't see any consuming above 8kB. Just posting it here for reference. grep Private_Dirty /proc/991/smaps Private_Dirty: 0 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 8 kB Private_Dirty: 0 kB Private_Dirty: 0 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 0 kB Private_Dirty: 0 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 0 kB Private_Dirty: 0 kB Private_Dirty: 8 kB Private_Dirty: 8 kB Private_Dirty: 8 kB Private_Dirty: 0 kB Private_Dirty: 0 kB Private_Dirty: 8 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 0 kB Private_Dirty: 8 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 4 kB Private_Dirty: 0 kB Private_Dirty: 0 kB cat /proc/991/smaps 0001-00014000 r-xp 07:00 174/bin/s6-supervise 00023000-00024000 r--p 3000 07:00 174/bin/s6-supervise 00024000-00025000 rw-p 4000 07:00 174/bin/s6-supervise 00025000-00046000 rw-p 00:00 0 [heap] b6e1c000-b6e2d000 r-xp 07:00 3652 /lib/libpthread-2.31.so b6e2d000-b6e3c000 ---p 00011000 07:00 3652 /lib/libpthread-2.31.so b6e3c000-b6e3d000 r--p 0001 07:00 3652 /lib/libpthread-2.31.so b6e3d000-b6e3e000 rw-p 00011000 07:00 3652 /lib/libpthread-2.31.so b6e3e000-b6e4 rw-p 00:00 0 b6e4-b6e45000 r-xp 07:00 3656 /lib/librt-2.31.so b6e45000-b6e54000 ---p 5000 07:00 3656 /lib/librt-2.31.so b6e54000-b6e55000 r--p 4000 07:00 3656 /lib/librt-2.31.so b6e55000-b6e56000 rw-p 5000 07:00 3656 /lib/librt-2.31.so b6e56000-b6f19000 r-xp 07:00 3613 /lib/libc-2.31.so b6f19000-b6f28000 ---p 000c3000 07:00 3613 /lib/libc-2.31.so b6f28000-b6f2a000 r--p 000c2000 07:00 3613 /lib/libc-2.31.so b6f2a000-b6f2c000 rw-p 000c4000 07:00 3613 /lib/libc-2.31.so b6f2c000-b6f2e000 rw-p 00:00 0 b6f2e000-b6f4d000 r-xp 07:00 3665 /lib/libskarnet.so.2.9.2.1 b6f4d000-b6f5c000 ---p 0001f000 07:00 3665 /lib/libskarnet.so.2.9.2.1 b6f5c000-b6f5e000 r--p 0001e000 07:00 3665 /lib/libskarnet.so.2.9.2.1 b6f5e000-b6f5f000 rw-p 0002 07:00 3665 /lib/libskarnet.so.2.9.2.1 b6f5f000-b6f6b000 rw-p 00:00 0 b6f6b000-b6f81000 r-xp 07:00 3605 /lib/ld-2.31.so b6f87000-b6f89000 rw-p 00:00 0 b6f91000-b6f92000 r--p 00016000 07:00 3605 /lib/ld-2.31.so b6f92000-b6f93000 rw-p 00017000 07:00 3605 /lib/ld-2.31.so beaf8000-beb19000 rw-p 00:00 0 [stack] Size:132 kB Rss: 4 kB Pss: 4 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 4 kB Referenced:4 kB Anonymous: 4 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize:4 kB MMUPageSize: 4 kB Locked:0 kB VmFlags: rd wr mr mw me gd ac becd5000-becd6000 r-xp 00:00 0 [sigpage] -1000 r-xp 00:00 0 [vectors] Sorry I am not able to post the whole data considering the mail size. On my Linux system, ps -axw -o pid,vsz,rss,time,comm | grep s6 1 1732 1128 00:00:06 s6-svscan 900 1736 452 00:00:00 s6-supervise 901 1736 480 00:00:00 s6-supervise 902 1736 444 00:00:00 s6-supervise 903 1736 444 00:00:00 s6-supervise 907 1744 496 00:00:00 s6-log . And I don't think ps_mem is lying , I just compared it with smem as well. Clear data on ps_mem: Private + Shared = RAM used Program 4.8 MiB + 786.0 KiB = 5.5 MiB s6-log (46) 12.2 MiB + 2.1 MiB = 14.3 MiB s6-supervise (129) smem: PID User Command Swap USS PSS RSS 1020 root s6-supervise wpa_supplicant0 96 98 996 2001 root s6-log -F wpa_supplicant.lo0 104 106 1128 Same(almost) amount of PSS/RSS are used by other s6-supervise and s6-log processes. I have tried with flag "--enable-allstatic" and unfortunately I don't see any improvement. If you were mentioning about shared memory, then yes we are good here. It is using 2.1 MiB for 129 instances, but the private memory is around 12.2 MiB. I am not sure whether this is the normal value or not. If possible, can you please share us a reference smap and ps_mem data on s6-supervise. That would really help. Dewayne, even though we pipe it to a file, we will be having a s6-supervisor for the log service. Maybe I didn't understand it well. Sorry about it. Ple
Re: Query on s6-log and s6-supervise
Thanks Laurent, that's really interesting. By comparison, my FBSD system uses: # ps -axw -o pid,vsz,rss,time,comm | grep s6 virt KB resident cpu total 38724 10904 1600 0:00.02 s6-log 41848 10788 1552 0:00.03 s6-log 42138 10848 1576 0:00.01 s6-log 4 10888 1596 0:00.02 s6-log 45878 10784 1516 0:00.00 s6-svscan 54453 10792 1544 0:00.00 s6-supervise ... lots ... 67937 10792 1540 0:00.00 s6-supervise 76442 10724 1484 0:00.01 s6-ipcserverd 76455 11364 1600 0:00.01 s6-fdholderd 84229 10896712 0:00.01 s6-log Processes pull-in both ld-elf and libc.so, from procstat -v start end path 0x1021000 0x122a000 /usr/local/bin/s6-supervise 0x801229000 0x80124f000 /libexec/ld-elf.so.1 0x801272000 0x80144c000 /lib/libc.so.7 Yes - libc is ... large. Arjun, if you want to reduce the number of s6-log processes perhaps consider piping them to a file which s6-log reads from. For example we maintain various web servers, the accesses are unique and of interest to customers, but they don't (really) care about the errors so we aggregate this with one s6-log. Works very well :)
Suspend s6-log writing to log file and resume it back after sometime
Hello, I am running s6 on an embedded platform and use s6-log for logging processes’ output to hard disk. I have a requirement where I need to pause logging to hard disk when the device is put to standby/sleep mode (basically to reduce hard disk usage and spin it down) and once the device is back UP from sleep, I want the logging to resume since the point it paused. Basically during the sleep, s6-log should just hold the logs in memory and when device is UP it should start dumping the logs as usual. Any pointers on how I can go about this? Is there any hack or tricks that could be done in s6-log to achieve this? Thanks in advance -- Deva
Re: Suspend s6-log writing to log file and resume it back after sometime
Any pointers on how I can go about this? Is there any hack or tricks that could be done in s6-log to achieve this? Sorry, but no, nothing comes to mind - s6-log was not designed for this. I don't think expecting services to keep running while not logging to disk, whether or not in standby/sleep mode, is reasonable: if logs keep coming up, memory fills up. What do you do if the machine doesn't wake up before the memory is full? The logger will die and you will lose all your carefully accumulated logs. Ideally, you would have dynamic verbosity in the service, and switch it to zero when going into standby/sleep mode, so it would stop producing logs, so you'd never wake up the disk. Of course, unless it's the wakeup event listener, the concept of still having a service running when in standby mode is weird: it defeats the very purpose of standby mode, which is saving energy. The best way to not have your disk spin is to have nothing to make it spin in the first place. :P s6-svc -p all your services when entering standby mode! (Except the wakeup event listener.) :D Sorry for being unhelpful, and good luck, -- Laurent
Re: Query on s6-log and s6-supervise
1. Why do we need to have separate supervisors for producer and consumer long run services? Is it possible to have one supervisor for both producer and consumer, because anyhow the consumer service need not to run when the producer is down. I can understand that s6 supervisor is meant to monitor only one service, but why not monitor a couple of services when it is logically valid if I am not wrong. Hi Arjun, The logic of the supervisor is already complex enough when it has to monitor one process. It would be quadratically as complex if it had to monitor two. In all likeliness, the first impact of such a change would be more bugs, because the logic would be a lot more difficult to understand and maintain. The amount of memory used by the s6 logic itself would not change (or would *increase* somewhat) if the code was organized in a different way in order to reduce the amount of processes, and you would see an overall decrease in code quality. Worsening the design to offset operational costs is not a good trade-off - it is not "logically valid", as you put it. I would not do it even if the high amount of memory consumed by your processes was due to s6 itself. But it is not the case: your operational costs are due to something else. See below. 2. Is it possible to have a single supervisor for a bundle of services? Like, one supervisor for a bundle (consisting of few services)? Again, there would be no engineering benefit to that. You would likely see operational benefits, yes, but s6 is the wrong place to try and get those benefits, because it is not the cause of your operational costs. 3. Generally how many instances of s6-supervise can run? We are running into a problem where we have 129 instances of s6-supervise that leads to higher memory consumption. We are migrating from systemd to s6 init system considering the light weight, but we have a lot of s6-log and s6-supervise instances that results in higher memory usage compared to systemd. Is it fine to have this many number of s6-supervise instances?ps_mem data - 5.5 MiB s6-log (46) , 14.3 MiB s6-supervise (129) It is normally totally fine to have this many number of s6-supervise instances (and of s6-log instances), and it is the intended usage. The skarnet.org server only has 256 MB of RAM, and currently sports 93 instances of s6-supervise (and 44 instances of s6-log) without any trouble. It could triple that amount without breaking a sweat. The real problem here is that your instances appear to use so much memory: *that* is not normal. Every s6-supervise process should use at most 4 pages (16k) of private dirty memory, so for 129 processes I would expect the memory usage to be around 2.1 MB. Your reported total shows 7 times as much, which sounds totally out of bounds to me, and even accounting for normal operational overhead, a factor of 7 is *completely bonkers*. There are two possible explanations here: - Either ps_mem is not accurately tallying the memory used by a given set of processes; - Or you are using a libc with an incredible amount of overhead, and your libc (and in particular, I suspect, dynamic linking management in your libc) is the culprit for the insane amount of memory that the s6-supervise processes seem to be eating. The easiest way to understand what's going on is to find a s6-supervise process's pid, and to perform # cat /proc/$pid/smaps_rollup That will tell you what's going on for the chosen s6-supervise process (they're all similar, so the number for the other s6-supervise processes won't be far off). In particular, look at the Private_Dirty line: that is the "real" amount of uncompressible memory used by that process. It should be around 16k, tops. Anything over that is overhead from your libc. If the value is not too much over 16k, then ps_mem is simply lying to you and there is nothing to worry about, except that you should use another tool to tally memory usage. But if the value is much higher, then it is time to diagnose deeper: # cat /proc/$pid/smaps That will show you all the mappings performed by your libc, and the amount of memory that each of these mappings uses. Again, the most important lines are the Private_Dirty ones - these are the values that add up for every s6-supervise instance. My hunch is that you will see *a lot* of mappings, each using 4k or 8k, or even in some cases 12k, of Private_Dirty memory. If it is the case, unfortunately there is nothing I can do about it, because that overhead is entirely caused by your libc. However, there is something *you* can do about it: - If "ldd /bin/s6-supervise" gives you a line mentioning libs6.so or libskarnet.so, try recompiling s6 with --enable-allstatic. This will link against the static version of libs6 and libskarnet, which will alleviate the costs of dynamic linking. (The price is that the *text* of s6-supervise will be a little bigger, but it doesn't ma
Query on s6-log and s6-supervise
Hi Team, I would like to hear from you for a few queries. Please help. 1. Why do we need to have separate supervisors for producer and consumer long run services? Is it possible to have one supervisor for both producer and consumer, because anyhow the consumer service need not to run when the producer is down. I can understand that s6 supervisor is meant to monitor only one service, but why not monitor a couple of services when it is logically valid if I am not wrong. 2. Is it possible to have a single supervisor for a bundle of services? Like, one supervisor for a bundle (consisting of few services)? 3. Generally how many instances of s6-supervise can run? We are running into a problem where we have 129 instances of s6-supervise that leads to higher memory consumption. We are migrating from systemd to s6 init system considering the light weight, but we have a lot of s6-log and s6-supervise instances that results in higher memory usage compared to systemd. Is it fine to have this many number of s6-supervise instances?ps_mem data - 5.5 MiB s6-log (46) , 14.3 MiB s6-supervise (129) Thanks, Arjun
