Re: Log rotation issue with runit
El jue., 27 dic. 2018 a las 6:36, Dmitry Bogatov escribió: > > And this issue happens not only on crash, it happens after every > termination of svlogd, due any signal. I would agree that SIGKILL is > crash, but issue reproduces with SIGINT and SIGTERM. SIGTERM should make svlogd exit cleanly, are you sure? It does for me, and when restarted, does not create any .u file (using Gentoo's runit, which is upstream runit plus a minor patch to the makefile). IMO, the bug report shows way too many .u files, this should be a rare occurence. Ideally, there should be none. And according to the timestamps, at least two or sometimes all three loggers are producing a .u file at about the same time, as if they had been started together. > Init: runit (via /run/runit.stopit) > [...] > Versions of packages runit recommends: > ii runit-init 2.1.2-19 > [...] > -- Configuration Files: > /etc/runit/3 changed [not included] What does this mean? Is this Debian with runit as the init system and a modified /etc/runit/3 instead of the one supplied with the Debian package? I'm going to make a guess here and say that the supervision tree is not being teared down properly, resulting in svlogd processes not exiting cleanly and producing lots of .u files, compounded with the fact that they are never deleted during rotations because of the fix to bug #878476... G.
Re: Log rotation issue with runit
Laurent Bercot: If loggers kept appending to "current" instead, it could have corrupt information in the middle of a "current" file, which would them be archived as a .s (intact file), which wouldn't be good. Examples: * https://unix.stackexchange.com/questions/237321/ * https://askubuntu.com/questions/356981/
Re: Log rotation issue with runit
El jue., 27 dic. 2018 a las 10:47, Steve Litt escribió: > > [...] Simply run a program, invoked by cron, > that deletes .u files over a certain age, as long as it doesn't delete > the latest .u file. Daemontools-style loggers like svlogd delete files themselves during a rotation for a reason,.. > [...] If Debian doesn't yet have an s6 package, > perhaps a packager could be found, and a Debian s6 (and s6-rc) package > can be made. It does. In buster and sid. Alas, no s6-rc (yet). G.
Re: Log rotation issue with runit
[2018-12-26 00:05] Steve Litt > part text/plain1664 > On Tue, 25 Dec 2018 13:39:17 + > Dmitry Bogatov wrote: > > > Hello! > > > > I am Debian maintainer of `runit' supervision suite. Recently, I > > received bug report [^1] about stray .u logfiles. After some > > investigation I found code, that caused issue, but it seems that it > > was written with some purpose, yet I fail to understand that purpose. > > > > Author Gerrit Pape directed me here. Any clarification would be > > welcome! Please keep bug in thread. > > > > [^1] https://bugs.debian.org/916230 > [...] > > This seems to be an intermittent problem (not reproducible at will), > and very hard to reproduce by anyone but the bug reporter. If the > preceding sentence is true, it's possible it will never be solved. In > such a situation, a workaround becomes a legitimate tactic. No, it is reproducible. See end of bug thread.
Re: Log rotation issue with runit
[2018-12-26 03:58] Alex Efros > Hi! > > I'm not sure is it good idea to include .u files in usual rotation process > at all, and especially handle them just like .s files. If several crashes > happens for some reason in a short period of time this will result in > deletion of all log files except last (say) 10 .u files, which are usually > small and may contain just one line. Replacing 10MB of last logs with 10 > last log lines doesn't sounds like a nice idea. > > Possible "right" solution will be to keep same amount of last .u files as > configured for .s files, i.e. if we've configured to keep last 10 files > then we may have at most double amount (10 .u files and 10 .s files). > (I didn't checked mentioned patches, so maybe they already works this way.) Why would it be wrong to just keep appending to `current' instead of moving it to `.u' file? (see my patch at end of bug thread) And this issue happens not only on crash, it happens after every termination of svlogd, due any signal. I would agree that SIGKILL is crash, but issue reproduces with SIGINT and SIGTERM.
Re: Log rotation issue with runit
Why would it be wrong to just keep appending to `current' instead of moving it to `.u' file? (see my patch at end of bug thread) On a crash, it is possible that a file you're writing to gets corrupted. For the integrity of your logs, it's better to move that potentially corrupted file to another place and start over. If loggers kept appending to "current" instead, it could have corrupt information in the middle of a "current" file, which would them be archived as a .s (intact file), which wouldn't be good. Please don't change that behaviour - it's correct. Loggers should not be repeatedly crashing and wiping out your existing logs by excessive rotations. If you have a repeatedly crashing logger, something is wrong in your installation (or the way you built the logger). However, if you cannot trust your installation, a workaround is to use s6-log, which is very similar to svlogd and understands a "S" directive, meaning "total size of archived files in the directory". Using a large "n" value (maximum number of archived files) and an appropriate "S" value will keep all your logs even in the case of repeated crashes - which will not happen with s6-log. -- Laurent
Re: Log rotation issue with runit
Alex Efros: I'm not sure is it good idea to include .u files in usual rotation process at all, and especially handle them just like .s files. If several crashes happens for some reason in a short period of time this will result in deletion of all log files except last (say) 10 .u files, which are usually small and may contain just one line. Replacing 10MB of last logs with 10 last log lines doesn't sounds like a nice idea. Then cap the log directory by total file size rather than file count. M. Bercot and I both provide tools with this feature. "I want to keep the last 10MiB of logs." is exactly what it is for. * http://jdebp.eu./Softwares/nosh/guide/cyclog.html * http://skarnet.org./software/s6/s6-log.html
