Re: s6-log not responding to signals
And the timeout is only going to start during exit, right? Naturally. :) -- Laurent
Re: s6-log not responding to signals
"Laurent Bercot" writes: >>While that would make s6-log nicer to integrate with s6-rc, I still >>think that the current behavior of potentially blocking SIGTERM forever >>is undesirable, so some kind of timeout in s6-log could still be a good >>idea. > > That's why I was suggesting a timeout. And since logging a partial line > as a complete line is always strictly better than dropping the partial > line, once you have the timeout feature, you don't need anything else: > set the timeout to 1 ms if you want to exit immediately even with > partial lines. And the timeout is only going to start during exit, right? /Esben
Re: s6-log not responding to signals
While that would make s6-log nicer to integrate with s6-rc, I still think that the current behavior of potentially blocking SIGTERM forever is undesirable, so some kind of timeout in s6-log could still be a good idea. That's why I was suggesting a timeout. And since logging a partial line as a complete line is always strictly better than dropping the partial line, once you have the timeout feature, you don't need anything else: set the timeout to 1 ms if you want to exit immediately even with partial lines. -- Laurent
Re: s6-log not responding to signals
"Laurent Bercot" writes: >> >>The goal is to never write partial lines. So if the process is sent a >>signal to exit while a partial line have been received, simply exit >>without writing anything to file. > > One of the goals is not to write a partial line if it can be avoided; > but it defers to the more important goal of not losing any data. > Your suggestion goes against that more important goal. I understand. I guess I am questioning the importance of the goal of not loosing data. In general, that is a very important goal when handling data. But not all data is equal. While adding more options carry it's own burden, it might be worth considering some kind of option configuring s6-log to do what is best if loosing data is not of the highest importance. What do you think? >>I would vote for simply dropping it. And as we are shutting down, the >>whole thing is a kind of race anyway, so the first part of the line >>could just as well have been not received at all, so I think we can >>safely just throw it away without even waiting for it. > > Nope. Not happening. > > Certainly, on shutdown, it doesn't matter whether you get that last > log line or not. But loggers don't only get killed on shutdown. Good point. > There are other, good, reasons why you would want to kill (and > restart) an s6-log process, and not losing any data is important in > these cases. Any chance that this could be handled by handling SIGHUP and SIGTERM differently? Thinking out loud... Keep the current behavior of SIGTERM, but change SIGHUP to throw away a partial line in buffer. System integraters can then play with the use of SIGHUP and SIGTERM to get the desired effect. With s6-rc, you can then configure down-signal to use SIGHUP instead of SIGTERM if you want to get out quickly. Or you can use timeout-kill to ensure you don't hang around forever if you stay with SIGTERM. While that would make s6-log nicer to integrate with s6-rc, I still think that the current behavior of potentially blocking SIGTERM forever is undesirable, so some kind of timeout in s6-log could still be a good idea. /Esben
Re: s6-log not responding to signals
The goal is to never write partial lines. So if the process is sent a signal to exit while a partial line have been received, simply exit without writing anything to file. One of the goals is not to write a partial line if it can be avoided; but it defers to the more important goal of not losing any data. Your suggestion goes against that more important goal. I would vote for simply dropping it. And as we are shutting down, the whole thing is a kind of race anyway, so the first part of the line could just as well have been not received at all, so I think we can safely just throw it away without even waiting for it. Nope. Not happening. Certainly, on shutdown, it doesn't matter whether you get that last log line or not. But loggers don't only get killed on shutdown. There are other, good, reasons why you would want to kill (and restart) an s6-log process, and not losing any data is important in these cases. -- Laurent
Re: s6-log not responding to signals
How are you thinking changes to termination behaviour will interact with the existing -p option? There would be no specific interaction. -p only makes s6-log ignore SIGTERM. The signal is received, but does nothing. The new timeout option would make it wait on receipt of an exit signal, be it SIGTERM or SIGHUP. So, with -p, it would only trigger the new behaviour on SIGHUP, and keep doing nothing on SIGTERM. As suggested by the documentation, when s6-log is waiting for a newline to arrive, its behaviour could be influenced by a) EOF on stdin, b) termination signal. Are you thinking of adding the timeout only if there is a termination signal, but EOF has not yet been detected? There are two exit conditions for s6-log: 1. it reads EOF on stdin; 2. it receives a SIGTERM (unless -p), or a SIGHUP. On EOF, s6-log exits *immediately*. If it has a partial line in its buffer, it will process and log it as a full line before exiting. It does not wait because there is no reason to: the producer closed the data stream, so s6-log is never getting any more data to finish the line. On a termination signal, the producer isn't necessarily done sending logs; the signal comes from a third party (the administrator). s6-log's goal is to exit asap but without losing any data, and on a line boundary. If there's nothing in its buffer, it exits immediately, but if there's a partial line, it will wait for the producer to send it the rest of the line, process this line, and then exit without reading anything more. (If the producer has more to send, it can do so if the pipe to s6-log is being fd-held; the next s6-log incarnation then resumes where the old one has stopped. If the pipe isn't being fd-held, then the producer gets a broken pipe error, but knows exactly what it has successfully sent and what it has not: no data has been lost in a buffer.) My suggestion is to add a timeout in the only case s6-log doesn't exit immediately: when it gets a termination signal and there is a partial line in the buffer. The wait is meant to give some leeway for the producer to send the rest of the line before s6-log exits, but if no such rest of the line is coming, it would be better for s6-log not to wait forever. -- Laurent
Re: s6-log not responding to signals
"Laurent Bercot" writes: >>The problem is that until a new-line is received, s6-log will not >>respond to SIGHUP and SIGTERM. I assume this is not as expected. > > This is expected; the goal is to finish reading partial lines > before existing. This is useful with services that are writing a > large amount of logs, where the buffer length does not necessarily > align with a newline: after receiving the signal, the logger reads > until the next newline, processes the line, then exits. I proper a slightly different approach. The goal is to never write partial lines. So if the process is sent a signal to exit while a partial line have been received, simply exit without writing anything to file. > No service should ever write a partial line at the end of their > lifetime. True. No service SHOULD do that. None the less. I don't think that s6-log should behave badly if a service does do this. Yes, the service is bad, but s6-log needs to be robust. > However, I agree that the situation you're describing is not ideal > and s6-log should be more robust. I'm thinking of adding a timeout: > if s6-log hasn't received the end of a partial line n milliseconds > after receiving a terminating signal, then it should process the > partial line anyway and exit. What do you think? I would vote for simply dropping it. And as we are shutting down, the whole thing is a kind of race anyway, so the first part of the line could just as well have been not received at all, so I think we can safely just throw it away without even waiting for it. Fast shutdown is a nice feature. Waiting for broken services, or waiting for a log line, that could just as well be missed anyway, is not really that important IMHO. /Esben
Re: s6-log not responding to signals
Laurent, How are you thinking changes to termination behaviour will interact with the existing -p option? > -p : protect against SIGTERM. Do not exit on receipt of a SIGTERM; > only exit on a SIGHUP or when reading EOF on stdin. This is useful for a > logger that you really do not want to lose even if automated administration > (e.g. the downing of a supervision tree) would kill it. As suggested by the documentation, when s6-log is waiting for a newline to arrive, its behaviour could be influenced by a) EOF on stdin, b) termination signal. Are you thinking of adding the timeout only if there is a termination signal, but EOF has not yet been detected? Earl On Thursday, May 25, 2023, 16:20, Laurent Bercot wrote: The problem is that until a new-line is received, s6-log will not respond to SIGHUP and SIGTERM. I assume this is not as expected. This is expected; the goal is to finish reading partial lines before existing. This is useful with services that are writing a large amount of logs, where the buffer length does not necessarily align with a newline: after receiving the signal, the logger reads until the next newline, processes the line, then exits. No service should ever write a partial line at the end of their lifetime. However, I agree that the situation you're describing is not ideal and s6-log should be more robust. I'm thinking of adding a timeout: if s6-log hasn't received the end of a partial line n milliseconds after receiving a terminating signal, then it should process the partial line anyway and exit. What do you think? -- Laurent
Re: s6-log not responding to signals
The problem is that until a new-line is received, s6-log will not respond to SIGHUP and SIGTERM. I assume this is not as expected. This is expected; the goal is to finish reading partial lines before existing. This is useful with services that are writing a large amount of logs, where the buffer length does not necessarily align with a newline: after receiving the signal, the logger reads until the next newline, processes the line, then exits. No service should ever write a partial line at the end of their lifetime. However, I agree that the situation you're describing is not ideal and s6-log should be more robust. I'm thinking of adding a timeout: if s6-log hasn't received the end of a partial line n milliseconds after receiving a terminating signal, then it should process the partial line anyway and exit. What do you think? -- Laurent
