Re: [systemd-devel] Why are the priorities of stdout and stderr the same
On Wed, 30 Aug 2023 at 17:55, Dave Howorth wrote: > On Tue, 29 Aug 2023 23:05:28 +0200 > Nils Kattenbeck wrote: > > No, you can use systemd-cat to then invoke your script which applies > > to every output of it. > > The solution to problems with gmail is simple. Stop using gmail and > find another provider. There are many. > Actually, the solution is not to be lazy. Just hit reply all and edit out the non-list address.
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
On Tue, 29 Aug 2023 23:05:28 +0200 Nils Kattenbeck wrote: > No, you can use systemd-cat to then invoke your script which applies > to every output of it. > > Also, you can just use reply-all in gmail (or even set it as the > default in the settings) to have the correct behaviour of sending > mails also the the mailing list. That's not the correct behaviour! The correct behaviour is to reply only to the list and not pollute posters' inboxes with extra copies. The solution to problems with gmail is simple. Stop using gmail and find another provider. There are many.
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
Op di 29 aug 2023 om 23:05 schreef Nils Kattenbeck :
> No, you can use systemd-cat to then invoke your script which applies
> to every output of it.
>
OK, I will dive into it.
> Also, you can just use reply-all in gmail (or even set it as the
> default in the settings) to have the correct behaviour of sending
> mails also the the mailing list.
>
I will do that then. Only problem is that the original poster will get the
message two times.
Except when I remove the To. But forgetting to remove that is not as bad as
not sending it to the mailing list.
> On Tue, Aug 29, 2023 at 10:25 PM Cecil Westerhof
> wrote:
> >
> > Aargh, forgot again that gmail works differently when replying. :'-{
> >
> > Op di 29 aug 2023 om 21:07 schreef Cecil Westerhof <
> cldwester...@gmail.com>:
> >>
> >> Op di 29 aug 2023 om 19:47 schreef Nils Kattenbeck <
> nilskem...@gmail.com>:
> >>>
> >>> Hi, At least for simple cases you can use systemd-cat which allows
> >>> setting different priorities for stdout and stderr. It even explicitly
> >>> states that doing so will lose the ordering guarantees which are only
> >>> possible when attaching stdout and stderr to the same fd (as Lennart
> >>> said).
> >>
> >>
> >> If I understand it correctly, that is on a statement basis, not for the
> complete script.
>
--
Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
No, you can use systemd-cat to then invoke your script which applies
to every output of it.
Also, you can just use reply-all in gmail (or even set it as the
default in the settings) to have the correct behaviour of sending
mails also the the mailing list.
On Tue, Aug 29, 2023 at 10:25 PM Cecil Westerhof wrote:
>
> Aargh, forgot again that gmail works differently when replying. :'-{
>
> Op di 29 aug 2023 om 21:07 schreef Cecil Westerhof :
>>
>> Op di 29 aug 2023 om 19:47 schreef Nils Kattenbeck :
>>>
>>> Hi, At least for simple cases you can use systemd-cat which allows
>>> setting different priorities for stdout and stderr. It even explicitly
>>> states that doing so will lose the ordering guarantees which are only
>>> possible when attaching stdout and stderr to the same fd (as Lennart
>>> said).
>>
>>
>> If I understand it correctly, that is on a statement basis, not for the
>> complete script.
>
>
> --
> Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
Aargh, forgot again that gmail works differently when replying. :'-{
Op di 29 aug 2023 om 21:07 schreef Cecil Westerhof :
> Op di 29 aug 2023 om 19:47 schreef Nils Kattenbeck :
>
>> Hi, At least for simple cases you can use systemd-cat which allows
>> setting different priorities for stdout and stderr. It even explicitly
>> states that doing so will lose the ordering guarantees which are only
>> possible when attaching stdout and stderr to the same fd (as Lennart
>> said).
>>
>
> If I understand it correctly, that is on a statement basis, not for the
> complete script.
>
--
Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
Hi, At least for simple cases you can use systemd-cat which allows setting different priorities for stdout and stderr. It even explicitly states that doing so will lose the ordering guarantees which are only possible when attaching stdout and stderr to the same fd (as Lennart said). Greetings Nils On Tue, Aug 29, 2023 at 2:10 PM Cecil Westerhof wrote: > > Op di 29 aug 2023 om 11:58 schreef Lennart Poettering > : >> >> On Di, 29.08.23 11:56, Cecil Westerhof (cldwester...@gmail.com) wrote: >> > > I agree with that usecase, and we have discussed this many times >> > > before, but we couldn#t come up with a nice way to make everything >> > > work: proper ordering and distintion of stdout/stderr. >> > >> > I agree that the default behaviour is the right one. But why not give >> > people the possibility to override this behaviour? When they override it >> > themselves, they cannot complain that they lose ordering. >> >> We are generally conservative when providing mechanisms that are too >> glaringly broken. Even if they are opt-in. > > > OK. Thanks for your patience. > > -- > Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
Op di 29 aug 2023 om 11:58 schreef Lennart Poettering < lenn...@poettering.net>: > On Di, 29.08.23 11:56, Cecil Westerhof (cldwester...@gmail.com) wrote: > > > I agree with that usecase, and we have discussed this many times > > > before, but we couldn#t come up with a nice way to make everything > > > work: proper ordering and distintion of stdout/stderr. > > > > I agree that the default behaviour is the right one. But why not give > > people the possibility to override this behaviour? When they override it > > themselves, they cannot complain that they lose ordering. > > We are generally conservative when providing mechanisms that are too > glaringly broken. Even if they are opt-in. > OK. Thanks for your patience. -- Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
On Di, 29.08.23 11:56, Cecil Westerhof (cldwester...@gmail.com) wrote: > > I agree with that usecase, and we have discussed this many times > > before, but we couldn#t come up with a nice way to make everything > > work: proper ordering and distintion of stdout/stderr. > > I agree that the default behaviour is the right one. But why not give > people the possibility to override this behaviour? When they override it > themselves, they cannot complain that they lose ordering. We are generally conservative when providing mechanisms that are too glaringly broken. Even if they are opt-in. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
Op di 29 aug 2023 om 11:36 schreef Lennart Poettering < lenn...@poettering.net>: > On Di, 29.08.23 11:20, Cecil Westerhof (cldwester...@gmail.com) wrote: > > > Also: everything has a timestamp, so there is in my opinion when you > choose > > to take them apart no big problem. > > For stream connections like those used for stdout/stderr, lines do not > come with timestamps. We add them on the reception side, which is too late. > I did not know that. I understand it now. > > > > To get what is send to stderr I had to do: > > > > journalctl -p 6 -u aptCacheUsage.service > > > > > > > > which gave beside a lot of other things the things send to stdout. > > > > > > > > Now I have two different statements I can do: > > > > journalctl -p 3 -u aptCacheUsage.service > > > > > > > > But it would be nice if I did not need two different statements (and > the > > > > logic around that) for that. > > > > > > Still not getting what you are trying to say here. > > > > > > > Often I am only interested in what is sent to stderr and do not want what > > is sent to stdout. When both have the same log level I can not really > > filter on messages sent to stderr. At the moment I want to see the > messages > > sent to stderr, I will also get the messages sent to stdout because they > > have the same error level. > > I agree with that usecase, and we have discussed this many times > before, but we couldn#t come up with a nice way to make everything > work: proper ordering and distintion of stdout/stderr. > I agree that the default behaviour is the right one. But why not give people the possibility to override this behaviour? When they override it themselves, they cannot complain that they lose ordering. -- Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
On Di, 29.08.23 11:20, Cecil Westerhof (cldwester...@gmail.com) wrote: > Also: everything has a timestamp, so there is in my opinion when you choose > to take them apart no big problem. For stream connections like those used for stdout/stderr, lines do not come with timestamps. We add them on the reception side, which is too late. > > > To get what is send to stderr I had to do: > > > journalctl -p 6 -u aptCacheUsage.service > > > > > > which gave beside a lot of other things the things send to stdout. > > > > > > Now I have two different statements I can do: > > > journalctl -p 3 -u aptCacheUsage.service > > > > > > But it would be nice if I did not need two different statements (and the > > > logic around that) for that. > > > > Still not getting what you are trying to say here. > > > > Often I am only interested in what is sent to stderr and do not want what > is sent to stdout. When both have the same log level I can not really > filter on messages sent to stderr. At the moment I want to see the messages > sent to stderr, I will also get the messages sent to stdout because they > have the same error level. I agree with that usecase, and we have discussed this many times before, but we couldn#t come up with a nice way to make everything work: proper ordering and distintion of stdout/stderr. The closest I cam was using two distinct SOCK_DGRAM sockets connect()ed to the same target socket (instead of the current approach of using SOCK_STREAM). This would give us two benefits: for each deliverd datagram we would get a source socket address reported to us, and it will tell us which of the two source sockets it was, hence hence if stdout or stderr. Moreover, we would get a kernel-supplied kernel timestamp on each datagram if we want. This however has a fairly big problem too: if programs write too much data into their stdout/stderr at once they would get EMSGSIZE back, which programs generally don't expect (i.e. if write()'s size is larger than datagram max size you get EMSGSIZE). Programs trying to write too much usually expect blocking behaviour... Thus this approach is not really an option. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
Op di 29 aug 2023 om 10:13 schreef Lennart Poettering <
lenn...@poettering.net>:
> On Sa, 26.08.23 06:14, Cecil Westerhof (cldwester...@gmail.com) wrote:
>
> Please keep mails like this on the mailing list.
>
Sorry, I am used to that when I am responding to a list, my reply goes to
the list. Gmail sadly works differently. Most I did catch, but not all. :'-(
> > We should not "assume the worst", hence given that the stderr stream
> > > is typically used for all kinds of informational messages we should
> > > not always assume its an error, because quite often its just
> > > informational.
> > >
> >
> > You have a very good point. When tcl opens a process for reading, it is
> an
> > error when there is something to read on stderr, except when you overrule
> > it. But that you can overrule it proves your point.
> >
> > > Hence, we use LOG_INFO if we have no clue simply because that's the
> > > "best assumption".
> > >
> >
> > I agree, but I would suggest a very simple solution.
> > There is SyslogLevel which sets the syslog level for stdout and stderr. I
> > would suggest adding SyslogLevelStderr. SyslogLevel would still set it
> for
> > both except when there is also SyslogLevelStderr.
>
> When journal redirection of both stdout + stderr is enabled for
> systemd services we'll connect a single pipe to both fds, in order to
> guarantee ordering, i.e. ensure that if something is written to
> stdout, and then something to stderr, we'll definitely process it in
> this order too. This however means, that on the receiving side we
> cannot distinguish stdout/stderr anymore, it's all one stream. Hence
> we can only choose between: guarantee correct ordering OR ability to
> distinguish stdout/stderr. We opted for the former, as corrupted
> ordering between stdout/stderr is just too confusing for users.
>
I understand your point. But that could be the default behaviour. At the
moment someone uses SyslogLevelStderr it is a conscious choice to take them
apart.
Also: everything has a timestamp, so there is in my opinion when you choose
to take them apart no big problem.
> > We generally recommend apps to use syslog() or sd-journal APIs to
> > > generate their log messages and specify the log level for each message
> > > explicitly, to avoid any doubts. Many programming language's logging
> > > frameworks natively have support for these.
> >
> > The script I use can be run from the command-line and from a service.
> > Because of that I have to use:
> > logMsg --simple "${message}" >&2
> > and:
> > echo "<3>$(logMsg --simple "${message}")" >&2
> >
> > doable but inconvenient.
> >
> > > Now when I want the things send to stderr I also get the things send to
> > > > stdout.
> > >
> > > I can't parse that.
> > >
> >
> > To get what is send to stderr I had to do:
> > journalctl -p 6 -u aptCacheUsage.service
> >
> > which gave beside a lot of other things the things send to stdout.
> >
> > Now I have two different statements I can do:
> > journalctl -p 3 -u aptCacheUsage.service
> >
> > But it would be nice if I did not need two different statements (and the
> > logic around that) for that.
>
> Still not getting what you are trying to say here.
>
Often I am only interested in what is sent to stderr and do not want what
is sent to stdout. When both have the same log level I can not really
filter on messages sent to stderr. At the moment I want to see the messages
sent to stderr, I will also get the messages sent to stdout because they
have the same error level.
--
Cecil Westerhof
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
On Sa, 26.08.23 06:14, Cecil Westerhof (cldwester...@gmail.com) wrote:
Please keep mails like this on the mailing list.
> > We should not "assume the worst", hence given that the stderr stream
> > is typically used for all kinds of informational messages we should
> > not always assume its an error, because quite often its just
> > informational.
> >
>
> You have a very good point. When tcl opens a process for reading, it is an
> error when there is something to read on stderr, except when you overrule
> it. But that you can overrule it proves your point.
>
> > Hence, we use LOG_INFO if we have no clue simply because that's the
> > "best assumption".
> >
>
> I agree, but I would suggest a very simple solution.
> There is SyslogLevel which sets the syslog level for stdout and stderr. I
> would suggest adding SyslogLevelStderr. SyslogLevel would still set it for
> both except when there is also SyslogLevelStderr.
When journal redirection of both stdout + stderr is enabled for
systemd services we'll connect a single pipe to both fds, in order to
guarantee ordering, i.e. ensure that if something is written to
stdout, and then something to stderr, we'll definitely process it in
this order too. This however means, that on the receiving side we
cannot distinguish stdout/stderr anymore, it's all one stream. Hence
we can only choose between: guarantee correct ordering OR ability to
distinguish stdout/stderr. We opted for the former, as corrupted
ordering between stdout/stderr is just too confusing for users.
> > We generally recommend apps to use syslog() or sd-journal APIs to
> > generate their log messages and specify the log level for each message
> > explicitly, to avoid any doubts. Many programming language's logging
> > frameworks natively have support for these.
>
> The script I use can be run from the command-line and from a service.
> Because of that I have to use:
> logMsg --simple "${message}" >&2
> and:
> echo "<3>$(logMsg --simple "${message}")" >&2
>
> doable but inconvenient.
>
> > Now when I want the things send to stderr I also get the things send to
> > > stdout.
> >
> > I can't parse that.
> >
>
> To get what is send to stderr I had to do:
> journalctl -p 6 -u aptCacheUsage.service
>
> which gave beside a lot of other things the things send to stdout.
>
> Now I have two different statements I can do:
> journalctl -p 3 -u aptCacheUsage.service
>
> But it would be nice if I did not need two different statements (and the
> logic around that) for that.
Still not getting what you are trying to say here.
Lennart
--
Lennart Poettering, Berlin
Re: [systemd-devel] Why are the priorities of stdout and stderr the same
On Do, 24.08.23 16:31, Cecil Westerhof (cldwester...@gmail.com) wrote: > Normally in a script when something is send to stdout it is seen as an > error has occurred. > But in systemd both get a priority of 6 (info). > Why does stderr not get a priority of 3 (err), or at least lower as > stdout? stderr is a bit of a misnomer, it's not just for errors, it's also for progress output, informational output and so, basically everything that is not considered the primary output contents that one would want to propagate in pipelines. We should not "assume the worst", hence given that the stderr stream is typically used for all kinds of informational messages we should not always assume its an error, because quite often its just informational. Hence, we use LOG_INFO if we have no clue simply because that's the "best assumption". We generally recommend apps to use syslog() or sd-journal APIs to generate their log messages and specify the log level for each message explicitly, to avoid any doubts. Many programming language's logging frameworks natively have support for these. > Now when I want the things send to stderr I also get the things send to > stdout. I can't parse that. Lennart -- Lennart Poettering, Berlin
[systemd-devel] Why are the priorities of stdout and stderr the same
Normally in a script when something is send to stdout it is seen as an error has occurred. But in systemd both get a priority of 6 (info). Why does stderr not get a priority of 3 (err), or at least lower as stdout? Now when I want the things send to stderr I also get the things send to stdout. -- Cecil Westerhof
