Re: Proposal: More structured logging

2022-04-07 Thread Michael Paquier
On Mon, Jan 31, 2022 at 05:46:29PM -0500, Greg Stark wrote: > It looks strange to me that the errorTag struct has a "const char > *tagname" but a "char *tagvalue". I think this is a side effect of the > code and not actually a sensible way to define the struct. Surely they > should both be const?

Re: Proposal: More structured logging

2022-01-31 Thread Greg Stark
1) I would like an interface which more or less guarantees that *every* parameter of the log message is included in the structured data. Ideally there should be no actual need to generate the formatted messages for destinations like elastic search, just record the message id and the parameters.

Re: Proposal: More structured logging

2022-01-27 Thread Ronan Dunklau
Le jeudi 27 janvier 2022, 08:15:01 CET Michael Paquier a écrit : > On Tue, Jan 18, 2022 at 06:46:03AM +0100, Ronan Dunklau wrote: > > Hum, there was a missing import in csvlog.c from the fix above. Sorry > > about > > that. > > + > > > You are also forgetting that the table listing all the

Re: Proposal: More structured logging

2022-01-26 Thread Michael Paquier
On Tue, Jan 18, 2022 at 06:46:03AM +0100, Ronan Dunklau wrote: > Hum, there was a missing import in csvlog.c from the fix above. Sorry about > that. + You are also forgetting that the table listing all the jsonlog fields needs a refresh with this new key called "tags", and that it has a JSON

Re: Proposal: More structured logging

2022-01-17 Thread Ronan Dunklau
Le lundi 17 janvier 2022, 09:18:04 CET Ronan Dunklau a écrit : > Le samedi 15 janvier 2022, 07:09:59 CET Julien Rouhaud a écrit : > > Hi, > > > > On Tue, Jan 11, 2022 at 11:05:26AM +0100, Ronan Dunklau wrote: > > > Done, and I added anoher commit per your suggestion to add this comment. > > > >

Re: Proposal: More structured logging

2022-01-17 Thread Ronan Dunklau
Le samedi 15 janvier 2022, 07:09:59 CET Julien Rouhaud a écrit : > Hi, > > On Tue, Jan 11, 2022 at 11:05:26AM +0100, Ronan Dunklau wrote: > > Done, and I added anoher commit per your suggestion to add this comment. > > The cfbot reports that the patchset doesn't apply anymore: > >

Re: Proposal: More structured logging

2022-01-14 Thread Julien Rouhaud
Hi, On Tue, Jan 11, 2022 at 11:05:26AM +0100, Ronan Dunklau wrote: > > Done, and I added anoher commit per your suggestion to add this comment. The cfbot reports that the patchset doesn't apply anymore: http://cfbot.cputube.org/patch_36_3293.log === Applying patches on top of PostgreSQL commit

Re: Proposal: More structured logging

2022-01-11 Thread Ronan Dunklau
Le mercredi 29 décembre 2021, 14:59:16 CET Justin Pryzby a écrit : > > Subject: [PATCH v3 2/3] Add test module for the new tag functionality. > > ... > > > +test_logging(PG_FUNCTION_ARGS) > > +{ > > ... > > > +(errmsg("%s", message), > > + ({ > > + forboth(lk, keys, lv,

Re: Proposal: More structured logging

2021-12-29 Thread Justin Pryzby
> Subject: [PATCH v3 2/3] Add test module for the new tag functionality. ... > +test_logging(PG_FUNCTION_ARGS) > +{ ... > + (errmsg("%s", message), > + ({ > + forboth(lk, keys, lv, values) > + { > + (errtag(lfirst(lk), "%s", (char *)

Re: Proposal: More structured logging

2021-09-09 Thread Ronan Dunklau
Le mercredi 8 septembre 2021, 11:51:31 CEST Peter Eisentraut a écrit : > On 01.09.21 10:00, Ronan Dunklau wrote: > > In-core it would open up the possibility to split log messages into > > different fields, for example the different statistics reported in the > > logs by VACUUM / ANALYZE VERBOSE

Re: Proposal: More structured logging

2021-09-08 Thread Peter Eisentraut
On 01.09.21 10:00, Ronan Dunklau wrote: In-core it would open up the possibility to split log messages into different fields, for example the different statistics reported in the logs by VACUUM / ANALYZE VERBOSE and make it easier to consume the output without having to parse the message.

Re: Proposal: More structured logging

2021-09-01 Thread Ronan Dunklau
Le mercredi 1 septembre 2021, 09:36:50 CEST Peter Eisentraut a écrit : > On 13.08.21 15:23, Ronan Dunklau wrote: > > The logging system already captures a lot of information in the ErrorData. > > But at present there is no way for a log message authors to include more > > metadata about the log

Re: Proposal: More structured logging

2021-09-01 Thread Peter Eisentraut
On 13.08.21 15:23, Ronan Dunklau wrote: The logging system already captures a lot of information in the ErrorData. But at present there is no way for a log message authors to include more metadata about the log outside of the log message itself. For example, including the extension name which

Re: Proposal: More structured logging

2021-09-01 Thread Peter Eisentraut
On 23.08.21 11:33, Magnus Hagander wrote: In short, I would also support the presence of JSON log format in core. (but as a proper log_destination of course -- or if it's time to actually split that into a separaet thing, being one parameter for log_destination and another for log_format) It

Re: Proposal: More structured logging

2021-08-31 Thread Michael Paquier
On Tue, Aug 31, 2021 at 10:46:30AM -0400, Sehrope Sarkuni wrote: > It needed some cleanup due to bit rot, but it now builds and works atop > master. I'll post it in its own thread. Thanks. -- Michael signature.asc Description: PGP signature

Re: Proposal: More structured logging

2021-08-31 Thread Sehrope Sarkuni
On Tue, Aug 24, 2021 at 7:22 PM Michael Paquier wrote: > From a code perspective, and while on it, we could split a bit elog.c > and move the log entries generated for each format into their own > file. That would be cleaner for CSV and JSON. As a whole I don't > have an objection with moving

Re: Proposal: More structured logging

2021-08-24 Thread Michael Paquier
On Mon, Aug 23, 2021 at 11:33:09AM +0200, Magnus Hagander wrote: > As long as it's optional, I don't think that drawback holds as an > argument. The same argument could be made against the cvs logs in the > first place -- they add information to every row that a lot of people > don't need. But

Re: Proposal: More structured logging

2021-08-23 Thread Magnus Hagander
On Sat, Aug 21, 2021 at 2:37 AM Michael Paquier wrote: > > On Fri, Aug 20, 2021 at 11:35:29AM +0200, Ronan Dunklau wrote: > > Michael, your jsonlog module already fullfills this need. Is it something > > that > > should be merged into our tree ? > > Yes, there is nothing technically preventing

Re: Proposal: More structured logging

2021-08-23 Thread Ronan Dunklau
Le vendredi 20 août 2021, 11:31:21 CEST Ronan Dunklau a écrit : > Le jeudi 19 août 2021, 15:04:30 CEST Alvaro Herrera a écrit : > > On 2021-Aug-13, Ronan Dunklau wrote: > > > ereport(NOTICE, > > > > > > (errmsg("My log message")), > > > (errtag("EMITTER", "MYEXTENSION")), > > >

Re: Proposal: More structured logging

2021-08-20 Thread Michael Paquier
On Fri, Aug 20, 2021 at 11:35:29AM +0200, Ronan Dunklau wrote: > Michael, your jsonlog module already fullfills this need. Is it something > that > should be merged into our tree ? Yes, there is nothing technically preventing to have this stuff in core, of course, and that would even take care

Re: Proposal: More structured logging

2021-08-20 Thread Ronan Dunklau
Le jeudi 19 août 2021, 16:50:10 CEST Alvaro Herrera a écrit : > On 2021-Aug-19, Magnus Hagander wrote: > > Another thing I've noticed in more and more other products is to be > > able to log as json, which is then later thrown into a central logging > > system somewhere. Basically like csv, but

Re: Proposal: More structured logging

2021-08-20 Thread Ronan Dunklau
Le jeudi 19 août 2021, 15:04:30 CEST Alvaro Herrera a écrit : > On 2021-Aug-13, Ronan Dunklau wrote: > > ereport(NOTICE, > > > > (errmsg("My log message")), > > (errtag("EMITTER", "MYEXTENSION")), > > (errtag("MSG-ID", "%d", error_message_id)) > > > > ); > > Interesting idea. I agree this

Re: Proposal: More structured logging

2021-08-19 Thread Alvaro Herrera
On 2021-Aug-19, Magnus Hagander wrote: > Another thing I've noticed in more and more other products is to be > able to log as json, which is then later thrown into a central logging > system somewhere. Basically like csv, but with the schema defined in > each row. Yes, a lot more overhead, but

Re: Proposal: More structured logging

2021-08-19 Thread Magnus Hagander
On Thu, Aug 19, 2021 at 3:04 PM Alvaro Herrera wrote: > > On 2021-Aug-13, Ronan Dunklau wrote: > > > ereport(NOTICE, > > (errmsg("My log message")), > > (errtag("EMITTER", "MYEXTENSION")), > > (errtag("MSG-ID", "%d", error_message_id)) > > ); > > Interesting idea. I agree this would be

Re: Proposal: More structured logging

2021-08-19 Thread Alvaro Herrera
On 2021-Aug-13, Ronan Dunklau wrote: > ereport(NOTICE, > (errmsg("My log message")), > (errtag("EMITTER", "MYEXTENSION")), > (errtag("MSG-ID", "%d", error_message_id)) > ); Interesting idea. I agree this would be useful. > Please find attached a very small POC patch to better demonstrate

Proposal: More structured logging

2021-08-13 Thread Ronan Dunklau
Hello, The logging system already captures a lot of information in the ErrorData. But at present there is no way for a log message authors to include more metadata about the log outside of the log message itself. For example, including the extension name which can be useful for filtering /