Re: [HACKERS] WIP: Separate log file for extension
Magnus Hagander wrote: > > On Fri, Aug 25, 2017 at 12:12 AM, Antonin Houska wrote: > > I like this idea in general. > > Then it's supposed to change some of its attributes > > > adjust_log_stream_attr(&stream->filename, "my_extension.log"); > > This, however, seems to be wrong. > > The logfile name does not belong in the extension, it belongs in the > configuration file. I think the extension should set it's "stream id" or > whatever you want to call it, and then it should be possible to control in > postgresql.conf where that log is sent. Doesn't the last paragraph of https://www.postgresql.org/message-id/11412.1503912190%40localhost address your concerns? > Also, what if this extension is loaded on demand in a session and not via > shared_preload_libraries? It looks like the syslogger only gets the list of > configured streams when it starts? Yes, the syslogger gets the list of streams only when it starts, so the extension that wants to use this feature needs to provide the file information via shared_preload_libraries. I consider it sufficient because various existing logging-related GUCs also can't be changed on-the-fly. > In short, I think the solution should be more generic, and not "just for > extensions". o.k. Any idea about dividing the streams into categories? Should they for example correspond somehow to categories of GUC variables? > I completely missed this thread when I did my quick-wip at > https://www.postgresql.org/message-id/flat/cabuevexztl0gorywm9s4tr_ft3fmjbraxqdxj+bqzjpvmru...@mail.gmail.com#cabuevexztl0gorywm9s4tr_ft3fmjbraxqdxj+bqzjpvmru...@mail.gmail.com > -- some of the changes made were close enough that I got the two confused :) > Based on the feedback of that one, have you done any performance checks? I don't expect mere routing of messages into multiple files to bring any overhead. I'll run some tests, just out of curiosity. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Separate log file for extension
Tomas Vondra wrote: > On 08/28/2017 11:23 AM, Antonin Houska wrote: > > Craig Ringer wrote: > > > >> On 25 August 2017 at 15:12, Antonin Houska wrote: > >> > >> How will this play with syslog? csvlog? etc? > > > > There's nothing special about csvlog: the LogStream structure has a > > "destination" field, so if particular extension wants this kind of output, > > it > > simply sets the LOG_DESTINATION_CSVLOG bit here. > > > > I assume Craig's point was more about CSVLOG requiring log_collector=on. > So what will happen if the PostgreSQL is started without the collector, > and an extension attempts to use LOG_DESTINATION_CSVLOG? Or will it > start a separate collector for each such extension? The patch does not try to change this behavior. So there'll be no csvlog file and only the plain (not csv-formatted) message will be written to console. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Separate log file for extension
On Fri, Aug 25, 2017 at 12:12 AM, Antonin Houska wrote:
> Attached is a draft patch to allow extension to write log messages to a
> separate file. It introduces a concept of a "log stream". The extension's
> shared library gets its stream assigned by calling this function from
> _PG_init()
> my_stream_id = get_log_stream("my_extension", &my_log_stream);
>
I like this idea in general.
> Then it's supposed to change some of its attributes
>
> adjust_log_stream_attr(&stream->filename, "my_extension.log");
>
This, however, seems to be wrong.
The logfile name does not belong in the extension, it belongs in the
configuration file. I think the extension should set it's "stream id" or
whatever you want to call it, and then it should be possible to control in
postgresql.conf where that log is sent.
Also, what if this extension is loaded on demand in a session and not via
shared_preload_libraries? It looks like the syslogger only gets the list of
configured streams when it starts?
In short, I think the solution should be more generic, and not "just for
extensions".
I completely missed this thread when I did my quick-wip at
https://www.postgresql.org/message-id/flat/cabuevexztl0gorywm9s4tr_ft3fmjbraxqdxj+bqzjpvmru...@mail.gmail.com#cabuevexztl0gorywm9s4tr_ft3fmjbraxqdxj+bqzjpvmru...@mail.gmail.com
-- some of the changes made were close enough that I got the two confused
:) Based on the feedback of that one, have you done any performance checks?
--
Magnus Hagander
Me: https://www.hagander.net/
Re: [HACKERS] WIP: Separate log file for extension
On 08/28/2017 11:23 AM, Antonin Houska wrote: > Craig Ringer wrote: > >> On 25 August 2017 at 15:12, Antonin Houska wrote: >> >> How will this play with syslog? csvlog? etc? > > There's nothing special about csvlog: the LogStream structure has a > "destination" field, so if particular extension wants this kind of output, it > simply sets the LOG_DESTINATION_CSVLOG bit here. > I assume Craig's point was more about CSVLOG requiring log_collector=on. So what will happen if the PostgreSQL is started without the collector, and an extension attempts to use LOG_DESTINATION_CSVLOG? Or will it start a separate collector for each such extension? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Separate log file for extension
Masahiko Sawada wrote: > On Fri, Aug 25, 2017 at 4:12 PM, Antonin Houska wrote: > > Attached is a draft patch to allow extension to write log messages to a > > separate file. > > Does it allow postgres core code to write log messages to multiple log > files as well? I can imagine a use case where we want to write log > messages to separate log files by error level or purpose (server log > and SQL log etc). At low level it should work as long as several log streams are reserved for the core code. However, as every single stream needs multiple variables, I have no idea how to configure those additional streams w/o adding many new GUC variables to the core. Also the discussion what should (not) be logged separate would probably be difficult. Specifically to log SQL statements to a separate file, I think (but not verified) that you can write an extension that reserves a stream for itself and also uses emit_log_hook to recognize SQL statements among the logged messages. If the hook adjusts the syslogger_stream field of ErrorData accordingly, the message should end up in the separate file. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Separate log file for extension
On Fri, Aug 25, 2017 at 4:12 PM, Antonin Houska wrote: > Attached is a draft patch to allow extension to write log messages to a > separate file. Does it allow postgres core code to write log messages to multiple log files as well? I can imagine a use case where we want to write log messages to separate log files by error level or purpose (server log and SQL log etc). Regards, -- Masahiko Sawada NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Separate log file for extension
Craig Ringer wrote: > On 25 August 2017 at 15:12, Antonin Houska wrote: > > How will this play with syslog? csvlog? etc? There's nothing special about csvlog: the LogStream structure has a "destination" field, so if particular extension wants this kind of output, it simply sets the LOG_DESTINATION_CSVLOG bit here. LOG_DESTINATION_SYSLOG is a problem because (AFAIK) a single process can maintain no more than one connection to the syslog server. I don't like the idea of closing the current connection and opening a different one whenever the next ereport() is associated with a different stream than the previous was. As for LOG_DESTINATION_EVENTLOG, I haven't checked yet. > I wonder if a level of indirection is appropriate here, where extensions (or > postgres subsystems, even) provide a log stream label. Then the logging > backed takes care of using that appropriately for the logging mechanism in > use; for logging to file that'd generally be separate files. Same for > CSVlog. Other mechanisms could be left as stubs initially. > > So the outcome would be the same, just without the assumption of specific > file name and output mechanism baked in. The example I shown in my previous email was the simplest case I could think of. But it does not mean that the file name has to be hard-coded in the extension. Instead of setting the LogStream.filename field, you can pass a pointer to this field to DefineCustomStringVariable() function, so the specific GUC can control the value. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at -- Sent via pgsql-hackers mailing list ([email protected]) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: Separate log file for extension
On 25 August 2017 at 15:12, Antonin Houska wrote: > Attached is a draft patch to allow extension to write log messages to a > separate file. I like the idea a lot. I'm not so sure about the approach. How will this play with syslog? csvlog? etc? I wonder if a level of indirection is appropriate here, where extensions (or postgres subsystems, even) provide a log stream label. Then the logging backed takes care of using that appropriately for the logging mechanism in use; for logging to file that'd generally be separate files. Same for CSVlog. Other mechanisms could be left as stubs initially. So the outcome would be the same, just without the assumption of specific file name and output mechanism baked in. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
