Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Fri, Feb 11, 2011 at 11:23 AM, Stephen Frost <sfr...@snowman.net> wrote: > > Updated patch attached, full git log below.
Thanks again for the review. Updated patch attached.
> The documentation for csvlog_fields should probably use <literal>
> around the default value.
Fixed.
> The sentence that begins "For details on what these fields are" should
> hyperlink the referenced sections of the documentation.
Fixed.
> The function prototype you added to elog.c is misformatted - the type
> should be on the line preceding the function name only for the
> definition, not for prototypes.
Fixed.
> The code for log_line_prefix = 'u' is indented wrong. Also, an else
> clause that has only an if hanging off of it can be turned into an
> "else if" for better readability.
Fixed.
> This part kind of concerns me:
>
> + * This is more of a 'safety valve' than anything else,
> + * since GUC processing really should happen before we do any error logging.
> + * We might even want to change this eventually to just not log CSV format
> logs
> + * if this ever happens, to avoid a discrepency in the CSV log file which
> would
> + * make it difficult to load into PG.
>
> I'm not really convinced that making the CSV log format variable is a
> good thing. One of the reasons we added that format in the first
> place is to make sure that we could generate log output in an easily
> parseable format, and this seems like a big step backwards for not
> much, especially if we can't even guarantee that we're not going to
> inject random differently-formatted lines during startup.
Comment, function, and whole issue removed.
> Stylistically, build_default_csvlog_list and assign_csvlog_fields
> ought to be loops driven off an array, rather than hand-coded.
Done, added CSVFieldNames and modiffied assign_csvlog_fileds to use it
(build_default_csvlog_list was removed).
> I think this was discussed before, and I hate to remention it, but
> would it make sense to reuse the single-character codes from
> log_line_prefix rather than inventing new codes for the same fields?
I'd rather ditch the log_line_prefix idea of single-letter codes since
it's never going to scale. Perhaps making log_line_prefix with work
%csvlog_name instead of just the %<single-letter> codes might work. I
don't see a solution which doesn't involve changing log_line_prefix
though, in any case.
> It would be awfully nice if we could inject something into the csvlog
> output format that would let client programs know which fields are
> present. This would be especially useful for people writing tools
> that are intended to work on ANY PG installation, rather than just,
> say, their own. I'm not sure if there's a clean way to do that,
> though.
Added csvlog_header GUC to allow the user to ask for the header to be
printed at the top of each log file. If and when an option is added to
PG's COPY to respect the header, this should resolve that issue.
Also updated to HEAD.
Full git log below, patch attached.
Thanks,
Stephen
commit 592c256ffff4ffde77fc29ff28fdedd2c9f2dafd
Merge: 33639eb cebbaa1
Author: Stephen Frost <sfr...@snowman.net>
Date: Sun Feb 13 21:11:44 2011 -0500
Merge branch 'master' of git://git.postgresql.org/git/postgresql into
log_csv_options
commit 33639ebfe67b0dd58a0a89161e9f0d5237830ed4
Author: Stephen Frost <sfr...@snowman.net>
Date: Sun Feb 13 21:08:08 2011 -0500
Add csvlog_header GUC, other cleanup
This patch adds a csvlog_header option which will start each CSV
log file with a header which matches the GUC (and hence the format
of the CSV log file generated).
Numerous other whitespace clean-ups, removed build_default_csvlog_list(),
since it wasn't actually necessary or useful. Added an array which
lists the text strings of the various CSVLOG options to simplify
assign_csvlog_fields().
commit 6bd2b9f1d2bc3b166a3e5598ee590e25159c61a5
Author: Stephen Frost <sfr...@snowman.net>
Date: Fri Feb 11 11:16:17 2011 -0500
Rename log_csv_fields GUC to csvlog_fields
This patch renames the log_csv_fileds GUC to csvlog_fields, to better
match the other csvlog_* options.
Also cleaned up the CSV generation code a bit by moving the comma-adding
code out of the switch() statement.
commit a281ca611e6181339e92b488c815e0cb8c1298d2
Merge: d8dddd1 183d3cf
Author: Stephen Frost <sfr...@snowman.net>
Date: Fri Feb 11 08:37:27 2011 -0500
Merge branch 'master' of git://git.postgresql.org/git/postgresql into
log_csv_options
commit d8dddd1c425a4c320540769084ceeb7d23bc3662
Author: Stephen Frost <sfr...@snowman.net>
Date: Sun Feb 6 14:02:05 2011 -0500
Change log_csv_options listing to a table
This patch changes the listing of field options available to
log_csv_options into a table, which will hopefully both look
better and be clearer.
commit f9851cdfaeb931f01c015f5651b72d16957c7114
Merge: 3e71e33 5ed45ac
Author: Stephen Frost <sfr...@snowman.net>
Date: Sun Feb 6 13:26:17 2011 -0500
Merge branch 'master' of git://git.postgresql.org/git/postgresql into
log_csv_options
commit 3e71e338a2b9352d730f59a989027e33d99bea50
Author: Stephen Frost <sfr...@snowman.net>
Date: Fri Jan 28 22:44:33 2011 -0500
Cleanup log_csv_options patch
Clean up of various function declarations to hopefully be correct
and clean and matching PG conventions. Also move TopMemoryContext
usage to later, since the local variables don't need to be in
TopMemoryContext. Lastly, ensure that a comma is not produced
after the last CSV field, and that one is produced if
application_name is not the last field.
Review by Itagaki Takahiro, thanks!
commit 1825def11badd661d219fa4c516f06e0ad423443
Merge: ff249ae 847e8c7
Author: Stephen Frost <sfr...@snowman.net>
Date: Wed Jan 19 06:50:03 2011 -0500
Merge branch 'master' of git://git.postgresql.org/git/postgresql into
log_csv_options
commit ff249aeac7216da623bf77840380d5e767f681fc
Author: Stephen Frost <sfr...@snowman.net>
Date: Wed Jan 19 00:26:52 2011 -0500
Add log_csv_fields GUC for CSV output & curr_role
This patch adds a new GUC called 'log_csv_fields'. This GUC allows
the user to control the set of fields written to the CSV output as
well as the order in which they are written. The default set of
fields remains those that were included in 9.0, to avoid breaking
existing user configurations.
In passing, update 'user_name' for log_line_prefix and log_csv_fields
to mean 'session user' (which could be reset by a superuser with
set session authorization), and add a 'role_name' option (%U) to
log_line_prefix and log_csv_fields, to allow users to log the
current role (as set by SET ROLE- not impacted by SECURITY DEFINER
functions).
*** a/doc/src/sgml/config.sgml
--- b/doc/src/sgml/config.sgml
***************
*** 3542,3548 **** local0.* /var/log/postgresql
</row>
<row>
<entry><literal>%u</literal></entry>
! <entry>User name</entry>
<entry>yes</entry>
</row>
<row>
--- 3542,3561 ----
</row>
<row>
<entry><literal>%u</literal></entry>
! <entry>Session user name, typically the user name which was used
! to authenticate to <productname>PostgreSQL</productname> with,
! but can be changed by a superuser, see <command>SET SESSION
! AUTHORIZATION</></entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>%U</literal></entry>
! <entry>Current role name, when set with <command>SET ROLE</>;
! the current role identifier is relevant for permission checking;
! Returns 'none' if the current role matches the session user.
! Note: Log messages from inside <literal>SECURITY DEFINER</>
! functions will show the calling role, not the effective role
! inside the <literal>SECURITY DEFINER</> function</entry>
<entry>yes</entry>
</row>
<row>
***************
*** 3659,3664 **** FROM pg_stat_activity;
--- 3672,3717 ----
</listitem>
</varlistentry>
+ <varlistentry id="guc-csvlog-fields" xreflabel="csvlog_fields">
+ <term><varname>csvlog_fields</varname> (<type>string</type>)</term>
+ <indexterm>
+ <primary><varname>csvlog_fields</> configuration parameter</primary>
+ </indexterm>
+ <listitem>
+ <para>
+ Controls the set and order of the fields which are written out in
+ the CSV-format log file.
+
+ The default is:
+ <literal>log_time, user_name, database_name, process_id,
+ connection_from, session_id, session_line_num, command_tag,
+ session_start_time, virtual_transaction_id, transaction_id,
+ error_severity, sql_state_code, message, detail, hint,
+ internal_query, internal_query_pos, context, query, query_pos,
+ location, application_name</literal>
+
+ For details on what these fields are, refer to the
+ <varname>log_line_prefix</varname> and
+ <xref linkend="runtime-config-logging-csvlog"> documentation.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-csvlog-header" xreflabel="csvlog_header">
+ <term><varname>csvlog_header</varname> (<type>boolean</type>)</term>
+ <indexterm>
+ <primary><varname>csvlog_header</> configuration parameter</primary>
+ </indexterm>
+ <listitem>
+ <para>
+ Controls if a header should be output for each file logged through
+ the CSV-format logging.
+
+ The default is: <literal>false</literal>, for backwards compatibility.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-log-lock-waits" xreflabel="log_lock_waits">
<term><varname>log_lock_waits</varname> (<type>boolean</type>)</term>
<indexterm>
***************
*** 3766,3799 **** FROM pg_stat_activity;
Including <literal>csvlog</> in the <varname>log_destination</> list
provides a convenient way to import log files into a database table.
This option emits log lines in comma-separated-values
! (<acronym>CSV</>) format,
! with these columns:
! timestamp with milliseconds,
! user name,
! database name,
! process ID,
! client host:port number,
! session ID,
! per-session line number,
! command tag,
! session start time,
! virtual transaction ID,
! regular transaction ID,
! error severity,
! SQLSTATE code,
! error message,
! error message detail,
! hint,
! internal query that led to the error (if any),
! character count of the error position therein,
! error context,
! user query that led to the error (if any and enabled by
! <varname>log_min_error_statement</>),
! character count of the error position therein,
! location of the error in the PostgreSQL source code
! (if <varname>log_error_verbosity</> is set to <literal>verbose</>),
! and application name.
! Here is a sample table definition for storing CSV-format log output:
<programlisting>
CREATE TABLE postgres_log
--- 3819,3971 ----
Including <literal>csvlog</> in the <varname>log_destination</> list
provides a convenient way to import log files into a database table.
This option emits log lines in comma-separated-values
! (<acronym>CSV</>) format. The following table defines the fields
! which can be included in the CSV output, their meanings, and if they
! are included in the default CSV layout (the default ordering matches
! the order of this table).
!
! <informaltable>
! <tgroup cols="3">
! <thead>
! <row>
! <entry>CSV Field Name</entry>
! <entry>Definition</entry>
! <entry>Included by Default</entry>
! </row>
! </thead>
! <tbody>
! <row>
! <entry><literal>log_time</literal></entry>
! <entry>timestamp with milliseconds</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>user_name</literal></entry>
! <entry>session user name</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>role_name</literal></entry>
! <entry>current role name</entry>
! <entry>no</entry>
! </row>
! <row>
! <entry><literal>database_name</literal></entry>
! <entry>name of database connected to</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>process_id</literal></entry>
! <entry>process ID of the backend PG process</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>connection_from</literal></entry>
! <entry>client host/IP and port number</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>session_id</literal></entry>
! <entry>ID of the session</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>session_line_number</literal></entry>
! <entry>per-session line number</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>command_tag</literal></entry>
! <entry>Command tag of the logged command</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>session_start_time</literal></entry>
! <entry>Start time of the current session</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>virtual_transaction_id</literal></entry>
! <entry>Virtual Transaction ID</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>transaction_id</literal></entry>
! <entry>Regular Transaction ID</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>error_severity</literal></entry>
! <entry>Error severity code of the log message</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>sql_state_code</literal></entry>
! <entry>SQLSTATE code of the command being logged</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>message</literal></entry>
! <entry>Error message</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>detail</literal></entry>
! <entry>Error message detail</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>hint</literal></entry>
! <entry>Error message hint</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>internal_query</literal></entry>
! <entry>internal query that led to the error (if any)</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>internal_query_pos</literal></entry>
! <entry>character count of the error position of the internal query</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>context</literal></entry>
! <entry>error context</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>query</literal></entry>
! <entry>user query that led to the error (if any and enabled by <varname>log_min_error_statement</varname>)</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>query_pos</literal></entry>
! <entry>character count of the error position of the user query</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>location</literal></entry>
! <entry>location of the error in the PostgreSQL source code (if <varname>log_error_verbosity</varname> is set to <literal>verbose</literal>)</entry>
! <entry>yes</entry>
! </row>
! <row>
! <entry><literal>application_name</literal></entry>
! <entry>Name of the connecting application, if provided by the application</entry>
! <entry>yes</entry>
! </row>
! </tbody>
! </tgroup>
! </informaltable>
!
! The set of columns to be included, and their order, in the CSV
! output can be controlled using the <varname>csvlog_fields</varname> option.
!
! For additional details on the definition of the above columns, refer
! to the documentation for <varname>log_line_prefix</varname>.
!
! Here is a sample table definition for storing the default CSV-format
! log output:
<programlisting>
CREATE TABLE postgres_log
*** a/src/backend/commands/variable.c
--- b/src/backend/commands/variable.c
***************
*** 847,852 **** assign_session_authorization(const char *value, bool doit, GucSource source)
--- 847,857 ----
return result;
}
+ /*
+ * function to return the stored session username, needed because we
+ * can't do catalog lookups when possibly being called after an error,
+ * eg: from elog.c or part of GUC handling.
+ */
const char *
show_session_authorization(void)
{
***************
*** 972,977 **** assign_role(const char *value, bool doit, GucSource source)
--- 977,987 ----
return result;
}
+ /*
+ * function to return the stored role username, needed because we
+ * can't do catalog lookups when possibly being called after an error,
+ * eg: from elog.c or part of GUC handling.
+ */
const char *
show_role(void)
{
*** a/src/backend/postmaster/syslogger.c
--- b/src/backend/postmaster/syslogger.c
***************
*** 147,152 **** static char *logfile_getname(pg_time_t timestamp, const char *suffix);
--- 147,153 ----
static void set_next_rotation_time(void);
static void sigHupHandler(SIGNAL_ARGS);
static void sigUsr1Handler(SIGNAL_ARGS);
+ static void write_csvlog_header(FILE *out_fh);
/*
***************
*** 988,993 **** pipeThread(void *arg)
--- 989,1019 ----
#endif /* WIN32 */
/*
+ * Internal function for writing out the header of a CSV-style log file
+ * to the passed-in file handle.
+ */
+ static void
+ write_csvlog_header(FILE *out_fh)
+ {
+ int rc;
+ int header_length = strlen(csvlog_fields);
+
+ /* Write out the csvlog_fields GUC, which matches the CSV log format
+ * header, at least, if we did everything right. */
+ rc = fwrite(csvlog_fields, 1, header_length, out_fh);
+
+ /* can't use ereport here because of possible recursion */
+ if (rc != header_length)
+ write_stderr("could not write to new log file: %s\n", strerror(errno));
+
+ rc = fputc('\n', out_fh);
+ if (rc != '\n')
+ write_stderr("could not write to new log file: %s\n", strerror(errno));
+
+ return;
+ }
+
+ /*
* open the csv log file - we do this opportunistically, because
* we don't know if CSV logging will be wanted.
*/
***************
*** 995,1004 **** static void
open_csvlogfile(void)
{
char *filename;
filename = logfile_getname(time(NULL), ".csv");
! csvlogFile = logfile_open(filename, "a", false);
pfree(filename);
}
--- 1021,1037 ----
open_csvlogfile(void)
{
char *filename;
+ FILE *fh;
filename = logfile_getname(time(NULL), ".csv");
! fh = logfile_open(filename, "a", false);
!
! /* Check if we are asked to write out a header for the CSV file. */
! if (csvlog_header)
! write_csvlog_header(fh);
!
! csvlogFile = fh;
pfree(filename);
}
***************
*** 1165,1170 **** logfile_rotate(bool time_based_rotation, int size_rotation_for)
--- 1198,1207 ----
return;
}
+ /* Check if we are asked to write out a header for the CSV file. */
+ if (csvlog_header)
+ write_csvlog_header(fh);
+
fclose(csvlogFile);
csvlogFile = fh;
*** a/src/backend/utils/error/elog.c
--- b/src/backend/utils/error/elog.c
***************
*** 3,8 ****
--- 3,17 ----
* elog.c
* error logging and reporting
*
+ * A few comments about situations where error processing is called:
+ *
+ * We need to be cautious of both a performance hit when logging, since
+ * log messages can be generated at a huge rate if every command is being
+ * logged and we also need to watch out for what can happen when we are
+ * trying to log from an aborted transaction. Specifically, attempting to
+ * do SysCache lookups and possibly use other usually available backend
+ * systems will fail badly when logging from an aborted transaction.
+ *
* Some notes about recursion and errors during error processing:
*
* We need to be robust about recursive-error scenarios --- for example,
***************
*** 59,73 ****
--- 68,85 ----
#include "access/transam.h"
#include "access/xact.h"
+ #include "commands/variable.h"
#include "libpq/libpq.h"
#include "libpq/pqformat.h"
#include "mb/pg_wchar.h"
#include "miscadmin.h"
+ #include "nodes/pg_list.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "storage/ipc.h"
#include "storage/proc.h"
#include "tcop/tcopprot.h"
+ #include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
***************
*** 93,98 **** extern bool redirection_done;
--- 105,144 ----
int Log_error_verbosity = PGERROR_VERBOSE;
char *Log_line_prefix = NULL; /* format for extra log line info */
int Log_destination = LOG_DESTINATION_STDERR;
+ char *csvlog_fields = NULL;
+ bool csvlog_header = false;
+
+ static List *csvlog_field_list = NIL;
+
+ /* To add a CSV field option, you need to update the enum in elog.h, check
+ * if the last value in the enum changed and if so update MAX_CSVLOG_OPTS,
+ * add code to handle the option in write_csv(), and add it here. */
+ const char *CSVFieldNames[] = {
+ "log_time", /* CSVLOG_LOG_TIME */
+ "user_name", /* CSVLOG_USER_NAME */
+ "role_name", /* CSVLOG_ROLE_NAME */
+ "database_name", /* CSVLOG_DATABASE_NAME */
+ "process_id", /* CSVLOG_PROCESS_ID */
+ "connection_from", /* CSVLOG_CONNECTION_FROM */
+ "session_id", /* CSVLOG_SESSION_ID */
+ "session_line_num", /* CSVLOG_SESSION_LINE_NUM */
+ "command_tag", /* CSVLOG_COMMAND_TAG */
+ "session_start_time", /* CSVLOG_SESSION_START_TIME */
+ "virtual_transaction_id", /* CSVLOG_VIRTUAL_TRANSACTION_ID */
+ "transaction_id", /* CSVLOG_TRANSACTION_ID */
+ "error_severity", /* CSVLOG_ERROR_SEVERITY */
+ "sql_state_code", /* CSVLOG_SQL_STATE_CODE */
+ "message", /* CSVLOG_MESSAGE */
+ "detail", /* CSVLOG_DETAIL */
+ "hint", /* CSVLOG_HINT */
+ "internal_query", /* CSVLOG_INTERNAL_QUERY */
+ "internal_query_pos", /* CSVLOG_INTERNAL_QUERY_POS */
+ "context", /* CSVLOG_CONTEXT */
+ "query", /* CSVLOG_QUERY */
+ "query_pos", /* CSVLOG_QUERY_POS */
+ "location", /* CSVLOG_LOCATION */
+ "application_name" /* CSVLOG_APPLICATION_NAME */
+ };
#ifdef HAVE_SYSLOG
***************
*** 161,166 **** static void write_csvlog(ErrorData *edata);
--- 207,217 ----
static void setup_formatted_log_time(void);
static void setup_formatted_start_time(void);
+ /* extern'd and used from guc.c... */
+ const char *assign_csvlog_fields(const char *newval, bool doit,
+ GucSource source);
+
+
/*
* in_error_recursion_trouble --- are we at risk of infinite error recursion?
***************
*** 1817,1831 **** log_line_prefix(StringInfo buf, ErrorData *edata)
}
break;
case 'u':
- if (MyProcPort)
{
! const char *username = MyProcPort->user_name;
!
! if (username == NULL || *username == '\0')
! username = _("[unknown]");
! appendStringInfoString(buf, username);
}
break;
case 'd':
if (MyProcPort)
{
--- 1868,1891 ----
}
break;
case 'u':
{
! const char *session_auth = show_session_authorization();
!
! if (*session_auth != '\0')
! appendStringInfoString(buf, session_auth);
! else if (MyProcPort)
! {
! const char *username = MyProcPort->user_name;
!
! if (username == NULL || *username == '\0')
! username = _("[unknown]");
! appendStringInfoString(buf, username);
! }
}
break;
+ case 'U':
+ appendStringInfoString(buf, show_role());
+ break;
case 'd':
if (MyProcPort)
{
***************
*** 1921,1926 **** log_line_prefix(StringInfo buf, ErrorData *edata)
--- 1981,2066 ----
}
/*
+ * Called when the GUC csvlog_fields() option has been set
+ * (currently only allowed in postmaster.conf, on PG restart).
+ *
+ * Processes the list passed in from the GUC system and updates the
+ * csvlog_field_list variable, which will then be used to generate
+ * CSV log output.
+ */
+ const char *
+ assign_csvlog_fields(const char *newval, bool doit, GucSource source)
+ {
+ /* Verify the list is valid */
+ List *new_csv_fields = NIL;
+ List *column_list = NIL;
+ ListCell *l;
+ char *rawstring;
+ MemoryContext oldcontext;
+
+ /* Need a modifyable version to pass to SplitIdentifierString */
+ rawstring = pstrdup(newval);
+
+ /* Parse string into list of identifiers */
+ if (!SplitIdentifierString(rawstring, ',', &column_list))
+ {
+ list_free(column_list);
+ pfree(rawstring);
+ return NULL;
+ }
+
+ /* Empty isn't a valid option */
+ if (column_list == NIL)
+ {
+ pfree(rawstring);
+ return NULL;
+ }
+
+ /*
+ * We need the allocations done for the csvlog_field_list to
+ * be preserved, so allocate them in TopMemoryContext.
+ */
+ oldcontext = MemoryContextSwitchTo(TopMemoryContext);
+
+ /*
+ * Loop through all of the fields provided by the user and build
+ * up our new_csv_fields list which will be processed by write_csvlog
+ */
+ foreach(l, column_list)
+ {
+ int curr_option;
+
+ /* Loop through all of the valid field options to try and match the
+ * current entry in the list to one of them. */
+ for (curr_option = 0; curr_option < MAX_CSVLOG_OPTS; curr_option++)
+ if (pg_strcasecmp(lfirst(l),CSVFieldNames[curr_option]) == 0)
+ {
+ new_csv_fields = lappend_int(new_csv_fields,curr_option);
+ break;
+ }
+
+ /* check if no option matched, and if so, return error */
+ if (curr_option == MAX_CSVLOG_OPTS)
+ return NULL;
+ }
+
+ if (doit)
+ {
+ /* put new list in place */
+ List *old_list = csvlog_field_list;
+
+ csvlog_field_list = new_csv_fields;
+
+ list_free(old_list);
+ }
+
+ /* Switch back to the calling context */
+ MemoryContextSwitchTo(oldcontext);
+
+ return newval;
+ }
+
+ /*
* append a CSV'd version of a string to a StringInfo
* We use the PostgreSQL defaults for CSV, i.e. quote = escape = '"'
* If it's NULL, append nothing.
***************
*** 1946,1957 **** appendCSVLiteral(StringInfo buf, const char *data)
}
/*
! * Constructs the error message, depending on the Errordata it gets, in a CSV
! * format which is described in doc/src/sgml/config.sgml.
*/
static void
write_csvlog(ErrorData *edata)
{
StringInfoData buf;
bool print_stmt = false;
--- 2086,2099 ----
}
/*
! * Constructs the error message, depending on the Errordata it gets, in the CSV
! * format requested by the user, based on the csvlog_fields GUC.
*/
static void
write_csvlog(ErrorData *edata)
{
+ int num_fields;
+ bool first_field = true;
StringInfoData buf;
bool print_stmt = false;
***************
*** 1961,1966 **** write_csvlog(ErrorData *edata)
--- 2103,2115 ----
/* has counter been reset in current process? */
static int log_my_pid = 0;
+ ListCell *l;
+
+ const char *session_auth = show_session_authorization();
+
+ /* csvlog_field_list should never be empty when we reach here */
+ Assert(csvlog_field_list != NIL);
+
/*
* This is one of the few places where we'd rather not inherit a static
* variable's value from the postmaster. But since we will, reset it when
***************
*** 1977,2134 **** write_csvlog(ErrorData *edata)
initStringInfo(&buf);
/*
! * timestamp with milliseconds
! *
! * Check if the timestamp is already calculated for the syslog message,
! * and use it if so. Otherwise, get the current timestamp. This is done
! * to put same timestamp in both syslog and csvlog messages.
*/
! if (formatted_log_time[0] == '\0')
! setup_formatted_log_time();
! appendStringInfoString(&buf, formatted_log_time);
! appendStringInfoChar(&buf, ',');
! /* username */
! if (MyProcPort)
! appendCSVLiteral(&buf, MyProcPort->user_name);
! appendStringInfoChar(&buf, ',');
! /* database name */
! if (MyProcPort)
! appendCSVLiteral(&buf, MyProcPort->database_name);
! appendStringInfoChar(&buf, ',');
! /* Process id */
! if (MyProcPid != 0)
! appendStringInfo(&buf, "%d", MyProcPid);
! appendStringInfoChar(&buf, ',');
! /* Remote host and port */
! if (MyProcPort && MyProcPort->remote_host)
! {
! appendStringInfoChar(&buf, '"');
! appendStringInfoString(&buf, MyProcPort->remote_host);
! if (MyProcPort->remote_port && MyProcPort->remote_port[0] != '\0')
! {
! appendStringInfoChar(&buf, ':');
! appendStringInfoString(&buf, MyProcPort->remote_port);
! }
! appendStringInfoChar(&buf, '"');
! }
! appendStringInfoChar(&buf, ',');
! /* session id */
! appendStringInfo(&buf, "%lx.%x", (long) MyStartTime, MyProcPid);
! appendStringInfoChar(&buf, ',');
! /* Line number */
! appendStringInfo(&buf, "%ld", log_line_number);
! appendStringInfoChar(&buf, ',');
! /* PS display */
! if (MyProcPort)
! {
! StringInfoData msgbuf;
! const char *psdisp;
! int displen;
! initStringInfo(&msgbuf);
! psdisp = get_ps_display(&displen);
! appendBinaryStringInfo(&msgbuf, psdisp, displen);
! appendCSVLiteral(&buf, msgbuf.data);
! pfree(msgbuf.data);
! }
! appendStringInfoChar(&buf, ',');
! /* session start timestamp */
! if (formatted_start_time[0] == '\0')
! setup_formatted_start_time();
! appendStringInfoString(&buf, formatted_start_time);
! appendStringInfoChar(&buf, ',');
! /* Virtual transaction id */
! /* keep VXID format in sync with lockfuncs.c */
! if (MyProc != NULL && MyProc->backendId != InvalidBackendId)
! appendStringInfo(&buf, "%d/%u", MyProc->backendId, MyProc->lxid);
! appendStringInfoChar(&buf, ',');
! /* Transaction id */
! appendStringInfo(&buf, "%u", GetTopTransactionIdIfAny());
! appendStringInfoChar(&buf, ',');
! /* Error severity */
! appendStringInfoString(&buf, error_severity(edata->elevel));
! appendStringInfoChar(&buf, ',');
! /* SQL state code */
! appendStringInfoString(&buf, unpack_sql_state(edata->sqlerrcode));
! appendStringInfoChar(&buf, ',');
! /* errmessage */
! appendCSVLiteral(&buf, edata->message);
! appendStringInfoChar(&buf, ',');
! /* errdetail or errdetail_log */
! if (edata->detail_log)
! appendCSVLiteral(&buf, edata->detail_log);
! else
! appendCSVLiteral(&buf, edata->detail);
! appendStringInfoChar(&buf, ',');
! /* errhint */
! appendCSVLiteral(&buf, edata->hint);
! appendStringInfoChar(&buf, ',');
! /* internal query */
! appendCSVLiteral(&buf, edata->internalquery);
! appendStringInfoChar(&buf, ',');
! /* if printed internal query, print internal pos too */
! if (edata->internalpos > 0 && edata->internalquery != NULL)
! appendStringInfo(&buf, "%d", edata->internalpos);
! appendStringInfoChar(&buf, ',');
! /* errcontext */
! appendCSVLiteral(&buf, edata->context);
! appendStringInfoChar(&buf, ',');
! /* user query --- only reported if not disabled by the caller */
! if (is_log_level_output(edata->elevel, log_min_error_statement) &&
! debug_query_string != NULL &&
! !edata->hide_stmt)
! print_stmt = true;
! if (print_stmt)
! appendCSVLiteral(&buf, debug_query_string);
! appendStringInfoChar(&buf, ',');
! if (print_stmt && edata->cursorpos > 0)
! appendStringInfo(&buf, "%d", edata->cursorpos);
! appendStringInfoChar(&buf, ',');
!
! /* file error location */
! if (Log_error_verbosity >= PGERROR_VERBOSE)
! {
! StringInfoData msgbuf;
!
! initStringInfo(&msgbuf);
!
! if (edata->funcname && edata->filename)
! appendStringInfo(&msgbuf, "%s, %s:%d",
! edata->funcname, edata->filename,
! edata->lineno);
! else if (edata->filename)
! appendStringInfo(&msgbuf, "%s:%d",
! edata->filename, edata->lineno);
! appendCSVLiteral(&buf, msgbuf.data);
! pfree(msgbuf.data);
! }
! appendStringInfoChar(&buf, ',');
! /* application name */
! if (application_name)
! appendCSVLiteral(&buf, application_name);
appendStringInfoChar(&buf, '\n');
--- 2126,2374 ----
initStringInfo(&buf);
/*
! * Get the number of fields, so we make sure to *not* include a comma
! * after the last field.
*/
! num_fields = list_length(csvlog_field_list);
! /*
! * Loop through the fields requested by the user, in the order requested, in
! * the csvlog_fields GUC.
! */
! foreach(l, csvlog_field_list)
! {
! /* If this isn't the first field, prepend a comma to seperate this
! * field from the previous one */
! if (!first_field)
! appendStringInfoChar(&buf, ',');
! else
! first_field = false;
! switch (lfirst_int(l))
! {
! case CSVLOG_LOG_TIME:
! {
! /*
! * timestamp with milliseconds
! *
! * Check if the timestamp is already calculated for the syslog message,
! * and use it if so. Otherwise, get the current timestamp. This is done
! * to put same timestamp in both syslog and csvlog messages.
! */
! if (formatted_log_time[0] == '\0')
! setup_formatted_log_time();
!
! appendStringInfoString(&buf, formatted_log_time);
! }
! break;
! case CSVLOG_USER_NAME:
! {
! /* session username, as done for %u */
! if (*session_auth != '\0')
! appendCSVLiteral(&buf, session_auth);
! else
! /* username */
! if (MyProcPort)
! {
! const char *username = MyProcPort->user_name;
! if (username == NULL || *username == '\0')
! username = _("[unknown]");
! appendCSVLiteral(&buf, MyProcPort->user_name);
! }
! }
! break;
! case CSVLOG_ROLE_NAME:
! /* current role, not updated if someone renames it in another
! * session, of course */
! appendCSVLiteral(&buf, show_role());
! break;
! case CSVLOG_DATABASE_NAME:
! {
! /* database name */
! if (MyProcPort)
! appendCSVLiteral(&buf, MyProcPort->database_name);
! }
! break;
! case CSVLOG_PROCESS_ID:
! {
! /* Process id */
! if (MyProcPid != 0)
! appendStringInfo(&buf, "%d", MyProcPid);
! }
! break;
! case CSVLOG_CONNECTION_FROM:
! {
! /* Remote host and port */
! if (MyProcPort && MyProcPort->remote_host)
! {
! appendStringInfoChar(&buf, '"');
! appendStringInfoString(&buf, MyProcPort->remote_host);
! if (MyProcPort->remote_port && MyProcPort->remote_port[0] != '\0')
! {
! appendStringInfoChar(&buf, ':');
! appendStringInfoString(&buf, MyProcPort->remote_port);
! }
! appendStringInfoChar(&buf, '"');
! }
! }
! break;
! case CSVLOG_SESSION_ID:
! /* session id */
! appendStringInfo(&buf, "%lx.%x", (long) MyStartTime, MyProcPid);
! break;
! case CSVLOG_SESSION_LINE_NUM:
! /* Line number */
! appendStringInfo(&buf, "%ld", log_line_number);
! break;
! case CSVLOG_COMMAND_TAG:
! {
! /* PS display */
! if (MyProcPort)
! {
! StringInfoData msgbuf;
! const char *psdisp;
! int displen;
! initStringInfo(&msgbuf);
! psdisp = get_ps_display(&displen);
! appendBinaryStringInfo(&msgbuf, psdisp, displen);
! appendCSVLiteral(&buf, msgbuf.data);
! pfree(msgbuf.data);
! }
! }
! break;
! case CSVLOG_SESSION_START_TIME:
! {
! /* session start timestamp */
! if (formatted_start_time[0] == '\0')
! setup_formatted_start_time();
! appendStringInfoString(&buf, formatted_start_time);
! }
! break;
! case CSVLOG_VIRTUAL_TRANSACTION_ID:
! {
! /* Virtual transaction id */
! /* keep VXID format in sync with lockfuncs.c */
! if (MyProc != NULL && MyProc->backendId != InvalidBackendId)
! appendStringInfo(&buf, "%d/%u", MyProc->backendId, MyProc->lxid);
! }
! break;
! case CSVLOG_TRANSACTION_ID:
! /* Transaction id */
! appendStringInfo(&buf, "%u", GetTopTransactionIdIfAny());
! break;
! case CSVLOG_ERROR_SEVERITY:
! /* Error severity */
! appendStringInfoString(&buf, error_severity(edata->elevel));
! break;
! case CSVLOG_SQL_STATE_CODE:
! /* SQL state code */
! appendStringInfoString(&buf, unpack_sql_state(edata->sqlerrcode));
! break;
!
! case CSVLOG_MESSAGE:
! /* errmessage */
! appendCSVLiteral(&buf, edata->message);
! break;
!
! case CSVLOG_DETAIL:
! {
! /* errdetail or errdetail_log */
! if (edata->detail_log)
! appendCSVLiteral(&buf, edata->detail_log);
! else
! appendCSVLiteral(&buf, edata->detail);
! }
! break;
! case CSVLOG_HINT:
! /* errhint */
! appendCSVLiteral(&buf, edata->hint);
! break;
! case CSVLOG_INTERNAL_QUERY:
! /* internal query */
! appendCSVLiteral(&buf, edata->internalquery);
! break;
! case CSVLOG_INTERNAL_QUERY_POS:
! {
! /* if printed internal query, print internal pos too */
! if (edata->internalpos > 0 && edata->internalquery != NULL)
! appendStringInfo(&buf, "%d", edata->internalpos);
! }
! break;
! case CSVLOG_CONTEXT:
! /* errcontext */
! appendCSVLiteral(&buf, edata->context);
! break;
! case CSVLOG_QUERY:
! {
! /* user query --- only reported if not disabled by the caller */
! if (is_log_level_output(edata->elevel, log_min_error_statement) &&
! debug_query_string != NULL &&
! !edata->hide_stmt)
! print_stmt = true;
! if (print_stmt)
! appendCSVLiteral(&buf, debug_query_string);
! }
! break;
!
! case CSVLOG_QUERY_POS:
! {
! if (print_stmt && edata->cursorpos > 0)
! appendStringInfo(&buf, "%d", edata->cursorpos);
! }
! break;
! case CSVLOG_LOCATION:
! {
! /* file error location */
! if (Log_error_verbosity >= PGERROR_VERBOSE)
! {
! StringInfoData msgbuf;
!
! initStringInfo(&msgbuf);
!
! if (edata->funcname && edata->filename)
! appendStringInfo(&msgbuf, "%s, %s:%d",
! edata->funcname, edata->filename,
! edata->lineno);
! else if (edata->filename)
! appendStringInfo(&msgbuf, "%s:%d",
! edata->filename, edata->lineno);
! appendCSVLiteral(&buf, msgbuf.data);
! pfree(msgbuf.data);
! }
! }
! break;
!
! case CSVLOG_APPLICATION_NAME:
! {
! /* application name */
! if (application_name)
! appendCSVLiteral(&buf, application_name);
! }
! break;
! }
! }
appendStringInfoChar(&buf, '\n');
***************
*** 2139,2144 **** write_csvlog(ErrorData *edata)
--- 2379,2386 ----
write_pipe_chunks(buf.data, buf.len, LOG_DESTINATION_CSVLOG);
pfree(buf.data);
+
+ return;
}
/*
*** a/src/backend/utils/misc/guc.c
--- b/src/backend/utils/misc/guc.c
***************
*** 65,70 ****
--- 65,71 ----
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+ #include "utils/elog.h"
#include "utils/guc_tables.h"
#include "utils/memutils.h"
#include "utils/pg_locale.h"
***************
*** 191,196 **** static char *config_enum_get_options(struct config_enum * record,
--- 192,200 ----
const char *prefix, const char *suffix,
const char *separator);
+ /* Needs to be defined here because elog.h can't #include guc.h */
+ extern const char *assign_csvlog_fields(const char *newval,
+ bool doit, GucSource source);
/*
* Options for enum values defined in this module.
***************
*** 1034,1039 **** static struct config_bool ConfigureNamesBool[] =
--- 1038,1052 ----
false, NULL, NULL
},
{
+ {"csvlog_header", PGC_POSTMASTER, LOGGING_WHAT,
+ gettext_noop("Enables including a header on CSV log files."),
+ NULL,
+ },
+ &csvlog_header,
+ false, NULL, NULL
+ },
+
+ {
{"sql_inheritance", PGC_USERSET, COMPAT_OPTIONS_PREVIOUS,
gettext_noop("Causes subtables to be included by default in various commands."),
NULL
***************
*** 2326,2331 **** static struct config_string ConfigureNamesString[] =
--- 2339,2355 ----
},
{
+ {"csvlog_fields", PGC_POSTMASTER, LOGGING_WHAT,
+ gettext_noop("Controls fields logged to CSV logfiles."),
+ gettext_noop("If blank, the default set of fields is used."),
+ GUC_LIST_INPUT
+ },
+ &csvlog_fields,
+ "log_time, user_name, database_name, process_id, connection_from, session_id, session_line_num, command_tag, session_start_time, virtual_transaction_id, transaction_id, error_severity, sql_state_code, message, detail, hint, internal_query, internal_query_pos, context, query, query_pos, location, application_name",
+ assign_csvlog_fields, NULL
+ },
+
+ {
{"log_timezone", PGC_SIGHUP, LOGGING_WHAT,
gettext_noop("Sets the time zone to use in log messages."),
NULL
*** a/src/backend/utils/misc/postgresql.conf.sample
--- b/src/backend/utils/misc/postgresql.conf.sample
***************
*** 378,383 ****
--- 378,388 ----
# processes
# %% = '%'
# e.g. '<%u%%%d> '
+
+ # fields to include in the CSV log output
+ #csvlog_fields = 'log_time, user_name, database_name, process_id, connection_from, session_id, session_line_num, command_tag, session_start_time, virtual_transaction_id, transaction_id, error_severity, sql_state_code, message, detail, hint, internal_query, internal_query_pos, context, query, query_pos, location, application_name'
+ #csvlog_header = false # should csvlog files have a header?
+
#log_lock_waits = off # log lock waits >= deadlock_timeout
#log_statement = 'none' # none, ddl, mod, all
#log_temp_files = -1 # log temporary files equal or larger
*** a/src/include/utils/elog.h
--- b/src/include/utils/elog.h
***************
*** 330,337 **** typedef enum
--- 330,386 ----
extern int Log_error_verbosity;
extern char *Log_line_prefix;
+ extern char *csvlog_fields; /* List of fields to log with CSV logging */
+ extern bool csvlog_header; /* Whether to include a header on CSV log files */
extern int Log_destination;
+ /*
+ * Enum of the CSV fields we understand for CSV-based logging,
+ * if an new field is added, the enum has to be updated, the
+ * definition of field names in elog.c needs to be updated, and the
+ * new field needs to be handled in write_csv() in elog.c.
+ * Also be sure to update MAX_CSVLOG_OPTS if you change what the last
+ * option in the enum list is.
+ */
+ typedef enum LogCSVFields
+ {
+ CSVLOG_LOG_TIME,
+ CSVLOG_USER_NAME,
+ CSVLOG_ROLE_NAME,
+ CSVLOG_DATABASE_NAME,
+ CSVLOG_PROCESS_ID,
+ CSVLOG_CONNECTION_FROM,
+ CSVLOG_SESSION_ID,
+ CSVLOG_SESSION_LINE_NUM,
+ CSVLOG_COMMAND_TAG,
+ CSVLOG_SESSION_START_TIME,
+ CSVLOG_VIRTUAL_TRANSACTION_ID,
+ CSVLOG_TRANSACTION_ID,
+ CSVLOG_ERROR_SEVERITY,
+ CSVLOG_SQL_STATE_CODE,
+ CSVLOG_MESSAGE,
+ CSVLOG_DETAIL,
+ CSVLOG_HINT,
+ CSVLOG_INTERNAL_QUERY,
+ CSVLOG_INTERNAL_QUERY_POS,
+ CSVLOG_CONTEXT,
+ CSVLOG_QUERY,
+ CSVLOG_QUERY_POS,
+ CSVLOG_LOCATION,
+ CSVLOG_APPLICATION_NAME
+ } LogCSVFields;
+
+ /* Make sure to update this if you add CSV log options and change
+ * what the last CSVLOG option is */
+ #define MAX_CSVLOG_OPTS CSVLOG_APPLICATION_NAME+1
+
+ /*
+ * Array of the names of each of the CSV fields we allow for logging,
+ * if an new field is added, the enum has to be updated *and* the
+ * definition of field names in elog.c needs to be updated.
+ */
+ extern const char *CSVFieldNames[];
+
/* Log destination bitmap */
#define LOG_DESTINATION_STDERR 1
#define LOG_DESTINATION_SYSLOG 2
*** a/src/tools/pgindent/typedefs.list
--- b/src/tools/pgindent/typedefs.list
***************
*** 855,860 **** LockTagType
--- 855,861 ----
LockTupleMode
LockingClause
LogStmtLevel
+ LogCSVFields
LogicalTape
LogicalTapeSet
MAGIC
signature.asc
Description: Digital signature
