On 11/24/2011 06:33 PM, Theo Schlossnagle wrote: > I see the next steps being: > 1) agreeing that a problem exists (I know one does, but I suppose > consensus is req'd)
+1, high volume logging is also a huge problem here at Skype. Some of the issues that immediately come to mind: - extracting usage statistics from logs - extracting error messages from logs - pushing the logs to remote log server (and no, syslog is not actually usable for that ;) > 2) agreeing that "hooks" are the right approach, if not propose a > different approach. (fwiw, it's incredible common) +1 for hook based approach. While the whole logging system could possibly use an overhaul, this is not likely going to happen for 9.2. Meanwhile hooks provide the flexibility to implement custom log collectors for those who really need it. > 3) reworking the implementation to fit in the project; I assume the > implementation I proposed will, at best, vaguely resemble anything > that gets integrated. It was just a PoC. > I'd vote for a hook to be added to EmitErrorReport. That way all logged messages pass through the hook, which could then decide whether to actually write the message to server log or to discard it. As an added bonus, the message is broken down into pieces so there is no need to parse the messages for severity, context, etc. Patch attached. regards, Martin
*** a/src/backend/utils/error/elog.c --- b/src/backend/utils/error/elog.c *************** *** 136,141 **** static int errordata_stack_depth = -1; /* index of topmost active frame */ --- 136,143 ---- static int recursion_depth = 0; /* to detect actual recursion */ + emit_log_hook_type emit_log_hook = NULL; /* hook for log interception */ + /* buffers for formatted timestamps that might be used by both * log_line_prefix and csv logs. */ *************** *** 1276,1281 **** EmitErrorReport(void) --- 1278,1286 ---- CHECK_STACK_DEPTH(); oldcontext = MemoryContextSwitchTo(ErrorContext); + if (emit_log_hook) + emit_log_hook(edata); + /* Send to server log, if enabled */ if (edata->output_to_server) send_message_to_server_log(edata); *** a/src/include/utils/elog.h --- b/src/include/utils/elog.h *************** *** 327,332 **** typedef struct ErrorData --- 327,334 ---- int saved_errno; /* errno at entry */ } ErrorData; + typedef void (*emit_log_hook_type)(ErrorData *edata); + extern void EmitErrorReport(void); extern ErrorData *CopyErrorData(void); extern void FreeErrorData(ErrorData *edata); *************** *** 347,352 **** typedef enum --- 349,355 ---- extern int Log_error_verbosity; extern char *Log_line_prefix; extern int Log_destination; + extern emit_log_hook_type emit_log_hook; /* Log destination bitmap */ #define LOG_DESTINATION_STDERR 1
-- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers