On 11/24/2011 06:33 PM, Theo Schlossnagle wrote:
> I see the next steps being:
>  1) agreeing that a problem exists (I know one does, but I suppose
> consensus is req'd)

+1, high volume logging is also a huge problem here at Skype. Some of
the issues that immediately come to mind:

- extracting usage statistics from logs
- extracting error messages from logs
- pushing the logs to remote log server (and no, syslog is not actually
  usable for that ;)

>  2) agreeing that "hooks" are the right approach, if not propose a
> different approach. (fwiw, it's incredible common)

+1 for hook based approach. While the whole logging system could
possibly use an overhaul, this is not likely going to happen for 9.2.
Meanwhile hooks provide the flexibility to implement custom log
collectors for those who really need it.

>  3) reworking the implementation to fit in the project; I assume the
> implementation I proposed will, at best, vaguely resemble anything
> that gets integrated. It was just a PoC.
> 

I'd vote for a hook to be added to EmitErrorReport. That way all logged
messages pass through the hook, which could then decide whether to
actually write the message to server log or to discard it. As an added
bonus, the message is  broken down into pieces so there is no need to
parse the messages for severity, context, etc.

Patch attached.

regards,
Martin
*** a/src/backend/utils/error/elog.c
--- b/src/backend/utils/error/elog.c
***************
*** 136,141 **** static int	errordata_stack_depth = -1; /* index of topmost active frame */
--- 136,143 ----
  
  static int	recursion_depth = 0;	/* to detect actual recursion */
  
+ emit_log_hook_type	emit_log_hook = NULL;	/* hook for log interception */
+ 
  /* buffers for formatted timestamps that might be used by both
   * log_line_prefix and csv logs.
   */
***************
*** 1276,1281 **** EmitErrorReport(void)
--- 1278,1286 ----
  	CHECK_STACK_DEPTH();
  	oldcontext = MemoryContextSwitchTo(ErrorContext);
  
+ 	if (emit_log_hook)
+ 		emit_log_hook(edata);
+ 
  	/* Send to server log, if enabled */
  	if (edata->output_to_server)
  		send_message_to_server_log(edata);
*** a/src/include/utils/elog.h
--- b/src/include/utils/elog.h
***************
*** 327,332 **** typedef struct ErrorData
--- 327,334 ----
  	int			saved_errno;	/* errno at entry */
  } ErrorData;
  
+ typedef void (*emit_log_hook_type)(ErrorData *edata);
+ 
  extern void EmitErrorReport(void);
  extern ErrorData *CopyErrorData(void);
  extern void FreeErrorData(ErrorData *edata);
***************
*** 347,352 **** typedef enum
--- 349,355 ----
  extern int	Log_error_verbosity;
  extern char *Log_line_prefix;
  extern int	Log_destination;
+ extern emit_log_hook_type	emit_log_hook;
  
  /* Log destination bitmap */
  #define LOG_DESTINATION_STDERR	 1
-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply via email to