Re: [HACKERS] Sketch of a Hook into the Logging Collector

2012-12-10 Thread Daniel Farina
On Sat, Dec 8, 2012 at 10:40 AM, Daniel Farina dan...@heroku.com wrote:
 Hello all,

 I am approaching this from the angle of increasing power by exposing
 the log collector (syslogger) pipe protocol.

I just spotted a better, already-committed patch.  Thanks to Hannu for
pointing it out:

https://commitfest.postgresql.org/action/patch_view?id=717

I'll retract this patch, unless someone finds it interesting for some reason.

--
fdr


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Sketch of a Hook into the Logging Collector

2012-12-08 Thread Daniel Farina
Hello all,

I have some needs that seem to support changing Postgres slightly to
give user programs a lot more power over how to process logging output
that neither the log collector nor the syslog output can well-satisfy
as-is.

I am approaching this from the angle of increasing power by exposing
the log collector (syslogger) pipe protocol.

Included is a patch whose general aesthetic is to touch as little of
the logging code as possible while achieving maximum power in a
contrib and stand-alone sample program.  However, a more satisfying
treatment that may be slightly more invasive to the logging system is
very welcome.

The general idea here is that the logging collector pipe protocol is a
pretty reasonable one for non-Postgres programs to consume as-is
pretty easily.  The proof of concept consists of three things:

* A hook into the log collector.  It is small but a little weird
  compared to the other hooks in the system.

* A contrib, pg_logcollectdup.  This contrib lets one forward logs to
  a named pipe specified in postgresql.conf.

* A stand-alone program, pg_logcollect_sample, that renders protocol
  traffic newline separated and with its binary fields converted to
  readable text.

The patch can also be seen from https://github.com/fdr/postgres.git in
the branch 'logdup'.  I do rebase this at-will for the time being.

I have a few detailed dissatisfactions with this approach, but I'd
rather hear your own dissatisfactions.  I also don't like the name.

A demo of configuring all of the above and seeing some output follows.
I can paste this in one go on my machine.  It creates /tmp/pg-logdup
and /tmp/pgdata-logdup.  All spawned processes can be listed
afterwards with 'jobs'.

# Install postgres and pg_logcollectdup contrib
./configure --prefix=/tmp/pg-logdup
make -sj16 install
cd contrib/pg_logcollectdup/
make -s install
/tmp/pg-logdup/bin/initdb -D /tmp/pgdata-logdup
mkfifo /tmp/pgdata-logdup/log-pipe

# Configure postgresql.conf

echo logging_collector = on  \
 /tmp/pgdata-logdup/postgresql.conf

echo shared_preload_libraries = 'pg_logcollectdup'  \
/tmp/pgdata-logdup/postgresql.conf

echo logcollectdup.destination = '/tmp/pgdata-logdup/log-pipe'  \
/tmp/pgdata-logdup/postgresql.conf

# Set up destination pipe
mkfifo /tmp/pgdata-logdup/log-pipe

# Build sample pipe formatter
make pg_logcollect_sample

# Run it in the background
./pg_logcollect_sample\
 /tmp/pgdata-logdup/log-pipe \
 /tmp/pgdata-logdup/duplogs.txt 

# Run Postgres with a non-default port for convenience
/tmp/pg-logdup/bin/postgres -D /tmp/pgdata-logdup --port=2345 

# Prevent race against file creation, then look at the logs
sleep 1
cat /tmp/pgdata-logdup/duplogs.txt

--
fdr


log-collector-extension-v1.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Sketch of a Hook into the Logging Collector

2012-12-08 Thread Daniel Farina
On Sat, Dec 8, 2012 at 10:40 AM, Daniel Farina dan...@heroku.com wrote:
 * A contrib, pg_logcollectdup.  This contrib lets one forward logs to
   a named pipe specified in postgresql.conf.

I have revised this part in the attached patch.  It's some error
handling in a case of user error, and the previous demo script and
narrative precepts are still the same.

--
fdr


log-collector-extension-v2.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers