Re: [PATCHES] WIP CSV logs
Andrew Dunstan wrote: > > Further update attached. Includes some code cleanup, and now sets up a > session-id for all processes, including postmaster, syslogger etc, so we > can make a primary key on the log table of (session_id, line_number). > > TODOs: fix on Windows, docs. I'm not going to be able to look at this before Monday I'm afraid. Not sure if Magnus has more time before then. Regards, Dave. ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PATCHES] WIP CSV logs
Andrew Dunstan wrote:
Andrew Dunstan wrote:
Here is a WIP patch of the CSV logs work brought up to date with CVS
HEAD. One large change I made was to multiplex the selects on the
pipes - previously it waited on one then the other - this seems
almost to defeat the purpose of using select() :-)
It seems to work well on Unix - I will test later on Windows, which
I'm slightly worried about.
I also want to get an id for non-session processes. I think this can
just be start-time+pid, just like for session processes, but we'll
need to stash it somewhere (just for those cases). If we do that we
will be able to set a primary key on the log table when we read the
data in, which Greg Smith was worried about.
I hope to get this polished off in the next 15 hours or so - after
that I'm away for 12 days.
Here is a slightly updated version. It compiles on Windows, but it
doesn't work - the CSV log file gets created but doesn't get any
content. Dave, Magnus - can you see what I've done wrong? The strange
thing is that I tried to do exactly the same thing for CSV as for
stderr, and the stderr file gets content just fine.
Further update attached. Includes some code cleanup, and now sets up a
session-id for all processes, including postmaster, syslogger etc, so we
can make a primary key on the log table of (session_id, line_number).
TODOs: fix on Windows, docs.
cheers
andrew
Index: src/backend/postmaster/autovacuum.c
===
RCS file: /cvsroot/pgsql/src/backend/postmaster/autovacuum.c,v
retrieving revision 1.51
diff -c -r1.51 autovacuum.c
*** src/backend/postmaster/autovacuum.c 25 Jun 2007 16:09:03 - 1.51
--- src/backend/postmaster/autovacuum.c 29 Jun 2007 14:33:40 -
***
*** 386,391
--- 386,393
/* reset MyProcPid */
MyProcPid = getpid();
+ MyStartTime = time(NULL); /* get start time for logger session id */
+
/* Identify myself via ps */
init_ps_display("autovacuum launcher process", "", "", "");
***
*** 1399,1404
--- 1401,1408
/* reset MyProcPid */
MyProcPid = getpid();
+ MyStartTime = time(NULL); /* get start time for logger session id */
+
/* Identify myself via ps */
init_ps_display("autovacuum worker process", "", "", "");
Index: src/backend/postmaster/pgarch.c
===
RCS file: /cvsroot/pgsql/src/backend/postmaster/pgarch.c,v
retrieving revision 1.29
diff -c -r1.29 pgarch.c
*** src/backend/postmaster/pgarch.c 10 Feb 2007 14:58:54 - 1.29
--- src/backend/postmaster/pgarch.c 29 Jun 2007 14:33:40 -
***
*** 223,228
--- 223,230
MyProcPid = getpid(); /* reset MyProcPid */
+ MyStartTime = time(NULL); /* get start time for logger session id */
+
/*
* If possible, make this process a group leader, so that the postmaster
* can signal any child processes too.
Index: src/backend/postmaster/pgstat.c
===
RCS file: /cvsroot/pgsql/src/backend/postmaster/pgstat.c,v
retrieving revision 1.160
diff -c -r1.160 pgstat.c
*** src/backend/postmaster/pgstat.c 28 Jun 2007 00:02:38 - 1.160
--- src/backend/postmaster/pgstat.c 29 Jun 2007 14:33:42 -
***
*** 2163,2168
--- 2163,2170
MyProcPid = getpid(); /* reset MyProcPid */
+ MyStartTime = time(NULL); /* get start time for logger session id */
+
/*
* If possible, make this process a group leader, so that the postmaster
* can signal any child processes too. (pgstat probably never has
Index: src/backend/postmaster/postmaster.c
===
RCS file: /cvsroot/pgsql/src/backend/postmaster/postmaster.c,v
retrieving revision 1.528
diff -c -r1.528 postmaster.c
*** src/backend/postmaster/postmaster.c 25 Jun 2007 16:09:03 - 1.528
--- src/backend/postmaster/postmaster.c 29 Jun 2007 14:33:44 -
***
*** 336,343
--- 336,345
HANDLE PostmasterHandle;
HANDLE initial_signal_pipe;
HANDLE syslogPipe[2];
+ HANDLE csvlogPipe[2];
#else
int syslogPipe[2];
+ int csvlogPipe[2];
#endif
char my_exec_path[MAXPGPATH];
char pkglib_path[MAXPGPATH];
***
*** 381,386
--- 383,390
MyProcPid = PostmasterPid = getpid();
+ MyStartTime = time(NULL); /* get start time for logger session id */
+
IsPostmasterEnvironment = true;
/*
***
*** 1225,1231
}
/* If we have lost the system logger, try to start a new one */
! if (SysLoggerPID == 0 && Redirect_stderr)
SysLoggerPID = SysLogger_Start();
/*
--- 1229,1236
}
/* If we have lost the system logger, try to start a new one */
! if ( SysLoggerPID == 0 &&
! (Redirect_stderr || (Log_destination & LOG_DESTINATION_CSVLOG) ) )
Re: [PATCHES] WIP CSV logs
Andrew Dunstan wrote:
Here is a WIP patch of the CSV logs work brought up to date with CVS
HEAD. One large change I made was to multiplex the selects on the
pipes - previously it waited on one then the other - this seems almost
to defeat the purpose of using select() :-)
It seems to work well on Unix - I will test later on Windows, which
I'm slightly worried about.
I also want to get an id for non-session processes. I think this can
just be start-time+pid, just like for session processes, but we'll
need to stash it somewhere (just for those cases). If we do that we
will be able to set a primary key on the log table when we read the
data in, which Greg Smith was worried about.
I hope to get this polished off in the next 15 hours or so - after
that I'm away for 12 days.
Here is a slightly updated version. It compiles on Windows, but it
doesn't work - the CSV log file gets created but doesn't get any
content. Dave, Magnus - can you see what I've done wrong? The strange
thing is that I tried to do exactly the same thing for CSV as for
stderr, and the stderr file gets content just fine.
cheers
andrew
Index: src/backend/postmaster/postmaster.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/postmaster/postmaster.c,v
retrieving revision 1.528
diff -c -r1.528 postmaster.c
*** src/backend/postmaster/postmaster.c 25 Jun 2007 16:09:03 - 1.528
--- src/backend/postmaster/postmaster.c 29 Jun 2007 01:42:29 -
***
*** 336,343
--- 336,345
HANDLE PostmasterHandle;
HANDLE initial_signal_pipe;
HANDLE syslogPipe[2];
+ HANDLE csvlogPipe[2];
#else
int syslogPipe[2];
+ int csvlogPipe[2];
#endif
char my_exec_path[MAXPGPATH];
char pkglib_path[MAXPGPATH];
***
*** 1225,1231
}
/* If we have lost the system logger, try to start a new one */
! if (SysLoggerPID == 0 && Redirect_stderr)
SysLoggerPID = SysLogger_Start();
/*
--- 1227,1234
}
/* If we have lost the system logger, try to start a new one */
! if ( SysLoggerPID == 0 &&
! (Redirect_stderr || (Log_destination & LOG_DESTINATION_CSVLOG) ) )
SysLoggerPID = SysLogger_Start();
/*
***
*** 1775,1784
--- 1778,1795
if (syslogPipe[0] >= 0)
close(syslogPipe[0]);
syslogPipe[0] = -1;
+
+ if (csvlogPipe[0] >= 0)
+ close(csvlogPipe[0]);
+ csvlogPipe[0] = -1;
#else
if (syslogPipe[0])
CloseHandle(syslogPipe[0]);
syslogPipe[0] = 0;
+
+ if (csvlogPipe[0])
+ CloseHandle(csvlogPipe[0]);
+ csvlogPipe[0] = 0;
#endif
}
}
***
*** 3960,3965
--- 3971,3977
#endif
memcpy(¶m->syslogPipe, &syslogPipe, sizeof(syslogPipe));
+ memcpy(¶m->csvlogPipe, &csvlogPipe, sizeof(csvlogPipe));
strlcpy(param->my_exec_path, my_exec_path, MAXPGPATH);
***
*** 4161,4166
--- 4173,4179
#endif
memcpy(&syslogPipe, ¶m->syslogPipe, sizeof(syslogPipe));
+ memcpy(&csvlogPipe, ¶m->csvlogPipe, sizeof(csvlogPipe));
strlcpy(my_exec_path, param->my_exec_path, MAXPGPATH);
Index: src/backend/postmaster/syslogger.c
===
RCS file: /projects/cvsroot/pgsql/src/backend/postmaster/syslogger.c,v
retrieving revision 1.32
diff -c -r1.32 syslogger.c
*** src/backend/postmaster/syslogger.c 14 Jun 2007 01:48:51 - 1.32
--- src/backend/postmaster/syslogger.c 29 Jun 2007 01:42:30 -
***
*** 85,91
static pg_time_t next_rotation_time;
static bool redirection_done = false;
static bool pipe_eof_seen = false;
! static FILE *syslogFile = NULL;
static char *last_file_name = NULL;
/*
--- 85,91
static pg_time_t next_rotation_time;
static bool redirection_done = false;
static bool pipe_eof_seen = false;
! static FILE *syslogFile = NULL, *csvlogFile = NULL;
static char *last_file_name = NULL;
/*
***
*** 103,120
} save_buffer;
#define CHUNK_SLOTS 20
! static save_buffer saved_chunks[CHUNK_SLOTS];
/* These must be exported for EXEC_BACKEND case ... annoying */
#ifndef WIN32
int syslogPipe[2] = {-1, -1};
#else
HANDLE syslogPipe[2] = {0, 0};
#endif
#ifdef WIN32
! static HANDLE threadHandle = 0;
! static CRITICAL_SECTION sysfileSection;
#endif
/*
--- 103,123
} save_buffer;
#define CHUNK_SLOTS 20
! static save_buffer stderr_saved_chunks[CHUNK_SLOTS];
! static save_buffer csvlog_saved_chunks[CHUNK_SLOTS];
/* These must be exported for EXEC_BACKEND case ... annoying */
#ifndef WIN32
int syslogPipe[2] = {-1, -1};
+ int csvlogPipe[2] = {-1, -1};
#else
HANDLE syslogPipe[2] = {0, 0};
+ HANDLE csvlogPipe[2] = {0, 0};
#endif
#ifdef WIN32
! static HANDLE stderrThreadHandle = 0, csvThreadHandle = 0;
! static CRITICAL_SECTION sysfileSection, csvfileSection;
#endif
/*
*
Re: [PATCHES] WIP csv logs
Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: I wonder, if we were to use an LWLock to protect writing to the stderr pipe, would it be too contentious? Sorry, that's a nonstarter. 1. Not all our processes are connected to shared memory. Even the ones that are don't necessarily have PGPROCs. 2. If somebody crashes while holding the lock, the entire system including the postmaster soon freezes up. 3. Having the postmaster take LWLocks is verboten anyway, mainly because of problem #2. Ugh. Well, until this problem is solved I'm not prepared to commit a patch that purports to provide machine readable logs. The only non-locking solution I can think of is to have one set of pipes per backend rather than one set per cluster. That could get pretty ugly, though, and have all sorts of problems with portability. I'm starting to wonder if the CSVlog patch needs to be put on hold at least for 8.4. If we do, I think I should at least commit the piece that prevents rotation in the middle of a line, and possible even backpatch it. cheers andrew ---(end of broadcast)--- TIP 7: You can help support the PostgreSQL project by donating at http://www.postgresql.org/about/donate
Re: [PATCHES] WIP csv logs
Alvaro Herrera <[EMAIL PROTECTED]> writes: > I wonder, if we were to use an LWLock to protect writing to the stderr > pipe, would it be too contentious? Sorry, that's a nonstarter. 1. Not all our processes are connected to shared memory. Even the ones that are don't necessarily have PGPROCs. 2. If somebody crashes while holding the lock, the entire system including the postmaster soon freezes up. 3. Having the postmaster take LWLocks is verboten anyway, mainly because of problem #2. regards, tom lane ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
Re: [PATCHES] WIP csv logs
Alvaro Herrera wrote: Andrew Dunstan wrote: Attached is my WIP version of the CSV logs patch. It does not include docs changes. It fixes the CSV thinko I just posted about, and also implements the "safe to rotate" scheme I suggested yesterday, at quite a modest cost. As Tom rightly points out, that does not protect us against interleaving log lines from different backends, which is an open problem. I wonder, if we were to use an LWLock to protect writing to the stderr pipe, would it be too contentious? That was the first thing I wanted to try. I'll code it up and then people can test to see if we get a significant performance hit. cheers andrew ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [PATCHES] WIP csv logs
Andrew Dunstan wrote: > > Attached is my WIP version of the CSV logs patch. It does not include > docs changes. > > It fixes the CSV thinko I just posted about, and also implements the > "safe to rotate" scheme I suggested yesterday, at quite a modest cost. > As Tom rightly points out, that does not protect us against interleaving > log lines from different backends, which is an open problem. I wonder, if we were to use an LWLock to protect writing to the stderr pipe, would it be too contentious? -- Alvaro Herrera Valdivia, Chile ICBM: S 39º 49' 18.1", W 73º 13' 56.4" "El destino baraja y nosotros jugamos" (A. Schopenhauer) ---(end of broadcast)--- TIP 3: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faq
