Re: [HACKERS] COPYable logs status

2007-06-11 Thread FAST PostgreSQL
Andrew Dunstan wrote: > > The CSVlog pipe is a separate pipe from the stderr pipe. Anything that > goes to stderr now will continue to go to stderr, wherever that is. > > I like this scheme for a couple of reasons: > . it will include the ability to tell the real end of a message > . it will let u

Re: [HACKERS] COPYable logs status

2007-06-11 Thread Tom Lane
Andrew Dunstan <[EMAIL PROTECTED]> writes: > I'll try to get a patch out for just the stderr case, which should be > back-patchable, then adjust the CSVlog patch to use it. Sounds like a plan. > I'm thinking of handling the partial lines with a small dynahash of > StringInfo buffers, which get

Re: [HACKERS] COPYable logs status

2007-06-11 Thread Andrew Dunstan
Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: The idea of one pipe per process is not really workable, because it would mean having as many pipes as backends which does not sound very good. But how about a mixed approach -- like have the all the backends share a pipe, controll

Re: [HACKERS] COPYable logs status

2007-06-09 Thread Markus Schiltknecht
Tom Lane wrote: Markus Schiltknecht <[EMAIL PROTECTED]> writes: Tom Lane wrote: We *have* a log-writing process. The problem is in getting the data to it. Remember the imessages approach I'm using for Postgres-R? It passes messages around using shared memory and signals the receiver on inco

Re: [HACKERS] COPYable logs status

2007-06-09 Thread Tom Lane
"Matthew T. O'Connor" <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> We *have* a log-writing process. The problem is in getting the data to it. > By that I assume you mean the bgwriter, I thought that was for WAL data, No, I'm talking about src/backend/postmaster/syslogger.c

Re: [HACKERS] COPYable logs status

2007-06-09 Thread Tom Lane
Markus Schiltknecht <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> We *have* a log-writing process. The problem is in getting the data to it. > Remember the imessages approach I'm using for Postgres-R? It passes > messages around using shared memory and signals the receiver on incoming > data

Re: [HACKERS] COPYable logs status

2007-06-09 Thread Markus Schiltknecht
Hi, Tom Lane wrote: We *have* a log-writing process. The problem is in getting the data to it. Remember the imessages approach I'm using for Postgres-R? It passes messages around using shared memory and signals the receiver on incoming data. It's not perfect, sure, but it's a general soluti

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Matthew T. O'Connor
Tom Lane wrote: "Matthew T. O'Connor" <[EMAIL PROTECTED]> writes: How about creating a log-writing-process? Postmaster could write to the log files directly until the log-writer is up and running, then all processes can send their log output through the log-writer. We *have* a log-writing pr

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Tom Lane
"Matthew T. O'Connor" <[EMAIL PROTECTED]> writes: > How about creating a log-writing-process? Postmaster could write to the > log files directly until the log-writer is up and running, then all > processes can send their log output through the log-writer. We *have* a log-writing process. The p

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Matthew T. O'Connor
Andrew Dunstan wrote: The situation with this patch is that I now have it in a state where I think it could be applied, but there is one blocker, namely that we do not have a way of preventing the interleaving of log messages from different backends, which leads to garbled logs. This is an exis

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Andrew Sullivan
On Fri, Jun 08, 2007 at 10:29:03AM -0400, Tom Lane wrote: > > The only solution that I can see is to invent some sort of simple > protocol for the syslogger pipe. Perhaps having a look at the current IETF syslog discussion will be helpful in that case? (I know it's not directly relevant, but ma

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Andrew Dunstan
Tom Lane wrote: Alvaro Herrera <[EMAIL PROTECTED]> writes: Martijn van Oosterhout wrote: The whole semantics of PIPEBUF should prevent garbling, as long as each write is a complete set of lines and no more than PIPEBUF bytes long. Have we determined the actual cause of the garbling?

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > The idea of one pipe per process is not really workable, because it > would mean having as many pipes as backends which does not sound very > good. But how about a mixed approach -- like have the all the backends > share a pipe, controlled by an LWLock,

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Tom Lane
Alvaro Herrera <[EMAIL PROTECTED]> writes: > Martijn van Oosterhout wrote: >> The whole semantics of PIPEBUF should prevent garbling, as long as each >> write is a complete set of lines and no more than PIPEBUF bytes long. >> Have we determined the actual cause of the garbling? > No, that's the ma

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Alvaro Herrera
Martijn van Oosterhout wrote: > On Fri, Jun 08, 2007 at 08:31:54AM -0400, Andrew Dunstan wrote: > > The situation with this patch is that I now have it in a state where I > > think it could be applied, but there is one blocker, namely that we do > > not have a way of preventing the interleaving o

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Martijn van Oosterhout
On Fri, Jun 08, 2007 at 08:31:54AM -0400, Andrew Dunstan wrote: > The situation with this patch is that I now have it in a state where I > think it could be applied, but there is one blocker, namely that we do > not have a way of preventing the interleaving of log messages from > different backe

Re: [HACKERS] COPYable logs status

2007-06-08 Thread Alvaro Herrera
Andrew Dunstan wrote: > Unfortunately, there is no solution in sight for this problem, certainly > not one which I think can be devised and implemented simply at this > stage of the cycle. The solution we'd like to use, LWLocks, is not > workable in his context. In consequence, I don't think we

[HACKERS] COPYable logs status

2007-06-08 Thread Andrew Dunstan
[summarising discussion on -patches] The situation with this patch is that I now have it in a state where I think it could be applied, but there is one blocker, namely that we do not have a way of preventing the interleaving of log messages from different backends, which leads to garbled logs