Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Stefan Kaltenbrunner
On 12/01/2010 05:32 AM, Jeff Janes wrote: On 11/28/10, Robert Haasrobertmh...@gmail.com wrote: In a close race, I don't think we should get bogged down in micro-optimization here, both because micro-optimizations may not gain much and because what works well on one platform may not do much at

[HACKERS] Suggesting a libpq addition

2010-12-05 Thread Marc Balmer
I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and does connection re-establishing if the connection fails (it can be discussed if this already to much magic, maybe remove

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Pavel Stehule
2010/12/5 Marc Balmer m...@msys.ch: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and does connection re-establishing if the connection fails (it can be discussed if

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Magnus Hagander
On Sun, Dec 5, 2010 at 10:22, Marc Balmer m...@msys.ch wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and How is that not a horrible idea, compared to using

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Heikki Linnakangas
On 05.12.2010 12:10, Magnus Hagander wrote: On Sun, Dec 5, 2010 at 10:22, Marc Balmerm...@msys.ch wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and How is that

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Marc Balmer
Am 05.12.2010 um 11:57 schrieb Heikki Linnakangas heikki.linnakan...@enterprisedb.com: On 05.12.2010 12:10, Magnus Hagander wrote: On Sun, Dec 5, 2010 at 10:22, Marc Balmerm...@msys.ch wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt,

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Magnus Hagander
On Sun, Dec 5, 2010 at 11:57, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 05.12.2010 12:10, Magnus Hagander wrote: On Sun, Dec 5, 2010 at 10:22, Marc Balmerm...@msys.ch  wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt,

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Dmitriy Igrishin
Hey hackers, Varargs-exec is useful only when programmer calls it directly. It is useless when libpq is used to create a more flexible high-level library (e.g., for C++). PQexecParams (PQexecPrepared) are good for it. What about auto reconnect. There are PQreset already and PG_CONNECTION_OK

Re: [HACKERS] SQL/MED - file_fdw

2010-12-05 Thread Andrew Dunstan
On 12/04/2010 11:11 PM, Itagaki Takahiro wrote: On Sun, Dec 5, 2010 at 07:24, Andrew Dunstanand...@dunslane.net wrote: Looking at file_parser.c, it seems to be largely taken from copy.c. Wouldn't it be better to call those functions, or refactor them so they are callable if necessary? We

Re: [HACKERS] disk caching for writing log

2010-12-05 Thread flyusa2010 fly
Thanks for your reply. Yes, i mean disk may lie to os. On Fri, Dec 3, 2010 at 12:14 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 12/03/2010 06:43 PM, Heikki Linnakangas wrote: On 03.12.2010 13:49, flyusa2010 fly wrote: When writing log, dbms should synchronously flush log to

[HACKERS] what are clog and xlog?

2010-12-05 Thread flyusa2010 fly
Hi, folks, I'm a newbie to postgres. I'm confused with xlog and clog. To my initial understanding, xlog is the periodic checkpoint log for data, while clog is for commit log (very unclear for me ...) Thanks!

[HACKERS] Using different semaphore for locking

2010-12-05 Thread flyusa2010 fly
I found that postgres uses different semaphore system call on some different operating systems. For example, I found that on linux, System V semaphore (semop etc.) is used to implement locking, while on Darwin, POSIX semaphore (sem_wait, sem_post etc.) is used. linux and Darwin support both System

Re: [HACKERS] new patch of MERGE (merge_204) a question about duplicated ctid

2010-12-05 Thread David Fetter
On Sat, Dec 04, 2010 at 09:27:52PM +0800, Boxuan Zhai wrote: Dear Greg, I have updated the MERGE patch for two main problems. Please attach the actual patch :) Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype:

Re: [HACKERS] what are clog and xlog?

2010-12-05 Thread Kevin Grittner
flyusa2010 fly wrote: clog is for commit log (very unclear for me ...) It is an important part of tracking tuple visibility. You might want to start with this page: http://wiki.postgresql.org/wiki/Hint_Bits -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Andrew Dunstan
On 12/04/2010 07:12 PM, Robert Haas wrote: I wouldn't necessarily be opposed to official topic branches at some point in the future, but I think it's premature to speculate about whether it'd be useful here. I'd need a lot of convincing if it imposed an extra burden on people like Tom.

[HACKERS] serializable read only deferrable

2010-12-05 Thread Kevin Grittner
I'm reviving the discussion on the subject topic because I just had an epiphany which makes it seem simple to implement. The concept of this is that if you start a SERIALIZABLE READ ONLY transaction in an SSI environment when certain conditions are true, it doesn't need to acquire predicate locks

Re: [HACKERS] Using different semaphore for locking

2010-12-05 Thread Tom Lane
flyusa2010 fly flyusa2...@gmail.com writes: I found that postgres uses different semaphore system call on some different operating systems. For example, I found that on linux, System V semaphore (semop etc.) is used to implement locking, while on Darwin, POSIX semaphore (sem_wait, sem_post

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 12/04/2010 07:12 PM, Robert Haas wrote: I wouldn't necessarily be opposed to official topic branches at some point in the future, but I think it's premature to speculate about whether it'd be useful here. I'd need a lot of convincing if it

Re: [HACKERS] serializable read only deferrable

2010-12-05 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes: I'm reviving the discussion on the subject topic because I just had an epiphany which makes it seem simple to implement. The concept of this is that if you start a SERIALIZABLE READ ONLY transaction in an SSI environment when certain

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Andrew Dunstan
On 12/05/2010 12:10 PM, Tom Lane wrote: Andrew Dunstanand...@dunslane.net writes: On 12/04/2010 07:12 PM, Robert Haas wrote: I wouldn't necessarily be opposed to official topic branches at some point in the future, but I think it's premature to speculate about whether it'd be useful

[HACKERS] allow COPY routines to read arbitrary numbers of fields

2010-12-05 Thread Andrew Dunstan
Attached is a patch that allows CopyReadAttibutesText() and CopyReadAttributesCSV() to read arbitrary numbers of attributes. Underflowing attributes are recorded as null, and space is made for overflowing attributes on a line. This patch doesn't result in any user-visible behavior. The

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Greg Smith
Joachim Wieland wrote: Regarding snapshot cloning and dump consistency, I brought this up already several months ago and asked if the feature is considered useful even without snapshot cloning. In addition, Joachim submitted a synchronized snapshot patch that looks to me like it slipped

Re: [HACKERS] libpq changes for synchronous replication

2010-12-05 Thread Greg Smith
The one time this year top-posting seems appropriate...this patch seems stalled waiting for some sort of response to the concerns Alvaro raised here. Alvaro Herrera wrote: Excerpts from Fujii Masao's message of jue nov 25 10:47:12 -0300 2010: The attached patch s/CopyXLog/CopyBoth/g and

Re: [HACKERS] serializable read only deferrable

2010-12-05 Thread Kevin Grittner
Tom Lane wrote: Kevin Grittner writes: I'm reviving the discussion on the subject topic because I just had an epiphany which makes it seem simple to implement. The concept of this is that if you start a SERIALIZABLE READ ONLY transaction in an SSI environment when certain conditions are

[HACKERS] Re: new patch of MERGE (merge_204) a question about duplicated ctid

2010-12-05 Thread Greg Smith
Boxuan Zhai wrote: I have updated the MERGE patch for two main problems. The patch inside the .tar.gz file you attached isn't right; that extracts to a tiny file of junk characters. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services and

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes: In addition, Joachim submitted a synchronized snapshot patch that looks to me like it slipped through the cracks without being fully explored. ... The way I read that thread, there were two objections: 1) This mechanism isn't general enough for all

Re: [HACKERS] wCTE behaviour

2010-12-05 Thread Greg Smith
Marko Tiikkaja wrote: This is almost exactly the patch from 2010-02 without CommandCounterIncrement()s. It's still a bit rough around the edges and needs some more comments, but I'm posting it here anyway. This patch passes all regression tests, but feel free to try to break it, there are

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Josh Berkus
* no coordination of restarts/configuration changes between the cluster and the pooler * you have two pieces of config files to configure your pooling settings (having all that available say in a catalog in pg would be awesome) * you lose all of the advanced authentication features of pg

Re: [HACKERS] Patch to add a primary key using an existing index

2010-12-05 Thread Peter Eisentraut
On fre, 2010-12-03 at 15:27 -0500, Robert Haas wrote: On Fri, Dec 3, 2010 at 2:56 PM, r t pg...@xzilla.net wrote: What exactly was the objection to the following -- ALTER TABLE table_name ADD PRIMARY KEY (column_list) USING index_name; Is the objection that you might have been trying to

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 11:59 AM, Josh Berkus j...@agliodbs.com wrote: * no coordination of restarts/configuration changes between the cluster and the pooler * you have two pieces of config files to configure your pooling settings (having all that available say in a catalog in pg would be

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 12:45 PM, Rob Wultsch wult...@gmail.com wrote: One thing I would suggest that the PG community keeps in mind while talking about built in connection process caching, is that it is very nice feature for memory leaks caused by a connection to not exist for and continue

Re: [HACKERS] V3: Idle in transaction cancellation

2010-12-05 Thread Kevin Grittner
Kevin Grittner wrote: Andres Freund wrote: On Thursday 02 December 2010 00:48:53 Kevin Grittner wrote: Is there any provision for one backend to cause a *different* backend which is idle in a transaction to terminate cleanly when it attempts to process its next statement? You might want

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-05 Thread Greg Smith
Jeff Janes wrote: I've attached a tiny patch to apply over yours, to deal with this and with the case where no files are synced. Thanks for that. That obvious error eluded me because in most of the patch update testing I was doing (on ext3), the longest sync was always about the same

Re: [HACKERS] Spread checkpoint sync

2010-12-05 Thread Greg Smith
Heikki Linnakangas wrote: If you fsync() a file with one dirty page in it, it's going to return very quickly, but a 1GB file will take a while. That could be problematic if you have a thousand small files and a couple of big ones, as you would want to reserve more time for the big ones. I'm

Re: [HACKERS] [PATCH] Revert default wal_sync_method to fdatasync on Linux 2.6.33+

2010-12-05 Thread Greg Smith
Josh Berkus wrote: I modified test_fsync in two ways to run this; first, to make it support O_DIRECT, and second to make it run in the *current* directory. Patch please? I agree with the latter change; what test_fsync does is surprising. I suggested a while ago that we refactor test_fsync

Re: [HACKERS] Spread checkpoint sync

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 2:53 PM, Greg Smith g...@2ndquadrant.com wrote: Heikki Linnakangas wrote: If you fsync() a file with one dirty page in it, it's going to return very quickly, but a 1GB file will take a while. That could be problematic if you have a thousand small files and a couple of

[HACKERS] CommitFest 2010-11: Status report at 2/3 of scheduled time

2010-12-05 Thread Greg Smith
After an unfortunate two weeks where I was lost in Nyquil-land and not paying as much attention as I should have, I just finished a pass checking over all the open 26 CommitFest items. Just under 50% of the patches that were open at the start of the CF are still hanging around, with 6 in

Re: [HACKERS] pg_execute_from_file review

2010-12-05 Thread Tom Lane
Itagaki Takahiro itagaki.takah...@gmail.com writes: On Fri, Dec 3, 2010 at 18:02, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: My understanding is that the variadic form shadows the other one in a way that it's now impossible to call it from SQL level. That's the reason why I did the (text,

Re: [HACKERS] Spread checkpoint sync

2010-12-05 Thread Greg Smith
Rob Wultsch wrote: Forgive me, but is all of this a step on the slippery slope to direct io? And is this a bad thing I don't really think so. There's an important difference in my head between direct I/O, where the kernel is told write this immediately!, and what I'm trying to achive. I

Re: [HACKERS] pg_execute_from_file review

2010-12-05 Thread Itagaki Takahiro
On Mon, Dec 6, 2010 at 08:01, Tom Lane t...@sss.pgh.pa.us wrote: Why is there a variadic replace() in this patch at all? It seems just about entirely unrelated to the stated purpose of the patch, as well as being of dubious usefulness. As I wrote in the previous mail, the most important part

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 12:41 PM, Andrew Dunstan and...@dunslane.net wrote: Well, ISTM that amounts to not having official topic branches :-) I agree that this is supposed to be one of git's strengths (or more exactly a strength of distributed SCM's generally).  I don't really see any great

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 1:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: I'm wondering if we should reconsider the pass-it-through-the-client approach, because if we could make that work it would be more general and it wouldn't need any special privileges.  The trick seems to be to apply sufficient

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 3:17 PM, Rob Wultsch wult...@gmail.com wrote: On Sun, Dec 5, 2010 at 12:45 PM, Rob Wultsch wult...@gmail.com wrote: One thing I would suggest that the PG community keeps in mind while talking about built in connection process caching, is that it is very nice feature for

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 2:45 PM, Rob Wultsch wult...@gmail.com wrote: I think you have read a bit more into what I have said than is correct.  MySQL can deal with thousands of users and separate schemas on commodity hardware. There are many design decisions (some questionable) that have made

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Andrew Dunstan
On 12/05/2010 08:55 PM, Robert Haas wrote: On Sun, Dec 5, 2010 at 1:28 PM, Tom Lanet...@sss.pgh.pa.us wrote: I'm wondering if we should reconsider the pass-it-through-the-client approach, because if we could make that work it would be more general and it wouldn't need any special privileges.

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sat, Dec 4, 2010 at 8:04 PM, Jeff Janes jeff.ja...@gmail.com wrote: But who would be doing the passing?  For the postmaster to be doing that would probably go against the minimalist design.  It would have to keep track of which backend is available, and which db and user it is primed for.  

Re: [HACKERS] pg_execute_from_file review

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 6:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: Itagaki Takahiro itagaki.takah...@gmail.com writes: On Fri, Dec 3, 2010 at 18:02, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: My understanding is that the variadic form shadows the other one in a way that it's now

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan and...@dunslane.net wrote: Why not just say give me the snapshot currently held by process ? And please, not temp files if possible. As far as I'm aware, the full snapshot doesn't normally exist in shared memory, hence the need for publication

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 6:59 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Dec 5, 2010 at 2:45 PM, Rob Wultsch wult...@gmail.com wrote: I think you have read a bit more into what I have said than is correct.  MySQL can deal with thousands of users and separate schemas on commodity

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 9:35 PM, Rob Wultsch wult...@gmail.com wrote: On Sun, Dec 5, 2010 at 6:59 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Dec 5, 2010 at 2:45 PM, Rob Wultsch wult...@gmail.com wrote: I think you have read a bit more into what I have said than is correct.  MySQL can

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Andrew Chernow
On 12/5/2010 4:22 AM, Marc Balmer wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and does connection re-establishing if the connection fails (it can be discussed if

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Andrew Chernow
A varargs version of PQexecParams() would be handy, though. Imagine being able to do: PQexecVParams(SELECT * FROM mytable WHERE foo = $1 AND bar = $2, foovar, barvar); instead of constructing an array for the variables. http://libpqtypes.esilo.com/man3/PQexecf.html -- Andrew Chernow

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Joachim Wieland
On Sun, Dec 5, 2010 at 9:27 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan and...@dunslane.net wrote: Why not just say give me the snapshot currently held by process ? And please, not temp files if possible. As far as I'm aware, the full

Re: [HACKERS] Review: Extensions Patch

2010-12-05 Thread David E. Wheeler
On Dec 4, 2010, at 6:14 AM, Dimitri Fontaine wrote: Hi, Thanks for the review, that's quite one! :) I'm not sure to follow you all along, it seems like the reading is try it first then understand and comment again, so sometimes I'm not sure if you're saying that docs are missing the

Re: [HACKERS] Re: Proposed Windows-specific change: Enable crash dumps (like core files)

2010-12-05 Thread Craig Ringer
On 22/11/2010 7:37 PM, Magnus Hagander wrote: Finally getting to looking at this one - sorry about the very long delay. Ditto, I'm afraid. I agree with Heikki's earlier comment that it's better to have this included in the backend - but that's obviously not going to happen for

Re: [HACKERS] libpq changes for synchronous replication

2010-12-05 Thread Fujii Masao
On Mon, Dec 6, 2010 at 3:07 AM, Greg Smith g...@2ndquadrant.com wrote: The one time this year top-posting seems appropriate...this patch seems stalled waiting for some sort of response to the concerns Alvaro raised here. Sorry for the delay. I didn't have the time. I gave this a look. It

Re: [HACKERS] Re: Proposed Windows-specific change: Enable crash dumps (like core files)

2010-12-05 Thread Craig Ringer
On 6/12/2010 12:57 PM, Craig Ringer wrote: On 22/11/2010 7:37 PM, Magnus Hagander wrote: Finally getting to looking at this one - sorry about the very long delay. Ditto, I'm afraid. Oh, I forgot to mention in the patch email: I'm not sure I've taken the right approach in terms of how I've

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Koichi Suzuki
Thank you Joachim; Yes, and the current patch requires the original (publisher) transaction is alive to prevent RecentXmin updated. I hope this restriction is acceptable if publishing/subscribing is provided via functions, not statements. Cheers; -- Koichi Suzuki 2010/12/6 Joachim

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Dmitriy Igrishin
Hey Andrew, No, thanks. :-) And I don't think that libpq should follows it (libpqtypes). 2010/12/6 Andrew Chernow a...@esilo.com A varargs version of PQexecParams() would be handy, though. Imagine being able to do: PQexecVParams(SELECT * FROM mytable WHERE foo = $1 AND bar = $2, foovar,

Re: [HACKERS] Timeout and wait-forever in sync rep

2010-12-05 Thread Fujii Masao
On Fri, Oct 15, 2010 at 9:41 PM, Fujii Masao masao.fu...@gmail.com wrote: The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait -forever' (i.e., you set allow_standalone_master to false), the master should detect the standby crash as soon as possible by using the timeout. For

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Heikki Linnakangas
On 06.12.2010 02:55, Robert Haas wrote: On Sun, Dec 5, 2010 at 1:28 PM, Tom Lanet...@sss.pgh.pa.us wrote: I'm wondering if we should reconsider the pass-it-through-the-client approach, because if we could make that work it would be more general and it wouldn't need any special privileges. The

Re: [HACKERS] Timeout and wait-forever in sync rep

2010-12-05 Thread Heikki Linnakangas
On 06.12.2010 07:42, Fujii Masao wrote: On Fri, Oct 15, 2010 at 9:41 PM, Fujii Masaomasao.fu...@gmail.com wrote: The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait -forever' (i.e., you set allow_standalone_master to false), the master should detect the standby crash as soon

[HACKERS] Timeout for asynchronous replication Re: Timeout and wait-forever in sync rep

2010-12-05 Thread Fujii Masao
On Mon, Dec 6, 2010 at 3:42 PM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Oct 15, 2010 at 9:41 PM, Fujii Masao masao.fu...@gmail.com wrote: The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait -forever' (i.e., you set allow_standalone_master to false), the master