[HACKERS] Timeout for asynchronous replication Re: Timeout and wait-forever in sync rep

2010-12-05 Thread Fujii Masao
On Mon, Dec 6, 2010 at 3:42 PM, Fujii Masao wrote: > On Fri, Oct 15, 2010 at 9:41 PM, Fujii Masao wrote: >> The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait >> -forever' (i.e., you set allow_standalone_master to false), the master >> should detect the standby crash as soon a

Re: [HACKERS] Timeout and wait-forever in sync rep

2010-12-05 Thread Heikki Linnakangas
On 06.12.2010 07:42, Fujii Masao wrote: On Fri, Oct 15, 2010 at 9:41 PM, Fujii Masao wrote: The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait -forever' (i.e., you set allow_standalone_master to false), the master should detect the standby crash as soon as possible by using

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Heikki Linnakangas
On 06.12.2010 02:55, Robert Haas wrote: On Sun, Dec 5, 2010 at 1:28 PM, Tom Lane wrote: I'm wondering if we should reconsider the pass-it-through-the-client approach, because if we could make that work it would be more general and it wouldn't need any special privileges. The trick seems to be

Re: [HACKERS] Timeout and wait-forever in sync rep

2010-12-05 Thread Fujii Masao
On Fri, Oct 15, 2010 at 9:41 PM, Fujii Masao wrote: > The timeout doesn't oppose to 'wait-forever'. Even if you choose 'wait > -forever' (i.e., you set allow_standalone_master to false), the master > should detect the standby crash as soon as possible by using the > timeout. For example, imagine t

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Dmitriy Igrishin
Hey Andrew, No, thanks. :-) And I don't think that libpq should follows it (libpqtypes). 2010/12/6 Andrew Chernow > > A varargs version of PQexecParams() would be handy, though. Imagine being >> able >> to do: >> >> PQexecVParams("SELECT * FROM mytable WHERE foo = $1 AND bar = $2", foovar, >>

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Koichi Suzuki
Thank you Joachim; Yes, and the current patch requires the original (publisher) transaction is alive to prevent RecentXmin updated. I hope this restriction is acceptable if publishing/subscribing is provided via functions, not statements. Cheers; -- Koichi Suzuki 2010/12/6 Joachim Wie

Re: [HACKERS] Re: Proposed Windows-specific change: Enable crash dumps (like core files)

2010-12-05 Thread Craig Ringer
On 6/12/2010 12:57 PM, Craig Ringer wrote: On 22/11/2010 7:37 PM, Magnus Hagander wrote: Finally getting to looking at this one - sorry about the very long delay. Ditto, I'm afraid. Oh, I forgot to mention in the patch email: I'm not sure I've taken the right approach in terms of how I've

Re: [HACKERS] libpq changes for synchronous replication

2010-12-05 Thread Fujii Masao
On Mon, Dec 6, 2010 at 3:07 AM, Greg Smith wrote: > The one time this year top-posting seems appropriate...this patch seems > stalled waiting for some sort of response to the concerns Alvaro raised > here. Sorry for the delay. I didn't have the time. > I gave this a look. It seems good, but I'm

Re: [HACKERS] Re: Proposed Windows-specific change: Enable crash dumps (like core files)

2010-12-05 Thread Craig Ringer
On 22/11/2010 7:37 PM, Magnus Hagander wrote: Finally getting to looking at this one - sorry about the very long delay. Ditto, I'm afraid. I agree with Heikki's earlier comment that it's better to have this included in the backend - but that's obviously not going to happen for already-releas

Re: [HACKERS] Review: Extensions Patch

2010-12-05 Thread David E. Wheeler
On Dec 4, 2010, at 6:14 AM, Dimitri Fontaine wrote: > Hi, > > Thanks for the review, that's quite one! :) > > I'm not sure to follow you all along, it seems like the reading is "try > it first then understand and comment again", so sometimes I'm not sure > if you're saying that docs are missing

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Joachim Wieland
On Sun, Dec 5, 2010 at 9:27 PM, Robert Haas wrote: > On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan wrote: >> Why not just say give me the snapshot currently held by process ? >> >> And please, not temp files if possible. > > As far as I'm aware, the full snapshot doesn't normally exist in >

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Andrew Chernow
A varargs version of PQexecParams() would be handy, though. Imagine being able to do: PQexecVParams("SELECT * FROM mytable WHERE foo = $1 AND bar = $2", foovar, barvar); instead of constructing an array for the variables. http://libpqtypes.esilo.com/man3/PQexecf.html -- Andrew Chernow eSi

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Andrew Chernow
On 12/5/2010 4:22 AM, Marc Balmer wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and does connection re-establishing if the connection fails (it can be discussed if th

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 9:35 PM, Rob Wultsch wrote: > On Sun, Dec 5, 2010 at 6:59 PM, Robert Haas wrote: >> On Sun, Dec 5, 2010 at 2:45 PM, Rob Wultsch wrote: >>> I think you have read a bit more into what I have said than is >>> correct.  MySQL can deal with thousands of users and separate schem

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 6:59 PM, Robert Haas wrote: > On Sun, Dec 5, 2010 at 2:45 PM, Rob Wultsch wrote: >> I think you have read a bit more into what I have said than is >> correct.  MySQL can deal with thousands of users and separate schemas >> on commodity hardware. There are many design decisi

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan wrote: > Why not just say give me the snapshot currently held by process ? > > And please, not temp files if possible. As far as I'm aware, the full snapshot doesn't normally exist in shared memory, hence the need for publication of some sort. W

Re: [HACKERS] pg_execute_from_file review

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 6:01 PM, Tom Lane wrote: > Itagaki Takahiro writes: >> On Fri, Dec 3, 2010 at 18:02, Dimitri Fontaine >> wrote: >>> My understanding is that the variadic form shadows the other one in a >>> way that it's now impossible to call it from SQL level. That's the >>> reason why

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sat, Dec 4, 2010 at 8:04 PM, Jeff Janes wrote: > But who would be doing the passing?  For the postmaster to be doing > that would probably go against the minimalist design.  It would have > to keep track of which backend is available, and which db and user it > is primed for.  Perhaps a feature

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Andrew Dunstan
On 12/05/2010 08:55 PM, Robert Haas wrote: On Sun, Dec 5, 2010 at 1:28 PM, Tom Lane wrote: I'm wondering if we should reconsider the pass-it-through-the-client approach, because if we could make that work it would be more general and it wouldn't need any special privileges. The trick seems t

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 2:45 PM, Rob Wultsch wrote: > I think you have read a bit more into what I have said than is > correct.  MySQL can deal with thousands of users and separate schemas > on commodity hardware. There are many design decisions (some > questionable) that have made MySQL much bette

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 3:17 PM, Rob Wultsch wrote: > On Sun, Dec 5, 2010 at 12:45 PM, Rob Wultsch wrote: >> One thing I would suggest that the PG community keeps in mind while >> talking about built in connection process caching, is that it is very >> nice feature for memory leaks caused by a con

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 1:28 PM, Tom Lane wrote: > I'm wondering if we should reconsider the pass-it-through-the-client > approach, because if we could make that work it would be more general and > it wouldn't need any special privileges.  The trick seems to be to apply > sufficient sanity testing

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Robert Haas
On Sun, Dec 5, 2010 at 12:41 PM, Andrew Dunstan wrote: > Well, ISTM that amounts to not having "official topic branches" :-) I agree > that this is supposed to be one of git's strengths (or more exactly a > strength of distributed SCM's generally).  I don't really see any great > value in sanctify

Re: [HACKERS] pg_execute_from_file review

2010-12-05 Thread Itagaki Takahiro
On Mon, Dec 6, 2010 at 08:01, Tom Lane wrote: > Why is there a variadic replace() in this patch at all? It seems just > about entirely unrelated to the stated purpose of the patch, as well > as being of dubious usefulness. As I wrote in the previous mail, the most important part of the patch for

Re: [HACKERS] Spread checkpoint sync

2010-12-05 Thread Greg Smith
Rob Wultsch wrote: Forgive me, but is all of this a step on the slippery slope to direct io? And is this a bad thing I don't really think so. There's an important difference in my head between direct I/O, where the kernel is told "write this immediately!", and what I'm trying to achive. I w

Re: [HACKERS] pg_execute_from_file review

2010-12-05 Thread Tom Lane
Itagaki Takahiro writes: > On Fri, Dec 3, 2010 at 18:02, Dimitri Fontaine wrote: >> My understanding is that the variadic form shadows the other one in a >> way that it's now impossible to call it from SQL level. That's the >> reason why I did the (text, text, text, VARIADIC text) version before,

[HACKERS] CommitFest 2010-11: Status report at 2/3 of scheduled time

2010-12-05 Thread Greg Smith
After an unfortunate two weeks where I was lost in Nyquil-land and not paying as much attention as I should have, I just finished a pass checking over all the open 26 CommitFest items. Just under 50% of the patches that were open at the start of the CF are still hanging around, with 6 in "Read

Re: [HACKERS] Spread checkpoint sync

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 2:53 PM, Greg Smith wrote: > Heikki Linnakangas wrote: >> >> If you fsync() a file with one dirty page in it, it's going to return very >> quickly, but a 1GB file will take a while. That could be problematic if you >> have a thousand small files and a couple of big ones, as

Re: [HACKERS] [PATCH] Revert default wal_sync_method to fdatasync on Linux 2.6.33+

2010-12-05 Thread Greg Smith
Josh Berkus wrote: I modified test_fsync in two ways to run this; first, to make it support O_DIRECT, and second to make it run in the *current* directory. Patch please? I agree with the latter change; what test_fsync does is surprising. I suggested a while ago that we refactor test_fsync t

Re: [HACKERS] Spread checkpoint sync

2010-12-05 Thread Greg Smith
Heikki Linnakangas wrote: If you fsync() a file with one dirty page in it, it's going to return very quickly, but a 1GB file will take a while. That could be problematic if you have a thousand small files and a couple of big ones, as you would want to reserve more time for the big ones. I'm not

Re: [HACKERS] Instrument checkpoint sync calls

2010-12-05 Thread Greg Smith
Jeff Janes wrote: I've attached a tiny patch to apply over yours, to deal with this and with the case where no files are synced. Thanks for that. That obvious error eluded me because in most of the patch update testing I was doing (on ext3), the longest sync was always about the same leng

Re: [HACKERS] V3: Idle in transaction cancellation

2010-12-05 Thread Kevin Grittner
"Kevin Grittner" wrote: > Andres Freund wrote: >> On Thursday 02 December 2010 00:48:53 Kevin Grittner wrote: >> >>> Is there any provision for one backend to cause a *different* >>> backend which is idle in a transaction to terminate cleanly when >>> it attempts to process its next statement? >>

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 12:45 PM, Rob Wultsch wrote: > One thing I would suggest that the PG community keeps in mind while > talking about built in connection process caching, is that it is very > nice feature for memory leaks caused by a connection to not exist for > and continue growing forever.

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Rob Wultsch
On Sun, Dec 5, 2010 at 11:59 AM, Josh Berkus wrote: > >> * no coordination of restarts/configuration changes between the cluster >> and the pooler >> * you have two pieces of config files to configure your pooling settings >> (having all that available say in a catalog in pg would be awesome) >> *

Re: [HACKERS] Patch to add a primary key using an existing index

2010-12-05 Thread Peter Eisentraut
On fre, 2010-12-03 at 15:27 -0500, Robert Haas wrote: > On Fri, Dec 3, 2010 at 2:56 PM, r t wrote: > > What exactly was the objection to the following --> > > ALTER TABLE table_name ADD PRIMARY KEY (column_list) USING index_name; > > Is the objection that you might have been trying to specify a co

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Josh Berkus
* no coordination of restarts/configuration changes between the cluster and the pooler * you have two pieces of config files to configure your pooling settings (having all that available say in a catalog in pg would be awesome) * you lose all of the advanced authentication features of pg (becaus

Re: [HACKERS] wCTE behaviour

2010-12-05 Thread Greg Smith
Marko Tiikkaja wrote: This is almost exactly the patch from 2010-02 without CommandCounterIncrement()s. It's still a bit rough around the edges and needs some more comments, but I'm posting it here anyway. This patch passes all regression tests, but feel free to try to break it, there are pr

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Tom Lane
Greg Smith writes: > In addition, Joachim submitted a synchronized snapshot patch that looks > to me like it slipped through the cracks without being fully explored. > ... > The way I read that thread, there were two objections: > 1) This mechanism isn't general enough for all use-cases outsid

[HACKERS] Re: new patch of MERGE (merge_204) & a question about duplicated ctid

2010-12-05 Thread Greg Smith
Boxuan Zhai wrote: I have updated the MERGE patch for two main problems. The patch inside the .tar.gz file you attached isn't right; that extracts to a tiny file of junk characters. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services and Suppo

Re: [HACKERS] serializable read only deferrable

2010-12-05 Thread Kevin Grittner
Tom Lane wrote: > "Kevin Grittner" writes: >> I'm reviving the discussion on the subject topic because I just >> had an epiphany which makes it seem simple to implement. The >> concept of this is that if you start a SERIALIZABLE READ ONLY >> transaction in an SSI environment when certain conditio

Re: [HACKERS] libpq changes for synchronous replication

2010-12-05 Thread Greg Smith
The one time this year top-posting seems appropriate...this patch seems stalled waiting for some sort of response to the concerns Alvaro raised here. Alvaro Herrera wrote: Excerpts from Fujii Masao's message of jue nov 25 10:47:12 -0300 2010: The attached patch s/CopyXLog/CopyBoth/g and a

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Greg Smith
Joachim Wieland wrote: Regarding snapshot cloning and dump consistency, I brought this up already several months ago and asked if the feature is considered useful even without snapshot cloning. In addition, Joachim submitted a synchronized snapshot patch that looks to me like it slipped throug

[HACKERS] allow COPY routines to read arbitrary numbers of fields

2010-12-05 Thread Andrew Dunstan
Attached is a patch that allows CopyReadAttibutesText() and CopyReadAttributesCSV() to read arbitrary numbers of attributes. Underflowing attributes are recorded as null, and space is made for overflowing attributes on a line. This patch doesn't result in any user-visible behavior. The curre

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Andrew Dunstan
On 12/05/2010 12:10 PM, Tom Lane wrote: Andrew Dunstan writes: On 12/04/2010 07:12 PM, Robert Haas wrote: I wouldn't necessarily be opposed to official topic branches at some point in the future, but I think it's premature to speculate about whether it'd be useful here. I'd need a lot of

Re: [HACKERS] serializable read only deferrable

2010-12-05 Thread Tom Lane
"Kevin Grittner" writes: > I'm reviving the discussion on the subject topic because I just had > an epiphany which makes it seem simple to implement. The concept of > this is that if you start a SERIALIZABLE READ ONLY transaction in an > SSI environment when certain conditions are true, it doesn'

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Tom Lane
Andrew Dunstan writes: > On 12/04/2010 07:12 PM, Robert Haas wrote: >> I wouldn't necessarily be opposed to official topic branches at some point >> in the future, but I think it's premature to speculate about whether it'd be >> useful here. > I'd need a lot of convincing if it imposed an extr

Re: [HACKERS] Using different semaphore for locking

2010-12-05 Thread Tom Lane
flyusa2010 fly writes: > I found that postgres uses different semaphore system call on some different > operating systems. > For example, I found that on linux, System V semaphore (semop etc.) is used > to implement locking, while on Darwin, POSIX semaphore (sem_wait, sem_post > etc.) is used. > l

[HACKERS] serializable read only deferrable

2010-12-05 Thread Kevin Grittner
I'm reviving the discussion on the subject topic because I just had an epiphany which makes it seem simple to implement. The concept of this is that if you start a SERIALIZABLE READ ONLY transaction in an SSI environment when certain conditions are true, it doesn't need to acquire predicate locks

Re: [HACKERS] FK's to refer to rows in inheritance child

2010-12-05 Thread Andrew Dunstan
On 12/04/2010 07:12 PM, Robert Haas wrote: I wouldn't necessarily be opposed to official topic branches at some point in the future, but I think it's premature to speculate about whether it'd be useful here. I'd need a lot of convincing if it imposed an extra burden on people like Tom. T

Re: [HACKERS] what are clog and xlog?

2010-12-05 Thread Kevin Grittner
flyusa2010 fly wrote: > clog is for commit log (very unclear for me ...) It is an important part of tracking tuple visibility. You might want to start with this page: http://wiki.postgresql.org/wiki/Hint_Bits -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] new patch of MERGE (merge_204) & a question about duplicated ctid

2010-12-05 Thread David Fetter
On Sat, Dec 04, 2010 at 09:27:52PM +0800, Boxuan Zhai wrote: > Dear Greg, > > I have updated the MERGE patch for two main problems. Please attach the actual patch :) Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter

[HACKERS] Using different semaphore for locking

2010-12-05 Thread flyusa2010 fly
I found that postgres uses different semaphore system call on some different operating systems. For example, I found that on linux, System V semaphore (semop etc.) is used to implement locking, while on Darwin, POSIX semaphore (sem_wait, sem_post etc.) is used. linux and Darwin support both System

[HACKERS] what are clog and xlog?

2010-12-05 Thread flyusa2010 fly
Hi, folks, I'm a newbie to postgres. I'm confused with xlog and clog. To my initial understanding, xlog is the periodic checkpoint log for data, while clog is for commit log (very unclear for me ...) Thanks!

Re: [HACKERS] disk caching for writing log

2010-12-05 Thread flyusa2010 fly
Thanks for your reply. Yes, i mean disk may lie to os. On Fri, Dec 3, 2010 at 12:14 PM, Stefan Kaltenbrunner wrote: > On 12/03/2010 06:43 PM, Heikki Linnakangas wrote: > >> On 03.12.2010 13:49, flyusa2010 fly wrote: >> >>> When writing log, dbms should synchronously flush log to disk. I'm >>> w

Re: [HACKERS] SQL/MED - file_fdw

2010-12-05 Thread Andrew Dunstan
On 12/04/2010 11:11 PM, Itagaki Takahiro wrote: On Sun, Dec 5, 2010 at 07:24, Andrew Dunstan wrote: Looking at file_parser.c, it seems to be largely taken from copy.c. Wouldn't it be better to call those functions, or refactor them so they are callable if necessary? We could export private f

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Dmitriy Igrishin
Hey hackers, Varargs-exec is useful only when programmer calls it directly. It is useless when libpq is used to create a more flexible high-level library (e.g., for C++). PQexecParams (PQexecPrepared) are good for it. What about auto reconnect. There are PQreset already and PG_CONNECTION_OK (_BAD

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Magnus Hagander
On Sun, Dec 5, 2010 at 11:57, Heikki Linnakangas wrote: > On 05.12.2010 12:10, Magnus Hagander wrote: >> >> On Sun, Dec 5, 2010 at 10:22, Marc Balmer  wrote: >>> >>> I am suggesting adding a function to libpq: >>> >>> PGresult *PQvexec(PGconn *conn, const char *fmt, ...); >>> >>> It behaves simila

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Marc Balmer
Am 05.12.2010 um 11:57 schrieb Heikki Linnakangas : > On 05.12.2010 12:10, Magnus Hagander wrote: >> On Sun, Dec 5, 2010 at 10:22, Marc Balmer wrote: >>> I am suggesting adding a function to libpq: >>> >>> PGresult *PQvexec(PGconn *conn, const char *fmt, ...); >>> >>> It behaves similar to P

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Heikki Linnakangas
On 05.12.2010 12:10, Magnus Hagander wrote: On Sun, Dec 5, 2010 at 10:22, Marc Balmer wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and How is that not a horrible

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Magnus Hagander
On Sun, Dec 5, 2010 at 10:22, Marc Balmer wrote: > I am suggesting adding a function to libpq: > > PGresult *PQvexec(PGconn *conn, const char *fmt, ...); > > It behaves similar to PQexec, but it allows for printf style varargs and How is that not a horrible idea, compared to using PQexecParams()?

Re: [HACKERS] Suggesting a libpq addition

2010-12-05 Thread Pavel Stehule
2010/12/5 Marc Balmer : > I am suggesting adding a function to libpq: > > PGresult *PQvexec(PGconn *conn, const char *fmt, ...); > > It behaves similar to PQexec, but it allows for printf style varargs and > does connection re-establishing if the connection fails (it can be > discussed if this alre

[HACKERS] Suggesting a libpq addition

2010-12-05 Thread Marc Balmer
I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and does connection re-establishing if the connection fails (it can be discussed if this already to much magic, maybe remove this

Re: [HACKERS] profiling connection overhead

2010-12-05 Thread Stefan Kaltenbrunner
On 12/01/2010 05:32 AM, Jeff Janes wrote: On 11/28/10, Robert Haas wrote: In a close race, I don't think we should get bogged down in micro-optimization here, both because micro-optimizations may not gain much and because what works well on one platform may not do much at all on another. The