Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 08:35, Robert Haas wrote: >> I expect it would be fine to have a tool that pulls LCRs out of WAL to >> prepare that to be sent to remote locations.  Is that what you have in >> mind? > > Yes.  I think it should be possible to generate LCRs from WAL, but I > think that the on-the-w

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Heikki Linnakangas
On 20.06.2012 01:27, Kevin Grittner wrote: Andres Freund wrote: Yes, thats definitely a valid use-case. But that doesn't preclude the other - also not uncommon - use-case where you want to have different master which all contain up2date data. I agree. I was just saying that while one requir

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 11:26, Tom Lane wrote: > "Kevin Grittner" writes: >> Simon Riggs wrote: >>> The proposal is to use WAL to generate the logical change stream. >>> That has been shown in testing to be around x4 faster than having >>> a separate change stream, which must also be WAL logged (as Jan

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Amit Kapila
>> I've got a problem with the assumption that, when pg_control is trash, >> megabytes or gigabytes of WAL can still be relied on completely. >> >> I'm almost inclined to suggest that we not get next-LSN from WAL, but >> by scanning all the pages in the main data store and computing the max >>

Re: [HACKERS] return values of backend sub-main functions

2012-06-19 Thread Josh Kupershmidt
On Tue, Jun 19, 2012 at 4:31 AM, Peter Eisentraut wrote: > On ons, 2012-01-18 at 21:21 +0200, Peter Eisentraut wrote: >> On lör, 2012-01-07 at 16:41 -0500, Tom Lane wrote: >> > Peter Eisentraut writes: >> > > I suggest that we change PostgresMain(), PostmasterMain(), BackendRun(), >> > > WalSende

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Greg Smith
On 06/19/2012 08:41 PM, Robert Haas wrote: On Tue, Jun 19, 2012 at 7:33 PM, Josh Berkus wrote: To what version will we backport it? All supported branches, I would think. Yeah, I would feel a lot better if this was improved in 8.3 and up. I didn't bother trying to push it before now becau

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Amit Kapila
>> I believe if WAL files are proper as mentioned in Alvaro's mail, the purposed logic should generate >> correct values. >> Do you see any problem in logic purposed in my original mail. >> Can I resume my work on this feature? > Maybe I'm missing your point, but... why don't you just use PITR to

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Tom Lane
"Kevin Grittner" writes: > Simon Riggs wrote: >> The proposal is to use WAL to generate the logical change stream. >> That has been shown in testing to be around x4 faster than having >> a separate change stream, which must also be WAL logged (as Jan >> noted). > Sure, that's why I want it. I

Re: [HACKERS] performance regression in 9.2 when loading lots of small tables

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 10:56 PM, Jeff Janes wrote: > But in the 9.2 branch, the slow phenotype was re-introduced in > 1575fbcb795fc331f4, although perhaps the details of who is locking > what differs.  I haven't yet sorted that out. It very much does. That commit prevents people from creating a

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 4:22 PM, Andres Freund wrote: >> > 1. dllist.h has double the element overhead by having an inline value >> > pointer (which is not needed when embedding) and a pointer to the list >> > (which I have a hard time seing as being useful) >> > 2. only double linked list, mine p

Re: [HACKERS] performance regression in 9.2 when loading lots of small tables

2012-06-19 Thread Jeff Janes
On Tue, Jun 19, 2012 at 2:38 PM, Robert Haas wrote: > On Tue, Jun 19, 2012 at 4:33 PM, Robert Haas wrote: >> On Mon, Jun 18, 2012 at 8:42 PM, Jeff Janes wrote: >>> There was a regression introduced in 9.2 that effects the creation and >>> loading of lots of small tables in a single transaction.

Re: [HACKERS] psql tab completion for GRANT role

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 3:16 AM, Peter Eisentraut wrote: > On tor, 2012-06-14 at 13:38 -0400, Robert Haas wrote: >> On Sun, Jan 8, 2012 at 3:48 PM, Peter Eisentraut wrote: >> > psql tab completion currently only supports the form GRANT privilege ON >> > something TO someone (and the analogous REV

Re: [HACKERS] Libxml2 load error on Windows

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 6:36 AM, Alex Shulgin wrote: > Dave Page writes: >> On Tue, Jun 19, 2012 at 11:04 AM, Alex Shulgin >> wrote: >>> >>> In a real bug-tracking system we would create a new bug/ticket and set >>> it's target version to 'candidate for next minor release' or something >>> like

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 2:23 PM, Andres Freund wrote: >> Well, the words are fuzzy, but I would define logical replication to >> be something which is independent of the binary format in which stuff >> gets stored on disk.  If it's not independent of the disk format, then >> you can't do heterogen

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 7:33 PM, Josh Berkus wrote: > To what version will we backport it? All supported branches, I would think. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 6:14 PM, Andres Freund wrote: > I definitely agree that low-level apply is possible as a module. Sure change > extraction needs core support but I was talking about what you need to > implement it reusing the "plain" logical support... > > What I do not understand is how yo

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 5:59 PM, Christopher Browne wrote: > On Tue, Jun 19, 2012 at 5:46 PM, Robert Haas wrote: >>> Btw, what do you mean with "conflating" the stream? I don't really see that >>> being proposed. >> >> It seems to me that you are intent on using the WAL stream as the >> logical c

Re: [HACKERS] REVIEW: Optimize referential integrity checks (todo item)

2012-06-19 Thread Tom Lane
Dean Rasheed writes: > On 12 February 2012 02:06, Vik Reykja wrote: >> I decided to take a crack at the todo item created from the following post: >> http://archives.postgresql.org/pgsql-performance/2005-10/msg00458.php > Here's my review of this patch. I've marked this patch committed, althoug

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Josh Berkus
To what version will we backport it? -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Foreign keys in pgbench

2012-06-19 Thread Tom Lane
Jeff Janes writes: > I think that pgbench should it make it easy to assess the impact of > foreign key constraints. > The attached adds a --foreign-keys option to initialization mode which > creates all the relevant constraints between the default tables. I had need of this for testing what I'm

Re: [HACKERS] WAL format changes

2012-06-19 Thread Andres Freund
Hi, On Wednesday, June 20, 2012 12:24:54 AM Heikki Linnakangas wrote: > On 19.06.2012 18:46, Andres Freund wrote: > > On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote: > >> Well, that was easier than I thought. Attached is a patch to make > >> XLogRecPtr a uint64, on top of my other

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Kevin Grittner
Andres Freund wrote: > Yes, thats definitely a valid use-case. But that doesn't preclude > the other - also not uncommon - use-case where you want to have > different master which all contain up2date data. I agree. I was just saying that while one requires an origin_id, the other doesn't. An

Re: [HACKERS] WAL format changes

2012-06-19 Thread Heikki Linnakangas
On 19.06.2012 18:46, Andres Freund wrote: On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote: Well, that was easier than I thought. Attached is a patch to make XLogRecPtr a uint64, on top of my other WAL format patches. I think we should go ahead with this. Cool. You plan to merge X

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 11:39:46 PM Robert Haas wrote: > On Tue, Jun 19, 2012 at 5:33 PM, Greg Smith wrote: > > In January of 2011 Robert committed > > 7f242d880b5b5d9642675517466d31373961cf98 to try and compact the fsync > > queue when clients find it full. There's no visible behavior change,

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Andres Freund
On Wednesday, June 20, 2012 12:15:03 AM Kevin Grittner wrote: > Simon Riggs wrote: > > If we use WAL in this way, multi-master implies that the data will > > *always* be in a loop. So in any configuration we must be able to > > tell difference between changes made by one node and another. > > Onl

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Kevin Grittner
Simon Riggs wrote: > The proposal is to use WAL to generate the logical change stream. > That has been shown in testing to be around x4 faster than having > a separate change stream, which must also be WAL logged (as Jan > noted). Sure, that's why I want it. > If we use WAL in this way, mult

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 11:46:56 PM Robert Haas wrote: > On Tue, Jun 19, 2012 at 3:18 PM, Andres Freund wrote: > > More seriously: Even if we don't put MM in core I think putting the basis > > for it in core so that somebody can build such a solution reusing the > > existing infrastructure is a

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 00:11, Tom Lane wrote: > Andres Freund writes: >> On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote: ...  (If you are thinking of something sufficiently high-level that merging could possibly work, then it's not WAL, and we shouldn't be trying to make the WAL >>>

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 05:59, Christopher Browne wrote: > But it's undesirable to pull *all* the bulk of contents of WAL around > if it's only part of the data that is going to get applied.  On a > "physical streaming" replica, any logical data that gets captured will > be useless.  And on a "logical re

Re: [HACKERS] Transactions over pathological TCP connections

2012-06-19 Thread Leon Smith
I just realized this is essentially an instance of the Two General's Problem; which is something I feel should have been more obvious to me. On Tue, Jun 19, 2012 at 5:50 PM, Leon Smith wrote: > On Tue, Jun 19, 2012 at 11:59 AM, Robert Haas wrote: > >> On Tue, Jun 19, 2012 at 1:56 AM, Tom Lane

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 04:31, Kevin Grittner wrote: > I've done a lot of MM replication, > and so far have not had to use a topology which allowed loops. The proposal is to use WAL to generate the logical change stream. That has been shown in testing to be around x4 faster than having a separate change

Re: [HACKERS] use of int4/int32 in C code

2012-06-19 Thread Tom Lane
"Kevin Grittner" writes: > Tom Lane wrote: >> I have not looked to see how many places do that. If it's a reasonably >> small number of places, I'm OK with getting rid of int4 at the C level. >> (int2/int8 the same of course.) > $ find -name '*.h' -or -name '*.c' | egrep -v '/tmp_check/' | xar

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 10:58:44 PM Marko Kreen wrote: > On Mon, Jun 18, 2012 at 6:35 PM, Simon Riggs wrote: > > On 13 June 2012 19:28, Andres Freund wrote: > >> This adds a new configuration parameter multimaster_node_id which > >> determines the id used for wal originating in one cluster. > >

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 05:46, Robert Haas wrote: > It seems to me that you are intent on using the WAL stream as the > logical change stream.  I think that's a bad design.  Instead, you > should extract changes from WAL and then ship them around in a format > that is specific to logical replication. Th

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Christopher Browne
On Tue, Jun 19, 2012 at 5:46 PM, Robert Haas wrote: >> Btw, what do you mean with "conflating" the stream? I don't really see that >> being proposed. > > It seems to me that you are intent on using the WAL stream as the > logical change stream.  I think that's a bad design.  Instead, you > should

Re: [HACKERS] pg_restore logging inconsistency

2012-06-19 Thread Alvaro Herrera
Excerpts from Josh Kupershmidt's message of mié may 30 14:55:12 -0400 2012: > Hi all, > > Bosco Rama recently complained[1] about not seeing a message printed > by pg_restore for each LO to be restored. The culprit seems to be the > different level passed to ahlog() for this status message: > >

Re: [HACKERS] use of int4/int32 in C code

2012-06-19 Thread Kevin Grittner
Tom Lane wrote: > I have not looked to see how many places do that. If it's a reasonably > small number of places, I'm OK with getting rid of int4 at the C level. > (int2/int8 the same of course.) $ find -name '*.h' -or -name '*.c' | egrep -v '/tmp_check/' | xargs cat \ | egrep -c '\bint2\

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 19 June 2012 14:03, Tom Lane wrote: > "Every WAL record"?  Why in heck would you attach it to every record? > Surely putting it in WAL page headers would be sufficient.  We could > easily afford to burn a page switch (if not a whole segment switch) > when changing masters. This does appear to

Re: [HACKERS] Transactions over pathological TCP connections

2012-06-19 Thread Leon Smith
On Tue, Jun 19, 2012 at 11:59 AM, Robert Haas wrote: > On Tue, Jun 19, 2012 at 1:56 AM, Tom Lane wrote: > > The transaction would be committed before a command success report is > > delivered to the client, so I don't think delivered-and-not-marked is > > possible. > > ...unless you have configu

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar jun 19 17:39:46 -0400 2012: > On Tue, Jun 19, 2012 at 5:33 PM, Greg Smith wrote: > > In January of 2011 Robert committed 7f242d880b5b5d9642675517466d31373961cf98 > > to try and compact the fsync queue when clients find it full.  There's no > > visible be

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Christopher Browne
On Tue, Jun 19, 2012 at 5:39 PM, Robert Haas wrote: > On Tue, Jun 19, 2012 at 5:33 PM, Greg Smith wrote: >> In January of 2011 Robert committed 7f242d880b5b5d9642675517466d31373961cf98 >> to try and compact the fsync queue when clients find it full.  There's no >> visible behavior change, just a

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 3:18 PM, Andres Freund wrote: > More seriously: Even if we don't put MM in core I think putting the basis for > it in core so that somebody can build such a solution reusing the existing > infrastructure is a sensible idea. Imo the only thing that requires explicit > suppor

Re: [HACKERS] use of int4/int32 in C code

2012-06-19 Thread Tom Lane
Robert Haas writes: > On Tue, Jun 19, 2012 at 9:47 AM, Tom Lane wrote: >> I thought the general idea was to use int32 most places, but int4 in >> catalog declarations. I don't think it's tremendously important if >> somebody uses the other though. > I concur with Peter that TMTOWTDI is not the

Re: [HACKERS] Backport of fsync queue compaction

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 5:33 PM, Greg Smith wrote: > In January of 2011 Robert committed 7f242d880b5b5d9642675517466d31373961cf98 > to try and compact the fsync queue when clients find it full.  There's no > visible behavior change, just a substantial performance boost possible in > the rare but e

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Simon Riggs
On 20 June 2012 04:58, Marko Kreen wrote: > On Mon, Jun 18, 2012 at 6:35 PM, Simon Riggs wrote: >> On 13 June 2012 19:28, Andres Freund wrote: >>> This adds a new configuration parameter multimaster_node_id which determines >>> the id used for wal originating in one cluster. >> >> Looks good and

Re: [HACKERS] performance regression in 9.2 when loading lots of small tables

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 4:33 PM, Robert Haas wrote: > On Mon, Jun 18, 2012 at 8:42 PM, Jeff Janes wrote: >> There was a regression introduced in 9.2 that effects the creation and >> loading of lots of small tables in a single transaction. >> >> It affects the loading of a pg_dump file which has a

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Kevin Grittner
Andres Freund wrote: > The problem is just that to support basically arbitrary decoding > requirements you need to provide at least those pieces of > information in a transactionally consistent manner: > * the data > * table names > * column names > * type information > * replication configurati

Re: [HACKERS] Testing 9.2 in ~production environment

2012-06-19 Thread Peter Eisentraut
On tis, 2012-06-19 at 09:33 -0400, Tom Lane wrote: > Come to think of it, another possible factor is that LIKE can't use > ordinary indexes on text if the locale isn't C. But he reported that the plans are the same. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make c

Re: [HACKERS] Near-duplicate RI NO ACTION and RESTRICT triggers

2012-06-19 Thread Dean Rasheed
On 19 June 2012 17:48, Tom Lane wrote: > I think that the argument for having the RESTRICT triggers behave > like this is that the SQL spec envisions the RESTRICT check occurring > immediately when the individual PK row is updated/deleted, and so there > would be no opportunity for another PK row

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Marko Kreen
On Mon, Jun 18, 2012 at 6:35 PM, Simon Riggs wrote: > On 13 June 2012 19:28, Andres Freund wrote: >> This adds a new configuration parameter multimaster_node_id which determines >> the id used for wal originating in one cluster. > > Looks good and it seems this aspect at least is commitable in th

Re: [HACKERS] performance regression in 9.2 when loading lots of small tables

2012-06-19 Thread Robert Haas
On Mon, Jun 18, 2012 at 8:42 PM, Jeff Janes wrote: > There was a regression introduced in 9.2 that effects the creation and > loading of lots of small tables in a single transaction. > > It affects the loading of a pg_dump file which has a large number of > small tables (10,000 schemas, one table

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Kevin Grittner
Andres Freund wrote: > Robert Haas wrote: >> Tom Lane wrote: >>> However, if we're dead set on doing it that way, let us put >>> information that is only relevant to logical replication records >>> into only the logical replication records. >> Right. If we decide we need this, and if we did de

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 09:58:43 PM Robert Haas wrote: > On Tue, Jun 19, 2012 at 3:48 PM, Andres Freund wrote: > > Why is that code not used more widely? Quite a bit of our list usage > > should be replaced embedding list element in larger structs imo. There > > are also open- coded inline list

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Marko Kreen
On Tue, Jun 19, 2012 at 11:02 PM, Andres Freund wrote: > On Tuesday, June 19, 2012 09:59:48 PM Marko Kreen wrote: >> On Wed, Jun 13, 2012 at 2:28 PM, Andres Freund > wrote: >> > +/* >> > + * removes a node from a list >> > + * Attention: O(n) >> > + */ >> > +static inline void ilist_s_remove(ilis

Re: [HACKERS] [PATCH 01/16] Overhaul walsender wakeup handling

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 09:55:30 PM Robert Haas wrote: > On Wed, Jun 13, 2012 at 7:28 AM, Andres Freund wrote: > > From: Andres Freund > > > > The previous coding could miss xlog writeouts at several places. E.g. > > when wal was written out by the background writer or even after a commit > >

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Andres Freund
Hi, On Tuesday, June 19, 2012 09:59:48 PM Marko Kreen wrote: > On Wed, Jun 13, 2012 at 2:28 PM, Andres Freund wrote: > > +/* > > + * removes a node from a list > > + * Attention: O(n) > > + */ > > +static inline void ilist_s_remove(ilist_s_head *head, > > + ilist

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Marko Kreen
On Wed, Jun 13, 2012 at 2:28 PM, Andres Freund wrote: > +/* > + * removes a node from a list > + * Attention: O(n) > + */ > +static inline void ilist_s_remove(ilist_s_head *head, > +                                  ilist_s_node *node) > +{ > +       ilist_s_node *last = &head->head; > +       ili

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 3:48 PM, Andres Freund wrote: > Why is that code not used more widely? Quite a bit of our list usage should be > replaced embedding list element in larger structs imo. There are also open- > coded inline list manipulations around (check aset.c for example). Because we've g

Re: [HACKERS] [PATCH 01/16] Overhaul walsender wakeup handling

2012-06-19 Thread Robert Haas
On Wed, Jun 13, 2012 at 7:28 AM, Andres Freund wrote: > From: Andres Freund > > The previous coding could miss xlog writeouts at several places. E.g. when wal > was written out by the background writer or even after a commit if > synchronous_commit=off. > This could lead to delays in sending data

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 09:16:41 PM Robert Haas wrote: > On Wed, Jun 13, 2012 at 7:28 AM, Andres Freund wrote: > > Adds a single and a double linked list which can easily embedded into > > other datastructures and can be used without any additional allocations. > > dllist.h advertises that it'

Re: [HACKERS] return values of backend sub-main functions

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 7:31 AM, Peter Eisentraut wrote: > On ons, 2012-01-18 at 21:21 +0200, Peter Eisentraut wrote: >> On lör, 2012-01-07 at 16:41 -0500, Tom Lane wrote: >> > Peter Eisentraut writes: >> > > I suggest that we change PostgresMain(), PostmasterMain(), BackendRun(), >> > > WalSende

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 3:03 PM, Alvaro Herrera wrote: > That's what I thought.  I will commit to both branches soon, then. I think there might be three branches involved. > Mind you, this should have been an "open item", not a commitfest item. > (Actually not even an open item.  We should have

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 1:43 AM, Tom Lane wrote: > Amit Kapila writes: >>> AFAIR you can create pg_control from scratch already with pg_resetxlog. >>> The hard part is coming up with values for the counters, such as the >>> next WAL location.  Some of them such as next OID are pretty harmless >>>

Re: [HACKERS] use of int4/int32 in C code

2012-06-19 Thread Peter Geoghegan
On 19 June 2012 20:11, Robert Haas wrote: > On Tue, Jun 19, 2012 at 9:47 AM, Tom Lane wrote: >> Peter Eisentraut writes: >>> What is the latest theory on using int4 vs. int32 in C code? >>> (equivalently int2, int16) >> >> I thought the general idea was to use int32 most places, but int4 in >> c

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 07:24:13 PM Robert Haas wrote: > On Tue, Jun 19, 2012 at 12:11 PM, Tom Lane wrote: > > Andres Freund writes: > >> On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote: > ... (If you are thinking > of something sufficiently high-level that merging could possibl

Re: [HACKERS] [PATCH 04/16] Add embedded list interface (header only)

2012-06-19 Thread Robert Haas
On Wed, Jun 13, 2012 at 7:28 AM, Andres Freund wrote: > Adds a single and a double linked list which can easily embedded into other > datastructures and can be used without any additional allocations. dllist.h advertises that it's embeddable. Can you use that instead, or enhance it slightly to s

Re: [HACKERS] use of int4/int32 in C code

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 9:47 AM, Tom Lane wrote: > Peter Eisentraut writes: >> What is the latest theory on using int4 vs. int32 in C code? >> (equivalently int2, int16) > > I thought the general idea was to use int32 most places, but int4 in > catalog declarations.  I don't think it's tremendous

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar jun 19 11:36:41 -0400 2012: > > On Mon, Jun 18, 2012 at 3:30 PM, Alvaro Herrera > wrote: > > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > > > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > >> an

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Peter Geoghegan
On 19 June 2012 19:44, Peter Geoghegan wrote: > You could do that, and some people do use custom collations for > various reasons. That's obviously very much of minority interest > though. Most people will just use citext or something. However, since > citext is itself a client of varstr_cmp(), th

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Tom Lane
"Kevin Grittner" writes: > I wasn't aware that en_US.UTF-8 doesn't have equivalence without > equality. I guess that surprising result in my last post is just > plain inevitable with that collation then. Bummer. Is there > actually anyone who finds that to be a useful behavior? For a > collati

Re: [HACKERS] Event Triggers reduced, v1

2012-06-19 Thread Robert Haas
On Fri, Jun 15, 2012 at 4:27 PM, Dimitri Fontaine wrote: > Allow me to open the new season of the DML trigger series, named > pg_event_trigger. This first episode is all about setting up the drama, > so that next ones make perfect sense. Comments: 1. I still think we ought to get rid of the noti

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Peter Geoghegan
On 19 June 2012 18:57, Kevin Grittner wrote: > We weren't using en_US.UTF-8 collation (or any other "proper" > collation) on Sybase -- I'm not sure whether they even supported > proper collation sequences on the versions we used.  I'm thinking of > when we were using their "case insensitive" sorti

Re: [HACKERS] [RFC][PATCH] Logical Replication/BDR prototype and architecture

2012-06-19 Thread Andres Freund
Hi, The most important part, even for people not following my discussion with Robert is at the bottom where the possible wal decoding strategies are laid out. On Tuesday, June 19, 2012 03:20:58 AM Robert Haas wrote: > On Sat, Jun 16, 2012 at 7:43 AM, Andres Freund wrote: > >> > Hm. Yes, you c

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Kevin Grittner
Peter Geoghegan wrote: > Kevin Grittner wrote: >> I'm pretty sure that when I was using Sybase ASE the order for >> non-equal values was always predictable, and it behaved in the >> manner I describe below. I'm less sure about any other product. > > Maybe it used a physical row identifier as

Re: [HACKERS] Do we want a xmalloc or similar function in the Backend?

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 07:35:53 PM Robert Haas wrote: > On Tue, Jun 19, 2012 at 10:17 AM, Andres Freund wrote: > > There are 70+ calls of malloc in the backend in the form of > > > > type* foo = malloc(sizeof(...)); > > if(!foo) > > elog(ERROR, "could not allocate memory"); > > > > which i

Re: [HACKERS] Do we want a xmalloc or similar function in the Backend?

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 10:17 AM, Andres Freund wrote: > There are 70+ calls of malloc in the backend in the form of > > type* foo = malloc(sizeof(...)); > if(!foo) >   elog(ERROR, "could not allocate memory"); > > which is a bit annoying to write at times. Would somebody argue against > introduci

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Alex Hunsaker
On Mon, Jun 18, 2012 at 1:30 PM, Alvaro Herrera wrote: > > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr >> and SvPVUTF8() when turning a perl string into a cstring. > > Hmm, this patch belongs i

Re: [HACKERS] initdb and fsync

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 07:22:02 PM Jeff Davis wrote: > On Mon, 2012-06-18 at 21:41 +0200, Andres Freund wrote: > > It calls pg_flush_data inside of copy_file which does the > > posix_fadvise... So maybe just put the sync_file_range in pg_flush_data? > > The functions in fd.c aren't linked to in

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 12:11 PM, Tom Lane wrote: > Andres Freund writes: >> On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote: ...  (If you are thinking of something sufficiently high-level that merging could possibly work, then it's not WAL, and we shouldn't be trying to make

Re: [HACKERS] initdb and fsync

2012-06-19 Thread Jeff Davis
On Mon, 2012-06-18 at 21:41 +0200, Andres Freund wrote: > It calls pg_flush_data inside of copy_file which does the posix_fadvise... So > maybe just put the sync_file_range in pg_flush_data? The functions in fd.c aren't linked to initdb, so it's a challenge to share that code (I remember now: tha

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Peter Geoghegan
On 19 June 2012 17:45, Kevin Grittner wrote: > Peter Geoghegan wrote: >> Are you sure that they actually have a tie-breaker, and don't just >> make the distinction between equality and equivalence (if only >> internally)? > > I'm pretty sure that when I was using Sybase ASE the order for > non-eq

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-19 Thread Alvaro Herrera
Excerpts from Boszormenyi Zoltan's message of mar jun 19 12:44:04 -0400 2012: > OK, all 4 Check* functions are now moved back into proc.c, > nothing outside of timeout.c touches anything in it. New patches > are attached. Yeah, I like this one better, thanks. It seems to me that the "check" fun

[HACKERS] Near-duplicate RI NO ACTION and RESTRICT triggers

2012-06-19 Thread Tom Lane
Our current interpretation of the difference between foreign keys with ON UPDATE/DELETE NO ACTION and those with ON UPDATE/DELETE RESTRICT is that they mean the same thing but RESTRICT checks are not deferrable. It follows from this that the trigger code ought to be the same for NO ACTION and RESTR

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Kevin Grittner
Peter Geoghegan wrote: > Kevin Grittner wrote: >> Since we appear to be questioning everything in this area, I'll >> raise something which has been bugging me for a while: in some >> other systems I've used, the "tie-breaker" comparison for >> equivalent values comes after equivalence sorting o

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Andres Freund
Hi, On Tuesday, June 19, 2012 06:11:20 PM Tom Lane wrote: > Andres Freund writes: > > On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote: > >>> ... (If you are thinking > >>> of something sufficiently high-level that merging could possibly work, > >>> then it's not WAL, and we shouldn't be try

Re: [HACKERS] WIP Patch: Selective binary conversion of CSV file foreign tables

2012-06-19 Thread Kohei KaiGai
Hi Fujita-san, Could you rebase this patch towards the latest tree? It was unavailable to apply the patch cleanly. I looked over the patch, then noticed a few points. At ProcessCopyOptions(), defel->arg can be NIL, isn't it? If so, cstate->convert_binary is not a suitable flag to check redundant

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Peter Geoghegan
On 19 June 2012 16:17, Kevin Grittner wrote: > Peter Geoghegan wrote: > >> So, just to give a bit more weight to my argument that we should >> recognise that equivalent strings ought to be treated identically > > Since we appear to be questioning everything in this area, I'll > raise something wh

Re: [HACKERS] pgsql_fdw in contrib

2012-06-19 Thread Merlin Moncure
On Tue, Jun 19, 2012 at 10:15 AM, Kohei KaiGai wrote: > Let me push the pgsql_fdw in core from different perspective. > > Right now, FDW is a feature that will take many enhancement in > the near future like join-pushdown, writable APIs and so on. > If we would not have a FDW extension in core tha

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-19 Thread Tom Lane
Andres Freund writes: > On Tuesday, June 19, 2012 04:30:59 PM Tom Lane wrote: >>> ... (If you are thinking >>> of something sufficiently high-level that merging could possibly work, >>> then it's not WAL, and we shouldn't be trying to make the WAL >>> representation cater for it.) > Do you reall

Re: [HACKERS] Transactions over pathological TCP connections

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 1:56 AM, Tom Lane wrote: > The transaction would be committed before a command success report is > delivered to the client, so I don't think delivered-and-not-marked is > possible. ...unless you have configured synchronous_commit=off, or fsync=off. Or unless your disk mel

Re: [HACKERS] Skip checkpoint on promoting from streaming replication

2012-06-19 Thread Fujii Masao
On Tue, Jun 19, 2012 at 5:30 PM, Kyotaro HORIGUCHI wrote: > Thank you. > >> What happens if the server skips an end-of-recovery checkpoint, >> is promoted to the master, runs some write transactions, >> crashes and restarts automatically before it completes >> checkpoint? In this case, the server

Re: [HACKERS] WAL format changes

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas wrote: > Well, that was easier than I thought. Attached is a patch to make XLogRecPtr > a uint64, on top of my other WAL format patches. I think we should go ahead > with this. +1. > The LSNs on pages are still stored in the old format, to avoi

Re: [HACKERS] Pg default's verbosity?

2012-06-19 Thread Robert Haas
On Tue, Jun 19, 2012 at 2:15 AM, Tom Lane wrote: > Robert Haas writes: >> There might be something to the idea of demoting a few of the things >> we've traditionally had as NOTICEs, though.  IME, the following two >> messages account for a huge percentage of the chatter: > >> NOTICE:  CREATE TABL

Re: [HACKERS] WAL format changes

2012-06-19 Thread Andres Freund
On Tuesday, June 19, 2012 10:14:08 AM Heikki Linnakangas wrote: > On 18.06.2012 21:08, Heikki Linnakangas wrote: > > On 18.06.2012 21:00, Robert Haas wrote: > >> On Thu, Jun 14, 2012 at 5:58 PM, Andres Freund > >> > >> wrote: > 1. Use a 64-bit segment number, instead of the log/seg combinatio

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-19 Thread Robert Haas
On Mon, Jun 18, 2012 at 3:30 PM, Alvaro Herrera wrote: > Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > >> Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr >> and SvPVUTF8() when turning a perl string into a cstring. > > Hmm, this patch belongs int

Re: [HACKERS] libpq compression

2012-06-19 Thread Robert Haas
On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout wrote: > On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: >> The fly in the ointment with any of these ideas is that the "configure >> list" is not a list of exact cipher names, as per Magnus' comment that >> the current default incl

Re: [HACKERS] initdb and fsync

2012-06-19 Thread David Fetter
On Mon, Jun 18, 2012 at 09:34:30PM +0300, Peter Eisentraut wrote: > On mån, 2012-06-18 at 18:05 +0200, Andres Freund wrote: > > - defaulting to initdb -N in the regression suite is not a good imo, > > because that way the buildfarm won't catch problems in that area... > > > The regression test sui

Re: [HACKERS] sortsupport for text

2012-06-19 Thread Kevin Grittner
Peter Geoghegan wrote: > So, just to give a bit more weight to my argument that we should > recognise that equivalent strings ought to be treated identically Since we appear to be questioning everything in this area, I'll raise something which has been bugging me for a while: in some other sys

Re: [HACKERS] pgsql_fdw in contrib

2012-06-19 Thread Kohei KaiGai
2012/6/19 Merlin Moncure : > On Mon, Jun 18, 2012 at 11:01 PM, Robert Haas wrote: >> On Mon, Jun 18, 2012 at 12:24 PM, Merlin Moncure wrote: >>> I can't help but wonder (having been down the contrib/core/extension >>> road myself) if it isn't better to improve the facilities to register >>> and s

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-06-19 Thread Alvaro Herrera
Excerpts from Boszormenyi Zoltan's message of mar jun 19 04:44:35 -0400 2012: > > 2012-06-18 19:46 keltezéssel, Alvaro Herrera írta: > > Excerpts from Boszormenyi Zoltan's message of vie may 11 03:54:13 -0400 > > 2012: > >> Hi, > >> > >> another rebasing and applied the GIT changes in > >> ada8f

  1   2   >