Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
> I think you're missing the point. There is no possible way to guarantee > database > consistency after applying pg_resetxlog, unless the database had been cleanly > shut > down beforehand. The reset will lose the xlog information that was needed to > restore > consistency. So arguing fro

Re: [HACKERS] Backport of fsync queue compaction

2012-06-20 Thread Greg Smith
I don't want to take a bunch of time away from the active CF talking about this, just wanted to pass along some notes: -Back branch release just happening a few weeks ago. Happy to have this dropped until the CF is over. -Attached is a working backport of this to 8.4, with standard git comm

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
> Just so I understand correctly, the aim of this is to "fix" the > situation where out of the thousands of files and 100s of GB of data > in my pg directory, the *only* corruption is that a single file > pg_control file is missing? This is just an example I have used to explain what should be the

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Amit Kapila
>> AFAIK PITR can be used in a scenario where there is a base back-up and we >> have archived >> the WAL files after that, now it can use WAL files to apply on base-backup. > Yes. If you want to recover the database from the media crash like the > corruption of pg_control file, you basically shoul

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Peter Geoghegan
On 21 June 2012 01:22, Peter Geoghegan wrote: > I've written a very small C++ program, which I've attached, that > basically proves that this can still make a fairly large difference - > I hope it's okay that that it's C++, but that allowed me to write the > program quickly, with no dependencies f

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Peter Geoghegan
On 20 June 2012 17:41, Tom Lane wrote: > Peter Geoghegan writes: >> No, I'm suggesting it would probably be at least a bit of a win here >> to cache the constant, and only have to do a strxfrm() + strcmp() per >> comparison. > > Um, have you got any hard evidence to support that notion?  The > tr

Re: [HACKERS] [PATCH 08/16] Introduce the ApplyCache module which can reassemble transactions from a stream of interspersed changes

2012-06-20 Thread Steve Singer
On 12-06-13 07:28 AM, Andres Freund wrote: From: Andres Freund The individual changes need to be identified by an xid. The xid can be a subtransaction or a toplevel one, at commit those can be reintegrated by doing a k-way mergesort between the individual transaction. Callbacks for apply_begin,

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun20, 2012, at 22:40 , Marko Kreen wrote: > On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug wrote: >> I'm starting to think that relying on SSL/TLS for compression of >> unencrypted connections might not be such a good idea after all. We'd >> be using the protocol in a way it quite clearly nev

Re: [HACKERS] Testing 9.2 in ~production environment

2012-06-20 Thread Kevin Grittner
James Cloos wrote: >create index mb_name_own_idx on mb ( lower(name), ownerid ); > WHERE lower(name) = lower('foo@bar') AND ownerid=7; If you expect to be using an equality test on ownerid, you should put that first in the index. BTW, this is starting to sound more like something for

Re: [HACKERS] Testing 9.2 in ~production environment

2012-06-20 Thread James Cloos
As a followup, I find that I can avoid the seq scan by adding an index to that table as: create index mb_name_own_idx on mb ( lower(name), ownerid ); and changing the query from using the idiom: WHERE name ILIKE 'foo@bar' AND ownerid=7; to using: WHERE lower(name) = lower('foo@bar') AND

Re: [HACKERS] pg_prewarm

2012-06-20 Thread Cédric Villemain
Le mercredi 20 juin 2012 21:53:43, Peter Eisentraut a écrit : > On tis, 2012-04-10 at 13:29 +0200, Cédric Villemain wrote: > > I have no problem deprecating overlapping features from pgfincore as > > soon as I can do a «depend:pg_prewarm[os_warm]» :) ...It would have > > been better to split pgfi

Re: [HACKERS] Remove support in ri_triggers.c for zero-column foreign keys?

2012-06-20 Thread Kevin Grittner
Andres Freund wrote: > On Wednesday, June 20, 2012 11:15:44 PM Tom Lane wrote: >> There is a nontrivial amount of code in ri_triggers.c that is >> concerned with supporting foreign key constraints having zero >> columns. ... >> >> I think we should rip all that code out and instead put one test

Re: [HACKERS] Remove support in ri_triggers.c for zero-column foreign keys?

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 11:15:44 PM Tom Lane wrote: > There is a nontrivial amount of code in ri_triggers.c that is concerned > with supporting foreign key constraints having zero columns. ... > > I think we should rip all that code out and instead put one test into > ri_FetchConstraintInfo i

[HACKERS] Remove support in ri_triggers.c for zero-column foreign keys?

2012-06-20 Thread Tom Lane
There is a nontrivial amount of code in ri_triggers.c that is concerned with supporting foreign key constraints having zero columns. There is of course no support for this concept in the standard; the semantics given to the case in the code are made up out of whole cloth. (Basically, it says that

Re: [HACKERS] pg_prewarm

2012-06-20 Thread Cédric Villemain
> The biggest problem with pgfincore from my point of view is that it > only works under Linux, whereas I use a MacOS X machine for my > development, and there is also Windows to think about. Even if that > were fixed, though, I feel we ought to have something in the core > distribution. This pat

Re: [HACKERS] Event Triggers reduced, v1

2012-06-20 Thread Dimitri Fontaine
Robert Haas writes: >> Cant you just put it alone in a test group in the parallel_schedule? Several >> other tests do that? > > Yeah, that seems like it should work. If not, I'd say the tests > themselves are broken. I completely missed that we could do that. I don't feel bright. Of course it ju

Re: [HACKERS] CREATE FOREGIN TABLE LACUNA

2012-06-20 Thread Alvaro Herrera
The patch uses literals such as 'r' to identify the relkind values. This should be using RELKIND_RELATION et al instead -- see src/include/catalog/pg_class.h. Other than that, it seems simple enough ... -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Co

Re: [HACKERS] libpq compression

2012-06-20 Thread Marko Kreen
On Wed, Jun 20, 2012 at 10:05 PM, Florian Pflug wrote: > I'm starting to think that relying on SSL/TLS for compression of > unencrypted connections might not be such a good idea after all. We'd > be using the protocol in a way it quite clearly never was intended to > be used... Maybe, but what is

Re: [HACKERS] Event Triggers reduced, v1

2012-06-20 Thread Dimitri Fontaine
Hi, Robert Haas writes: > 1. I still think we ought to get rid of the notion of BEFORE or AFTER > (i.e. pg_event_trigger.evttype) and just make that detail part of the > event name (e.g. pg_event_trigger.evtevent). Many easily forseeable > event types will be more like "during" rather than "befo

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 10:12:46 PM Aidan Van Dyk wrote: > On Wed, Jun 20, 2012 at 3:49 PM, Andres Freund wrote: > > On Wednesday, June 20, 2012 09:41:03 PM Aidan Van Dyk wrote: > >> On Wed, Jun 20, 2012 at 3:27 PM, Andres Freund > > > > wrote: > >> >> OK, so in this case, I still don't see

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Aidan Van Dyk
On Wed, Jun 20, 2012 at 3:49 PM, Andres Freund wrote: > On Wednesday, June 20, 2012 09:41:03 PM Aidan Van Dyk wrote: >> On Wed, Jun 20, 2012 at 3:27 PM, Andres Freund > wrote: >> >> OK, so in this case, I still don't see how the "origin_id" is even >> >> enough. >> >> >> >> C applies the change o

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Greg Smith
On 06/20/2012 03:22 PM, Peter Geoghegan wrote: The situation would be made a lot better if we could just find a way to generalise pgbench a little bit more. I'm thinking about a facility for specifying new tables in scripts, with a moderate degree of flexibility as to their definition, data, and

Re: [HACKERS] pg_prewarm

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 3:53 PM, Peter Eisentraut wrote: > I'm worried about the overlap with pgfincore.  It's pretty well > established in this space, and has about 73% feature overlap with > pg_prewarm, while having more features all together.  So it would seem > to me that it would be better to

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 2:57 PM, Heikki Linnakangas wrote: > On 20.06.2012 21:41, Peter Geoghegan wrote: >> On 20 June 2012 18:42, Robert Haas  wrote: >>> On Wed, Jun 20, 2012 at 3:48 AM, Simon Riggs >>>  wrote: I'm sure Jeff submitted this because of the need for a standard test, rather

Re: [HACKERS] pg_prewarm

2012-06-20 Thread Peter Eisentraut
On tis, 2012-04-10 at 13:29 +0200, Cédric Villemain wrote: > I have no problem deprecating overlapping features from pgfincore as > soon as I can do a «depend:pg_prewarm[os_warm]» :) ...It would have > been better to split pgfincore analyze and warming parts times > ago, anyway. So pg_prewarm

Re: [HACKERS] Event Triggers reduced, v1

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 3:39 PM, Andres Freund wrote: > On Wednesday, June 20, 2012 09:33:26 PM Dimitri Fontaine wrote: >> > Hmm, I don't like the idea of a test that only runs in serial mode. >> > Maybe we can find some way to make it work in parallel mode as well. >> >> I don't see how we can ma

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 09:41:03 PM Aidan Van Dyk wrote: > On Wed, Jun 20, 2012 at 3:27 PM, Andres Freund wrote: > >> OK, so in this case, I still don't see how the "origin_id" is even > >> enough. > >> > >> C applies the change originally from A (routed through B, because it's > >> faster).

Re: [HACKERS] WAL format changes

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 20:43, Fujii Masao wrote: On Wed, Jun 20, 2012 at 8:19 PM, Magnus Hagander wrote: On Tue, Jun 19, 2012 at 5:57 PM, Robert Haas wrote: On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas wrote: Well, that was easier than I thought. Attached is a patch to make XLogRecPtr a uint

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Aidan Van Dyk
On Wed, Jun 20, 2012 at 3:27 PM, Andres Freund wrote: >> OK, so in this case, I still don't see how the "origin_id" is even enough. >> >> C applies the change originally from A (routed through B, because it's >> faster).  But when it get's the change directly from A, how does it >> know to *not*

Re: [HACKERS] Event Triggers reduced, v1

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 09:33:26 PM Dimitri Fontaine wrote: > > Hmm, I don't like the idea of a test that only runs in serial mode. > > Maybe we can find some way to make it work in parallel mode as well. > > I don't see how we can manage to do that, as adding an event trigger to > some comman

Re: [HACKERS] Event Triggers reduced, v1

2012-06-20 Thread Dimitri Fontaine
Alvaro Herrera writes: > Did you try REASSIGN OWNED and DROP OWNED with a role that has defined > some event triggers? Not yet. Will add to regression tests, good catch. > Hmm, I don't like the idea of a test that only runs in serial mode. > Maybe we can find some way to make it work in parallel

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 09:23:34 PM Heikki Linnakangas wrote: > > And then we just put the originid on each heap record for MMR, in some > > manner, discussed later. > > I reserve the right to object to that, too :-). Others raised the > concern that a 16-bit integer is not a very intuitive i

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 09:24:29 PM Aidan Van Dyk wrote: > On Wed, Jun 20, 2012 at 3:15 PM, Andres Freund wrote: > > To recap why we think origin_id is a sensible design choice: > > > > There are many sensible replication topologies where it does make sense > > that you want to receive chang

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 03:23, Heikki Linnakangas wrote: >> And then we just put the originid on each heap record for MMR, in some >> manner, discussed later. > > > I reserve the right to object to that, too :-). OK. But that would be only for MMR, using special record types. > Others raised the concer

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 03:13, Heikki Linnakangas wrote: > On 20.06.2012 21:51, Simon Riggs wrote: >> >> On 21 June 2012 02:32, Heikki Linnakangas >>  wrote: >>> >>> I'm not saying that we need to implement all possible conflict resolution >>> >>> algorithms right now - on the contrary I think conflict r

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Aidan Van Dyk
On Wed, Jun 20, 2012 at 3:15 PM, Andres Freund wrote: > To recap why we think origin_id is a sensible design choice: > > There are many sensible replication topologies where it does make sense that > you want to receive changes (on node C) from one node (say B) that originated > from some other n

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 22:11, Simon Riggs wrote: On 21 June 2012 02:56, Simon Riggs wrote: I think allowing rmgrs to redefine the wasted bytes in the header is the best idea. Hmm, I think the best idea is to save 2 bytes off the WAL header for all records, so there are no wasted bytes on 64bit or 32b

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Tom Lane
Josh Berkus writes: >> >> 4) MAP_HEADER ('column 1'-> 'col_1', 'Date' -> 'input_date' ...) >> >> to cover the case when column names do not match. > Personally, I think that's going way beyond what we want to do with > COPY. I agree --- if you know that much about the incoming data, you might

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Peter Geoghegan
On 20 June 2012 19:57, Heikki Linnakangas wrote: > Yeah, this sounds like a good approach. A library of standard workload > scripts seems very useful. I've been using custom scripts to benchmark WAL > insertion scalability lately, that also seems like a kind of a thing to put > in such a library.

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Simon Riggs
On 21 June 2012 02:57, Heikki Linnakangas wrote: > On 20.06.2012 21:41, Peter Geoghegan wrote: >> >> On 20 June 2012 18:42, Robert Haas  wrote: >>> >>> On Wed, Jun 20, 2012 at 3:48 AM, Simon Riggs >>>  wrote: I'm sure Jeff submitted this because of the need for a standard test, rath

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
Hi, On Wednesday, June 20, 2012 08:32:53 PM Heikki Linnakangas wrote: > On 20.06.2012 17:35, Simon Riggs wrote: > > On 20 June 2012 16:23, Heikki Linnakangas > > > > wrote: > >> On 20.06.2012 11:17, Simon Riggs wrote: > >>> On 20 June 2012 15:45, Heikki Linnakangas > >>> > >>> wrote: > >>>

Re: [HACKERS] pl/perl and utf-8 in sql_ascii databases

2012-06-20 Thread Alvaro Herrera
Excerpts from Alex Hunsaker's message of vie feb 10 16:53:05 -0300 2012: > Seems like we missed the fact that we still did SvUTF8_on() in sv2cstr > and SvPVUTF8() when turning a perl string into a cstring. Right. So I played a bit with this patch, and touched it a bit mainly just to add some mor

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 21:51, Simon Riggs wrote: On 21 June 2012 02:32, Heikki Linnakangas wrote: I'm not saying that we need to implement all possible conflict resolution algorithms right now - on the contrary I think conflict resolution belongs outside core It's a pretty standard requirement to hav

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 02:56, Simon Riggs wrote: > I think allowing rmgrs to redefine the wasted bytes in the header is > the best idea. Hmm, I think the best idea is to save 2 bytes off the WAL header for all records, so there are no wasted bytes on 64bit or 32bit. That way the potential for use goes

Re: [HACKERS] Testing 9.2 in ~production environment

2012-06-20 Thread James Cloos
Updating pg_database to set datctype='C' did solve the speed issues with the two largs dbs. Presumably, since LC_CTYPE=en_US.UTF-8 was in the env when I ran pg_restore, it overrode the ctype setting in the dump files. Some of the slow selects do use ilike; even w/ datctype='C' the indices are ski

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun20, 2012, at 18:42 , Magnus Hagander wrote: > That is a very good point. Before we design *another* feature that > relies on it, we should verify if the syntax is compatible in the > other libraries that would be interesting (gnutls and NSS primarily), > and if it's not that at least the *fun

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 21:41, Peter Geoghegan wrote: On 20 June 2012 18:42, Robert Haas wrote: On Wed, Jun 20, 2012 at 3:48 AM, Simon Riggs wrote: I'm sure Jeff submitted this because of the need for a standard test, rather than the wish to actually modify pgbench itself. Can I suggest that we includ

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 02:45, Heikki Linnakangas wrote: > On 20.06.2012 16:46, Simon Riggs wrote: >> >> The proposal now includes flag bits that would allow the addition of a >> variable length header, should that ever become necessary. So the >> unused space in the fixed header is not being "used up" as

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 02:32, Heikki Linnakangas wrote: > On 20.06.2012 17:35, Simon Riggs wrote: >> >> On 20 June 2012 16:23, Heikki Linnakangas >>  wrote: >>> >>> On 20.06.2012 11:17, Simon Riggs wrote: On 20 June 2012 15:45, Heikki Linnakangas    wrote: > > > So, if t

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 16:46, Simon Riggs wrote: The proposal now includes flag bits that would allow the addition of a variable length header, should that ever become necessary. So the unused space in the fixed header is not being "used up" as you say. In any case, the fixed header still has 4 wasted byt

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 07:50:37 PM Robert Haas wrote: > On Wed, Jun 20, 2012 at 1:40 PM, Andres Freund wrote: > >> I realized a problem with that idea this morning: it might work for > >> reading things, but if anyone attempts to write data you've got big > >> problems. Maybe we could get a

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Peter Geoghegan
On 20 June 2012 18:42, Robert Haas wrote: > On Wed, Jun 20, 2012 at 3:48 AM, Simon Riggs wrote: >> I'm sure Jeff submitted this because of the need for a standard test, >> rather than the wish to actually modify pgbench itself. >> >> Can I suggest that we include a list of standard scripts with p

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Heikki Linnakangas
On 20.06.2012 17:35, Simon Riggs wrote: On 20 June 2012 16:23, Heikki Linnakangas wrote: On 20.06.2012 11:17, Simon Riggs wrote: On 20 June 2012 15:45, Heikki Linnakangas wrote: So, if the origin id is not sufficient for some conflict resolution mechanisms, what extra information do yo

Re: [HACKERS] proposal and patch : support INSERT INTO...RETURNING with partitioned table using rule

2012-06-20 Thread Kevin Grittner
John Lumby wrote: > I attach patch based on clone of postgresql.git as of yesterday Please read about the CommitFest process: http://wiki.postgresql.org/wiki/CommitFest and add your patch to the open CF: https://commitfest.postgresql.org/action/commitfest_view/open This will ensure th

Re: [HACKERS] Too frequent message of pgbench -i?

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 4:04 AM, Tatsuo Ishii wrote: > Currently pgbench -i prints following message every 10k tuples created. > >                        fprintf(stderr, "%d tuples done.\n", j); > > I think it's long time ago when the frequency of message seemed to be > appropriate because compute

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Alvaro Herrera
Excerpts from Andrew Dunstan's message of mié jun 20 12:56:52 -0400 2012: > > On 06/20/2012 12:50 PM, Alvaro Herrera wrote: > > Another related case: you have a file with headers and columns (n, t, > > x, y, z) but your table only has n and t. How would you tell COPY to > > discard the junk col

Re: [HACKERS] [ADMIN] pg_basebackup blocking all queries with horrible performance

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 7:18 AM, Magnus Hagander wrote: You agreed to add something like NOSYNC option into START_REPLICATION command? >>> >>> I'm on the fence. I was hoping somebody else would chime in with an >>> opinion as well. >> >> +1 > > Nobody else with any opinion on this? :(

Re: [HACKERS] Pg default's verbosity?

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 11:25 AM, Peter Eisentraut wrote: > I don't like these messages any more than the next guy, but why drop > only those, and not any of the other NOTICE-level messages?  The meaning > of NOTICE is pretty much, if this is the first time you're using > PostgreSQL, let me tell y

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 01:06, Christopher Browne wrote: > I guess I'm not seeing the purpose to having the origin node id in the > WAL stream either. > > We have it in the Slony sl_log_* stream, however there is a crucial > difference, in that sl_log_* is expressly a shared structure.  In > contrast, WA

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
Hi Chris! On Wednesday, June 20, 2012 07:06:28 PM Christopher Browne wrote: > On Wed, Jun 20, 2012 at 11:50 AM, Andres Freund wrote: > > On Wednesday, June 20, 2012 05:34:42 PM Kevin Grittner wrote: > >> Simon Riggs wrote: > >> > This is not transaction metadata, it is WAL record metadata > >>

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 21 June 2012 01:40, Andres Freund wrote: >> I think extraction is a very sensible place to start; actually, I >> think it's the best possible place to start.  But this particular >> thread is about adding origin_ids to WAL, which I think is definitely >> not the best place to start. > Yep. I

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 1:40 PM, Andres Freund wrote: >> I realized a problem with that idea this morning: it might work for >> reading things, but if anyone attempts to write data you've got big >> problems.  Maybe we could get away with forbidding that, not sure. > Hm, why is writing a problem?

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 20 June 2012 23:56, Robert Haas wrote: > On Wed, Jun 20, 2012 at 10:08 AM, Simon Riggs wrote: > But I think getting even > single-master logical replication working well in a single release > cycle is going to be a job and a half. OK, so your estimate is 1.5 people to do

Re: [HACKERS] WAL format changes

2012-06-20 Thread Fujii Masao
On Wed, Jun 20, 2012 at 8:19 PM, Magnus Hagander wrote: > On Tue, Jun 19, 2012 at 5:57 PM, Robert Haas wrote: >> On Tue, Jun 19, 2012 at 4:14 AM, Heikki Linnakangas >> wrote: >>> Well, that was easier than I thought. Attached is a patch to make XLogRecPtr >>> a uint64, on top of my other WAL for

Re: [HACKERS] pgbench--new transaction type

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 3:48 AM, Simon Riggs wrote: > I'm sure Jeff submitted this because of the need for a standard test, > rather than the wish to actually modify pgbench itself. > > Can I suggest that we include a list of standard scripts with pgbench > for this purpose? These can then be copi

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 07:17:57 PM Robert Haas wrote: > On Wed, Jun 20, 2012 at 12:53 PM, Andres Freund wrote: > > I would prefer the event trigger way because that seems to be the most > > extensible/reusable. It would allow a fully replicated databases and > > catalog only instances. > > I

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Josh Berkus
> > 4) MAP_HEADER ('column 1'-> 'col_1', 'Date' -> 'input_date' ...) > > to cover the case when column names do not match. Personally, I think that's going way beyond what we want to do with COPY. At that point, check out the CSV-array FDW. Of course, if someone writes a WIP patch which imple

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Peter Geoghegan
On 20 June 2012 17:41, Tom Lane wrote: > Peter Geoghegan writes: >> No, I'm suggesting it would probably be at least a bit of a win here >> to cache the constant, and only have to do a strxfrm() + strcmp() per >> comparison. > > Um, have you got any hard evidence to support that notion?  The > tr

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Marc Mamin
> -Ursprüngliche Nachricht- > Von: pgsql-hackers-ow...@postgresql.org im Auftrag von Josh Berkus > Gesendet: Mi 6/20/2012 7:06 > An: pgsql-hackers@postgresql.org > Betreff: Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER ... > (1) is valuable > for backwards com

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 12:41 PM, Tom Lane wrote: >> The fact is that this is likely to be a fairly significant >> performance win, because strxfrm() is quite simply the way you're >> supposed to do collation-aware sorting, and is documented as such. For >> that reason, C standard library implemen

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 12:53 PM, Andres Freund wrote: > I would prefer the event trigger way because that seems to be the most > extensible/reusable. It would allow a fully replicated databases and catalog > only instances. > I think we need to design event triggers in a way you cannot simply cir

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Christopher Browne
On Wed, Jun 20, 2012 at 11:50 AM, Andres Freund wrote: > On Wednesday, June 20, 2012 05:34:42 PM Kevin Grittner wrote: >> Simon Riggs wrote: >> > This is not transaction metadata, it is WAL record metadata >> > required for multi-master replication, see later point. > >> > We need to add informat

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Josh Berkus
> In the past people have asked me to have copy use the CSV header line in > place of supplying a column list in the COPY command. I can certainly > see some utility in that, and I think it might achieve what David wants. > Using that scenario it would be an error to supply an explicit column > li

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Simon Riggs
On 20 June 2012 23:34, Kevin Grittner wrote: > Simon Riggs wrote: >> Kevin Grittner wrote: Heikki Linnakangas  wrote: >>> I don't like the idea of adding the origin id to the record header. It's only required in some occasions, and on some record types. >>> >>> Right. >> >> W

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Andrew Dunstan
On 06/20/2012 12:50 PM, Alvaro Herrera wrote: Another related case: you have a file with headers and columns (n, t, x, y, z) but your table only has n and t. How would you tell COPY to discard the junk columns? Currently it just complains that they are there. That's one of the main use ca

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Andrew Dunstan
On 06/20/2012 12:18 PM, Tom Lane wrote: David Fetter writes: OK, new proposal: COPY FROM (Thanks, Andrew! Must not post while asleep...) should have an option which requires that HEADER be enabled and mandates that the column names in the header match the columns coming in. Has someone got a

Re: [HACKERS] foreign key locks

2012-06-20 Thread Tom Lane
Jaime Casanova writes: > On Thu, Jun 14, 2012 at 11:41 AM, Alvaro Herrera > wrote: >> This is v12 of the foreign key locks patch. > Just noticed that this patch needs a rebase because of the refactoring > Tom did in ri_triggers.c Hold on a bit before you work on that code --- I've got one more

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
Hi, On Wednesday, June 20, 2012 05:44:09 PM Robert Haas wrote: > On Wed, Jun 20, 2012 at 10:02 AM, Andres Freund wrote: > > Were not the only ones here that are performing scope creep though... I > > think about all people who have posted in the whole thread except maybe > > Tom and Marko are gu

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Alvaro Herrera
Excerpts from David Fetter's message of mié jun 20 12:43:31 -0400 2012: > On Wed, Jun 20, 2012 at 12:18:45PM -0400, Tom Lane wrote: > > In your original proposal, I was rather wondering what should happen > > if the incoming file didn't have the same set of columns called out > > in the COPY comm

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread David Fetter
On Wed, Jun 20, 2012 at 12:18:45PM -0400, Tom Lane wrote: > David Fetter writes: > > OK, new proposal: > > > COPY FROM (Thanks, Andrew! Must not post while asleep...) should have > > an option which requires that HEADER be enabled and mandates that the > > column names in the header match the co

Re: [HACKERS] libpq compression

2012-06-20 Thread Magnus Hagander
On Wed, Jun 20, 2012 at 12:35 PM, Florian Pflug wrote: > On Jun19, 2012, at 17:36 , Robert Haas wrote: >> On Mon, Jun 18, 2012 at 1:42 PM, Martijn van Oosterhout >> wrote: >>> On Sun, Jun 17, 2012 at 12:29:53PM -0400, Tom Lane wrote: The fly in the ointment with any of these ideas is that th

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Tom Lane
Peter Geoghegan writes: > No, I'm suggesting it would probably be at least a bit of a win here > to cache the constant, and only have to do a strxfrm() + strcmp() per > comparison. Um, have you got any hard evidence to support that notion? The traditional advice is that strcoll is faster than us

Re: [HACKERS] libpq compression

2012-06-20 Thread Alvaro Herrera
Excerpts from Tom Lane's message of mié jun 20 11:49:51 -0400 2012: > > Alvaro Herrera writes: > > I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain > > the list of ciphers starting from the stringified configuration > > parameter and iterate on them. > > Do you mean that

[HACKERS] proposal and patch : support INSERT INTO...RETURNING with partitioned table using rule

2012-06-20 Thread John Lumby
--- Problem I'm trying to solve:     For partitioned tables,  make it possible to use RETURNING clause on INSERT INTO    together with DO INSTEAD rule [  Note  -  wherever I say INSERT I also mean UPDATE and DELETE ] --- Cu

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Tom Lane
David Fetter writes: > OK, new proposal: > COPY FROM (Thanks, Andrew! Must not post while asleep...) should have > an option which requires that HEADER be enabled and mandates that the > column names in the header match the columns coming in. > Has someone got a better name for this option than

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread David Fetter
On Wed, Jun 20, 2012 at 11:47:14AM -0400, Tom Lane wrote: > David Fetter writes: > > Rather than being totally ignored in the COPY OUT (CSV HEADER) > > case, the header line in should be parsed to establish which > > columns are where and rearranging the output if needed. > > This is not "fix a P

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Fujii Masao
On Wed, Jun 20, 2012 at 12:40 PM, Amit Kapila wrote: >>> I believe if WAL files are proper as mentioned in Alvaro's mail, the > purposed logic should generate >>> correct values. >>> Do you see any problem in logic purposed in my original mail. >>> Can I resume my work on this feature? > >> Maybe

Re: [HACKERS] sortsupport for text

2012-06-20 Thread Peter Geoghegan
On 20 June 2012 15:55, Tom Lane wrote: > Peter Geoghegan writes: >> I think that this change may have made the difference between the >> Hungarians getting away with it and not getting away with it. Might it >> have been that for text, they were using some operator that wasn't '=' >> (perhaps one

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 10:08 AM, Simon Riggs wrote: But I think getting even single-master logical replication working well in a single release cycle is going to be a job and a half. >>> >>> OK, so your estimate is 1.5 people to do that. And if we have more >>> people, should they

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Andres Freund
On Wednesday, June 20, 2012 05:34:42 PM Kevin Grittner wrote: > Simon Riggs wrote: > > This is not transaction metadata, it is WAL record metadata > > required for multi-master replication, see later point. > > We need to add information to every WAL record that is used as the > > source for gene

Re: [HACKERS] libpq compression

2012-06-20 Thread Tom Lane
Alvaro Herrera writes: > I looked at the code (apps/ciphers.c) and it looks pretty easy to obtain > the list of ciphers starting from the stringified configuration > parameter and iterate on them. Do you mean that it will produce an expansion of the set of ciphers meeting criteria like "!aNULL"?

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Andrew Dunstan
On 06/20/2012 11:02 AM, David Fetter wrote: Folks, A co-worker filed a bug against file_fdw where the columns in a FOREIGN TABLE were scrambled on SELECT. It turned out that this comes from the (yes, it's documented, but since it's documented in a place not obviously linked to the bug, it's p

Re: [HACKERS] Nasty, propagating POLA violation in COPY CSV HEADER

2012-06-20 Thread Tom Lane
David Fetter writes: > Rather than being totally ignored in the COPY OUT (CSV HEADER) case, > the header line in should be parsed to establish which columns are > where and rearranging the output if needed. This is not "fix a POLA violation". This is a non-backwards-compatible new feature, which

Re: [HACKERS] libpq compression

2012-06-20 Thread Florian Pflug
On Jun20, 2012, at 17:34 , Tom Lane wrote: > Florian Pflug writes: >> I wonder though if shouldn't restrict the allowed ciphers list to being >> a simple list of supported ciphers. If our goal is to support multiple >> SSL libraries transparently then surely having openssl-specific syntax >> in th

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Robert Haas
On Wed, Jun 20, 2012 at 10:02 AM, Andres Freund wrote: > Were not the only ones here that are performing scope creep though... I think > about all people who have posted in the whole thread except maybe Tom and > Marko are guilty of doing so. > > I still think its rather sensible to focus on exact

Re: [HACKERS] [PATCH 10/16] Introduce the concept that wal has a 'origin' node

2012-06-20 Thread Kevin Grittner
Simon Riggs wrote: > Kevin Grittner wrote: >>> Heikki Linnakangas wrote: >> >>> I don't like the idea of adding the origin id to the record >>> header. It's only required in some occasions, and on some record >>> types. >> >> Right. > > Wrong, as explained. The point is not wrong; you are sim

Re: [HACKERS] libpq compression

2012-06-20 Thread Tom Lane
Florian Pflug writes: > I wonder though if shouldn't restrict the allowed ciphers list to being > a simple list of supported ciphers. If our goal is to support multiple > SSL libraries transparently then surely having openssl-specific syntax > in the config file isn't exactly great anyway... No,

Re: [HACKERS] Pg default's verbosity?

2012-06-20 Thread Peter Eisentraut
On tis, 2012-06-19 at 02:15 -0400, Tom Lane wrote: > Robert Haas writes: > > There might be something to the idea of demoting a few of the things > > we've traditionally had as NOTICEs, though. IME, the following two > > messages account for a huge percentage of the chatter: > > > NOTICE: CREAT

Re: [HACKERS] Release versioning inconsistency

2012-06-20 Thread Tom Lane
Peter Eisentraut writes: > On ons, 2012-06-20 at 13:26 +0200, Magnus Hagander wrote: >> That might actually be a good idea. We can't really change the way we >> named the betas, but it's not too late to consider naming the actual >> release as 9.2.0... > The final release was always going to be c

Re: [HACKERS] Allow WAL information to recover corrupted pg_controldata

2012-06-20 Thread Tom Lane
Amit Kapila writes: >> I'm almost inclined to suggest that we not get next-LSN from WAL, but >> by scanning all the pages in the main data store and computing the max >> observed LSN. This is clearly not very attractive from a performance >> standpoint, but it would avoid the obvious failure mode

  1   2   >