Re: [HACKERS] pg_controldata gobbledygook

2013-04-26 Thread Daniel Farina
On Thu, Apr 25, 2013 at 9:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: I think I've heard of scripts grepping the output of pg_controldata for this that or the other. Any rewording of the labels would break that. While I'm not

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-26 Thread Heikki Linnakangas
On 26.04.2013 07:02, Kyotaro HORIGUCHI wrote: I am uncertain a bit weather it is necessary to move curFileTLI to anywhere randomly read . On a short glance, the random access occurs also for reading checkpoint-related records. I didn't understand that. Also I don't have clear distinction

Re: [HACKERS] event trigger API documentation?

2013-04-26 Thread Dimitri Fontaine
Peter Eisentraut pete...@gmx.net writes: It seems pretty straightforward and useful, so I'm not sure where your hesitation is coming from. If you're talking about my hesitation to consider what we have now as marketing material worthy, it comes from the fact that I don't have a use case where I

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-26 Thread Amit Langote
What would happen if a recycled segment gets assigned a newer timeline than the one we are currently recovering from? In the reported erroneous behavior, that is what happens causing XLogFileReadAnyTLI() to return such bogus segment causing the error. Since, expectedTLIs contains a newer timeline

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-26 Thread Heikki Linnakangas
On 26.04.2013 07:47, Amit Langote wrote: How would code after applying this patch behave if a recycled segment gets renamed using the newest timeline (say 3) while we are still recovering from a lower timeline (say 2)? In that case, since XLogFileReadAnyTLI returns that recycled segment as the

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Ants Aasma
On Apr 25, 2013 10:38 PM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-04-23 at 11:44 +0300, Ants Aasma wrote: I will try to reword. Did you have a chance to clarify this, as well as some of the other documentation issues Simon mentioned here?

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-26 Thread KONDO Mitsumasa
Hi, I discavered the problem cause. I think taht horiguchi's discovery is another problem... Problem has CreateRestartPoint. In recovery mode, PG should not WAL record. Because PG does not know latest WAL file location. But in this problem case, PG(standby) write WAL file at RestartPoint in

Re: [HACKERS] pg_controldata gobbledygook

2013-04-26 Thread Gavin Flower
On 26/04/13 18:53, Daniel Farina wrote: On Thu, Apr 25, 2013 at 9:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Tom Lane wrote: I think I've heard of scripts grepping the output of pg_controldata for this that or the other. Any rewording of the

Re: [HACKERS] pg_controldata gobbledygook

2013-04-26 Thread Andres Freund
On 2013-04-25 23:07:02 -0400, Peter Eisentraut wrote: I'm not sure who is supposed to be able to read this sort of stuff: Latest checkpoint's NextXID: 0/7575 Latest checkpoint's NextOID: 49152 Latest checkpoint's NextMultiXactId: 7 Latest checkpoint's NextMultiOffset: 13

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 18 April 2013 19:11, Heikki Linnakangas hlinnakan...@vmware.com wrote: I just found out that if you use continuous archiving and online backups, it's surprisingly difficult to restore a backup, without replaying any more WAL than necessary. I didn't add it myself because I don't see the

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Heikki Linnakangas
On 26.04.2013 12:16, Simon Riggs wrote: On 18 April 2013 19:11, Heikki Linnakangashlinnakan...@vmware.com wrote: I just found out that if you use continuous archiving and online backups, it's surprisingly difficult to restore a backup, without replaying any more WAL than necessary. I didn't

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Florian Pflug
On Apr26, 2013, at 10:28 , Ants Aasma ants.aa...@eesti.ee wrote: On Apr 25, 2013 10:38 PM, Jeff Davis pg...@j-davis.com wrote: On Tue, 2013-04-23 at 11:44 +0300, Ants Aasma wrote: I will try to reword. Did you have a chance to clarify this, as well as some of the other documentation

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Andres Freund
On 2013-04-26 13:11:00 +0200, Florian Pflug wrote: The unresolved code issue that I know of is moving the compiler flags behind a configure check. I would greatly appreciate it if you could take a look at that. My config-fu is weak and it would take me some time to figure out how to do

Re: [HACKERS] pg_controldata gobbledygook

2013-04-26 Thread Bernd Helmle
--On 25. April 2013 23:19:14 -0400 Tom Lane t...@sss.pgh.pa.us wrote: I think I've heard of scripts grepping the output of pg_controldata for this that or the other. Any rewording of the labels would break that. While I'm not opposed to improving the labels, I would vote against your second,

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 26 April 2013 11:29, Heikki Linnakangas hlinnakan...@vmware.com wrote: But there is also an operation to simply restore a backup. Just because a tool supports an imprecise definition of a restore, doesn't mean Postgres should encourage and support that. Restore a backup is more suited to

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Magnus Hagander
On Fri, Apr 26, 2013 at 1:47 PM, Simon Riggs si...@2ndquadrant.com wrote: On 26 April 2013 11:29, Heikki Linnakangas hlinnakan...@vmware.com wrote: But there is also an operation to simply restore a backup. Just because a tool supports an imprecise definition of a restore, doesn't mean

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 26 April 2013 12:54, Magnus Hagander mag...@hagander.net wrote: That said, maybe the easier choice for a *system* (such as v-thingy) would be to simply to the full backup using pg_basebackup -x (or similar), therefor not needing the log archive at all when restoring. Yes, it makes the base

Re: [HACKERS] pg_controldata gobbledygook

2013-04-26 Thread Robert Haas
On Fri, Apr 26, 2013 at 5:08 AM, Andres Freund and...@2ndquadrant.com wrote: I have to admit I don't see the point. None of those values is particularly interesting to anybody without implementation level knowledge and those will likely deal with them just fine. And I find the version with the

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Heikki Linnakangas
On 26.04.2013 14:54, Magnus Hagander wrote: On Fri, Apr 26, 2013 at 1:47 PM, Simon Riggssi...@2ndquadrant.com wrote: On 26 April 2013 11:29, Heikki Linnakangashlinnakan...@vmware.com wrote: But there is also an operation to simply restore a backup. Just because a tool supports an

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2013-04-26 13:11:00 +0200, Florian Pflug wrote: The unresolved code issue that I know of is moving the compiler flags behind a configure check. I would greatly appreciate it if you could take a look at that. My config-fu is weak and it would

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes: That said, maybe the easier choice for a *system* (such as v-thingy) would be to simply to the full backup using pg_basebackup -x (or similar), therefor not needing the log archive at all when restoring. Yes, it makes the base backup slightly larger,

Re: [HACKERS] Functional dependencies and GROUP BY - for subqueries

2013-04-26 Thread Ashutosh Bapat
Hi All, If group by clause has primary key, the targetlist may have columns which are not part of the aggregate and not part of group by clause. The relevant commit is e49ae8d3bc588294d07ce1a1272b31718cfca5ef and relevant mail thread has subject Functional dependencies and GROUP BY. As a result,

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 26 April 2013 14:48, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: That said, maybe the easier choice for a *system* (such as v-thingy) would be to simply to the full backup using pg_basebackup -x (or similar), therefor not needing the log archive at all

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Simon Riggs
On 26 April 2013 14:40, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: On 2013-04-26 13:11:00 +0200, Florian Pflug wrote: The unresolved code issue that I know of is moving the compiler flags behind a configure check. I would greatly appreciate it if you could

Re: [HACKERS] Functional dependencies and GROUP BY - for subqueries

2013-04-26 Thread Tom Lane
Ashutosh Bapat ashutosh.ba...@enterprisedb.com writes: The reason being, it doesn't look into the subqueries (in FROM clause) to infer that p.product_id is essentially product.product_id which is a primary key. Right. Attached find a crude patch to infer the same by traversing subqueries. I

Re: [HACKERS] libpq COPY handling

2013-04-26 Thread Robert Haas
On Thu, Apr 25, 2013 at 1:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, the documentation in libpq.sgml is a bit bogus too, because it counsels trying the PQputCopyEnd() call again, which will not work (since we already changed the asyncStatus). We could make that say a zero result is

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Robert Haas
On Fri, Apr 26, 2013 at 10:05 AM, Simon Riggs si...@2ndquadrant.com wrote: Restore points are definitely the way to go here, this is what they were created for. Stopping at a labelled location has a defined meaning for the user, which is much better than just stop anywhere convenient, which I

Re: [HACKERS] libpq COPY handling

2013-04-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Apr 25, 2013 at 1:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, the documentation in libpq.sgml is a bit bogus too, because it counsels trying the PQputCopyEnd() call again, which will not work (since we already changed the asyncStatus).

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Magnus Hagander
On Apr 26, 2013 4:38 PM, Robert Haas robertmh...@gmail.com wrote: On Fri, Apr 26, 2013 at 10:05 AM, Simon Riggs si...@2ndquadrant.com wrote: Restore points are definitely the way to go here, this is what they were created for. Stopping at a labelled location has a defined meaning for the

Re: [HACKERS] Failing start-up archive recovery at Standby mode in PG9.2.4

2013-04-26 Thread Mitsumasa KONDO
I explain more detail about this problem. This problem was occurred by RestartPoint create illegal WAL file in during archive recovery. But I cannot recognize why illegal WAL file was created in CreateRestartPoint(). My attached patch is really plain… In problem case at XLogFileReadAnyTLI(),

Re: [HACKERS] libpq COPY handling

2013-04-26 Thread Robert Haas
On Fri, Apr 26, 2013 at 10:48 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Apr 25, 2013 at 1:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, the documentation in libpq.sgml is a bit bogus too, because it counsels trying the PQputCopyEnd() call

Re: [HACKERS] Functional dependencies and GROUP BY - for subqueries

2013-04-26 Thread Ashutosh Bapat
On Fri, Apr 26, 2013 at 7:54 PM, Tom Lane t...@sss.pgh.pa.us wrote: Ashutosh Bapat ashutosh.ba...@enterprisedb.com writes: The reason being, it doesn't look into the subqueries (in FROM clause) to infer that p.product_id is essentially product.product_id which is a primary key. Right.

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 26 April 2013 15:38, Robert Haas robertmh...@gmail.com wrote: On Fri, Apr 26, 2013 at 10:05 AM, Simon Riggs si...@2ndquadrant.com wrote: Restore points are definitely the way to go here, this is what they were created for. Stopping at a labelled location has a defined meaning for the user,

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Robert Haas
On Fri, Apr 26, 2013 at 11:35 AM, Simon Riggs si...@2ndquadrant.com wrote: Given that I was describing how we might implement Heikki's suggestion, I find this comment confusing. Please explain. Heikki's suggestion is simply to have a mode that stops as soon as consistency is reached. The

Re: [HACKERS] danger of stats_temp_directory = /dev/shm

2013-04-26 Thread Robert Haas
On Thu, Apr 25, 2013 at 12:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Jeff Janes escribió: With the stats file split patch 187492b6c2e8cafc5 introduced in 9.3dev, now after a crash the postmaster will try to delete all files in the directory

Re: [HACKERS] [ADMIN] Simultaneous index creates on different schemas cause deadlock?

2013-04-26 Thread Paul Hinze
On Thu, Apr 25, 2013 at 12:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: The cause is that each one will wait for all older snapshots to be gone --- and it does that before dropping its own snapshot, so that the other ones will see it as something to be waited out too. This makes sense. Thank you

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 26 April 2013 16:38, Robert Haas robertmh...@gmail.com wrote: On Fri, Apr 26, 2013 at 11:35 AM, Simon Riggs si...@2ndquadrant.com wrote: Given that I was describing how we might implement Heikki's suggestion, I find this comment confusing. Please explain. Heikki's suggestion is simply to

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Heikki Linnakangas
On 26.04.2013 19:05, Simon Riggs wrote: On 26 April 2013 16:38, Robert Haasrobertmh...@gmail.com wrote: On Fri, Apr 26, 2013 at 11:35 AM, Simon Riggssi...@2ndquadrant.com wrote: Given that I was describing how we might implement Heikki's suggestion, I find this comment confusing. Please

Re: [HACKERS] pg_controldata gobbledygook

2013-04-26 Thread Jeff Janes
On Fri, Apr 26, 2013 at 2:08 AM, Andres Freund and...@2ndquadrant.comwrote: On 2013-04-25 23:07:02 -0400, Peter Eisentraut wrote: I'm not sure who is supposed to be able to read this sort of stuff: Latest checkpoint's NextXID: 0/7575 Latest checkpoint's NextOID: 49152

Re: [HACKERS] Functional dependencies and GROUP BY - for subqueries

2013-04-26 Thread Tom Lane
Ashutosh Bapat ashutosh.ba...@enterprisedb.com writes: On Fri, Apr 26, 2013 at 7:54 PM, Tom Lane t...@sss.pgh.pa.us wrote: A larger point is that the patch as proposed doesn't fix the stated problem, because it only descends into written-out subqueries. It would only succeed at looking into

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Robert Haas
On Fri, Apr 26, 2013 at 12:25 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Doing it the other way means you need to add a new kind of recovery target to the API just for this. recovery_target_immediate = on Sounds good to me. Yeah, I don't have a problem with that, at all.

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Simon Riggs
On 26 April 2013 17:25, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 26.04.2013 19:05, Simon Riggs wrote: On 26 April 2013 16:38, Robert Haasrobertmh...@gmail.com wrote: On Fri, Apr 26, 2013 at 11:35 AM, Simon Riggssi...@2ndquadrant.com wrote: Given that I was describing how we

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Magnus Hagander
On Fri, Apr 26, 2013 at 6:43 PM, Simon Riggs si...@2ndquadrant.com wrote: On 26 April 2013 17:25, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 26.04.2013 19:05, Simon Riggs wrote: On 26 April 2013 16:38, Robert Haasrobertmh...@gmail.com wrote: On Fri, Apr 26, 2013 at 11:35 AM, Simon

Re: [HACKERS] Recovery target 'immediate'

2013-04-26 Thread Heikki Linnakangas
On 26.04.2013 19:50, Magnus Hagander wrote: On Fri, Apr 26, 2013 at 6:43 PM, Simon Riggssi...@2ndquadrant.com wrote: On 26 April 2013 17:25, Heikki Linnakangashlinnakan...@vmware.com wrote: Actually, from a usability point of view I think would be nice to have just one setting,

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-04-26 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/08/2013 08:34 AM, Dimitri Fontaine wrote: Joe Conway m...@joeconway.com writes: OK, maybe I'll try to take a look in the meantime. That would be awesome :) Did you have any comment on the other pg_dump patch (reviewed by Vibhor)?

Re: [HACKERS] pg_dump with postgis extension dumps rules separately

2013-04-26 Thread Dimitri Fontaine
Joe Conway m...@joeconway.com writes: Committed back to 9.1 Thanks, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Bug Fix: COLLATE with multiple ORDER BYs in aggregates

2013-04-26 Thread Tom Lane
David Fetter da...@fetter.org writes: While testing the upcoming FILTER clause for aggregates, Erik Rijkers uncovered a long-standing bug in $subject, namely that this case wasn't handled. Please find attached a patch by Andrew Gierth and myself which fixes this issue and adds a regression

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Jeff Davis
be specified by the user by configuring with CFLAGS_EXTRA=-msse4.1. I don't know of any more required changes, aside from documentation improvements. Regards, Jeff Davis fnv-jeff-20130426.patch Description: Binary data fnv-jeff-20130426-cflags-extra.patch Description: Binary data -- Sent

Re: [HACKERS] libpq COPY handling

2013-04-26 Thread Gavin Flower
On 27/04/13 02:48, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Apr 25, 2013 at 1:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, the documentation in libpq.sgml is a bit bogus too, because it counsels trying the PQputCopyEnd() call again, which will not work (since we

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Greg Smith
On 4/26/13 3:57 PM, Jeff Davis wrote: The second patch adds the configure-time check for the right compilation flags, and uses them when compiling checksum.c. I called the new variable CFLAGS_EXTRA, for lack of a better idea, so feel free to come up with a new name. It doesn't check for, or use,

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Jeff Davis
On Fri, 2013-04-26 at 16:40 -0400, Greg Smith wrote: I think I need to do two baselines: master without checksums, and master with extra optimizations but still without checksums. It may be the case that using better compile time optimizations gives a general speedup that's worth

[HACKERS] pg_ctl non-idempotent behavior change

2013-04-26 Thread Jeff Janes
After 87306184580c9c49717, if the postmaster dies without cleaning up (i.e. power outage), running pg_ctl start just gives this message and then exits: pg_ctl: another server might be running Under the old behavior, it would try to start the server anyway, and succeed, then go through recovery

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Andres Freund
On 2013-04-26 12:57:09 -0700, Jeff Davis wrote: I updated the patch and split it into two parts (attached). The second patch adds the configure-time check for the right compilation flags, and uses them when compiling checksum.c. I called the new variable CFLAGS_EXTRA, for lack of a better

[HACKERS] Re: [COMMITTERS] pgsql: Fix collation assignment for aggregates with ORDER BY.

2013-04-26 Thread David Fetter
On Fri, Apr 26, 2013 at 07:49:47PM +, Tom Lane wrote: Fix collation assignment for aggregates with ORDER BY. ORDER BY expressions were being treated the same as regular aggregate arguments for purposes of collation determination, but really they should not affect the aggregate's

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Jeff Davis
On Sat, 2013-04-27 at 00:20 +0200, Andres Freund wrote: CFLAGS_VECTORIZATION? EXTRA sounds to generic to me. I went with CFLAGS_VECTOR to be a little shorter while still keeping some meaning. I think it would be better to have a PGAC_PROG_CC_VAR_OPT or so which assigns the flag to some passed

Re: [HACKERS] Re: [COMMITTERS] pgsql: Fix collation assignment for aggregates with ORDER BY.

2013-04-26 Thread Jeff Davis
On Fri, 2013-04-26 at 16:46 -0700, David Fetter wrote: On Fri, Apr 26, 2013 at 07:49:47PM +, Tom Lane wrote: Given this risk and the lack of field complaints about the issue, it doesn't seem prudent to back-patch. ... This needs back-patching to 9.1, where the bug was introduced. It

Re: [HACKERS] Re: [COMMITTERS] pgsql: Fix collation assignment for aggregates with ORDER BY.

2013-04-26 Thread David Fetter
On Fri, Apr 26, 2013 at 05:52:03PM -0700, Jeff Davis wrote: On Fri, 2013-04-26 at 16:46 -0700, David Fetter wrote: On Fri, Apr 26, 2013 at 07:49:47PM +, Tom Lane wrote: Given this risk and the lack of field complaints about the issue, it doesn't seem prudent to back-patch. ...

[HACKERS] exactly what is COPY BOTH mode supposed to do in case of an error?

2013-04-26 Thread Robert Haas
It seems the backend and libpq don't agree. The backend makes no special provision to wait for a CopyDone message if an error occurs during copy-both. It simply sends an ErrorResponse and that's it. libpq, on the other hand, treats either CopyDone or ErrorResponse as a cue to transition to

[HACKERS] Assert's vs elog ERROR vs elog FATAL

2013-04-26 Thread Daniel Wood
These two questions are about the correct coding practice in Postgresql vs the specifics of xact.c Is the main difference between: if (s-blockState != TBLOCK_SUBINPROGESS) elog(*FATAL*, ... vs Assert(s-blockState == TBLOCK_SUBINPROGRESS); the fact that in both cases: a)

Re: [HACKERS] Assert's vs elog ERROR vs elog FATAL

2013-04-26 Thread Tom Lane
Daniel Wood dw...@salesforce.com writes: Is the main difference between: if (s-blockState != TBLOCK_SUBINPROGESS) elog(*FATAL*, ... vs Assert(s-blockState == TBLOCK_SUBINPROGRESS); the fact that in both cases: a) the situation is unexpected, as in no user code can