Re: [HACKERS] Performance problem in PLPgSQL

2013-07-24 Thread Amit Kapila
On Wednesday, July 24, 2013 4:23 AM Tom Lane wrote: I wrote: Marc Cousin cousinm...@gmail.com writes: The example below is of course extremely simplified, and obviously not what we are really doing in the database, but it exhibits the slowdown between 9.1.9 and 9.2.4. Hm. Some

[HACKERS] Backup throttling

2013-07-24 Thread Antonin Houska
Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. // Antonin Houska (Tony) diff --git a/doc/src/sgml/ref/pg_basebackup.sgml b/doc/src/sgml/ref/pg_basebackup.sgml index eb0c1d6..3b7ecfd 100644 --- a/doc/src/sgml/ref/pg_basebackup.sgml

Re: [HACKERS] Performance problem in PLPgSQL

2013-07-24 Thread Amit Kapila
On Wednesday, July 24, 2013 11:38 AM Amit Kapila wrote: On Wednesday, July 24, 2013 4:23 AM Tom Lane wrote: I wrote: Marc Cousin cousinm...@gmail.com writes: The example below is of course extremely simplified, and obviously not what we are really doing in the database, but it

[HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread MauMau
Hello, I've encountered a bug of PITR that corrupts the database. I'm willing to submit the patch to fix it, but I'm wondering what approach is appropriate. Could you give me your opinions? [Problem] I cannot connect to the database after performing the following steps: 1. CREATE DATABASE

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Andres Freund
On 2013-07-24 19:30:09 +0900, MauMau wrote: I've encountered a bug of PITR that corrupts the database. I'm willing to submit the patch to fix it, but I'm wondering what approach is appropriate. Could you give me your opinions? [Problem] I cannot connect to the database after performing the

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Andres Freund
On 2013-07-24 12:59:43 +0200, Andres Freund wrote: Approach 2 Like the DROP TABLE/INDEX case, piggyback the directory deletion record on the transaction commit record, and eliminate the directory deletion record altogether. I don't think burdening commit records with that makes sense.

Re: [HACKERS] Backup throttling

2013-07-24 Thread Fujii Masao
On Wed, Jul 24, 2013 at 4:20 PM, Antonin Houska antonin.hou...@gmail.com wrote: Hello, the purpose of this patch is to limit impact of pg_backup on running server. Feedback is appreciated. Interesting. Please add this patch to the next commitfeat.

[HACKERS] DATE type output does not follow datestyle parameter

2013-07-24 Thread MauMau
Hello, The description of datestyle parameter does not seem to match the actual behavior. Is this a bug to be fixed? Which do you think should be corrected, the program or the manual? The manual says: DateStyle (string) Sets the display format for date and time values, as well as the

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Heikki Linnakangas
Andres Freund and...@2ndquadrant.com wrote: On 2013-07-24 12:59:43 +0200, Andres Freund wrote: Approach 2 Like the DROP TABLE/INDEX case, piggyback the directory deletion record on the transaction commit record, and eliminate the directory deletion record altogether. I don't think

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Andres Freund
On 2013-07-24 15:45:52 +0300, Heikki Linnakangas wrote: Andres Freund and...@2ndquadrant.com wrote: On 2013-07-24 12:59:43 +0200, Andres Freund wrote: Approach 2 What we imo could do would be to drop the tablespaces in a *separate* transaction *after* the transaction that removed the

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Heikki Linnakangas
Andres Freund and...@2ndquadrant.com wrote: On 2013-07-24 15:45:52 +0300, Heikki Linnakangas wrote: Andres Freund and...@2ndquadrant.com wrote: On 2013-07-24 12:59:43 +0200, Andres Freund wrote: Approach 2 What we imo could do would be to drop the tablespaces in a *separate* transaction

Re: [HACKERS] Suggestion for concurrent index creation using a single full scan operation

2013-07-24 Thread Tim Kane
Wow.. thanks guys, really appreciate the detailed analysis. Tim On Wed, Jul 24, 2013 at 4:08 AM, Noah Misch n...@leadboat.com wrote: On Tue, Jul 23, 2013 at 01:06:26PM +0100, Tim Kane wrote: I haven't given this a lot of thought, but it struck me that when rebuilding tables (be it for a

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: That's no different from CREATE TABLE / INDEX and DROP TABLE / INDEX. E.g. If you crash after CREATE TABLE but before COMMIT, the file is leaked. But it's just a waste of space, everything still works. Well, it is different, because if you

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Andres Freund
On 2013-07-24 16:15:59 +0300, Heikki Linnakangas wrote: For DROP DATABASE, without something like incomplete actions, piggybacking on the commit record doesn't solve the issue of CHECKPOINTS either, because the commit record you piggybacked on could have committed before a checkpoint, while

Re: [HACKERS] [GENERAL] Insert result does not match record count

2013-07-24 Thread Vik Fearing
On 07/22/2013 06:20 PM, Jeff Janes wrote: On Fri, Jul 19, 2013 at 3:20 PM, Natalie Wenz nataliew...@ebureau.com wrote: Hi all, I am moving some data from one table to another in 9.2.4, and keep seeing this strange scenario: insert into newtable select data from oldtable where proc_date = x

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Tom Lane
I wrote: The only thing here that really bothers me is that a crash during DROP DATABASE/TABLESPACE could leave us with a partially populated db/ts that's still accessible through the system catalogs. ... I guess one thing we could do is create a flag file, say dead.dont.use, in the

Re: [HACKERS] [bug fix] PITR corrupts the database cluster

2013-07-24 Thread Andres Freund
On 2013-07-24 10:57:14 -0400, Tom Lane wrote: I wrote: The only thing here that really bothers me is that a crash during DROP DATABASE/TABLESPACE could leave us with a partially populated db/ts that's still accessible through the system catalogs. ... I guess one thing we could do is

[HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Tom Lane
So I went off to implement the SPITupleTable tracking discussed in 6553.1374424...@sss.pgh.pa.us, and thought it would be cool to use the slist infrastructure defined in lib/ilist.h rather than creating a separate List node for each SPITupleTable struct. However, I soon ran into a problem: there's

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Andres Freund
On 2013-07-24 11:26:00 -0400, Tom Lane wrote: So I went off to implement the SPITupleTable tracking discussed in 6553.1374424...@sss.pgh.pa.us, and thought it would be cool to use the slist infrastructure defined in lib/ilist.h rather than creating a separate List node for each SPITupleTable

Re: [HACKERS] [GENERAL] Insert result does not match record count

2013-07-24 Thread Vik Fearing
On 07/24/2013 04:04 PM, Vik Fearing wrote: On 07/22/2013 06:20 PM, Jeff Janes wrote: On Fri, Jul 19, 2013 at 3:20 PM, Natalie Wenz nataliew...@ebureau.com wrote: Hi all, I am moving some data from one table to another in 9.2.4, and keep seeing this strange scenario: insert into newtable

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2013-07-24 11:26:00 -0400, Tom Lane wrote: So I'm going to end up hand-implementing the same kind of manipulation we frequently use with traditional Lists, namely keep a second variable that's the preceding list element (not the next one) so I

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Alvaro Herrera
Andres Freund wrote: On 2013-07-24 11:26:00 -0400, Tom Lane wrote: So I'm going to end up hand-implementing the same kind of manipulation we frequently use with traditional Lists, namely keep a second variable that's the preceding list element (not the next one) so I can unlink and

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Andres Freund
On 2013-07-24 11:53:39 -0400, Alvaro Herrera wrote: Andres Freund wrote: On 2013-07-24 11:26:00 -0400, Tom Lane wrote: So I'm going to end up hand-implementing the same kind of manipulation we frequently use with traditional Lists, namely keep a second variable that's the preceding

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Alvaro Herrera
Andres Freund wrote: On 2013-07-24 11:53:39 -0400, Alvaro Herrera wrote: Andres Freund wrote: Amusingly, this will mean ForgetBackgroundWorker() will need to have a slist_mutable_iter * argument if it wants to use the new function ... The only alternative I see would be to pass both the

[HACKERS] dynamic background workers, round two

2013-07-24 Thread Robert Haas
The dynamic background workers patch that I submitted for CF1 was generally well-received, but several people commented on a significant limitation: there's currently no way for a backend that requests a new background worker to know whether that background worker was successfully started. If

Re: [HACKERS] Design proposal: fsync absorb linear slider

2013-07-24 Thread Robert Haas
On Tue, Jul 23, 2013 at 12:13 PM, Greg Smith g...@2ndquadrant.com wrote: On 7/23/13 10:56 AM, Robert Haas wrote: On Mon, Jul 22, 2013 at 11:48 PM, Greg Smith g...@2ndquadrant.com wrote: We know that a 1GB relation segment can take a really long time to write out. That could include up to 128

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Robert Haas
On Tue, Jul 23, 2013 at 9:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: If it weren't that we've been speculating for years about deprecating SRFs-in-tlists once we had LATERAL, I would personally consider this patch DOA in this form. If we do think we'll probably deprecate that feature, then not

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jul 23, 2013 at 9:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: If it weren't that we've been speculating for years about deprecating SRFs-in-tlists once we had LATERAL, I would personally consider this patch DOA in this form. I guess I'd sort of

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Robert Haas
On Fri, Jul 19, 2013 at 1:50 PM, Greg Stark st...@mit.edu wrote: My only reservation with this patch is whether the WITH_ORDINALITY parser hack is the way we want to go. The precedent was already set with WITH TIME ZONE though and I think this was the best option. I share this reservation.

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Robert Haas
On Wed, Jul 24, 2013 at 1:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jul 23, 2013 at 9:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: If it weren't that we've been speculating for years about deprecating SRFs-in-tlists once we had LATERAL, I would

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Greg Stark
On Wed, Jul 24, 2013 at 6:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: That seems to me to be unlikely to happen, because it would be impossible to preserve the current (admittedly bad) semantics. If we're going to change the behavior at all we might as well just drop the feature, IMO. It would

Re: [HACKERS] [GENERAL] Insert result does not match record count

2013-07-24 Thread Tom Lane
Vik Fearing vik.fear...@dalibo.com writes: Also worth mentioning is bug #7766. http://www.postgresql.org/message-id/e1tlli5-0007tr...@wrigleys.postgresql.org Yeah, did you read that whole thread? The real issue here is going to be whether client-side code falls over on wider-than-32-bit

Re: [HACKERS] Patch to add support of IF NOT EXISTS to others CREATE statements

2013-07-24 Thread Karol Trzcionka
Hello, patch works fine but is there any reason to comparing each ifNotExists in different way? i.e. ProcedureCreate if (!ifNotExists) ... else { ... return } TypeCreate if (ifNotExists) { ... return } ... --- Shouldn't it be more consistent? Regards, Karol -- Sent via pgsql-hackers

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Greg Stark
On Wed, Jul 24, 2013 at 6:39 PM, Robert Haas robertmh...@gmail.com wrote: This patch will introduce, without documentation, a fifth class of keyword. ORDINALITY will need to be quoted when, and only when, it immediately follows WITH. Without some change to our deparsing code, this is a

Re: [HACKERS] Adding Zigzag Merge Join to Index Nested Loops Join

2013-07-24 Thread Robert Haas
On Tue, Jul 23, 2013 at 3:40 PM, tubadzin tubad...@o2.pl wrote: I want add Zigzag Merge join to Index Nested Loops Join alghoritm. http://www.cs.berkeley.edu/~fox/summaries/database/query_eval_5-8.html Which files are responsible for Index nested loops join ? (not simple nested loops join

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Andres Freund
Hi, On 2013-07-24 11:49:20 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: This will require another member variable in slist_mutable_iter which obviously will need to be maintained, but that seems fine to me since it will reduce the cost of actually deleting

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Tom Lane
Greg Stark st...@mit.edu writes: On Wed, Jul 24, 2013 at 6:39 PM, Robert Haas robertmh...@gmail.com wrote: This patch will introduce, without documentation, a fifth class of keyword. ORDINALITY will need to be quoted when, and only when, it immediately follows WITH. Without some change to

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Andres Freund
On 2013-07-24 13:36:39 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Tue, Jul 23, 2013 at 9:38 PM, Tom Lane t...@sss.pgh.pa.us wrote: If it weren't that we've been speculating for years about deprecating SRFs-in-tlists once we had LATERAL, I would personally consider

Re: [HACKERS] [GENERAL] Insert result does not match record count

2013-07-24 Thread Andres Freund
On 2013-07-24 13:48:23 -0400, Tom Lane wrote: Vik Fearing vik.fear...@dalibo.com writes: Also worth mentioning is bug #7766. http://www.postgresql.org/message-id/e1tlli5-0007tr...@wrigleys.postgresql.org Yeah, did you read that whole thread? The real issue here is going to be whether

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: The first attached patch adds slist_delete_current(), updates the comments addressing your points and converts the bgworker code to pass the iterator around (it's more efficient which might actually matter with a few hundred bgworkers). I found the

Re: [HACKERS] [GENERAL] Insert result does not match record count

2013-07-24 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I think fixing this for 9.4 is fine, but due to the compat issues I think it's to late for 9.3. Agreed -- this is effectively a protocol change, albeit a pretty minor one, so I can't see back-patching it. regards, tom lane

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Andres Freund
Andres Freund and...@2ndquadrant.com writes: The first attached patch adds slist_delete_current(), updates the comments addressing your points and converts the bgworker code to pass the iterator around (it's more efficient which might actually matter with a few hundred bgworkers). I

Re: [HACKERS] dynamic background workers, round two

2013-07-24 Thread Andres Freund
Hi, On 2013-07-24 12:46:21 -0400, Robert Haas wrote: The attached patch attempts to remedy this problem. When you register a background worker, you can obtain a handle that can subsequently be used to query for the worker's PID. If you additionally initialize bgw_notify_pid = MyProcPid,

Re: [HACKERS] Review: UNNEST (and other functions) WITH ORDINALITY

2013-07-24 Thread Andrew Gierth
Tom Lane said: If we did it with a WithOrdinality expression node, the result would always be of type RECORD, and we'd have to use blessed tuple descriptors to keep track of exactly which record type a particular instance emits. This is certainly do-able (see RowExpr for precedent). Maybe

Re: [HACKERS] ilist.h is not useful as-is

2013-07-24 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: The first attached patch adds slist_delete_current(), updates the comments addressing your points and converts the bgworker code to pass the iterator around (it's more efficient which might actually matter with a few hundred bgworkers). Applied

Re: [HACKERS] dynamic background workers, round two

2013-07-24 Thread Michael Paquier
On Thu, Jul 25, 2013 at 6:34 AM, Andres Freund and...@2ndquadrant.comwrote: Hi, On 2013-07-24 12:46:21 -0400, Robert Haas wrote: The attached patch attempts to remedy this problem. When you register a background worker, you can obtain a handle that can subsequently be used to query for

Re: [HACKERS] dynamic background workers, round two

2013-07-24 Thread Michael Paquier
On Thu, Jul 25, 2013 at 6:34 AM, Andres Freund and...@2ndquadrant.comwrote: --- a/contrib/worker_spi/worker_spi.c +++ b/contrib/worker_spi/worker_spi.c Btw, I've posted a minimal regression test for bworkers/worker_spi in 20130724175742.gd10...@alap2.anarazel.de - I'd like to see some

Re: [HACKERS] dynamic background workers, round two

2013-07-24 Thread Andres Freund
On 2013-07-25 08:03:17 +0900, Michael Paquier wrote: On Thu, Jul 25, 2013 at 6:34 AM, Andres Freund and...@2ndquadrant.comwrote: --- a/contrib/worker_spi/worker_spi.c +++ b/contrib/worker_spi/worker_spi.c Btw, I've posted a minimal regression test for bworkers/worker_spi in

[HACKERS] Re: proposal 9.4 plpgsql: allows access to call stack from GET DIAGNOSTICS statement

2013-07-24 Thread Stephen Frost
All, * Rushabh Lathia (rushabh.lat...@gmail.com) wrote: Latest patch looks good to me. I've committed this, with some pretty serious clean-up. In the future, please don't simply copy/paste code w/o any updates to the comments included, particularly when the comments no longer make sense in the

Re: [HACKERS] Preventing tuple-table leakage in plpgsql

2013-07-24 Thread Tom Lane
I wrote: Hmm ... good point. The other plan I'd been considering was to add explicit tracking inside spi.c of all tuple tables created within the current procedure, and then have AtEOSubXact_SPI flush any that were created inside a failed subxact. The tables would still be children of the

Re: [HACKERS] dynamic background workers, round two

2013-07-24 Thread Michael Paquier
On Thu, Jul 25, 2013 at 8:05 AM, Andres Freund and...@2ndquadrant.comwrote: On 2013-07-25 08:03:17 +0900, Michael Paquier wrote: On Thu, Jul 25, 2013 at 6:34 AM, Andres Freund and...@2ndquadrant.com wrote: --- a/contrib/worker_spi/worker_spi.c +++ b/contrib/worker_spi/worker_spi.c

[HACKERS] Re: proposal 9.4 plpgsql: allows access to call stack from GET DIAGNOSTICS statement

2013-07-24 Thread Pavel Stehule
Hello 2013/7/25 Stephen Frost sfr...@snowman.net: All, * Rushabh Lathia (rushabh.lat...@gmail.com) wrote: Latest patch looks good to me. I've committed this, with some pretty serious clean-up. In the future, please don't simply copy/paste code w/o any updates to the comments included,