Re: [HACKERS] Timing events WIP v1

2012-12-12 Thread Pavel Stehule
2012/12/12 Greg Smith g...@2ndquadrant.com: On 11/20/12 8:02 PM, Craig Ringer wrote: On 11/20/2012 04:48 PM, Pavel Stehule wrote: I don't like to see current hstore in core - It doesn't support nesting, it doesn't support different datatypes, it is not well supported by plpgsql ... or

Re: [HACKERS] [WIP PATCH] for Performance Improvement in Buffer Management

2012-12-12 Thread Amit Kapila
On Wednesday, December 12, 2012 5:23 AM Greg Smith wrote: On 11/23/12 5:57 AM, Amit kapila wrote: Let us try to see by example: Total RAM - 22G Database size - 16G ... Case -2 (Shared Buffers - 10G) a. Load all the files in OS buffers. In best case OS buffers can contain10-12G data as

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-11 21:05:51 -0500, Joachim Wieland wrote: On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund and...@2ndquadrant.com wrote: One problem I see is that while exporting a snapshot solves the visibility issues of the table's contents it does not protect against schema changes. I am not

[HACKERS] bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
COPY IN loops in heap_multi_insert() extending the table until it fills the disk when trying to insert a wide row into a table with a low fill-factor. Internally fill-factor is implemented by reserving some space space on a page. For large enough rows and small enough fill-factor

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-11 22:20:18 -0500, Tom Lane wrote: Joachim Wieland j...@mcknight.de writes: On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund and...@2ndquadrant.com wrote: One problem I see is that while exporting a snapshot solves the visibility issues of the table's contents it does not

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-11 22:39:14 -0500, Steve Singer wrote: On 12-12-11 06:52 PM, Andres Freund wrote: Hi, Problem 1: One problem I see is that while exporting a snapshot solves the visibility issues of the table's contents it does not protect against schema changes. I am not sure whether thats a

[HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Andres Freund
On 2012-12-12 03:04:19 -0800, David Gould wrote: COPY IN loops in heap_multi_insert() extending the table until it fills the disk when trying to insert a wide row into a table with a low fill-factor. Internally fill-factor is implemented by reserving some space space on a page. For large

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Heikki Linnakangas
On 12.12.2012 13:27, Andres Freund wrote: On 2012-12-12 03:04:19 -0800, David Gould wrote: One more point, in the case where we don't insert any rows, we still do all the dirtying and logging work even though we did not modify the page. I have tried skip all this if no rows are added (nthispage

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
On Wed, 12 Dec 2012 12:27:11 +0100 Andres Freund and...@2ndquadrant.com wrote: On 2012-12-12 03:04:19 -0800, David Gould wrote: COPY IN loops in heap_multi_insert() extending the table until it fills the Heh. Nice one. Did you hit that in practice? Yeah, with a bunch of hosts that run

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Heikki Linnakangas
On 12.12.2012 14:17, David Gould wrote: On Wed, 12 Dec 2012 12:27:11 +0100 Andres Freundand...@2ndquadrant.com wrote: On 2012-12-12 03:04:19 -0800, David Gould wrote: COPY IN loops in heap_multi_insert() extending the table until it fills the Heh. Nice one. Did you hit that in practice?

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
On Wed, 12 Dec 2012 13:56:08 +0200 Heikki Linnakangas hlinnakan...@vmware.com wrote: However, RelationGetBufferForTuple() won't return such a page, it guarantees that the first tuple does indeed fit on the page it returns. For the same reason, the later check that at least one tuple was

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Heikki Linnakangas
On 12.12.2012 14:24, David Gould wrote: I don't know if this is the same thing. At least in the comments I was reading trying to figure this out there was some concern that someone else could change the space on the page. Does RelationGetBufferForTuple() guarantee against this too? Yeah,

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread David Gould
On Wed, 12 Dec 2012 14:23:12 +0200 Heikki Linnakangas hlinnakan...@vmware.com wrote: The bug's been fixed now, but note that huge tuples like this will always cause the table to be extended. Even if there are completely empty pages in the table, after a vacuum. Even a completely empty

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-12-12 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: Yeah, I had forgotten about that loose end, but we definitely need something of the sort. Committed with additional documentation. (I put something in the CREATE EVENT TRIGGER ref page, but do we need anything about it in chapter 37?) Thanks! I guess we

Re: [HACKERS] [v9.3] writable foreign tables

2012-12-12 Thread Erik Rijkers
On Wed, December 12, 2012 14:45, Kohei KaiGai wrote: pgsql-v9.3-writable-fdw-poc.v8.part-2.patch 151 k pgsql-v9.3-writable-fdw-poc.v8.part-1.patch 70 k I wanted to have a look at this, and tried to apply part 1, en then part 2 on top of that (that's the idea, right?) part 1 applies and

Re: [HACKERS] [v9.3] writable foreign tables

2012-12-12 Thread Albe Laurenz
Erik Rijkers wrote: pgsql-v9.3-writable-fdw-poc.v8.part-2.patch 151 k pgsql-v9.3-writable-fdw-poc.v8.part-1.patch 70 k I wanted to have a look at this, and tried to apply part 1, en then part 2 on top of that (that's the idea, right?) part 1 applies and then part 2 does not. Part 2

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-12-12 Thread Tom Lane
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: BTW, is the example in the CREATE EVENT TRIGGER ref page meant as an IQ test for unwary readers? Maybe we could just append to it how to remove such an event trigger, which is easy to do because you can't

Re: [HACKERS] Multiple --table options for other commands

2012-12-12 Thread Karl O. Pinc
On 12/11/2012 10:25:43 PM, Josh Kupershmidt wrote: On Tue, Dec 11, 2012 at 11:46 AM, Karl O. Pinc k...@meme.com wrote: Yes, the current pg_restore silently ignores multiple --table arguments, and seems to use the last one. You are introducing a backwards incompatible change here.

Re: [HACKERS] [BUG?] lag of minRecoveryPont in archive recovery

2012-12-12 Thread Fujii Masao
On Wed, Dec 12, 2012 at 2:03 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 11.12.2012 17:04, Fujii Masao wrote: The patch looks good. I confirmed that the PANIC error didn't happen, with the patch. Ok, committed, and also moved the CheckRecoveryConsistency call. Please take a

Re: [HACKERS] Re: bulk_multi_insert infinite loops with large rows and small fill factors

2012-12-12 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: The bug's been fixed now, but note that huge tuples like this will always cause the table to be extended. Even if there are completely empty pages in the table, after a vacuum. Even a completely empty existing page is not considered

Re: [HACKERS] PageIsAllVisible()'s trustworthiness in Hot Standby

2012-12-12 Thread Pavan Deolasee
On Wed, Dec 12, 2012 at 1:35 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Dec 4, 2012 at 12:10 PM, Pavan Deolasee pavan.deola...@gmail.com wrote: Hmm. Yeah, I do not have guts to prove that either. I'll probably write up a comment for your consideration to explain why we don't trust

Re: [HACKERS] [PATCH] PL/Python: Add spidata to all spiexceptions

2012-12-12 Thread Jan Urbański
Though not the original patch autor, I did modify and submit it to the CF app, so I'll reply here :) On 10/12/12 19:20, Karl O. Pinc wrote: On 12/09/2012 10:33:59 PM, Karl O. Pinc wrote: I've gone ahead and signed up to review this patch. Thanks! There were 2 outstanding issue raised:

[HACKERS] PRIVATE columns

2012-12-12 Thread Simon Riggs
Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some cases we have data that is private and we do not wish others to see it, such as patient names. This becomes more important when we have row security.

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Jan Wieck
On 12/12/2012 1:12 PM, Simon Riggs wrote: Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some cases we have data that is private and we do not wish others to see it, such as patient names. This becomes

Re: [HACKERS] [PATCH] PL/Python: Add spidata to all spiexceptions

2012-12-12 Thread Karl O. Pinc
Hi Jan and Oskari, On 12/12/2012 11:36:54 AM, Jan Urbański wrote: On 10/12/12 19:20, Karl O. Pinc wrote: On 12/09/2012 10:33:59 PM, Karl O. Pinc wrote: There were 2 outstanding issue raised: Is this useful enough/proceeding in the right direction to commit now? I believe the

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Simon Riggs
On 12 December 2012 19:13, Jan Wieck janwi...@yahoo.com wrote: On 12/12/2012 1:12 PM, Simon Riggs wrote: Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some cases we have data that is private and we do

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Joshua D. Drake
On 12/12/2012 12:12 PM, Simon Riggs wrote: Would protecting it the same way, we protect the passwords in pg_authid, be sufficient? The user backend does need to be able to access the stats data during optimization. It's hard to have data accessible and yet impose limits on the uses to which

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-12-12 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: Maybe we could just append to it how to remove such an event trigger, which is easy to do because you can't target an event trigger related command with event triggers, so the following is not disabled: DROP EVENT TRIGGER abort_ddl; Oh really? The

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. Only if you have appropriate privileges. In some cases we have data that is private and we do not wish others to

[HACKERS] Use gcc built-in atomic inc/dec in lock.c

2012-12-12 Thread Mikko Tiihonen
Hi, I noticed a XXX: It might be worth considering using an atomic fetch-and-add instruction here, on architectures where that is supported. in lock.c Here is my first try at using it. The patch adds a configure check for gcc 4.7 __atomic_add_fetch as well as the older __sync_add_and_fetch

Re: [HACKERS] Use gcc built-in atomic inc/dec in lock.c

2012-12-12 Thread Peter Geoghegan
On 12 December 2012 22:11, Mikko Tiihonen mikko.tiiho...@nitorcreations.com wrote: noticed a XXX: It might be worth considering using an atomic fetch-and-add instruction here, on architectures where that is supported. in lock.c Here is my first try at using it. That's interesting, but I have

Re: [HACKERS] Enabling Checksums

2012-12-12 Thread Greg Smith
On 12/5/12 6:49 PM, Simon Riggs wrote: * Zeroing pages, making pages all 1s * Transposing pages * Moving chunks of data sideways in a block * Flipping bits randomly * Flipping data endianness * Destroying particular catalog tables or structures I can take this on, as part of the QA around

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-12 12:13:44 +0100, Andres Freund wrote: On 2012-12-11 22:20:18 -0500, Tom Lane wrote: Joachim Wieland j...@mcknight.de writes: On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund and...@2ndquadrant.com wrote: One problem I see is that while exporting a snapshot solves the

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-12 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2012-12-12 12:13:44 +0100, Andres Freund wrote: This morning I wondered whether we couldn't protect against that by acquiring share locks on the catalog rows pg_dump reads, that would result in could not serialize access due to concurrent update

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-12 Thread Andres Freund
On 2012-12-12 18:52:33 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2012-12-12 12:13:44 +0100, Andres Freund wrote: This morning I wondered whether we couldn't protect against that by acquiring share locks on the catalog rows pg_dump reads, that would result in

Re: [HACKERS] logical changeset generation v3 - git repository

2012-12-12 Thread Peter Geoghegan
On 9 December 2012 19:14, Andres Freund and...@2ndquadrant.com wrote: I pushed a new version which - is rebased ontop of master - is based ontop of the new xlogreader (biggest part) - is base ontop of the new binaryheap.h - some fixes - some more comments I decided to take another look at

Re: [HACKERS] [PERFORM] encouraging index-only scans

2012-12-12 Thread Bruce Momjian
On Wed, Dec 12, 2012 at 05:27:39PM -0500, Andrew Dunstan wrote: On 12/12/2012 05:12 PM, Andrew Dunstan wrote: On 12/12/2012 04:32 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: A client is testing a migration from 9.1 to 9.2, and has found that a large number of queries

Re: [HACKERS] [PERFORM] encouraging index-only scans

2012-12-12 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Wed, Dec 12, 2012 at 05:27:39PM -0500, Andrew Dunstan wrote: Actually, the table had been analysed but not vacuumed, so this kinda begs the question what will happen to this value on pg_upgrade? Will people's queries suddenly get slower until

Re: [HACKERS] PRIVATE columns

2012-12-12 Thread Jan Wieck
On 12/12/2012 3:12 PM, Simon Riggs wrote: On 12 December 2012 19:13, Jan Wieck janwi...@yahoo.com wrote: On 12/12/2012 1:12 PM, Simon Riggs wrote: Currently, ANALYZE collects data on all columns and stores these samples in pg_statistic where they can be seen via the view pg_stats. In some

Re: [HACKERS] [PERFORM] encouraging index-only scans

2012-12-12 Thread Pavan Deolasee
On Thu, Dec 13, 2012 at 9:21 AM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: On Wed, Dec 12, 2012 at 05:27:39PM -0500, Andrew Dunstan wrote: Actually, the table had been analysed but not vacuumed, so this kinda begs the question what will happen to this value on

Re: [HACKERS] Multiple --table options for other commands

2012-12-12 Thread Josh Kupershmidt
On Wed, Dec 12, 2012 at 8:14 AM, Karl O. Pinc k...@meme.com wrote: On 12/11/2012 10:25:43 PM, Josh Kupershmidt wrote: On Tue, Dec 11, 2012 at 11:46 AM, Karl O. Pinc k...@meme.com wrote: I believe you need ellipses behind --table in the syntax summaries of the command reference docs. Hrm, I

Re: [HACKERS] Multiple --table options for other commands

2012-12-12 Thread Karl O. Pinc
Hi Josh, The good news is that there's only this little bit of documentation formatting to fix On 12/12/2012 11:04:53 PM, Josh Kupershmidt wrote: On Wed, Dec 12, 2012 at 8:14 AM, Karl O. Pinc k...@meme.com wrote: On 12/11/2012 10:25:43 PM, Josh Kupershmidt wrote: On Tue, Dec 11, 2012 at