Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
Hi, On 2014-01-02 05:26:26 +, Greg Stark wrote: 2) refetching a row could conceivably end up retrieving different data than was present when the row was originally read. (In some cases that might actually be the intended behaviour) That's possible with system columns as well. In the

Re: [HACKERS] more psprintf() use

2014-01-02 Thread Andres Freund
On 2014-01-02 09:49:48 +0200, Heikki Linnakangas wrote: On 01/02/2014 05:14 AM, Peter Eisentraut wrote: diff --git a/contrib/hstore/hstore_io.c b/contrib/hstore/hstore_io.c index 772a5ca..8331a56 100644 --- a/contrib/hstore/hstore_io.c +++ b/contrib/hstore/hstore_io.c @@ -1114,11 +1114,7 @@

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2014-01-02 Thread Andres Freund
On 2013-12-31 13:56:53 -0800, Peter Geoghegan wrote: ISTM that you should be printing just the value and the unique index there, and not any information about the tuple proper. Doing any less could be highly confusing. I agree that the message needs improvement, but I don't agree that we

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2014-01-02 Thread Andres Freund
On 2013-12-31 11:36:36 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On 31 December 2013 09:12, Christian Kruse christ...@2ndquadrant.com wrote: Output with patch: LOG: process 24774 acquired ShareLock on transaction 696 after 11688.720 ms CONTEXT: relation

Re: [HACKERS] What exactly is drop-index-concurrently-1.spec trying to test?

2014-01-02 Thread Andres Freund
On 2013-12-31 17:14:11 -0500, Tom Lane wrote: Peter pointed out in http://www.postgresql.org/message-id/527c0fe9.7000...@gmx.net that Kyotaro-san's patch to treat unique indexes as satisfying any sort condition that they are a prefix of broke the drop-index-concurrently-1 isolation test. The

Re: [HACKERS] proposal: multiple read-write masters in a cluster with wal-streaming synchronization

2014-01-02 Thread Andres Freund
On 2013-12-31 13:51:08 -0800, Mark Dilger wrote: The BDR documentation http://wiki.postgresql.org/images/7/75/BDR_Presentation_PGCon2012.pdf says,     Physical replication forces us to use just one node: multi-master required for write scalability     Physical replication provides

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-02 Thread Andres Freund
On 2013-12-31 13:37:59 +0100, Pavel Stehule wrote: We use the namespace ext to the internal code (src/backend/access/common/reloptions.c) skip some validations and store the custom GUC. Do you think we don't need to use the ext namespace? yes - there be same mechanism as we use for

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-02 Thread Andres Freund
On 2014-01-01 21:15:46 -0500, Robert Haas wrote: [ sensible reasoning ] However, I'm not sure it's really worth it. I think what people really care about is knowing whether the bitmap lossified or not, and generally how much got lossified. The counts of exact and lossy pages are sufficient

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2014-01-02 Thread Peter Geoghegan
On Thu, Jan 2, 2014 at 12:56 AM, Andres Freund and...@2ndquadrant.com wrote: I agree that the message needs improvement, but I don't agree that we shouldn't lock the tuple's location. If you manually investigate the situation that's where you'll find the conflicting tuple - I don't see what we

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Andres Freund
On 2013-12-27 14:11:44 -0800, Peter Geoghegan wrote: On Fri, Dec 27, 2013 at 12:57 AM, Andres Freund and...@2ndquadrant.com wrote: I don't think the current syntax the feature implements can be used as the sole argument what the feature should be able to support. If you think from the

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2014-01-02 Thread Andres Freund
On 2014-01-02 01:40:38 -0800, Peter Geoghegan wrote: On Thu, Jan 2, 2014 at 12:56 AM, Andres Freund and...@2ndquadrant.com wrote: I agree that the message needs improvement, but I don't agree that we shouldn't lock the tuple's location. If you manually investigate the situation that's where

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
On Thu, Jan 2, 2014 at 1:49 AM, Andres Freund and...@2ndquadrant.com wrote: Well, you're not totally on your own for something like that with this feature. You can project the conflicter's tid, and possibly do a more sophisticated recovery, like inspecting the locked row and iterating. Yea,

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-02 Thread Fabrízio de Royes Mello
On Thu, Jan 2, 2014 at 7:19 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-31 13:37:59 +0100, Pavel Stehule wrote: We use the namespace ext to the internal code (src/backend/access/common/reloptions.c) skip some validations and store the custom GUC. Do you think we

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Andres Freund
On 2014-01-02 02:20:02 -0800, Peter Geoghegan wrote: On Thu, Jan 2, 2014 at 1:49 AM, Andres Freund and...@2ndquadrant.com wrote: Well, you're not totally on your own for something like that with this feature. You can project the conflicter's tid, and possibly do a more sophisticated

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-02 Thread Andres Freund
On 2014-01-02 08:26:20 -0200, Fabrízio de Royes Mello wrote: On Thu, Jan 2, 2014 at 7:19 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-31 13:37:59 +0100, Pavel Stehule wrote: We use the namespace ext to the internal code (src/backend/access/common/reloptions.c) skip

Re: [HACKERS] Patch: show relation and tuple infos of a lock to acquire

2014-01-02 Thread Christian Kruse
Hi, On 02/01/14 10:02, Andres Freund wrote: Christian's idea of a context line seems plausible to me. I don't care for this implementation too much --- a global variable? Ick. Yea, the data should be stored in ErrorContextCallback.arg instead. Fixed. I also palloc() the

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread David Rowley
On Fri, Dec 27, 2013 at 1:36 AM, David Rowley dgrowle...@gmail.com wrote: From what I can tell adding an inverse transition function to support AVG for numeric does not affect the number of trailing zeros in the results, so I've attached a patch which now has inverse transition functions to

[HACKERS] [PATCH] Remove some duplicate if conditions

2014-01-02 Thread David Rowley
I've attached a simple patch which removes some duplicate if conditions that seemed to have found their way into the code. These are per PVS-Studio's warnings. Regards David Rowley duplicate_if_test.patch Description: Binary data -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Fixing pg_basebackup with tablespaces found in $PGDATA

2014-01-02 Thread Magnus Hagander
On Wed, Jan 1, 2014 at 11:53 PM, Dimitri Fontaine dimi...@2ndquadrant.frwrote: Hi, As much as I've seen people frown upon $subject, it still happens in the wild, and Magnus seems to agree that the current failure mode of our pg_basebackup tool when confronted to the situation is a bug. So

[HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Rushabh Lathia
Hi All, Test case: drop table if exists t; create table t(c text); insert into t values ('x'), (repeat(md5('abcdefghijklmnop'), 1)); select pg_column_size(c), pg_column_size(c || '') FROM t; CREATE OR REPLACE FUNCTION copy_toast_out() RETURNS VOID AS $$ declare v text; BEGIN

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a, pg_tuple_header(a.tableoid, a.ctid) x; Two points: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 07:35:11 -0500, Robert Haas wrote: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a, pg_tuple_header(a.tableoid, a.ctid) x; Two points: 1) it's a bit weird to go to this

Re: [HACKERS] Show lossy heap block info in EXPLAIN ANALYZE for bitmap heap scan

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 4:27 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-01 21:15:46 -0500, Robert Haas wrote: [ sensible reasoning ] However, I'm not sure it's really worth it. I think what people really care about is knowing whether the bitmap lossified or not, and generally

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Robert Haas
On Tue, Dec 31, 2013 at 4:12 AM, Peter Geoghegan p...@heroku.com wrote: On Tue, Dec 31, 2013 at 12:52 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: 1. PromiseTupleInsertionLockAcquire(my xid) 2. Insert heap tuple 3. Insert index tuples 4. Check if conflict happened. Kill the

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread Nicolas Barbier
2013/12/15 David Rowley dgrowle...@gmail.com: I've been working on speeding up aggregate functions when used in the context of a window's with non fixed frame heads. 1. Fully implement negative transition functions for SUM and AVG. I would like to mention that this functionality is also

Re: [HACKERS] Fixing pg_basebackup with tablespaces found in $PGDATA

2014-01-02 Thread Dimitri Fontaine
Magnus Hagander mag...@hagander.net writes: We can't get away with just comparing the relative part of the pathname. Because it will fail if there is another path with exactly the same length, containing the tablespace. Actually… yeah. I think we might want to store a value in the

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 7:40 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-01-02 07:35:11 -0500, Robert Haas wrote: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: I fail to see why. What is so ugly about this: select x.* from pgbench_accounts a,

Re: [HACKERS] more psprintf() use

2014-01-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:49:48 +0200, Heikki Linnakangas wrote: Is it legal to return a constant with PG_RETURN_CSTRING? Grepping around, I don't see that being done anywhere else, but there are places that do PG_RETURN_CSTRING(pstrdup(constant))... I

Re: [HACKERS] Fixing pg_basebackup with tablespaces found in $PGDATA

2014-01-02 Thread Bernd Helmle
--On 1. Januar 2014 23:53:46 +0100 Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Hi, As much as I've seen people frown upon $subject, it still happens in the wild, and Magnus seems to agree that the current failure mode of our pg_basebackup tool when confronted to the situation is a bug.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme that depends on having a system column -- ctid At any rate, my goal isn't really to get rid of

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 12:26 AM, Greg Stark st...@mit.edu wrote: 1) it's a bit weird to go to this effort to eliminate system columns by using a scheme that depends on having a system column -- ctid

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Heikki Linnakangas
On 01/02/2014 02:53 PM, Robert Haas wrote: On Tue, Dec 31, 2013 at 4:12 AM, Peter Geoghegan p...@heroku.com wrote: On Tue, Dec 31, 2013 at 12:52 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: 1. PromiseTupleInsertionLockAcquire(my xid) 2. Insert heap tuple 3. Insert index tuples 4.

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread Erik Rijkers
On Thu, January 2, 2014 13:36, Erik Rijkers wrote: On Thu, January 2, 2014 13:05, David Rowley wrote: here's a slightly updated patch [inverse_transition_functions_v1.8.patch.gz ] patch applies, and compiles (although with new warnings). But make check complains loudly: see attached.

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 11:08 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/02/2014 02:53 PM, Robert Haas wrote: On Tue, Dec 31, 2013 at 4:12 AM, Peter Geoghegan p...@heroku.com wrote: On Tue, Dec 31, 2013 at 12:52 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: 1.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Actually, I thought the function approach was a good proposal. You are right that func(tab.*) isn't going to work, because it's going to get a Datum-ified tuple not a pointer to raw on-disk storage.

Re: [HACKERS] [PATCH] Remove some duplicate if conditions

2014-01-02 Thread Tom Lane
David Rowley dgrowle...@gmail.com writes: I've attached a simple patch which removes some duplicate if conditions that seemed to have found their way into the code. These are per PVS-Studio's warnings. -1. If PVS-Studio is complaining about this type of coding, to hell with it; it should

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Robert Haas
On Tue, Dec 31, 2013 at 12:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: Mark Dilger markdil...@yahoo.com writes: In src/include/pgstat.h, the PGSTAT_NUM_TABENTRIES macro attempts to subtract off the size of the PgStat_MsgTabstat struct up to the m_entry[] field. This macro was correct up until

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Dec 31, 2013 at 12:20 PM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should look for a less breakable way to manage the message size limit; if Robert missed this issue in that patch, other people are going to miss it in

Re: [HACKERS] proposal: multiple read-write masters in a cluster with wal-streaming synchronization

2014-01-02 Thread Mark Dilger
My original email was mostly a question about whether WAL data could be merged from multiple servers, or whether I was overlooking some unsolvable difficulty.  I'm still mostly curious about that question. I anticipated that my proposal would require partitioning the catalogs. For instance,

Re: [HACKERS] RFC: Async query processing

2014-01-02 Thread Claudio Freire
On Wed, Dec 18, 2013 at 1:50 PM, Florian Weimer fwei...@redhat.com wrote: On 11/04/2013 02:51 AM, Claudio Freire wrote: On Sun, Nov 3, 2013 at 3:58 PM, Florian Weimer fwei...@redhat.com wrote: I would like to add truly asynchronous query processing to libpq, enabling command pipelining.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 12:46 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Actually, I thought the function approach was a good proposal. You are right that func(tab.*) isn't going to work, because it's going

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 12:46 PM, Tom Lane t...@sss.pgh.pa.us wrote: In any case, upon further reflection I'm not convinced that doing this with a SELECT-based query is the right thing, no matter whether the query looks at a function or a system column;

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread Tom Lane
Erik Rijkers e...@xs4all.nl writes: The TRACE_POSTGRESQL_SORT_DONE warnings were not from your patch; sorry about that. They occur on HEAD too (with a debug compile). tuplesort.c:935:44: warning: comparison between pointer and integer [enabled by default]

Re: [HACKERS] more psprintf() use

2014-01-02 Thread Alvaro Herrera
Peter Eisentraut wrote: psprintf() in place of hardcoded palloc(N) + sprintf() and the like. + values[j++] = psprintf(%d, stat.blkno); + values[j++] = psprintf(%c, stat.type); + values[j++] = psprintf(%d, stat.live_items); + values[j++] = psprintf(%d, stat.dead_items); +

Re: [HACKERS] proposal: multiple read-write masters in a cluster with wal-streaming synchronization

2014-01-02 Thread Andres Freund
On 2014-01-02 10:18:52 -0800, Mark Dilger wrote: I anticipated that my proposal would require partitioning the catalogs. For instance, autovacuum could only run on locally owned tables, and would need to store the analyze stats data in a catalog partition belonging to the local server, but

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Heikki Linnakangas
On 01/02/2014 02:24 PM, Rushabh Lathia wrote: Hi All, Test case: drop table if exists t; create table t(c text); insert into t values ('x'), (repeat(md5('abcdefghijklmnop'), 1)); select pg_column_size(c), pg_column_size(c || '') FROM t; CREATE OR REPLACE FUNCTION copy_toast_out() RETURNS

Re: [HACKERS] proposal: multiple read-write masters in a cluster with wal-streaming synchronization

2014-01-02 Thread Merlin Moncure
On Tue, Dec 31, 2013 at 3:51 PM, Mark Dilger markdil...@yahoo.com wrote: The BDR documentation http://wiki.postgresql.org/images/7/75/BDR_Presentation_PGCon2012.pdf says, Physical replication forces us to use just one node: multi-master required for write scalability Physical

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 12:46:34 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 09:40:54 -0500, Tom Lane wrote: Actually, I thought the function approach was a good proposal. You are right that func(tab.*) isn't going to work, because it's going to get a

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: The simplest fix would be to just detoast everything on assignment but that was rejected on performance grounds in that previous thread. I don't see any other realistic way to fix this, however, so maybe we should just bite the bullet and

Re: [HACKERS] proposal: multiple read-write masters in a cluster with wal-streaming synchronization

2014-01-02 Thread Mark Dilger
Thanks to both of you for all the feedback.  Your reasoning about why it is not worth implementing, what the problems with it would be, etc., are helpful. Sorry about using the word multimaster where it might have been better to say sharded. BTW, since the space shuttle has already left orbit,

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 2:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: I both agree and disagree with this. I think that pgstattuple is useful, and I further agree that adding a stat to it about how much of the heap is frozen would be worthwhile. However, an aggregate number isn't always what

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Andres Freund
On 2014-01-02 21:21:15 +0200, Heikki Linnakangas wrote: I don't see any other realistic way to fix this, however, so maybe we should just bite the bullet and do it anyway. We could remember the subtransaction a variable was created in and error out if it the creating subtransaction aborted and

Re: [HACKERS] proposal: multiple read-write masters in a cluster with wal-streaming synchronization

2014-01-02 Thread Andres Freund
On 2014-01-02 11:35:57 -0800, Mark Dilger wrote: BTW, since the space shuttle has already left orbit, as you metaphorically put it, maybe there should be more visibility to the wider world about this?  You can go to postgresql.org and find diddly squat about it.  I grant you that it is not a

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 12:46:34 -0500, Tom Lane wrote: For real forensics work, you need to be able to see all tuples, which makes me think that something akin to pgstattuple is the right API; that is return a set of the header info for all tuples on

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: As I commented to Robert, the page-at-a-time behavior of pageinspect is not an API detail we'd want to copy for this. I envision something like select hdr.*, foo.* from tuple_header_details('foo'::regclass)

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: We could certainly add a function that returns SETOF record, taking e.g. regclass as an argument, but it doesn't seem a stretch to me to think that you might want to get tuple header information for some but not all tuples in the relation, and I don't

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread David Rowley
On Fri, Jan 3, 2014 at 5:33 AM, Erik Rijkers e...@xs4all.nl wrote: *** /var/data1/pg_stuff/pg_sandbox/pgsql.inverse/src/ test/regress/expected/window.out 2014-01-02 16:19:48.0 +0100 --- /var/data1/pg_stuff/pg_sandbox/pgsql.inverse/src/test/regress/results/window.out 2014-01-02

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: As I commented to Robert, the page-at-a-time behavior of pageinspect is not an API detail we'd want to copy for this. I envision something like select hdr.*, foo.* from

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
I decided to make at least a cursory attempt to measure or characterize the performance of each of our approaches to value locking. Being fair here is a non-trivial matter, because of the fact that upserts can behave quite differently based on the need to insert or update, lock contention and so

[HACKERS] Streaming replication bug in 9.3.2, WAL contains references to invalid pages

2014-01-02 Thread Christophe Pettus
Greetings, We've had two clients experience a crash on the secondary of a streaming replication pair, running PostgreSQL 9.3.2. In both cases, the messages were close to this example: 2013-12-30 18:08:00.464 PST,,,23869,,52ab4839.5d3d,16,,2013-12-13 09:47:37 PST,1/0,0,WARNING,01000,page

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 21:21:15 +0200, Heikki Linnakangas wrote: I don't see any other realistic way to fix this, however, so maybe we should just bite the bullet and do it anyway. We could remember the subtransaction a variable was created in and error

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 2:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Thu, Jan 2, 2014 at 2:44 PM, Tom Lane t...@sss.pgh.pa.us wrote: As I commented to Robert, the page-at-a-time behavior of pageinspect is not an API detail we'd want to copy for this.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Well, that's fair enough. I don't mind having two functions. Should the whole-table function also include invisible tuples? Certainly, that's exactly why I was proposing it. You can do a join if you want to suppress them.

Re: [HACKERS] preserving forensic information when we freeze

2014-01-02 Thread Andres Freund
On 2014-01-02 14:44:34 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 12:46:34 -0500, Tom Lane wrote: For real forensics work, you need to be able to see all tuples, which makes me think that something akin to pgstattuple is the right API; that is return

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Andres Freund
On 2014-01-02 15:00:58 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-01-02 21:21:15 +0200, Heikki Linnakangas wrote: I don't see any other realistic way to fix this, however, so maybe we should just bite the bullet and do it anyway. We could remember the

Re: [HACKERS] CLUSTER FREEZE

2014-01-02 Thread Robert Haas
On Mon, Dec 23, 2013 at 6:53 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-22 20:45:02 -0500, Robert Haas wrote: I suspect we ought to extend this to rewriting variants of ALTER TABLE as well, but a little thought is needed there. ATRewriteTables() appears to just call

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 3:19 PM, Andres Freund and...@2ndquadrant.com wrote: I was wondering if we could somehow arrange to not release the subtransaction's AccessShareLock on the table, as long as it was protecting toasted references someplace. Sounds fairly ugly... I think the only

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Andres Freund
On 2014-01-02 16:05:09 -0500, Robert Haas wrote: On Thu, Jan 2, 2014 at 3:19 PM, Andres Freund and...@2ndquadrant.com wrote: I was wondering if we could somehow arrange to not release the subtransaction's AccessShareLock on the table, as long as it was protecting toasted references

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
On Thu, Jan 2, 2014 at 8:08 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Yeah, it seems like PromiseTupleInsertionLockAcquire should be locking the tuple, rather than the XID. Well, that would be ideal, because we already have tuple locks. It would be nice to use the same concept for

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Tom Lane
I wrote: It occurs to me that, rather than trying to improve the struct definition methodology, maybe we should just add static asserts to catch any inconsistency here. It wouldn't be that hard: #define PGSTAT_MAX_MSG_SIZE 1000 #define PGSTAT_MSG_PAYLOAD(PGSTAT_MAX_MSG_SIZE -

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
On Thu, Jan 2, 2014 at 2:37 AM, Andres Freund and...@2ndquadrant.com wrote: Locking the definitely visible row only works if there's a row matching the index's columns. If the values of the new row don't have corresponding values in all the indexes you have the same old race conditions again.

Re: [HACKERS] Planning time in explain/explain analyze

2014-01-02 Thread Andreas Karlsson
On 01/02/2014 04:08 AM, Robert Haas wrote: I'm wondering whether the time should be stored inside the PlannedStmt node instead of passing it around separately. One possible problem with the way you've done things here is that, in the case of a prepared statement, EXPLAIN ANALYZE will emit the

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread Erik Rijkers
On Thu, January 2, 2014 17:33, Erik Rijkers wrote: On Thu, January 2, 2014 13:36, Erik Rijkers wrote: On Thu, January 2, 2014 13:05, David Rowley wrote: here's a slightly updated patch [inverse_transition_functions_v1.8.patch.gz ] patch applies, and compiles (although with new warnings).

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Mark Dilger
I still don't understand why this case in src/include/pgstat.h is different from cases elsewhere in the code.  Taken from src/include/access/heapam_xlog.h: typedef struct xl_heap_header {     uint16  t_infomask2;     uint16  t_infomask;     uint8   t_hoff; } xl_heap_header; #define

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread Erik Rijkers
On Fri, January 3, 2014 00:09, Erik Rijkers wrote: connection to server was lost So, to repeat, this runs fine on a server compiled for speed. I forgot to append the log messages: 2014-01-03 00:19:17.073 CET 14054 LOG: database system is ready to accept connections TRAP:

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Andres Freund
On 2014-01-02 15:15:58 -0800, Mark Dilger wrote: I still don't understand why this case in src/include/pgstat.h is different from cases elsewhere in the code.  Taken from src/include/access/heapam_xlog.h: typedef struct xl_heap_header {     uint16  t_infomask2;     uint16 

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Tom Lane
Mark Dilger markdil...@yahoo.com writes: I still don't understand why this case in src/include/pgstat.h is different from cases elsewhere in the code. The reason why I'm exercised about it is that (a) somebody actually made a mistake of this type, and (b) it wasn't caught by any automated

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Mark Dilger
I completely understand the padding issues that you are dealing with.  I was mostly curious why Tom wanted to use asserts to double-check the code in one place, while happily not doing so in what seemed the same kind of situation elsewhere. He has since made the reason for that clear. On

Re: [HACKERS] Streaming replication bug in 9.3.2, WAL contains references to invalid pages

2014-01-02 Thread MauMau
From: Christophe Pettus x...@thebuild.com We've had two clients experience a crash on the secondary of a streaming replication pair, running PostgreSQL 9.3.2. In both cases, the messages were close to this example: 2013-12-30 18:08:00.464 PST,,,23869,,52ab4839.5d3d,16,,2013-12-13 09:47:37

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-02 Thread Peter Geoghegan
On Thu, Jan 2, 2014 at 11:58 AM, Peter Geoghegan p...@heroku.com wrote: My executive summary is that the exclusion patch performs about the same on lower client counts, presumably due to not having the additional window of btree lock contention. By 8 clients, the exclusion patch does

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Mark Dilger
The mechanism that occurs to me (and I'm not wedded to this idea) is: typedef uint8 T_HOFF_TYPE; typedef struct xl_heap_header {     uint16  t_infomask2;     uint16  t_infomask;     T_HOFF_TYPE t_hoff; } xl_heap_header; #define SizeOfHeapHeader   

Re: [HACKERS] [PATCH] Store Extension Options

2014-01-02 Thread Robert Haas
On Thu, Jan 2, 2014 at 4:19 AM, Andres Freund and...@2ndquadrant.com wrote: On 2013-12-31 13:37:59 +0100, Pavel Stehule wrote: We use the namespace ext to the internal code (src/backend/access/common/reloptions.c) skip some validations and store the custom GUC. Do you think we don't

Re: [HACKERS] fix_PGSTAT_NUM_TABENTRIES_macro patch

2014-01-02 Thread Tom Lane
Mark Dilger markdil...@yahoo.com writes: The mechanism that occurs to me (and I'm not wedded to this idea) is: typedef uint8 T_HOFF_TYPE; typedef struct xl_heap_header {     uint16  t_infomask2;     uint16  t_infomask;     T_HOFF_TYPE t_hoff; }

Re: [HACKERS] truncating pg_multixact/members

2014-01-02 Thread Robert Haas
On Mon, Dec 30, 2013 at 10:59 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: One problem I see is length of time before freezing multis: they live for far too long, causing the SLRU files to eat way too much disk space. I ran burnmulti in a loop, creating multis of 3 members each, with a

Re: [HACKERS] [PATCH] Make various variables read-only (const)

2014-01-02 Thread Wim Lewis
By an odd coincidence, I also decided to try to const-ify libpq recently, and noticed this thread as I was cleaning up my patch for submission. For what it's worth, I've attached my patch to this message. It doesn't move as much data into the text segment as Oskari Saarenmaa's patch does, but it

Re: [HACKERS] [bug fix] connection service file doesn't take effect with ECPG apps

2014-01-02 Thread MauMau
From: Michael Meskes mes...@postgresql.org However, I'd prefer to solve the problem slightly differently by not creating an empty host variable instead of checking for it after the fact. But I take it you don't mind that. Fixed in HEAD and all back branches. Thanks for the report. Thank you

Re: [HACKERS] ERROR: missing chunk number 0 for toast value

2014-01-02 Thread Amit Kapila
On Fri, Jan 3, 2014 at 12:51 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 01/02/2014 02:24 PM, Rushabh Lathia wrote: Hi All, Test case: drop table if exists t; create table t(c text); insert into t values ('x'), (repeat(md5('abcdefghijklmnop'), 1)); select

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-02 Thread David Rowley
On Fri, Jan 3, 2014 at 12:23 PM, Erik Rijkers e...@xs4all.nl wrote: On Fri, January 3, 2014 00:09, Erik Rijkers wrote: connection to server was lost So, to repeat, this runs fine on a server compiled for speed. I forgot to append the log messages: 2014-01-03 00:19:17.073 CET 14054

Re: [HACKERS] [PATCH] Remove some duplicate if conditions

2014-01-02 Thread David Rowley
On Fri, Jan 3, 2014 at 6:59 AM, Tom Lane t...@sss.pgh.pa.us wrote: David Rowley dgrowle...@gmail.com writes: I've attached a simple patch which removes some duplicate if conditions that seemed to have found their way into the code. These are per PVS-Studio's warnings. -1. If PVS-Studio