Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-03 Thread Andreas Seltenreich
Alvaro Herrera writes: Why not just use an unsigned 64 bit variable? Also, perhaps palloc_huge() avoids the whole problem in the first place ... Ja, that crossed my mind too, but the current limit is already far beyond anything that is usually configured for per-backend memory use, so I

Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-03 Thread Andreas Seltenreich
Piotr Stefaniak writes: s/int/Size/ doesn't fix anything on 32-bit machines. Postgres requires twos-complement representation, so that the assumption that signed integer types wrap around on overflow can be safely made. Thanks for the clarification! -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Alvaro Herrera
Andres Freund wrote: On 2015-06-03 15:01:46 -0300, Alvaro Herrera wrote: One idea I had was: what if the oldestMulti pointed to another multi earlier in the same 0046 file, so that it is read-as-zeroes (and the file is created), and then a subsequent multixact truncate tries to read a

Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-03 Thread Andres Freund
On June 3, 2015 9:42:21 PM GMT+02:00, Andreas Seltenreich andreas.seltenre...@credativ.de wrote: Piotr Stefaniak writes: s/int/Size/ doesn't fix anything on 32-bit machines. Postgres requires twos-complement representation, so that the assumption that signed integer types wrap around on

Re: [HACKERS] Re: [COMMITTERS] pgsql: Map basebackup tablespaces using a tablespace_map file

2015-06-03 Thread Andrew Dunstan
On 06/02/2015 11:55 PM, Amit Kapila wrote: On Tue, Jun 2, 2015 at 10:26 PM, Andrew Dunstan and...@dunslane.net mailto:and...@dunslane.net wrote: On 05/15/2015 02:21 AM, Amit Kapila wrote: Find the patch which gets rid of rmtree usage. I have made it as a separate

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Robert Haas
On Wed, Jun 3, 2015 at 8:24 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Jun 2, 2015 at 5:22 PM, Andres Freund and...@anarazel.de wrote: Hm. If GetOldestMultiXactOnDisk() gets the starting point by scanning the disk it'll always get one at a segment boundary, right? I'm not sure

Re: [HACKERS] [PATCH] Document that directly callable functions may use fn_extra

2015-06-03 Thread Jim Nasby
On 5/29/15 10:21 AM, Peter Eisentraut wrote: On 5/28/15 10:15 PM, Craig Ringer wrote: I was a puzzled by src/backend/utils/fmgr/README and fmgr.h's descriptions of fcinfo-flinfo-fn_extra (FmgrInfo.fn_extra) as they seem to conflict with actual usage. The docs suggest that fl_extra is for the

[HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Thomas Munro
On Mon, Jun 1, 2015 at 4:55 PM, Noah Misch n...@leadboat.com wrote: While testing this (with inconsistent-multixact-fix-master.patch applied, FWIW), I noticed a nearby bug with a similar symptom. TruncateMultiXact() omits the nextMXact==oldestMXact special case found in each other

Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-03 Thread Michael Paquier
On Thu, Jun 4, 2015 at 1:32 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Andreas Seltenreich wrote: Tom Lane t...@sss.pgh.pa.us writes: Andreas Seltenreich andreas.seltenre...@credativ.de writes: The scary one is due to an integer overflow the attached patch also fixes.

Re: [HACKERS] pg_xlog - pg_xjournal?

2015-06-03 Thread Jim Nasby
On 6/2/15 4:58 PM, David Steele wrote: On 5/31/15 1:46 PM, Tom Lane wrote: Hm. I think the impact on third-party backup tools would be rather bad, but there's a simple modification of the idea that might fix that: just always create pg_xlog as a symlink to pg_xjournal during initdb. Anybody

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Alvaro Herrera
Robert Haas wrote: So here's a patch taking a different approach. I tried to apply this to 9.3 but it's messy because of pgindent. Anyone would have a problem with me backpatching a pgindent run of multixact.c? Also, you have a new function SlruPageExists, but we already have

Re: [HACKERS] xid wrap / optimize frozen tables?

2015-06-03 Thread Jeff Janes
On May 24, 2015 6:42 AM, Nils Goroll sl...@schokola.de wrote: Hi Jeff and all, On 23/05/15 22:13, Jeff Janes wrote: Are you sure it is the read IO that causes the problem? Yes. Trouble is here that we are talking about a 361 GB table List of relations

Re: [HACKERS] Restore-reliability mode

2015-06-03 Thread Geoff Winkless
On 3 June 2015 at 14:50, Noah Misch n...@leadboat.com wrote: I ​ ​ would define the subject matter as bug fixes, testing and review, not restructuring, testing and review. Different code structures are clearest to different hackers. Restructuring, on average, adds bugs even more quickly

[HACKERS] Restore-reliability mode

2015-06-03 Thread Noah Misch
Subject changed from Re: [CORE] postpone next week's release. On Sat, May 30, 2015 at 10:48:45PM -0400, Bruce Momjian wrote: Well, I think we stop what we are doing, focus on restructuring, testing, and reviewing areas that historically have had problems, and when we are done, we can look to

Re: [HACKERS] Restore-reliability mode

2015-06-03 Thread Andres Freund
On 2015-06-03 09:50:49 -0400, Noah Misch wrote: Second, I would define the subject matter as bug fixes, testing and review, not restructuring, testing and review. Different code structures are clearest to different hackers. Restructuring, on average, adds bugs even more quickly than feature

Re: [HACKERS] Restore-reliability mode

2015-06-03 Thread Joshua D. Drake
On 06/03/2015 07:18 AM, Andres Freund wrote: On 2015-06-03 09:50:49 -0400, Noah Misch wrote: Second, I would define the subject matter as bug fixes, testing and review, not restructuring, testing and review. Different code structures are clearest to different hackers. Restructuring, on

Re: [HACKERS] auto_explain sample rate

2015-06-03 Thread Craig Ringer
On 3 June 2015 at 20:04, Andres Freund and...@anarazel.de wrote: On 2015-06-03 18:54:24 +0800, Craig Ringer wrote: OK, here we go. Hm. Wouldn't random sampling be better than what you do? If your queries have a pattern to them (e.g. you always issue the same 10 queries in succession), this

Re: [HACKERS] [CORE] Restore-reliability mode

2015-06-03 Thread Josh Berkus
On 06/03/2015 06:50 AM, Noah Misch wrote: Subject changed from Re: [CORE] postpone next week's release. On Sat, May 30, 2015 at 10:48:45PM -0400, Bruce Momjian wrote: If we have to totally stop feature development until we are all happy with the code we have, so be it. If people feel they

Re: [HACKERS] [PATCH] Add error handling to byteaout.

2015-06-03 Thread Alvaro Herrera
Andreas Seltenreich wrote: Tom Lane t...@sss.pgh.pa.us writes: Andreas Seltenreich andreas.seltenre...@credativ.de writes: The scary one is due to an integer overflow the attached patch also fixes. s/int/Size/ doesn't fix anything on 32-bit machines. Well, it changes the signedness

Re: [HACKERS] anole: assorted stability problems

2015-06-03 Thread Alvaro Herrera
Tom Lane wrote: Andres Freund and...@anarazel.de writes: Uh. I'm pretty sure there were some back when that patch went in. And there definitely used to be a couple earlier. I guess itanium really is dying (mixed bad: It's a horrible architecture, but more coverage would still be good).

Re: [HACKERS] Further issues with jsonb semantics, documentation

2015-06-03 Thread Peter Geoghegan
On Wed, Jun 3, 2015 at 7:02 PM, Peter Geoghegan p...@heroku.com wrote: Consider this case: postgres=# select '{c:5, a:6, b:7}'::jsonb - 1; ?column? -- {a: 6, c: 5} (1 row) Clearly anyone expecting the value a to be removed here would be in for a surprise. Moreover,

[HACKERS] Further issues with jsonb semantics, documentation

2015-06-03 Thread Peter Geoghegan
I've noticed some more issues with the jsonb documentation, and the new jsonb stuff generally. I didn't set out to give Andrew feedback on the semantics weeks after feature freeze, but unfortunately this feels like another discussion that we need to have now rather than later. operator jsonb -

Re: [HACKERS] Re: [COMMITTERS] pgsql: Map basebackup tablespaces using a tablespace_map file

2015-06-03 Thread Amit Kapila
On Thu, Jun 4, 2015 at 1:52 AM, Andrew Dunstan and...@dunslane.net wrote: On 06/02/2015 11:55 PM, Amit Kapila wrote: On Tue, Jun 2, 2015 at 10:26 PM, Andrew Dunstan and...@dunslane.net mailto:and...@dunslane.net wrote: Well, it seems to me the new function is being altogether way too

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Alvaro Herrera
Alvaro Herrera wrote: Really, the whole question of how this code goes past the open() failure in SlruPhysicalReadPage baffles me. I don't see any possible way for the file to be created ... Hmm, the checkpointer can call TruncateMultiXact when in recovery, on restartpoints. I wonder if in

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Andres Freund
On 2015-06-03 15:01:46 -0300, Alvaro Herrera wrote: Andres Freund wrote: That's not necessarily the case though, given how the code currently works. In a bunch of places the SLRUs are accessed *before* having been made consistent by WAL replay. Especially if several checkpoints/vacuums

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Andres Freund
On 2015-06-03 00:42:55 -0300, Alvaro Herrera wrote: Thomas Munro wrote: On Tue, Jun 2, 2015 at 9:30 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My guess is that the file existed, and perhaps had one or more pages, but the wanted page doesn't exist, so we tried to read but got 0

Re: [HACKERS] [CORE] Restore-reliability mode

2015-06-03 Thread Andres Freund
On 2015-06-03 10:21:28 -0700, Josh Berkus wrote: So, historically, this is what the period between feature freeze and beta1 was for; the consolidation phase was supposed to deal with this. The problem over the last few years, by my observation, has been that consolidation has been left to

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Alvaro Herrera
Thomas Munro wrote: I have finally reproduced that error! See attached repro shell script. The conditions are: 1. next multixact == oldest multixact (no active multixacts, pointing past the end) 2. next multixact would be the first item on a new page (multixact % 2048 == 0) 3. the

Re: [CORE] [HACKERS] postpone next week's release

2015-06-03 Thread Stefan Kaltenbrunner
On 05/31/2015 03:51 AM, David Steele wrote: On 5/30/15 8:38 PM, Joshua D. Drake wrote: On 05/30/2015 03:48 PM, David Steele wrote: On 5/30/15 2:10 PM, Robert Haas wrote: What, in this release, could break things badly? RLS? Grouping sets? Heikki's WAL format changes? That last one sounds

Re: [HACKERS] Re: [GENERAL] 9.4.1 - 9.4.2 problem: could not access status of transaction 1

2015-06-03 Thread Alvaro Herrera
Andres Freund wrote: On 2015-06-03 00:42:55 -0300, Alvaro Herrera wrote: Thomas Munro wrote: On Tue, Jun 2, 2015 at 9:30 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: My guess is that the file existed, and perhaps had one or more pages, but the wanted page doesn't exist, so