Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Simon Riggs
On 8 November 2012 23:58, Jeff Janes jeff.ja...@gmail.com wrote: On Thu, Nov 8, 2012 at 2:50 PM, Hannu Krosing ha...@2ndquadrant.com wrote: On 11/08/2012 11:40 PM, Simon Riggs wrote: On 8 November 2012 20:36, Jeff Janes jeff.ja...@gmail.com wrote: It does not seem outrageous to me that there

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 8 November 2012 23:20, Hannu Krosing ha...@2ndquadrant.com wrote: On 11/08/2012 08:51 PM, Simon Riggs wrote: On 8 November 2012 17:07, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 7, 2012 at 10:34 AM, Simon Riggs si...@2ndquadrant.com wrote: For 9.2 we discussed having COPY

[HACKERS] ERROR: lock 9 is not held

2012-11-09 Thread er
9.3devel / AMD FX-8120 8-core / centos 6.2 / Linux 2.6.32-279.11.1.el6.x86_64 Hi, I made a test setup of 9.3devel synchronous replication (git-master as of a few hours ago): two instances on 1 machine: master: 93_1 port 6664 slave: 93_2 port 6665 I do nothing on either server. I get

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Hannu Krosing
On 11/09/2012 09:34 AM, Simon Riggs wrote: On 8 November 2012 23:20, Hannu Krosing ha...@2ndquadrant.com wrote: On 11/08/2012 08:51 PM, Simon Riggs wrote: On 8 November 2012 17:07, Robert Haas robertmh...@gmail.com wrote: On Wed, Nov 7, 2012 at 10:34 AM, Simon Riggs si...@2ndquadrant.com

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Marti Raudsepp
On Wed, Nov 7, 2012 at 5:34 PM, Simon Riggs si...@2ndquadrant.com wrote: I was unhappy with changing the behaviour of TRUNCATE, and still am. So the proposal here is to have a specific modifier on TRUNCATE command that makes it MVCC safe by throwing a serialization error. That new behaviour

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 10:28, Marti Raudsepp ma...@juffo.org wrote: On Wed, Nov 7, 2012 at 5:34 PM, Simon Riggs si...@2ndquadrant.com wrote: I was unhappy with changing the behaviour of TRUNCATE, and still am. So the proposal here is to have a specific modifier on TRUNCATE command that makes it

Re: [HACKERS] [BUG] False indication in pg_stat_replication.sync_state

2012-11-09 Thread Fujii Masao
On Fri, Nov 9, 2012 at 4:06 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Fujii Masao escribió: On Fri, Oct 19, 2012 at 10:29 PM, Fujii Masao masao.fu...@gmail.com wrote: However, I've forgotten to treat other three portions in walsender.c and syncrep.c also does

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Robert Haas
On Thu, Nov 8, 2012 at 6:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 8 November 2012 20:36, Jeff Janes jeff.ja...@gmail.com wrote: It does not seem outrageous to me that there would be real-world conditions in which invalidations would be sent more

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Robert Haas
On Fri, Nov 9, 2012 at 8:22 AM, Simon Riggs si...@2ndquadrant.com wrote: Personally I think the behavior should be dictated by the *reader*. The one doing the truncation may not know about the consistency requirements of particular readers. Especially when you do the truncate via pg_restore or

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Simon Riggs
On 9 November 2012 13:42, Robert Haas robertmh...@gmail.com wrote: On Thu, Nov 8, 2012 at 6:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 8 November 2012 20:36, Jeff Janes jeff.ja...@gmail.com wrote: It does not seem outrageous to me that there would be

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Robert Haas
On Fri, Nov 9, 2012 at 9:02 AM, Simon Riggs si...@2ndquadrant.com wrote: So, do we need a sinval overrun or just a sinval message to provoke starvation? The former would be bad but the latter would be really, really bad. IIRC the queue has 4K entries, and IIRC a single DDL operation might

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 14:01, Robert Haas robertmh...@gmail.com wrote: I think the question that hasn't really been adequately answered is: where and how are we going to track conflicts? Your previous patch involved storing an XID in pg_class, but I think we both found that a bit grotty - it'd

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Markus Wanner
Jeff, On 11/09/2012 02:01 AM, Jeff Davis wrote: For the sake of simplicity (implementation as well as usability), it seems like there is agreement that checksums should be enabled or disabled for the entire instance, not per-table. Agreed. I've quickly thought about making it a per-database

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Simon Riggs
On 9 November 2012 14:16, Robert Haas robertmh...@gmail.com wrote: On Fri, Nov 9, 2012 at 9:02 AM, Simon Riggs si...@2ndquadrant.com wrote: So, do we need a sinval overrun or just a sinval message to provoke starvation? The former would be bad but the latter would be really, really bad. IIRC

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Markus Wanner
On 11/09/2012 06:18 AM, Jesper Krogh wrote: I would definately stuff our system in state = 2 in your description if it was available. Hm.. that's an interesting statement. What's probably worst when switching from OFF to ON is the VACUUM run that needs to touch every page (provided you haven't

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Kevin Grittner
Simon Riggs wrote: Robert Haas robertmh...@gmail.com wrote: One of us is confused, because IIUC Tom just fixed this this morning, and I'm trying to figure out how many users will be affected by it, and how seriously. Like, do we need an immediate minor release? You asked what provokes

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So, do we need a sinval overrun or just a sinval message to provoke starvation? The former would be bad but the latter would be really, really bad. The former. Actually, a signal will be sent as soon as a backend is determined to be unreasonably far

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Marti Raudsepp
On Fri, Nov 9, 2012 at 4:24 PM, Simon Riggs si...@2ndquadrant.com wrote: We need a fast lookup structure that is expandable to accommodate arbitrary numbers of truncates. Shared hash table, with some form of overflow mechanism. Surely you only need to remember the last completed truncate for

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Kevin Grittner
Robert Haas wrote: What I've been wondering since this last came up is whether we could use some variant of the SIREAD locks Kevin introduced for SSI to handle this case - essentially have the transaction doing the TRUNCATE make an entry in the lock table that will force a serialization

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Alvaro Herrera
Tom Lane escribió: Robert Haas robertmh...@gmail.com writes: IIRC the queue has 4K entries, and IIRC a single DDL operation might provoke a couple of sinvals, but I'm thinking that somebody would probably have to be creating 1024 temp tables a minute to overrun the queue, which is very

Re: [HACKERS] AutoVacuum starvation from sinval messages

2012-11-09 Thread Robert Haas
On Fri, Nov 9, 2012 at 10:08 AM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Tom Lane escribió: Robert Haas robertmh...@gmail.com writes: IIRC the queue has 4K entries, and IIRC a single DDL operation might provoke a couple of sinvals, but I'm thinking that somebody would probably have

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 14:55, Marti Raudsepp ma...@juffo.org wrote: On Fri, Nov 9, 2012 at 4:24 PM, Simon Riggs si...@2ndquadrant.com wrote: We need a fast lookup structure that is expandable to accommodate arbitrary numbers of truncates. Shared hash table, with some form of overflow mechanism.

Re: [HACKERS] WIP checksums patch

2012-11-09 Thread Robert Haas
On Thu, Nov 8, 2012 at 9:17 PM, Christopher Browne cbbro...@gmail.com wrote: I see one thing to be concerned about, there... I imagine it would not be a totally happy thing if the only way to switch it on/off was to use Slony or Londiste to replicate into a database with the opposite setting.

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 14:22, Kevin Grittner kgri...@mail.com wrote: Robert Haas wrote: What I've been wondering since this last came up is whether we could use some variant of the SIREAD locks Kevin introduced for SSI to handle this case - essentially have the transaction doing the TRUNCATE

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Merlin Moncure
On Fri, Nov 9, 2012 at 8:22 AM, Kevin Grittner kgri...@mail.com wrote: Robert Haas wrote: What I've been wondering since this last came up is whether we could use some variant of the SIREAD locks Kevin introduced for SSI to handle this case - essentially have the transaction doing the

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Kevin Grittner
Merlin Moncure wrote: Kevin Grittner kgri...@mail.com wrote: Robert Haas wrote: It seems to me that the goal would be to make this semantically idential to the behavior users would see if an unqualified DELETE were run against the table rather than a TRUNCATE. but, triggers would not

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 15:34, Kevin Grittner kgri...@mail.com wrote: If we're not talking about making conflicts with other transactions behave just the same as an unqualified DELETE from a user perspective, I'm not sure what the goal is, exactly. Reasonable question. My goal is to allow COPY to

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 15:46, Simon Riggs si...@2ndquadrant.com wrote: Altering TRUNCATE so it behaves perfectly from an MVCC/Serializable perspective is a much bigger, and completely different goal, as well as something I don't see as desirable anyway for at least 2 good reasons, as explained.

Re: [HACKERS] DEALLOCATE IF EXISTS

2012-11-09 Thread Vik Reykja
On Tue, Oct 9, 2012 at 4:44 PM, Vik Reykja vikrey...@gmail.com wrote: On Tue, Oct 9, 2012 at 4:09 PM, Tom Lane t...@sss.pgh.pa.us wrote: =?ISO-8859-1?Q?S=E9bastien_Lardi=E8re?= slardi...@hi-media.com writes: Indeed, brackets was not correct, it's better now (I think), and correct some

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Kevin Grittner
Simon Riggs wrote: This seems like a subset of the issues which one might want to address by making DDL statement behave in a more strictly MVCC fashion. Does it make sense to pick those off one at a time, or should something like this be done only in the context of an overall plan to deal

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Kevin Grittner
Simon Riggs wrote: My goal is to allow COPY to load frozen tuples without causing MVCC violations. OK. That wasn't initially clear to me. Forcing a tightly scoped proposal into a much wider one will just kill this and leave it blocked. The goal is important enough and narrow enough to

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 9 November 2012 15:34, Kevin Grittner kgri...@mail.com wrote: If we're not talking about making conflicts with other transactions behave just the same as an unqualified DELETE from a user perspective, I'm not sure what the goal is, exactly.

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Robert Haas
On Fri, Nov 9, 2012 at 9:22 AM, Kevin Grittner kgri...@mail.com wrote: (1) Any attempt to read from the truncated table would not block. The pg_class version included in the transaction's snapshot would determine which heap and indexes were accessed. Well, the thing is, you can't actually do

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Robert Haas
On Fri, Nov 9, 2012 at 11:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: My goal is to allow COPY to load frozen tuples without causing MVCC violations. If that's the goal, I question why you're insisting on touching TRUNCATE's behavior. We already have the principle that TRUNCATE is like

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Josh Berkus
Jeff, I don't think a GUC entirely makes sense (in its current form, anyway). We basically care about 3 states: Huh? Why would a GUC not make sense? How else would you make sure that checksums where on when you started the system? 1. Off: checksums are not written, nor are they verified.

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-09 Thread Jeff Janes
On Thu, Nov 8, 2012 at 9:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: Are sure the server you are dumping out of is head? I experimented a bit with dumping/restoring 16000 tables matching Bruce's test case (ie, one serial column apiece). The pg_dump

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Jeff Davis
On Fri, 2012-11-09 at 15:42 +0100, Markus Wanner wrote: On 11/09/2012 06:18 AM, Jesper Krogh wrote: I would definately stuff our system in state = 2 in your description if it was available. Hm.. that's an interesting statement. What's probably worst when switching from OFF to ON is the

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Jeff Davis
On Thu, 2012-11-08 at 23:33 -0300, Alvaro Herrera wrote: There's no such thing as a system-wide VACUUM. The most you can get is a database-wide VACUUM, which means you'd have to store the state per-database somewhere (presumably the pg_database catalog), and perhaps pg_control could have it

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Markus Wanner
On 11/09/2012 07:53 PM, Jeff Davis wrote: One problem is telling which pages are protected and which aren't. We can have a couple bits in the header indicating that a checksum is present, but it's a little disappointing to have only a few bits protecting a 16-bit checksum. Given your

Re: [HACKERS] My first patch! (to \df output)

2012-11-09 Thread Jon Erdman
On Oct 27, 2012, at 10:45 AM, Pavel Stehule pavel.steh...@gmail.com wrote: Hello 2012/10/27 Jon Erdman postgre...@thewickedtribe.net: Hello Hackers! So, currently the only way to see if a function is security definer or not is to directly query pg_proc. This is both irritating, and I

Re: [HACKERS] My first patch! (to \df output)

2012-11-09 Thread Jon Erdman
Oops! Here it is in the proper diff format. I didn't have my env set up correctly :( describe.patch Description: Binary data -- Jon T Erdman Postgresql Zealot On Nov 9, 2012, at 1:53 PM, Jon Erdman postgre...@thewickedtribe.net wrote: On Oct 27, 2012, at 10:45 AM, Pavel Stehule

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Just having an option to preload frozen tuples dodges all of these issues by throwing our hands up in the air, but it does have the advantage of being more general. Even if we do that I'm not sure it would be a bad thing to try to solve this issue in

Re: [HACKERS] TRUNCATE SERIALIZABLE and frozen COPY

2012-11-09 Thread Simon Riggs
On 9 November 2012 16:27, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: On 9 November 2012 15:34, Kevin Grittner kgri...@mail.com wrote: If we're not talking about making conflicts with other transactions behave just the same as an unqualified DELETE from a user

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-09 Thread Bruce Momjian
On Thu, Nov 8, 2012 at 08:59:21PM -0800, Jeff Janes wrote: On Thu, Nov 8, 2012 at 4:33 PM, Bruce Momjian br...@momjian.us wrote: On Thu, Nov 8, 2012 at 03:46:09PM -0800, Jeff Janes wrote: On Wed, Nov 7, 2012 at 6:17 PM, Bruce Momjian br...@momjian.us wrote: As a followup to Magnus's

Re: [HACKERS] Enabling Checksums

2012-11-09 Thread Jeff Davis
On Fri, 2012-11-09 at 20:48 +0100, Markus Wanner wrote: Given your description of option 2 I was under the impression that each page already has a bit indicating whether or not the page is protected by a checksum. Why do you need more bits than that? The bit indicating that a checksum is

Re: [HACKERS] WIP checksums patch

2012-11-09 Thread Jeff Davis
On Mon, 2012-11-05 at 12:19 -0500, Robert Haas wrote: Yeah. I definitely think that we could shed an enormous amount of complexity by deciding that this is, for now, an option that can only be selected at initdb time. That would remove approximately 85% of everything I've ever disliked about

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-09 Thread Andres Freund
On 2012-11-08 12:30:11 -0500, Peter Eisentraut wrote: On 11/7/12 9:17 PM, Bruce Momjian wrote: As a followup to Magnus's report that pg_upgrade was slow for many tables, I did some more testing with many tables, e.g.: CREATE TABLE test991 (x SERIAL); I ran it for 0, 1k, 2k, ...

Re: [HACKERS] WIP checksums patch

2012-11-09 Thread Jeff Davis
On Fri, 2012-11-09 at 10:18 -0500, Robert Haas wrote: Sure, I agree. I don't think it should stay that way forever, but removing the burden of dealing with this issue from the initial commit would likely allow that commit to happen this release cycle, perhaps even in the next CommitFest. And

[HACKERS] Inadequate thought about buffer locking during hot standby replay

2012-11-09 Thread Tom Lane
During normal running, operations such as btree page splits are extremely careful about the order in which they acquire and release buffer locks, if they're doing something that concurrently modifies multiple pages. During WAL replay, that all goes out the window. Even if an individual

Re: [HACKERS] WIP checksums patch

2012-11-09 Thread Jeff Davis
On Mon, 2012-10-01 at 10:22 -0700, Josh Berkus wrote: I think that's OK, because it's still protected by the WAL CRC, and there's no expectation that the checksum is correct in shared buffers, and the correct checksum should be set on the next checkpoint. Just an observation. We'd need

Re: [HACKERS] Inadequate thought about buffer locking during hot standby replay

2012-11-09 Thread Andres Freund
On 2012-11-09 18:24:25 -0500, Tom Lane wrote: I can't prove that this is the cause of bug #7648, but it's fairly easy to see that it could explain the symptom. You only need to assume that the page-being-split had been handled as a full-page image, and that the new right-hand page had gotten

Re: [HACKERS] Inadequate thought about buffer locking during hot standby replay

2012-11-09 Thread Daniel Farina
On Fri, Nov 9, 2012 at 3:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: During normal running, operations such as btree page splits are extremely careful about the order in which they acquire and release buffer locks, if they're doing something that concurrently modifies multiple pages. During

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-09 Thread Jeff Janes
On Thu, Nov 8, 2012 at 7:25 PM, Bruce Momjian br...@momjian.us wrote: I did some more research and realized that I was not using --schema-only like pg_upgrade uses. With that setting, things look like this: ... For profiling pg_dump in isolation, you should also specify --binary-upgrade. I

Re: [HACKERS] Inadequate thought about buffer locking during hot standby replay

2012-11-09 Thread Tom Lane
Andres Freund and...@anarazel.de writes: On 2012-11-09 18:24:25 -0500, Tom Lane wrote: I'm inclined to think that we need to fix this by getting rid of RestoreBkpBlocks per se, and instead having the per-WAL-record restore routines dictate when each full-page image is restored (and whether or

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-09 Thread Jeff Janes
On Fri, Nov 9, 2012 at 3:06 PM, Bruce Momjian br...@momjian.us wrote: On Thu, Nov 8, 2012 at 08:59:21PM -0800, Jeff Janes wrote: On Thu, Nov 8, 2012 at 4:33 PM, Bruce Momjian br...@momjian.us wrote: I am actually now dumping git head/9.3, so I assume all the problems we know about should

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL

2012-11-09 Thread Amit kapila
On Thursday, November 08, 2012 7:56 PM Amit Kapila On Thursday, November 08, 2012 1:45 AM Robert Haas wrote: On Wed, Nov 7, 2012 at 2:50 PM, Josh Berkus j...@agliodbs.com wrote: Well, Magnus' proposed implementation supposed that the existing values *have* been loaded into the current