Re: [HACKERS] Checksums, state of play

2012-03-08 Thread Peter Geoghegan
On 7 March 2012 20:56, Bruce Momjian br...@momjian.us wrote: Yep, good summary.  Giving ourselves a few months to think about it and consider other failure cases will make this a great 9.3 addition. Recent Intel processors that support SSE 4.2, including those based on the core

Re: [HACKERS] Checksums, state of play

2012-03-08 Thread Thom Brown
On 8 March 2012 20:55, Peter Geoghegan pe...@2ndquadrant.com wrote: On 7 March 2012 20:56, Bruce Momjian br...@momjian.us wrote: Yep, good summary.  Giving ourselves a few months to think about it and consider other failure cases will make this a great 9.3 addition. Recent Intel processors

Re: [HACKERS] Checksums, state of play

2012-03-07 Thread Robert Haas
On Tue, Mar 6, 2012 at 2:27 PM, Bruce Momjian br...@momjian.us wrote: The feature is no where near complete, and we should not be designing features at this stage. I agree, on both counts. Although Simon did a good job pulling together something that basically works in a short amount of time,

Re: [HACKERS] Checksums, state of play

2012-03-07 Thread Simon Riggs
On Wed, Mar 7, 2012 at 5:28 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Mar 6, 2012 at 2:27 PM, Bruce Momjian br...@momjian.us wrote: The feature is no where near complete, and we should not be designing features at this stage. I agree, on both counts.  Although Simon did a good job

Re: [HACKERS] Checksums, state of play

2012-03-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: So I think it's time to push this one out to 9.3. Frankly, I thought it was probably too late for 9.2 when it was first proposed, and the number of issues that have been identified since then just confirm that feeling. There is another point beyond the

Re: [HACKERS] Checksums, state of play

2012-03-07 Thread Robert Haas
On Wed, Mar 7, 2012 at 3:09 PM, Simon Riggs si...@2ndquadrant.com wrote: Neither do I. It's pretty clear from our last discussion that the fix proposed doesn't actually work fully so I don't think its going to be either more robust or more certain to give low false positives. So I don't think

Re: [HACKERS] Checksums, state of play

2012-03-07 Thread Bruce Momjian
On Wed, Mar 07, 2012 at 03:33:34PM -0500, Robert Haas wrote: On Wed, Mar 7, 2012 at 3:09 PM, Simon Riggs si...@2ndquadrant.com wrote: Neither do I. It's pretty clear from our last discussion that the fix proposed doesn't actually work fully so I don't think its going to be either more

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Mon, Mar 5, 2012 at 11:29 PM, Josh Berkus j...@agliodbs.com wrote: 3. Pages with checksums set need to have a version marking to show that they are a later version of the page layout. That version number needs to be extensible to many later versions. Pages of multiple versions need to

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Mon, Mar 5, 2012 at 10:03 AM, Simon Riggs si...@2ndquadrant.com wrote: 1. We don't need them because there will be something better in a later release. I don't think anybody disagrees that a better solution is possible in the future; doubts have been expressed as to what will be required

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 2:25 PM, Robert Haas robertmh...@gmail.com wrote: 4. Checksums are being removed, but some blocks may still have them. Thus, it's not an error for a block to have no checksum, but any still-remaining checksums should be correct (though possibly we ought not to complain

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 2:25 PM, Robert Haas robertmh...@gmail.com wrote: For the reasons stated above, I believe pd_tli is less useful than pd_pagesize_version.  I fear that if we repurpose pd_pagesize_version, we're going to make things very difficult for people who want to write

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 3:31 PM, Simon Riggs si...@2ndquadrant.com wrote: I'll keep an open mind for now about database/table level. I'm not sure how possible/desirable each is. Table level sounds great, but how will it work with recovery? We don't have a relcache in Startup process. So either

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 10:31 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 2:25 PM, Robert Haas robertmh...@gmail.com wrote: 4. Checksums are being removed, but some blocks may still have them. Thus, it's not an error for a block to have no checksum, but any

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 10:40 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 2:25 PM, Robert Haas robertmh...@gmail.com wrote: For the reasons stated above, I believe pd_tli is less useful than pd_pagesize_version.  I fear that if we repurpose pd_pagesize_version, we're

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 4:06 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 3:31 PM, Simon Riggs si...@2ndquadrant.com wrote: I'll keep an open mind for now about database/table level. I'm not sure how possible/desirable each is. Table level sounds great, but how will it

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 4:42 PM, Robert Haas robertmh...@gmail.com wrote: As to whether we should increment pd_pagesize_version, I'm not sure quite what you were saying about that (I think you may have an extra or missing word there), but I don't think it's necessary here. I said this... On

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 12:00 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 4:42 PM, Robert Haas robertmh...@gmail.com wrote: As to whether we should increment pd_pagesize_version, I'm not sure quite what you were saying about that (I think you may have an extra or

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Heikki Linnakangas
On 06.03.2012 19:00, Simon Riggs wrote: On Tue, Mar 6, 2012 at 4:42 PM, Robert Haasrobertmh...@gmail.com wrote: As to whether we should increment pd_pagesize_version, I'm not sure quite what you were saying about that (I think you may have an extra or missing word there), but I don't think

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 11:50 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 4:06 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 3:31 PM, Simon Riggs si...@2ndquadrant.com wrote: I'll keep an open mind for now about database/table level. I'm not sure

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 5:14 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Has EDB or anybody else you know of has used the pd_pagesize_version field for something else, so you'd rather I didn't touch that? The EDB page format is exactly the same as the community one.

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 5:14 PM, Robert Haas robertmh...@gmail.com wrote: Options (1) Recovery ignores checksums until db in consistent state (2) Recovery ignores checksums until all databases are enabled, when we set flag in pg_control (3) Recovery checks blocks marked as having a

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 12:23 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Mar 6, 2012 at 5:14 PM, Robert Haas robertmh...@gmail.com wrote: Options (1) Recovery ignores checksums until db in consistent state (2) Recovery ignores checksums until all databases are enabled, when we set

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Bruce Momjian
On Tue, Mar 06, 2012 at 09:25:17AM -0500, Robert Haas wrote: 2. Turning checksums on/off/on/off in rapid succession can cause false positive reports of checksum failure if crashes occur and are ignored. That may lead to the feature and PostgreSQL being held in disrepute. This I do think

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Bruce Momjian
On Mon, Mar 05, 2012 at 03:03:18PM +, Simon Riggs wrote: To avoid any confusion as to where this proposed feature is now, I'd like to summarise my understanding, make proposals and also request clear feedback on them. Checksums have a number of objections to them outstanding. 1. We

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 5:50 PM, Bruce Momjian br...@momjian.us wrote: One crazy idea would be to have a checksum _version_ number somewhere on the page and in pg_controldata.  When you turn on checksums, you increment that value, and all new checksum pages get that checksum version;  if you

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: That's not why I want to leave that field alone, though: I want to leave that field alone for backward and forward compatibility, so that any version of community PostgreSQL ever released - and any page inspection tools, current or future - can look at

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Bruce Momjian
On Tue, Mar 06, 2012 at 06:00:13PM +, Simon Riggs wrote: On Tue, Mar 6, 2012 at 5:50 PM, Bruce Momjian br...@momjian.us wrote: One crazy idea would be to have a checksum _version_ number somewhere on the page and in pg_controldata.  When you turn on checksums, you increment that

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Simon Riggs
On Tue, Mar 6, 2012 at 5:56 PM, Bruce Momjian br...@momjian.us wrote: On Mon, Mar 05, 2012 at 03:03:18PM +, Simon Riggs wrote: To avoid any confusion as to where this proposed feature is now, I'd like to summarise my understanding, make proposals and also request clear feedback on them.

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Bruce Momjian
On Tue, Mar 06, 2012 at 01:52:31PM -0500, Bruce Momjian wrote: On Tue, Mar 06, 2012 at 06:00:13PM +, Simon Riggs wrote: On Tue, Mar 6, 2012 at 5:50 PM, Bruce Momjian br...@momjian.us wrote: One crazy idea would be to have a checksum _version_ number somewhere on the page and in

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Bruce Momjian
On Tue, Mar 06, 2012 at 07:09:23PM +, Simon Riggs wrote: On Tue, Mar 6, 2012 at 5:56 PM, Bruce Momjian br...@momjian.us wrote: On Mon, Mar 05, 2012 at 03:03:18PM +, Simon Riggs wrote: To avoid any confusion as to where this proposed feature is now, I'd like to summarise my

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 12:50 PM, Bruce Momjian br...@momjian.us wrote: I think the turning checksums on/off/on/off is really a killer problem, and obviously many of the actions needed to make it safe make the checksum feature itself less useful. One crazy idea would be to have a checksum

Re: [HACKERS] Checksums, state of play

2012-03-06 Thread Marcin Mańk
On Tue, Mar 06, 2012 at 07:09:23PM +, Simon Riggs wrote: The problem is actually on/off/crash/on in quick succession which is much less likely. I must be missing something, but how about: if (!has_checksums page_loses_checksum_due_to_hint_bit_write) wal_log_the_hint_bit_write();

[HACKERS] Checksums, state of play

2012-03-05 Thread Simon Riggs
To avoid any confusion as to where this proposed feature is now, I'd like to summarise my understanding, make proposals and also request clear feedback on them. Checksums have a number of objections to them outstanding. 1. We don't need them because there will be something better in a later

Re: [HACKERS] Checksums, state of play

2012-03-05 Thread Josh Berkus
3. Pages with checksums set need to have a version marking to show that they are a later version of the page layout. That version number needs to be extensible to many later versions. Pages of multiple versions need to exist within the server to allow simple upgrades and migration. This is