Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-26 Thread Merlin Moncure
On Wed, Apr 25, 2012 at 9:28 AM, Merlin Moncure wrote: >> That patch actually has more than one optimization in it, I think, but >> the basic idea is that if we could figure out a way to set >> HEAP_XMIN_COMMITTED when loading data into a table created or >> truncated within the same transaction,

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-25 Thread Robert Haas
On Tue, Apr 24, 2012 at 8:52 PM, Greg Stark wrote: > On Tue, Apr 24, 2012 at 9:40 PM, Robert Haas wrote: >>  For three things, index pages >> have hint-type changes that are not single-bit changes. > > ? Just how big are these? Part of the reason hint bit updates are safe > is because one bit def

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-25 Thread Merlin Moncure
On Tue, Apr 24, 2012 at 3:40 PM, Robert Haas wrote: > On Sat, Apr 21, 2012 at 7:08 PM, Greg Stark wrote: >> The earlier consensus was to move all the hint bits to a dedicated >> area and exclude them from the checksum. I think double-write buffers >> seem to have become more fashionable but a sum

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-24 Thread Josh Berkus
On 4/21/12 2:40 PM, Jeff Davis wrote: > If we do use WAL for hint bit updates, that has an impact on Hot > Standby, because HS can't write WAL. So, it would seem that HS could not > set hint bits. If we're WAL-logging hint bits, then the standby would be receiving them, so it doesn't *need* to wri

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-24 Thread Greg Stark
On Tue, Apr 24, 2012 at 9:40 PM, Robert Haas wrote: >  For three things, index pages > have hint-type changes that are not single-bit changes. ? Just how big are these? Part of the reason hint bit updates are safe is because one bit definitely absolutely has to be entirely in one page. You can't

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-24 Thread Robert Haas
On Sat, Apr 21, 2012 at 7:08 PM, Greg Stark wrote: > The earlier consensus was to move all the hint bits to a dedicated > area and exclude them from the checksum. I think double-write buffers > seem to have become more fashionable but a summary that doesn't > describe the former is definitely inco

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-24 Thread Robert Haas
On Sat, Apr 21, 2012 at 5:40 PM, Jeff Davis wrote: > * In addition to detecting random garbage, we also need to be able to > detect zeroing of pages. Right now, a zero page is not considered > corrupt, so that's a problem. We'll need to WAL table extension > operations, and we'll need to mitigate

Re: [HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-21 Thread Greg Stark
On Sat, Apr 21, 2012 at 10:40 PM, Jeff Davis wrote: > * In addition to detecting random garbage, we also need to be able to > detect zeroing of pages. Right now, a zero page is not considered > corrupt, so that's a problem. We'll need to WAL table extension > operations, and we'll need to mitigate

[HACKERS] 9.3: summary of corruption detection / checksums / CRCs discussion

2012-04-21 Thread Jeff Davis
A lot of discussion took place regarding corruption detection, and I am attempting to summarize it in a useful way. Please excuse the lack of references; I'm hoping to agree on the basic problem space and the nature of the solutions offered, and then turn it into a wiki where we can get into the de