Re: [HACKERS] pg_filedump 9.3: checksums (and a few other fixes)

2013-06-14 Thread Jeff Davis
PageCalcChecksum16 in toto into checksum.c (or really now into checksum_impl.h), because that and not just checksum_block() is the functionality that is wanted. Agreed. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Patch for fail-back without fresh backup

2013-06-14 Thread Jeff Davis
On Fri, 2013-06-14 at 16:10 +0200, Andres Freund wrote: Jeff Davis has a patch pending (1365493015.7580.3240.camel@sussancws0025) that passes the buffer_std flag down to MarkBufferDirtyHint() for exactly that reason. I thought we were on track committing that, but rereading the thread

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
to Noah's comment) it seems like the change is safe. Are there any remaining questions or objections? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
, which is a weakness in the current checksums code. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] fallocate / posix_fallocate for new WAL file creation (etc...)

2013-06-14 Thread Jeff Davis
I had in mind for WAL creation overhead would benefit from it. Great, I'll wait on those results. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_filedump 9.3: checksums (and a few other fixes)

2013-06-10 Thread Jeff Davis
(infomask) \ (((infomask) HEAP_LOCK_MASK) == HEAP_XMAX_KEYSHR_LOCK) Presumably it'd be better to do something similar. I was hesitant to do too much interpretation of the bits. Do you think it would be better to just remove the test for XMAX_SHR_LOCK? Regards, Jeff Davis -- Sent

Re: [HACKERS] pg_filedump 9.3: checksums (and a few other fixes)

2013-06-10 Thread Jeff Davis
did for the CRC stuff recently, so that you only need access to a .h file for it? The CRC implementation is entirely in header files. Do you think we need to go that far, or is it fine to just put it in libpgport and link that to pg_filedump? Regards, Jeff Davis -- Sent via pgsql

Re: [HACKERS] Placing hints in line pointers

2013-06-09 Thread Jeff Davis
to 64kB if we used 8-byte alignment for tuples) I think that supporting 64KB blocks has merit. Interesting observation about the extra bits available. That would be an on-disk format change, so perhaps this is something to add to the list of waiting for a format change? Regards, Jeff Davis

[HACKERS] pg_filedump 9.3: checksums (and a few other fixes)

2013-06-09 Thread Jeff Davis
, because we removed page-level bits indicating the presence of a checksum. The patch is a bit ugly: I had to copy some code, and copy the entire checksum.c file (minus some Asserts, which don't work in an external program). Suggestions welcome. Regards, Jeff Davis diff -Nc pg_filedump

Re: [HACKERS] Eliminating PD_ALL_VISIBLE, take 2

2013-05-30 Thread Jeff Davis
to reuse that bit for something else, so maybe it's best to just ignore it entirely. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] visibilitymap_set and checksums

2013-05-29 Thread Jeff Davis
a better way. Agreed on all counts. Comment patches are welcome, of course. I'll add that if we remove PD_ALL_VISIBLE, this complexity disappears. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] getting rid of freezing

2013-05-29 Thread Jeff Davis
PD_ALL_VISIBLE, then this would be very simple, right? We would just follow normal logging rules for setting the visible or frozen bit. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] getting rid of freezing

2013-05-29 Thread Jeff Davis
, but it seems like it may be possible -- it may even simplify things. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] removing PD_ALL_VISIBLE

2013-05-29 Thread Jeff Davis
only need to emit XLOG_HEAP_VISIBLE for every page in the heap, rather than XLOG_HEAP_FREEZE. Agreed. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Planning incompatibilities for Postgres 10.0

2013-05-25 Thread Jeff Davis
need. That being said, I agree with you that planning in advance is important here, so that everyone knows when they need to get format-breaking changes in by. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-10 Thread Jeff Davis
that it sounds unlikely that blocks 100 and 102 would be written, but not 101. But perhaps that's more likely in systems like ZFS where the physical blocks might be in very different places. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-09 Thread Jeff Davis
streaming replication. I haven't tried it, but I'm assuming that it gives a useful error message. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-09 Thread Jeff Davis
fsync()s of the auxiliary file, should offer a lot of coverage against problems. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-08 Thread Jeff Davis
. Ideas welcome. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-07 Thread Jeff Davis
either way; merely to present the options. Not sure exactly what you mean about the code simplification, but I agree that anything more substantial than this patch should be left for 9.4. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-06 Thread Jeff Davis
On Mon, 2013-05-06 at 15:31 -0400, Robert Haas wrote: On Wed, May 1, 2013 at 3:04 PM, Jeff Davis pg...@j-davis.com wrote: Regardless, you have a reasonable claim that my patch had effects that were not necessary. I have attached a draft patch to remedy that. Only rudimentary testing

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-03 Thread Jeff Davis
On Fri, 2013-05-03 at 19:52 +0100, Simon Riggs wrote: On 1 May 2013 20:40, Jeff Davis pg...@j-davis.com wrote: Looks easy. There is no additional logic for checksums, so there's no third complexity. So we either have * cleanup info with vismap setting info * cleanup info only

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-01 Thread Jeff Davis
to be bounded to me. Regardless, you have a reasonable claim that my patch had effects that were not necessary. I have attached a draft patch to remedy that. Only rudimentary testing was done. Regards, Jeff Davis *** a/src/backend/access/heap/heapam.c --- b/src/backend/access/heap/heapam.c

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-01 Thread Jeff Davis
to MarkBufferDirty so that we could force it to do what we want in each case, but decided against it: http://www.postgresql.org/message-id/1357798000.5162.38.camel@jdavis-laptop Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-05-01 Thread Jeff Davis
the conditional logic that Robert is referring to. That being said, there may be a simpler way to achieve the same result, and that may be what you have in mind. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-30 Thread Jeff Davis
was just code complexity/readability. I tried preserving the previous behavior, and it's not so bad, but it seemed unnecessarily ugly for the benefit of a rare case. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Jeff Davis
be specified by the user by configuring with CFLAGS_EXTRA=-msse4.1. I don't know of any more required changes, aside from documentation improvements. Regards, Jeff Davis fnv-jeff-20130426.patch Description: Binary data fnv-jeff-20130426-cflags-extra.patch Description: Binary data -- Sent

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Jeff Davis
. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-26 Thread Jeff Davis
algorithm is unchanged). Regards, Jeff Davis *** a/config/c-compiler.m4 --- b/config/c-compiler.m4 *** *** 242,247 undefine([Ac_cachevar])dnl --- 242,272 + # PGAC_PROG_CC_VAR_OPT + # --- + # Given a variable name and a string, check

Re: [HACKERS] Re: [COMMITTERS] pgsql: Fix collation assignment for aggregates with ORDER BY.

2013-04-26 Thread Jeff Davis
. It was explained in the commit message why it was not backported. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-25 Thread Jeff Davis
not sure if others are waiting on me for a new patch or not. I can give the documentation issues a try, but I was hesitant to do so because you've done the research. The problems that I can correct are fairly trivial. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Jeff Davis
On Wed, 2013-04-24 at 08:20 +0100, Simon Riggs wrote: On 24 April 2013 01:10, Jeff Davis pg...@j-davis.com wrote: I'd prefer that it was some kind of a checksum ID code -- e.g. 0 for no checksum, 1 for FNV-1a-SR3, etc. That would allow us to release 9.4 with a new algorithm without forcing

Re: [HACKERS] Enabling Checksums

2013-04-24 Thread Jeff Davis
On Wed, 2013-04-24 at 21:09 +0100, Simon Riggs wrote: On 24 April 2013 21:06, Jeff Davis pg...@j-davis.com wrote: What goal are you trying to accomplish with this patch? That we might need to patch the checksum version on a production release. Oh, I see. I don't think we need two output

[HACKERS] Substituting Checksum Algorithm (was: Enabling Checksums)

2013-04-23 Thread Jeff Davis
is both effective and safe. I'd lean toward simplicity and closer adherence to the published version of the algorithm rather than detecting a few more obscure error patterns. It looks like the modification slows down the algorithm, too. Regards, Jeff Davis *** a/src/backend/storage/page

Re: [HACKERS] Enabling Checksums

2013-04-23 Thread Jeff Davis
for no checksum, 1 for FNV-1a-SR3, etc. That would allow us to release 9.4 with a new algorithm without forcing existing users to change. initdb would have to take the code as an option, probably in string form. What do you think? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Jeff Davis
the existing CRC algorithm is just too slow. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Jeff Davis
the performance based on your analysis; nor am I worried about the error detection rate. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Jeff Davis
the compiler flags for a single file so we don't need to put it in its own directory? Regards, Jeff Davis *** a/src/backend/storage/page/bufpage.c --- b/src/backend/storage/page/bufpage.c *** *** 23,28 static char pageCopyData[BLCKSZ]; /* for checksum calculation */ --- 23,29

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Jeff Davis
: 251 vs 255. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Jeff Davis
On Thu, 2013-04-18 at 19:05 +0200, Florian Pflug wrote: On Apr18, 2013, at 19:04 , Jeff Davis pg...@j-davis.com wrote: On Wed, 2013-04-17 at 20:21 -0400, Greg Smith wrote: -Original checksum feature used Fletcher checksums. Its main problems, to quote wikipedia, include that it cannot

Re: [HACKERS] Enabling Checksums

2013-04-17 Thread Jeff Davis
, if someone has checksums enabled and wants to disable it, why is pg_upgrade the right time to do that? Wouldn't it make more sense to allow them to do that at any time? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Enabling Checksums

2013-04-17 Thread Jeff Davis
expect it to be useful until we do decide to break the page format. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-12 Thread Jeff Davis
it. It might be worth a memset beforehand just to be sure. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-12 Thread Jeff Davis
will not match the checksum any more. So, we need to be sure the original page hole is all-zero when we calculate the checksum. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Enabling Checksums

2013-04-12 Thread Jeff Davis
in for data pages. The support for changing the WAL checksums seems lukewarm, and there might be quite a few alternatives (e.g. optimizing the CRC for backup blocks as Heikki suggested) to achieve that performance goal. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] Enabling Checksums

2013-04-12 Thread Jeff Davis
pages now and leaving WAL optimization until later. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Jeff Davis
to be in a separate file (if the specialization makes it in 9.3). Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Jeff Davis
on some of these points. I assume changes to the WAL checksums are 9.4 material? I'm satisfied with SIMD data checksums in 9.3 and that we have a plan for using SIMD for WAL checksums later. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-09 Thread Jeff Davis
why we allow changing that while the server is on. Either the I/O system requires it for safety or not, right? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-09 Thread Jeff Davis
easily miss the assumption that we have a standard buffer. Regards, Jeff Davis *** a/src/backend/access/hash/hash.c --- b/src/backend/access/hash/hash.c *** *** 287,293 hashgettuple(PG_FUNCTION_ARGS) /* * Since this can be redone later if needed, mark as a hint

Re: [HACKERS] Enabling Checksums

2013-04-09 Thread Jeff Davis
that this is not suitable for WAL at all? Using SIMD for WAL is not a requirement at all; I just thought it might be a nice benefit for non-checksum-enabled users in some later release. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Enabling Checksums

2013-04-05 Thread Jeff Davis
still looking into SIMD? Right now, it's using the existing CRC implementation. Obviously we can't change it after it ships. Or is it too late to change it already? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Enabling Checksums

2013-04-05 Thread Jeff Davis
more people using a slightly flawed algorithm than fewer using it because it has too great a performance impact. Unless somebody tells me not to waste my time I'll go ahead and come up with a workable patch by Monday. Sounds great to me, thank you. Regards, Jeff Davis -- Sent via

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-05 Thread Jeff Davis
, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-05 Thread Jeff Davis
-but-not-flushed WAL data makes it to disk, which could cause a false positive and may be a fairly common case. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-05 Thread Jeff Davis
that patch be applied or are you waiting on the issue Jeff Janes raised to be resolved, as well? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
it in exclusive mode after we find out we'll need to do XLogInsert. It means that MarkBufferDirtyHint may end up blocking for longer, but would avoid the memcpy. I haven't really thought through the details. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
replication mitigates the problem somewhat, by being a second place to write WAL. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
a strong opinion. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-04 Thread Jeff Davis
for this particular solution, but I do concur with Jeff Janes that it's a little bit scary and that we can entertain some ideas about how to mitigate it. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] regression test failed when enabling checksum

2013-04-03 Thread Jeff Davis
is extended but the page is not written yet. So empty pages aren't necessarily an indication of corruption (though I'd like to fix that eventually, because sometimes zeroing is corruption). Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] regression test failed when enabling checksum

2013-04-03 Thread Jeff Davis
that is because it's a new page with tuples on it, so that means something in the insert/update path ended up not writing the FPI before writing the page. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-03 Thread Jeff Davis
another patch. Not sure if that could explain this problem or not. Regards, Jeff Davis *** a/src/backend/access/heap/pruneheap.c --- b/src/backend/access/heap/pruneheap.c *** *** 257,264 heap_page_prune(Relation relation, Buffer buffer, TransactionId OldestXmin, * point

Re: [HACKERS] Hash Join cost estimates

2013-04-01 Thread Jeff Davis
whether we'll accept the plan at all, rather than adding terms to the cost estimate. Sounds reasonable. Ideally, we'd have a way to continue executing even in that case; but that's a project by itself, and would make it even more difficult to cost accurately. Regards, Jeff Davis -- Sent

Re: [HACKERS] regression test failed when enabling checksum

2013-04-01 Thread Jeff Davis
. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Hash Join cost estimates

2013-03-29 Thread Jeff Davis
of is to model the cost of comparing the hashes themselves. Also, searching the archives turns up at least one other, but I think I've seen more: http://www.postgresql.org/message-id/a82128a6-4e3b-43bd-858d-21b129f7b...@richrelevance.com Regards, Jeff Davis -- Sent via pgsql-hackers mailing

Re: [HACKERS] Hash Join cost estimates

2013-03-29 Thread Jeff Davis
On Fri, 2013-03-29 at 16:37 -0400, Tom Lane wrote: Jeff Davis pg...@j-davis.com writes: Yes, I have run into this issue (or something very similar). I don't understand why the bucketsize even matters much -- assuming few hash collisions, we are not actually evaluating the quals any more

Re: [HACKERS] regression test failed when enabling checksum

2013-03-26 Thread Jeff Davis
closely later. Regards, Jeff Davis *** a/src/backend/access/heap/rewriteheap.c --- b/src/backend/access/heap/rewriteheap.c *** *** 273,286 end_heap_rewrite(RewriteState state) /* Write the last page, if any */ if (state-rs_buffer_valid) { - PageSetChecksumInplace

Re: [HACKERS] Limiting setting of hint bits by read-only queries; vacuum_delay

2013-03-25 Thread Jeff Davis
though, and the finer-grained control can be over VACUUM. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Jeff Davis
confidence. Do you have other links we should read about this approach, or possible weaknesses? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Jeff Davis
a transition path, since we no longer use header bits. Avoiding zero might help us there. Hopefully not necessary, but something we might find useful. Also, it would help us identify situations where the checksum is never set. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Default connection parameters for postgres_fdw and dblink

2013-03-22 Thread Jeff Davis
or something (similar to search_path). Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-21 Thread Jeff Davis
. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Jeff Davis
to miss a common form of hardware failure, although it would also reduce the number of possible checksums slightly (about 4% fewer than 2^16). I'm leaning toward this option now, or a CRC of some kind if the performance is reasonable. Regards, Jeff Davis -- Sent via pgsql-hackers mailing

Re: [HACKERS] Enabling Checksums

2013-03-19 Thread Jeff Davis
not sure. On another note, I think I found a bug with the current latest patch. Ugh. Great catch, thank you! Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] Enabling Checksums

2013-03-18 Thread Jeff Davis
. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-18 Thread Jeff Davis
On Sun, 2013-03-17 at 22:26 -0700, Daniel Farina wrote: as long as I am able to turn them off easily To be clear: you don't get the performance back by doing ignore_checksum_failure = on. You only get around the error itself, which allows you to dump/reload the good data. Regards, Jeff

Re: [HACKERS] Enabling Checksums

2013-03-13 Thread Jeff Davis
On Thu, 2013-03-07 at 13:45 -0800, Jeff Davis wrote: I need to do another self-review after these changes and some more extensive testing, so I might have missed a couple things. New patch attached. Aside from rebasing, I also found a problem with temp tables. At first I was going to fix

Re: [HACKERS] Enabling Checksums

2013-03-07 Thread Jeff Davis
missed a couple things. Regards, Jeff Davis checksums-20130307.patch.gz Description: GNU Zip compressed data replace-tli-with-checksums-20130307.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Enabling Checksums

2013-03-05 Thread Jeff Davis
post a new patch with these comments addressed, probably tomorrow so that I have some time to self-review and do some basic testing. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
are somewhat suspect anyway. Also, it's a little more challenging to test corruption on a filesystem, because you need to find the location of the file you want to corrupt, and corrupt it out from underneath the filesystem. Greg may have more comments on this matter. Regards, Jeff Davis

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
the materialized view patch. I will post a new version tonight that includes those fixes as well as something to address these recent comments (probably just another GUC). Further comment in another reply. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
, but the same is true for zero_damaged_pages. So we can just still allow the writes to proceed (including setting the checksum on write), and the system should be as available as it would be without checksums. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
commit the postgres checksums? On the other side of the coin, if btrfs with checksums is exactly the same speed as ext4 with no postgres checksums (i.e. checksums are free if we use btrfs), does that mean postgres checksums should be rejected? Regards, Jeff Davis -- Sent via pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
, and will poor settings mean a bunch of postgres is slow blog posts? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
don't think that's a reasonable requirement. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
a recovery feature. A boolean seems more appropriate than a slider. That's a good point about burying the messages with DEBUG, but I think it might be slightly over-engineering it. I am willing to change it if others want it, though. Regards, Jeff Davis -- Sent via pgsql-hackers mailing

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
On Mon, 2013-03-04 at 23:22 +0200, Heikki Linnakangas wrote: On 04.03.2013 23:00, Jeff Davis wrote: On Mon, 2013-03-04 at 22:27 +0200, Heikki Linnakangas wrote: Yeah, fragmentation will certainly hurt some workloads. But how badly, and which workloads, and how does that compare

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-03-04 Thread Jeff Davis
you for the test results. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-01-27 Thread Jeff Davis
seem to go very far. Or, we could abandon the whole thing and tell people to use ZFS/btrfs/NAS/SAN. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-01-25 Thread Jeff Davis
was to add the heap buffer to the WAL chain of the XLOG_HEAP2_VISIBLE wal record when doing log_heap_visible. It seemed simpler to understand than trying to add a bunch of options to MarkBufferDirty. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Enabling Checksums

2013-01-24 Thread Jeff Davis
On Wed, 2013-01-16 at 17:38 -0800, Jeff Davis wrote: New version of checksums patch. And another new version of both patches. Changes: * Rebased. * Rename SetBufferCommitInfoNeedsSave to MarkBufferDirtyHint. Now that it's being used more places, it makes sense to give it a more generic name

Re: [HACKERS] Removing PD_ALL_VISIBLE

2013-01-22 Thread Jeff Davis
On Mon, 2013-01-21 at 12:00 +0200, Heikki Linnakangas wrote: On 21.01.2013 11:10, Jeff Davis wrote: That confuses me. The testing was to show it didn't hurt other workloads (like scans or inserts/updates/deletes); so the best possible result is that they don't show signs either way. I

Re: [HACKERS] Removing PD_ALL_VISIBLE

2013-01-21 Thread Jeff Davis
is why I'm giving up on it. If people are concerned about the costs, then I can fix those; but there's nothing I can do to increase the benefits. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Removing PD_ALL_VISIBLE

2013-01-20 Thread Jeff Davis
On Sun, 2013-01-20 at 22:19 -0500, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Jan 18, 2013 at 3:31 AM, Jeff Davis pg...@j-davis.com wrote: So, I attached a new version of the patch that doesn't look at the VM for tables with fewer than 32 pages. That's the only change

Re: [HACKERS] Removing PD_ALL_VISIBLE

2013-01-20 Thread Jeff Davis
would like to be clear about what is a known practical problem, what is a theoretical problem that has eluded my testing capabilities, and what is no problem at all. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] gistchoose vs. bloat

2013-01-20 Thread Jeff Davis
a little questionable for performance (except in those cases where few penalties are identical anyway), but could plausibly be useful for a crash report or something. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Removing PD_ALL_VISIBLE

2013-01-18 Thread Jeff Davis
On Thu, 2013-01-17 at 14:53 -0800, Jeff Davis wrote: Test plan: 1. Take current patch (without skip VM check for small tables optimization mentioned above). 2. Create 500 tables each about 1MB. 3. VACUUM them all. 4. Start 500 connections (one for each table) 5. Time the running

<    1   2   3   4   5   6   7   8   9   10   >