Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Andres Freund
On 2013-04-17 18:16:36 -0700, Daniel Farina wrote: The original paper is often shorthanded Castagnoli 93, but it exists in the IEEE's sphere of influence and is hard to find a copy of. Luckily, a pretty interesting survey paper discussing some of the issues was written by Koopman in 2002 and

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Andres Freund
On 2013-04-18 00:44:02 +0300, Ants Aasma wrote: I went ahead and coded up both the parallel FNV-1a and parallel FNV-1a + srl1-xor variants and ran performance tests and detection rate tests on both. Performance results: Mul-add checksums: 12.9 bytes/s FNV-1a checksums: 13.5 bytes/s FNV-1a

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Simon Riggs
On 17 April 2013 22:36, Bruce Momjian br...@momjian.us wrote: I would like to know the answer of how an upgrade from checksum to no-checksum would behave so I can modify pg_upgrade to allow it. Why? 9.3 pg_upgrade certainly doesn't need it. When we get to 9.4, if someone has checksums

Re: [HACKERS] event trigger API documentation?

2013-04-18 Thread Dimitri Fontaine
Peter Eisentraut pete...@gmx.net writes: Offhand, that seems about enough, but I'm just beginning to explore. I'm interested into hearing about any such use case… Chances are, event triggers will end up somewhere near the top of the release announcements, so we should have a consistent

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Daniel Farina
On Wed, Apr 17, 2013 at 11:08 PM, Andres Freund and...@2ndquadrant.com wrote: On 2013-04-17 18:16:36 -0700, Daniel Farina wrote: The original paper is often shorthanded Castagnoli 93, but it exists in the IEEE's sphere of influence and is hard to find a copy of. Luckily, a pretty interesting

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Ants Aasma
On Thu, Apr 18, 2013 at 5:08 AM, Greg Smith g...@2ndquadrant.com wrote: On 4/17/13 8:56 PM, Ants Aasma wrote: Nothing from the two points, but the CRC calculation algorithm can be switched out for slice-by-4 or slice-by-8 variant. Speed up was around factor of 4 if I remember correctly...I

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Bruce Momjian
On Thu, Apr 18, 2013 at 09:17:39AM +0100, Simon Riggs wrote: On 17 April 2013 22:36, Bruce Momjian br...@momjian.us wrote: I would like to know the answer of how an upgrade from checksum to no-checksum would behave so I can modify pg_upgrade to allow it. Why? 9.3 pg_upgrade certainly

Re: [HACKERS] event trigger API documentation?

2013-04-18 Thread Alvaro Herrera
Dimitri Fontaine escribió: Peter Eisentraut pete...@gmx.net writes: Offhand, that seems about enough, but I'm just beginning to explore. I'm interested into hearing about any such use case… Chances are, event triggers will end up somewhere near the top of the release announcements, so

Re: [HACKERS] event trigger API documentation?

2013-04-18 Thread Peter Eisentraut
On 4/18/13 5:05 AM, Dimitri Fontaine wrote: Peter Eisentraut pete...@gmx.net writes: Offhand, that seems about enough, but I'm just beginning to explore. I'm interested into hearing about any such use case… Without going into too many details (because I don't have them yet), I was thinking

Re: [HACKERS] event trigger API documentation?

2013-04-18 Thread Dimitri Fontaine
Alvaro Herrera alvhe...@2ndquadrant.com writes: You can be as much silent as you want in marketing materials (though maybe Berkus will disagree with you about being silent there), but it is not admissible to be silent in the documentation or pretend the feature is not there. Whatever got

Re: [HACKERS] event trigger API documentation?

2013-04-18 Thread Dimitri Fontaine
Peter Eisentraut pete...@gmx.net writes: Without going into too many details (because I don't have them yet), I was thinking about triggering an external test suite whenever there is a schema change in the database. So if all you want to know about is that something did change in the schema to

Re: [HACKERS] event trigger API documentation?

2013-04-18 Thread Alvaro Herrera
Dimitri Fontaine escribió: Alvaro Herrera alvhe...@2ndquadrant.com writes: You can be as much silent as you want in marketing materials (though maybe Berkus will disagree with you about being silent there), but it is not admissible to be silent in the documentation or pretend the feature

Re: [HACKERS] (auto)vacuum truncate exclusive lock

2013-04-18 Thread Jan Wieck
On 4/12/2013 1:57 PM, Tom Lane wrote: Kevin Grittner kgri...@ymail.com writes: Tom Lane t...@sss.pgh.pa.us wrote: I think that the minimum appropriate fix here is to revert the hunk I quoted, ie take out the suppression of stats reporting and analysis. I'm not sure I understand -- are you

Re: [HACKERS] (auto)vacuum truncate exclusive lock

2013-04-18 Thread Jan Wieck
On 4/12/2013 2:08 PM, Alvaro Herrera wrote: Tom Lane escribió: Are you saying you intend to revert that whole concept? That'd be okay with me, I think. Otherwise we need some thought about how to inform the stats collector what's really happening. Maybe what we need is to consider table

Re: [HACKERS] (auto)vacuum truncate exclusive lock

2013-04-18 Thread Jan Wieck
On 4/18/2013 11:44 AM, Jan Wieck wrote: Yes, that was the rationale behind it combined with don't change function call sequences and more all over the place. function call signatures -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin --

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Ants Aasma
On Thu, Apr 18, 2013 at 5:57 PM, Ants Aasma a...@cybertec.at wrote: I'll generate an avalanche diagram for CRC32C too, but it will take a while even if I use a smaller dataset. Well that was useless... In CRC flipping each bit in the input flips preset pattern of bits in the output regardless

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Jeff Davis
On Wed, 2013-04-17 at 20:21 -0400, Greg Smith wrote: -Original checksum feature used Fletcher checksums. Its main problems, to quote wikipedia, include that it cannot distinguish between blocks of all 0 bits and blocks of all 1 bits. That is fairly easy to fix by using a different modulus:

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Florian Weimer
* Greg Smith: The TCP/IP checksum spec is at https://tools.ietf.org/html/rfc793 ; its error detection limitations are described at http://www.noahdavids.org/self_published/CRC_and_checksum.html ; and a good article about optimizing its code is at

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Florian Pflug
On Apr18, 2013, at 19:04 , Jeff Davis pg...@j-davis.com wrote: On Wed, 2013-04-17 at 20:21 -0400, Greg Smith wrote: -Original checksum feature used Fletcher checksums. Its main problems, to quote wikipedia, include that it cannot distinguish between blocks of all 0 bits and blocks of all 1

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Ants Aasma
On Thu, Apr 18, 2013 at 8:05 PM, Florian Pflug f...@phlo.org wrote: On Apr18, 2013, at 19:04 , Jeff Davis pg...@j-davis.com wrote: On Wed, 2013-04-17 at 20:21 -0400, Greg Smith wrote: -Original checksum feature used Fletcher checksums. Its main problems, to quote wikipedia, include that it

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Florian Pflug
On Apr18, 2013, at 18:48 , Ants Aasma a...@cybertec.at wrote: On Thu, Apr 18, 2013 at 5:57 PM, Ants Aasma a...@cybertec.at wrote: I'll generate an avalanche diagram for CRC32C too, but it will take a while even if I use a smaller dataset. Well that was useless... In CRC flipping each bit in

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Ants Aasma
On Thu, Apr 18, 2013 at 8:15 PM, Florian Pflug f...@phlo.org wrote: So either the CRC32-C polynomial isn't irreducible, or there something fishy going on. Could there be a bug in your CRC implementation? Maybe a mixup between big and little endian, or something like that? I'm suspecting an

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Jeff Davis
On Thu, 2013-04-18 at 19:05 +0200, Florian Pflug wrote: On Apr18, 2013, at 19:04 , Jeff Davis pg...@j-davis.com wrote: On Wed, 2013-04-17 at 20:21 -0400, Greg Smith wrote: -Original checksum feature used Fletcher checksums. Its main problems, to quote wikipedia, include that it cannot

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Ants Aasma
On Thu, Apr 18, 2013 at 8:24 PM, Ants Aasma a...@cybertec.at wrote: On Thu, Apr 18, 2013 at 8:15 PM, Florian Pflug f...@phlo.org wrote: So either the CRC32-C polynomial isn't irreducible, or there something fishy going on. Could there be a bug in your CRC implementation? Maybe a mixup between

[HACKERS] Recovery target 'immediate'

2013-04-18 Thread Heikki Linnakangas
I just found out that if you use continuous archiving and online backups, it's surprisingly difficult to restore a backup, without replaying any more WAL than necessary. If you don't set a recovery target, PostgreSQL will recover all the WAL it finds. You can set recovery target time to a

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Florian Pflug
On 18.04.2013, at 20:02, Ants Aasma a...@cybertec.at wrote: On Thu, Apr 18, 2013 at 8:24 PM, Ants Aasma a...@cybertec.at wrote: On Thu, Apr 18, 2013 at 8:15 PM, Florian Pflug f...@phlo.org wrote: So either the CRC32-C polynomial isn't irreducible, or there something fishy going on. Could there

Re: [HACKERS] Enabling Checksums

2013-04-18 Thread Greg Stark
On Thu, Apr 18, 2013 at 6:04 PM, Florian Weimer f...@deneb.enyo.de wrote: The TCP checksum is too weak to be practical. Every now an then, I see data transfers where the checksum is valid, but the content contains bit flips. Well of course, it's only a 16-bit checksum. 64k packets isn't very