On Tue, 2005-05-31 at 22:36 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
Hmmm. I seem to recall asking myself why xl_prev existed if it wasn't
used, but passed that by. Damn.
I couldn't believe it'd been overlooked this long, either. It's the
sort of thing that you assume
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 31 May 2005 17:27
To: Greg Stark
Cc: Mark Cave-Ayland (External); 'Manfred Koizar'; 'Bruce
Momjian'; pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
The odds
Simon Riggs [EMAIL PROTECTED] writes:
If I/O is a problem, then surely you will agree that PreAllocXLog is
still required and should not be run by a backend?
It is still required, but it isn't run by backends --- it's fired off
during checkpoints. I think there was some discussion recently
Mark Cave-Ayland [EMAIL PROTECTED] writes:
I'm still a little nervous about dropping down to CRC32 from CRC64 and so
was just wondering what the net saving would be using one CRC64 across the
whole WAL record?
None to speak of; the startup/teardown time is trivial. It's the
per-byte cost that
It occurs to me that at least on some OSes the WAL logs are being synced with
O_SYNC or its ilk. In those cases the writes should be guaranteed to be
written out in the order postgres wrote them. So if the tail end of the WAL
entry is there (is there any sort of footer?) then the entire entry
Greg Stark [EMAIL PROTECTED] writes:
It occurs to me that at least on some OSes the WAL logs are being synced with
O_SYNC or its ilk. In those cases the writes should be guaranteed to be
written out in the order postgres wrote them. So if the tail end of the WAL
entry is there (is there any
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 27 May 2005 17:49
To: Mark Cave-Ayland (External)
Cc: 'Manfred Koizar'; 'Greg Stark'; 'Bruce Momjian';
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
I went back
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Alternatively, we might say that 64-bit CRC was overkill from
day one, and we'd rather get the additional 10% or 20% or so
speedup. I'm kinda leaning in that direction, but only weakly.
What would you need to persuade you either way? I believe
Tom Lane [EMAIL PROTECTED] writes:
It's not really a matter of backstopping the hardware's error detection;
if we were trying to do that, we'd keep a CRC for every data page, which
we don't. The real reason for the WAL CRCs is as a reliable method of
identifying the end of WAL: when the
Greg Stark [EMAIL PROTECTED] writes:
Tom Lane [EMAIL PROTECTED] writes:
It's not really a matter of backstopping the hardware's error detection;
if we were trying to do that, we'd keep a CRC for every data page, which
we don't. The real reason for the WAL CRCs is as a reliable method of
Tom Lane [EMAIL PROTECTED] writes:
Is the random WAL data really the concern? It seems like a more reliable way
of dealing with that would be to just accompany every WAL entry with a
sequential id and stop when the next id isn't the correct one.
We do that, too (the xl_prev links and
Greg Stark [EMAIL PROTECTED] writes:
Tom Lane [EMAIL PROTECTED] writes:
Is the random WAL data really the concern? It seems like a more reliable way
of dealing with that would be to just accompany every WAL entry with a
sequential id and stop when the next id isn't the correct one.
We do
On Tue, 31 May 2005 12:07:53 +0100, Mark Cave-Ayland
[EMAIL PROTECTED] wrote:
Perhaps Manfred can tell us the generator
polynomial that was used to create the lookup tables?
32 26 23 22 16 12 11 10 8 7 5 4 2 1
X + X + X + X + X + X + X + X + X + X + X + X + X +
On Tue, 2005-05-31 at 12:27 -0400, Tom Lane wrote:
Greg Stark [EMAIL PROTECTED] writes:
Tom Lane [EMAIL PROTECTED] writes:
Is the random WAL data really the concern? It seems like a more reliable
way
of dealing with that would be to just accompany every WAL entry with a
sequential id
Simon Riggs [EMAIL PROTECTED] writes:
Hmmm. I seem to recall asking myself why xl_prev existed if it wasn't
used, but passed that by. Damn.
I couldn't believe it'd been overlooked this long, either. It's the
sort of thing that you assume got done the first time :-(
PreAllocXLog was already a
-Original Message-
From: Manfred Koizar [mailto:[EMAIL PROTECTED]
Sent: 25 May 2005 20:25
To: Manfred Koizar
Cc: Tom Lane; Greg Stark; Bruce Momjian; Mark Cave-Ayland
(External); pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut
-Original Message-
From: Manfred Koizar [mailto:[EMAIL PROTECTED]
Sent: 25 May 2005 20:25
To: Manfred Koizar
Cc: Tom Lane; Greg Stark; Bruce Momjian; Mark Cave-Ayland
(External); pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Opt 32 32a32b2x32 64 64a 64b
O14.91 4.86 5.43 6.00 11.4 11.3911.39
O24.96 4.94 4.69 5.18 15.86 18.7524.73
O34.82 4.83 4.64
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 27 May 2005 15:00
To: Mark Cave-Ayland (External)
Cc: 'Manfred Koizar'; 'Greg Stark'; 'Bruce Momjian';
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
Mark Cave-Ayland
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 27 May 2005 15:00
To: Mark Cave-Ayland (External)
Cc: 'Manfred Koizar'; 'Greg Stark'; 'Bruce Momjian';
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
Not sure I
Mark Cave-Ayland [EMAIL PROTECTED] writes:
I don't know whether gcc is just producing an inefficient CRC32 compared to
2x32 but the results seem very odd There must be something else we are
missing?
I went back and looked at the code, and see that I was misled by
terminology: what we've
Tom Lane wrote:
Alternatively, we might say that 64-bit CRC was overkill from day one,
and we'd rather get the additional 10% or 20% or so speedup. I'm kinda
leaning in that direction, but only weakly.
Yes, I lean in that direction too since the CRC calculation is showing
up in our profiling.
On Wed, 18 May 2005 13:50:22 +0200, I wrote:
The most important figure is, that at MaxSpeed (/O2) 2x32 is almost
twice as fast as CRC32 while only being marginally slower than CRC32.
^
Silly typo! That should have been:
The most important figure is, that at MaxSpeed (/O2)
On E, 2005-05-16 at 12:35 -0400, Tom Lane wrote:
Given the lack of impressive advantage to the 64-bit code even on 64-bit
architectures, it might be best to go with the 32-bit code everywhere,
but I also think we have grounds to file a gcc bug report.
Maybe on other platforms , but 20% on
On T, 2005-05-17 at 22:37 -0400, Tom Lane wrote:
Bruce Momjian pgman@candle.pha.pa.us writes:
I don't understand why we are testing 64-bit CRC when I thought we
agreed that 32-bit was good enough for our purposes.
Well, we need to understand exactly what is going on here. I'd not
like to
On K, 2005-05-18 at 10:24 +0300, Hannu Krosing wrote:
On T, 2005-05-17 at 22:37 -0400, Tom Lane wrote:
Bruce Momjian pgman@candle.pha.pa.us writes:
I don't understand why we are testing 64-bit CRC when I thought we
agreed that 32-bit was good enough for our purposes.
Well, we need to
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 18 May 2005 06:12
To: Greg Stark
Cc: Bruce Momjian; Manfred Koizar; Mark Cave-Ayland
(External); pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
Greg Stark [EMAIL
On Wed, 18 May 2005 01:12:26 -0400, Tom Lane [EMAIL PROTECTED] wrote:
Wait, par for 32-bit CRCs? Or for 64-bit CRCs calculated using 32-bit ints?
Right, the latter. We haven't actually tried to measure the cost of
plain 32bit CRCs... although I seem to recall that when we originally
decided to
(BI tested crctest in two machines and two versions of gcc.
(B
(BUltraSPARC III, gcc 2.95.3:
(Bgcc -O1 crctest.c 1.321517 s
(Bgcc -O2 crctest.c 1.099186 s
(Bgcc -O3 crctest.c 1.099330 s
(Bgcc -O1 crctest64.c
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 16 May 2005 17:36
To: Mark Cave-Ayland (External)
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
I did some experimentation and concluded that gcc is screwing
On Mon, 16 May 2005, Tom Lane wrote:
I did some experimentation and concluded that gcc is screwing up
big-time on optimizing the CRC64 code for 32-bit Intel. It does much
better on every other architecture though.
Anyone want to try it with non-gcc compilers?
Solaris 9 x86 - Sun
On Mon, 16 May 2005 12:35:35 -0400, Tom Lane [EMAIL PROTECTED] wrote:
Anyone want to try it with non-gcc compilers?
MS VC++ 6.0 with various predefined optimizer settings
2x3264
Default (without any /O) 0.828125 0.906250
MinSize (contains /O1)
Manfred Koizar wrote:
On Mon, 16 May 2005 12:35:35 -0400, Tom Lane [EMAIL PROTECTED] wrote:
Anyone want to try it with non-gcc compilers?
MS VC++ 6.0 with various predefined optimizer settings
2x3264
Default (without any /O) 0.828125 0.906250
Bruce Momjian pgman@candle.pha.pa.us writes:
I don't understand why we are testing 64-bit CRC when I thought we
agreed that 32-bit was good enough for our purposes.
Well, we need to understand exactly what is going on here. I'd not
like to think that we dropped back from 64 to 32 bit because
Tom Lane wrote:
Bruce Momjian pgman@candle.pha.pa.us writes:
I don't understand why we are testing 64-bit CRC when I thought we
agreed that 32-bit was good enough for our purposes.
Well, we need to understand exactly what is going on here. I'd not
like to think that we dropped back from
Bruce Momjian pgman@candle.pha.pa.us writes:
Tom Lane wrote:
Well, we need to understand exactly what is going on here. I'd not
like to think that we dropped back from 64 to 32 bit because of one
possibly-minor optimization bug in one compiler on one platform.
Even if that compiler+platform
To: Bruce Momjian
Cc: Manfred Koizar; Mark Cave-Ayland; pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
Bruce Momjian pgman@candle.pha.pa.us writes:
Tom Lane wrote:
Well, we need to understand exactly what is going on here. I'd not
like to think
Tom Lane [EMAIL PROTECTED] writes:
Do we know that? The results I showed put at least one fundamentally
32bit platform (the PowerBook I'm typing this on) at dead par for 32bit
and 64bit CRCs.
Wait, par for 32-bit CRCs? Or for 64-bit CRCs calculated using 32-bit ints?
--
greg
Greg Stark [EMAIL PROTECTED] writes:
Tom Lane [EMAIL PROTECTED] writes:
Do we know that? The results I showed put at least one fundamentally
32bit platform (the PowerBook I'm typing this on) at dead par for 32bit
and 64bit CRCs.
Wait, par for 32-bit CRCs? Or for 64-bit CRCs calculated
Tom Lane wrote:
Greg Stark [EMAIL PROTECTED] writes:
Tom Lane [EMAIL PROTECTED] writes:
Do we know that? The results I showed put at least one fundamentally
32bit platform (the PowerBook I'm typing this on) at dead par for 32bit
and 64bit CRCs.
Wait, par for 32-bit CRCs? Or for
Dann Corbit [EMAIL PROTECTED] writes:
Probably you already knew that, and probably the birthday paradox does
not apply.
I generally use 64 bit CRCs (UMAC) for just about anything that needs a
CRC.
http://www.cs.ucdavis.edu/~rogaway/umac/
The birthday paradox doesn't come up here. The CRC
-Original Message-
From: Simon Riggs [mailto:[EMAIL PROTECTED]
Sent: 12 May 2005 16:52
To: Mark Cave-Ayland (External)
Cc: 'Christopher Kings-Lynne'; 'Tom Lane'; 'Bruce Momjian';
pgsql-hackers@postgresql.org
Subject: RE: [HACKERS] Cost of XLogInsert CRC calculations
(cut
-Original Message-
From: Mark Cave-Ayland [mailto:[EMAIL PROTECTED]
Sent: 16 May 2005 09:04
To: 'Simon Riggs'
Cc: 'Christopher Kings-Lynne'; 'Tom Lane'; 'Bruce Momjian';
'pgsql-hackers@postgresql.org'
Subject: RE: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
The program
Mark Cave-Ayland [EMAIL PROTECTED] writes:
I didn't post the sources to the list originally as I wasn't sure if the
topic were of enough interest to warrant a larger email. I've attached the
two corrected programs as a .tar.gz - crctest.c uses uint32, whereas
crctest64.c uses uint64.
I did
: Re: [HACKERS] Cost of XLogInsert CRC calculations
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Sigh, so it would help if I had added the offset to the data
pointer... ;)
Would you post the corrected program so people can try it on
a few other architectures? No point in reinventing
On Mon, 2005-05-16 at 12:12 +0100, Mark Cave-Ayland wrote:
This now gives the following (correct) result on both platforms:
Win32: 1.8GHz P4, WinXP
Linux: 2.8GHz Xeon, FC1
Win32 UINT64: 0x782104059a01660 (crc0)
~158us
Win32 UINT32: 0x78210405 (crc1),
: [HACKERS] Cost of XLogInsert CRC calculations
Win32 UINT64: 0x782104059a01660 (crc0) ~158us
Win32 UINT32: 0x78210405 (crc1), 0x59a01660 (crc0) ~58us
FC1 UINT64: 0x782104059a01660 (crc0) ~76us
FC1 UINT32: 0x78210405 (crc1), 0x59a01660 (crc0) ~29us
It would
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Sigh, so it would help if I had added the offset to the data pointer... ;)
Would you post the corrected program so people can try it on a few other
architectures? No point in reinventing the wheel, even if it is a
pretty trivial wheel.
-Original Message-
From: Christopher Kings-Lynne [mailto:[EMAIL PROTECTED]
Sent: 12 May 2005 02:46
To: Simon Riggs
Cc: Mark Cave-Ayland (External); 'Tom Lane'; 'Bruce Momjian';
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
perhaps
On Thu, 2005-05-12 at 13:48 +0100, Mark Cave-Ayland wrote:
From: Christopher Kings-Lynne [mailto:[EMAIL PROTECTED]
perhaps the CRC-32 routines could be written in in-line assembler
If you can do this, step right up. :-)
Best Regards, Simon Riggs
Surely there's an
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: 10 May 2005 23:22
To: Simon Riggs
Cc: Bruce Momjian; Mark Cave-Ayland (External);
pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Cost of XLogInsert CRC calculations
(cut)
That's awfully vague --- can't you
On Wed, 2005-05-11 at 13:40 +0100, Mark Cave-Ayland wrote:
So unless we can guarantee a minimum of 1k data per Xlog record then
Adler-32 won't be suitable.
Most records are either 8KB or much less than 1KB. Is the benefit gained
from the 8KB records worth the loss on the more frequent smaller
perhaps the CRC-32 routines could be written in in-line assembler
If you can do this, step right up. :-)
Best Regards, Simon Riggs
Surely there's an open source code floating around somewhere?
Chris
---(end of broadcast)---
TIP 4: Don't 'kill -9'
-Original Message-
From: Mark Cave-Ayland [mailto:[EMAIL PROTECTED]
Sent: 07 March 2005 11:04
To: '[EMAIL PROTECTED]'
Cc: 'pgsql-hackers@postgreSQL.org'
Subject: Re: Cost of XLogInsert CRC calculations
(cut)
I suppose that the bulk of the CPU cycles being attributed to
Tom Lane wrote:
Mark Cave-Ayland [EMAIL PROTECTED] writes:
I was just researching some articles on compression (zlib) and I saw mention
of the Adler-32 algorithm which is supposed to be slightly less accurate
than an equivalent CRC calculation but significantly faster to compute. I
Mark Cave-Ayland [EMAIL PROTECTED] writes:
I was just researching some articles on compression (zlib) and I saw mention
of the Adler-32 algorithm which is supposed to be slightly less accurate
than an equivalent CRC calculation but significantly faster to compute. I
haven't located a good
On Tue, 2005-05-10 at 10:34 -0400, Bruce Momjian wrote:
Tom Lane wrote:
Mark Cave-Ayland [EMAIL PROTECTED] writes:
I was just researching some articles on compression (zlib) and I saw
mention
of the Adler-32 algorithm which is supposed to be slightly less accurate
than an
Simon Riggs [EMAIL PROTECTED] writes:
The cause of the performance problem has been attributed to it being a
64-bit rather than 32-bit calculation. That is certainly part of it, but
I have seen evidence that there is an Intel processor stall associated
with the use of a single byte constant
On Tue, 2005-05-10 at 18:22 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
The cause of the performance problem has been attributed to it being a
64-bit rather than 32-bit calculation. That is certainly part of it, but
I have seen evidence that there is an Intel processor
On Fri, 2005-03-11 at 19:31 +0100, Hans-Jrgen Schnig wrote:
One of the things I was thinking about was whether we could use up those
cycles more effectively. If we were to include a compression routine
before we calculated the CRC that would
- reduce the size of the blocks to be written,
One of the things I was thinking about was whether we could use up those
cycles more effectively. If we were to include a compression routine
before we calculated the CRC that would
- reduce the size of the blocks to be written, hence reduce size of xlog
- reduce the following CRC calculation
I
On Mon, 2005-03-07 at 20:50 -0500, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
Well, we're using the CRC in 3 separate places...
(1) for xlog records
(2) for complete blocks copied to xlog
(3) for control files
For (1), records are so short that probably CRC16 would be
Hi Tom,
I was profiling a case involving UPDATEs into a table with too many
indexes (brought to mind by mysql's sql-bench, about which more later) and
got this rather surprising result for routines costing more than 1% of the
total runtime:
(cut)
I suppose that the bulk of the CPU cycles
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Wow, a 64-bit CRC does seem excessive, especially when going back to Zmodem
days where a 50-100k file seemed to be easily protected by a 32-bit CRC. I'm
sure there are some error rates somewhere dependent upon the polynomial and
the types of error
Tom Lane wrote:
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Wow, a 64-bit CRC does seem excessive, especially when going back to Zmodem
days where a 50-100k file seemed to be easily protected by a 32-bit CRC. I'm
sure there are some error rates somewhere dependent upon the polynomial and
the
On Mon, 2005-03-07 at 09:39 -0500, Tom Lane wrote:
Mark Cave-Ayland [EMAIL PROTECTED] writes:
Wow, a 64-bit CRC does seem excessive, especially when going back to Zmodem
days where a 50-100k file seemed to be easily protected by a 32-bit CRC. I'm
sure there are some error rates somewhere
Simon Riggs [EMAIL PROTECTED] writes:
Well, we're using the CRC in 3 separate places...
(1) for xlog records
(2) for complete blocks copied to xlog
(3) for control files
For (1), records are so short that probably CRC16 would be sufficient
without increasing the error rate noticeably.
I
Tom Lane [EMAIL PROTECTED] writes:
Simon Riggs [EMAIL PROTECTED] writes:
For (1), records are so short that probably CRC16 would be sufficient
without increasing the error rate noticeably.
The control files are so short that CRC16 would be plenty.
It's not really the size of the data
On Sun, 6 Mar 2005, Tom Lane wrote:
I suppose that the bulk of the CPU cycles being attributed to XLogInsert
are going into the inlined CRC calculations. Maybe we need to think
twice about the cost/benefit ratio of using 64-bit CRCs to protect xlog
records that are often only a few dozen bytes.
On Sun, 2005-03-06 at 00:17 -0500, Tom Lane wrote:
I was profiling a case involving UPDATEs into a table with too many
indexes (brought to mind by mysql's sql-bench, about which more later)
and got this rather surprising result for routines costing more than
1% of the total runtime:
Each
70 matches
Mail list logo
