In the WAL we just need to be able to detect torn pages and stop
reading WAL at that point. That's easier and doesn't really need a
CRC. We could just adopt the Sybase strategy of storing a unique id
number every 512 bytes throughout the WAL page. If those numbers
don't
match then we
On Fri, 2006-10-27 at 10:54 +0200, Zeugswetter Andreas ADI SD wrote:
In the WAL we just need to be able to detect torn pages and stop
reading WAL at that point. That's easier and doesn't really need a
CRC. We could just adopt the Sybase strategy of storing a unique id
number every 512
On Fri, Oct 27, 2006 at 10:11:08AM +0100, Simon Riggs wrote:
Putting xl_prev to the end helps only for direct IO WAL sync methods,
else we would need it on every page.
[There is already an XLogRecPtr on each 8k page.]
Given that hardware sector size is still 512 bytes, should there be a
Putting xl_prev to the end helps only for direct IO WAL sync
methods, else we would need it on every page.
[There is already an XLogRecPtr on each 8k page.]
Given that hardware sector size is still 512 bytes, should
there be a way of detecting a missing 512 byte block in the
On Tue, 2006-10-24 at 15:42 -0400, Gregory Stark wrote:
Tom Lane [EMAIL PROTECTED] writes:
Simon is essentially arguing that if we are willing to assume no
incomplete write() we may as well assume it for WAL too. This seems
to me to be raising the risk significantly, but I admit that I
Simon Riggs [EMAIL PROTECTED] writes:
I've looked into this in more depth following your suggestion: I think
it seems straightforward to move the xl_prev field from being a header
to a trailer. That way when we do the test on the back pointer we will
be assured that there is no torn page
On Mon, Oct 23, 2006 at 05:23:27PM -0400, Tom Lane wrote:
Mark Kirkwood [EMAIL PROTECTED] writes:
Right - I think the regression is caused by libc and kernel being built
with gcc 3.4.6 and the test program being built with gcc 4.1.2.
Why do you think that? The performance of the CRC
On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
On Mon, 2006-10-23 at 13:52 -0400, Tom Lane wrote:
No can do --- we rely on the checksums to be able to tell when we've hit
the end of WAL during replay.
No we don't: Zero length records are the
Simon Riggs [EMAIL PROTECTED] writes:
On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
There are actually three checks used to detect end of WAL: zero record
length, invalid checksum, and incorrect back-pointer. Zero length is
the first and cleanest-looking test, but AFAICS we have to have
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
There are actually three checks used to detect end of WAL: zero record
length, invalid checksum, and incorrect back-pointer. Zero length is
the
On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
There are actually three checks used to detect end of WAL: zero record
length,
[EMAIL PROTECTED] writes:
The 5 year claim seems
decades too short unless they are talking about a newer technology.
I think what Simon is on about is CRCs being routinely used on the cable
between the disk drive and the CPU. When I was involved in this stuff
you usually only got a parity bit
On Tue, 2006-10-24 at 10:18 -0400, [EMAIL PROTECTED] wrote:
On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
No, because unlike tuples, WAL records can and do cross page boundaries.
But not that often, with full_page_writes =
On Tue, Oct 24, 2006 at 05:05:58PM +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 10:18 -0400, [EMAIL PROTECTED] wrote:
On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
No, because unlike tuples, WAL records can and do cross
On Tue, 2006-10-24 at 12:47 -0400, [EMAIL PROTECTED] wrote:
On Tue, Oct 24, 2006 at 05:05:58PM +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 10:18 -0400, [EMAIL PROTECTED] wrote:
On Tue, Oct 24, 2006 at 02:52:36PM +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane
On Tue, 2006-10-24 at 14:52 +0100, Simon Riggs wrote:
On Tue, 2006-10-24 at 09:37 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
On Mon, 2006-10-23 at 15:12 -0400, Tom Lane wrote:
There are actually three checks used to detect end of WAL: zero record
length, invalid
[EMAIL PROTECTED] wrote:
I'm on wrong on one of these assumptions, I'm open to being educated.
My opinion as of a few seconds ago, is that a write to a single disk
sector is safe, but that a write that extends across several sectors
is not.
Unless it's fsync'ed, which is what we do at
On 10/24/06, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote:
I wasn't aware that a system could protect against this. :-)
I write 8 Kbytes - how can I guarantee that the underlying disk writes
all 8 Kbytes before it loses power? And why isn't the CRC a valid means
of dealing with this? :-)
[snip]
Gregory Maxwell [EMAIL PROTECTED] writes:
I'm not aware of any other system which can guaranteed the atomicity
of 8k writes.
The reasoning for supporting full_page_writes = off is that if you have
a stable kernel and suitable backup power, battery backed write cache,
etc, your risk of a
Sorry for getting into the conversation so late... It was a long weekend in India.On 10/23/06, Tom Lane
[EMAIL PROTECTED]
wrote:I didn't particularly trust the timing calculations in your benchmark
program, Any particular reason? (why and what did you doubt in it?). I designed the prog. to be
Tom Lane [EMAIL PROTECTED] writes:
Simon is essentially arguing that if we are willing to assume no
incomplete write() we may as well assume it for WAL too. This seems
to me to be raising the risk significantly, but I admit that I can't
put my finger on why exactly.
Actually I think we can
On Tue, 2006-10-24 at 14:07 -0400, Tom Lane wrote:
Gregory Maxwell [EMAIL PROTECTED] writes:
I'm not aware of any other system which can guaranteed the atomicity
of 8k writes.
The reasoning for supporting full_page_writes = off is that if you have
a stable kernel and suitable backup
Gurjeet Singh [EMAIL PROTECTED] writes:
On 10/23/06, Tom Lane [EMAIL PROTECTED] wrote:
I didn't particularly trust the timing calculations in your benchmark
program,
Any particular reason? (why and what did you doubt in it?).
Well, the specific thing that set off my bogometer was
Tom Lane wrote:
Mark Kirkwood [EMAIL PROTECTED] writes:
Interesting that the slice-8 algorithm seems to work noticeably better
on FreeBSD than Linux -
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
On Sun, 2006-10-22 at 17:18 +0200, Martijn van Oosterhout wrote:
On Sun, Oct 22, 2006 at 08:10:56PM +0530, Gurjeet Singh wrote:
Hi all,
Michael Kounavis has given a green signal (please refer the forwarded
message).
I don't think that helps. The publishing date of this article was
On Mon, 23 Oct 2006 07:35:08 +0200, Tom Lane [EMAIL PROTECTED] wrote:
I realize I forgot to document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
PPC: gcc version 4.0.1 (Apple Computer, Inc. build 5363)
both Intels: gcc version 4.1.1 20060525 (Red Hat 4.1.1-1)
Interesting
Tom Lane wrote:
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
PPC: gcc version 4.0.1 (Apple Computer, Inc. build 5363)
both Intels: gcc version 4.1.1 20060525 (Red Hat 4.1.1-1)
Interesting that it's
On Mon, 23 Oct 2006, Mark Kirkwood wrote:
Tom Lane wrote:
Yah, I checked. Several times... but if anyone else wants to repeat
the experiment, please do. Or look for bugs in either my test case
or Gurjeet's.
Just for fun, I tried it out with both GCC and with Intel's C compiler
Simon Riggs [EMAIL PROTECTED] writes:
Slice-By-8 was first mentioned here:
Are you sure?
US patent 7,047,479 filed in 2002 sounds like it may be relevant:
http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1Sect2=HITOFFd=PALLp=1u=%2Fnetahtml%2FPTO%2Fsrchnum.htmr=1f=Gl=50s1=7047479.PN.
--
Mark Kirkwood wrote:
Tom Lane wrote:
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
Ah - good point, FreeBSD is using an older compiler:
FreeBSD: gcc (GCC) 3.4.6 [FreeBSD] 20060305
Linux: gcc (GCC) 4.1.1 (Gentoo 4.1.1)
Hmm - there is a
On Mon, 2006-10-23 at 05:22 -0400, Gregory Stark wrote:
Simon Riggs [EMAIL PROTECTED] writes:
Slice-By-8 was first mentioned here:
Are you sure?
US patent 7,047,479 filed in 2002 sounds like it may be relevant:
MK == Mark Kirkwood [EMAIL PROTECTED] writes:
MK Here are the results after building gcc 4.1.2 (repeating results
MK for gcc 3.4.6 for comparison). I suspect that performance is
MK probably impacted because gcc 4.1.2 (and also the rest of the
MK tool-chain) is built with gcc 3.4.6 - but it
Jeremy Drake [EMAIL PROTECTED] writes:
So at this point I realize that intel's compiler is optimizing the loop
away, at least for the std crc and probably for both. So I make mycrc an
array of 2, and substript mycrc[j1] in the loop.
That's not a good workaround, because making mycrc expensive
On Sun, 2006-10-22 at 18:06 -0400, Tom Lane wrote:
These numbers are um, not impressive. Considering that a large fraction
of our WAL records are pretty short, the fact that slice8 consistently
loses at short buffer lengths is especially discouraging. Much of that
ground could be made up
Simon Riggs [EMAIL PROTECTED] writes:
Instead, I'd like to include a parameter to turn off CRC altogether, for
heavily CPU bound operations and the WAL drive on trustworthy hardware.
No can do --- we rely on the checksums to be able to tell when we've hit
the end of WAL during replay. You may
On 10/23/06, Tom Lane [EMAIL PROTECTED] wrote:
It's not so much that I don't trust Intel as that a CRC algorithm is
exactly the sort of nice little self-contained thing that people love
to try to patent these days. What I am really afraid of is that someone
else has already invented this same
Bruce, Tom, All:
Given the numbers I posted earlier today, the proposal is dead in the
water anyway, quite aside from any legal considerations.
Agreed. I just wanted to point out we have other sharks in the water.
*IF* Slice-by-8 turned out to be a winner, I could get the legal issues
On Mon, 2006-10-23 at 13:52 -0400, Tom Lane wrote:
Simon Riggs [EMAIL PROTECTED] writes:
Instead, I'd like to include a parameter to turn off CRC altogether, for
heavily CPU bound operations and the WAL drive on trustworthy hardware.
No can do --- we rely on the checksums to be able to
In most cases, it would be foolish to avoid: but there are cases where
the data is CRC checked by the hardware/system already, so I'd like to
make an option to turn this off, defaulting to on, for safety.
How would we know? What are those cases?
Sounds like a foot gun to me.
Sincerely,
Simon Riggs [EMAIL PROTECTED] writes:
On Mon, 2006-10-23 at 13:52 -0400, Tom Lane wrote:
No can do --- we rely on the checksums to be able to tell when we've hit
the end of WAL during replay.
No we don't: Zero length records are the trigger for EOF.
Only if the file happens to be all-zero
On Mon, 23 Oct 2006, Tom Lane wrote:
Jeremy Drake [EMAIL PROTECTED] writes:
So at this point I realize that intel's compiler is optimizing the loop
away, at least for the std crc and probably for both. So I make mycrc an
array of 2, and substript mycrc[j1] in the loop.
That's not a good
Jeremy Drake [EMAIL PROTECTED] writes:
On Mon, 23 Oct 2006, Tom Lane wrote:
That's not a good workaround, because making mycrc expensive to access
means your inner loop timing isn't credible at all. Instead try making the
buffer array nonlocal --- malloc it, perhaps.
That did not make any
On Mon, 23 Oct 2006, Tom Lane wrote:
Hmm. Maybe store the CRCs into a global array somewhere?
uint32 results[NTESTS];
for ...
{
INIT/COMP/FIN_CRC32...
results[j] = mycrc;
}
This still adds a bit of overhead to the outer loop, but not
Benny Amorsen wrote:
MK == Mark Kirkwood [EMAIL PROTECTED] writes:
MK Here are the results after building gcc 4.1.2 (repeating results
MK for gcc 3.4.6 for comparison). I suspect that performance is
MK probably impacted because gcc 4.1.2 (and also the rest of the
MK tool-chain) is built with
Mark Kirkwood [EMAIL PROTECTED] writes:
Right - I think the regression is caused by libc and kernel being built
with gcc 3.4.6 and the test program being built with gcc 4.1.2.
Why do you think that? The performance of the CRC loop shouldn't depend
at all on either libc or the kernel, because
Tom Lane wrote:
Mark Kirkwood [EMAIL PROTECTED] writes:
Right - I think the regression is caused by libc and kernel being built
with gcc 3.4.6 and the test program being built with gcc 4.1.2.
Why do you think that? The performance of the CRC loop shouldn't depend
at all on either libc or
Hi all, Michael Kounavis has given a green signal (please refer the forwarded message).Best regards,-- [EMAIL PROTECTED][EMAIL PROTECTED] gmail | hotmail | yahoo }.com
-- Forwarded message --From: Kounavis, Michael E [EMAIL PROTECTED]
Date: Oct 20, 2006 10:43 PMSubject: RE: CRC
On 10/22/06, Gregory Maxwell [EMAIL PROTECTED] wrote:
On Opteron 265 his test code shows SB8 (the intel alg) is 2.48x fasterfor checksum and 1.95x faster for verify for the 800 * 1024 blocks of1 KB each workload. For 10 blocks of 8k I got simmlar results aswell.
I think you meant800*1024
On Sun, Oct 22, 2006 at 08:10:56PM +0530, Gurjeet Singh wrote:
Hi all,
Michael Kounavis has given a green signal (please refer the forwarded
message).
I don't think that helps. The publishing date of this article was March
2006. If this is really the first time this algorithm was
On 10/22/06, Martijn van Oosterhout kleptog@svana.org wrote:
On Sun, Oct 22, 2006 at 08:10:56PM +0530, Gurjeet Singh wrote:
Hi all,
Michael Kounavis has given a green signal (please refer the forwarded
message).
I don't think that helps. The publishing date of this article was March
Gurjeet Singh [EMAIL PROTECTED] writes:
If possible, people should test it on different platforms, so as to
ensure that it doesn't perform any worse than older implementation on any
supported platform (please post the results, if you do test it).
I didn't particularly trust the timing
Tom Lane wrote:
Bruce Momjian [EMAIL PROTECTED] writes:
However, I am not sure getting a clarification from the author even
helps us legally. Also, why are we more critical of an Intel-provided
idea than any other idea we get from the community?
Bitter experience with other companies.
Bruce Momjian [EMAIL PROTECTED] writes:
Tom Lane wrote:
Bruce Momjian [EMAIL PROTECTED] writes:
Also, why are we more critical of an Intel-provided
idea than any other idea we get from the community?
Bitter experience with other companies.
The problem is we have lots of companies
Tom Lane wrote:
Bruce Momjian [EMAIL PROTECTED] writes:
Tom Lane wrote:
Bruce Momjian [EMAIL PROTECTED] writes:
Also, why are we more critical of an Intel-provided
idea than any other idea we get from the community?
Bitter experience with other companies.
The problem is we have
On Sun, Oct 22, 2006 at 06:06:13PM -0400, Tom Lane wrote:
Intel Xeon EM64T (Fedora Core 5)
8192 bytes4.4208797.633120
1024 bytes0.5717940.819372
64 bytes 0.0473540.071906
Intel Pentium 4 (Fedora Core 5)
8192 bytes6.94232428.848572
[EMAIL PROTECTED] writes:
... My first speculation is
that you did your Intel numbers backwards. My second speculation is
that you already thought of that and confirmed before posting. :-)
Yah, I checked. Several times... but if anyone else wants to repeat
the experiment, please do. Or look
Tom Lane wrote:
Yah, I checked. Several times... but if anyone else wants to repeat
the experiment, please do. Or look for bugs in either my test case
or Gurjeet's.
FWIW - FreeBSD and Linux results using Tom's test program on almost
identical hardware[1]:
Std crc
Mark Kirkwood [EMAIL PROTECTED] writes:
Interesting that the slice-8 algorithm seems to work noticeably better
on FreeBSD than Linux -
Are you running similar gcc versions on both? I realize I forgot to
document what I was using:
HPPA: gcc version 2.95.3 20010315 (release)
PPC: gcc version
Gurjeet Singh wrote:
On 10/20/06, Alvaro Herrera [EMAIL PROTECTED] wrote:
It doesn't say anything about
being free of patents though.
The Sourceforge project referred to in the article (but for which no
link is given) seems to be this one:
Bruce Momjian [EMAIL PROTECTED] writes:
However, I am not sure getting a clarification from the author even
helps us legally. Also, why are we more critical of an Intel-provided
idea than any other idea we get from the community?
Bitter experience with other companies.
So unless we hear
On 10/21/06, Tom Lane [EMAIL PROTECTED] wrote:
[snip]
It hasn't even been tested. One thing I'd want to know about is the
performance effect on non-Intel machines.
On Opteron 265 his test code shows SB8 (the intel alg) is 2.48x faster
for checksum and 1.95x faster for verify for the 800 *
Resending... The previous 3 attempt haven't succeeded, probably because of size restrictions; so sending in two parts.On 10/18/06, Gurjeet Singh
[EMAIL PROTECTED] wrote:Hi all, Please refer to the following link for a new algorithm to calculate CRC, developed by Intel.
Gurjeet Singh [EMAIL PROTECTED] writes:
Please refer to the following link for a new algorithm to calculate
CRC, developed by Intel.
http://www.intel.com/technology/magazine/communications/slicing-by-8-0306.htm
I can't help wondering what the patent status of this algorithm is.
(This from a
Tom Lane wrote:
Gurjeet Singh [EMAIL PROTECTED] writes:
Please refer to the following link for a new algorithm to calculate
CRC, developed by Intel.
http://www.intel.com/technology/magazine/communications/slicing-by-8-0306.htm
I can't help wondering what the patent status of this
On 10/20/06, Alvaro Herrera [EMAIL PROTECTED] wrote:
It doesn't say anything aboutbeing free of patents though.The Sourceforge project referred to in the article (but for which nolink is given) seems to be this one:
http://sourceforge.net/projects/slicing-by-8Yes. I had already mentioned that link
65 matches
Mail list logo
