Dennis Kubes wrote:
It turns out that ECC memory did the trick. We replaced all memory on
our 50 node cluster with ECC memory and it has just completed a 50
Million page crawl and merge with 0 errors. Before we would have 10-20
errors or more on this job.
I still find it interesting that the non-ecc memory passed all burn-in
and hardware tests yet still failed randomly under production
conditions. I guess a good rule of thumb would be for production nutch
and hadoop systems ECC memory is always the way to go. Anyways, thanks
for all the help in getting this problem resolved.
This is good validation how important ECC memory is. Currently HDFS
client deletes a block when it notices a checksum error. After moving to
Block level CRCs soon, we should make Datanode re-validate the block
before deciding to delete it.
Raghu.
Dennis Kubes
Dennis Kubes wrote:
Doug Cutting wrote:
Dennis Kubes wrote:
Do we know if this is a hardware issue. If it is possibly a
software issue I can dedicate some resources to tracking down bugs.
I would just need a little guidance on where to start looking?
We don't know. The checksum mechanism is designed to catch hardware
problems. So one must certainly consider that as a likely cause. If
it is instead a software bug then it should be reproducible. Are you
seeing any consistent patterns? If not, then I'd lean towards hardware.
Michael Stack has some experience tracking down problems with flaky
memory. Michael, did you use a test program to validate the memory
on a node?
Again, do your nodes have ECC memory?
Sorry, I was checking on that. No, the nodes don't have ECC memory.
I just priced it out and it is only $20 more per Gig to go ECC so I
think that is what we are going to do. We are going to do some tests
and I will keep the list updated on the progress. Thanks for your help.
Dennis Kubes
Doug
