[
https://issues.apache.org/jira/browse/HADOOP-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105948#comment-13105948
]
Steve Loughran commented on HADOOP-7550:
----------------------------------------
[Stone 2000] is a great paper, but it doesn't imply that a cryptographically
secure checksum is needed
# the classic Ether CRC32 is known to have problems, as Stone shows. Jumbo
Packets use a different polynomial with a reduced risk.
# IPv4 has a per-packet CRC32, though that goes away in IPv6.
# TCP and UDP packets have a 16 bit accumulator that misses byte swaps and
other events.
# Post Stone: with TCP offload engines in the NICs, problems on the server bus
may not be picked up, which argues for checksums in the RAM-layer.
I would propose using CRC32, but with the same polynomial as Jumbo Ether
Packets. It's believed to be better than classic Ether frames, and makes a lot
more sense than making up our own value.
As well as adding the checksum, we should report all failures so that the ops
team can recognise that there is a problem with a specific Node's networking. I
will delegate the suggestions there to Allen.
The other thing to consider is whether there should be a way to turn this off
-though that will complicate Facebook's protocol version negotiation, and add
a new master/worker protocol mismatch. Why the option to turn it off? It helps
to measure the cost of the feature.
Further Reading, [Loughran 2011] : Data Integrity in Hadoop,
http://www.slideshare.net/steve_l/did-you-reallywantthatdata
> Need for Integrity Validation of RPC
> ------------------------------------
>
> Key: HADOOP-7550
> URL: https://issues.apache.org/jira/browse/HADOOP-7550
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc
> Reporter: Dave Thompson
> Assignee: Dave Thompson
>
> Some recent investigation of network packet corruption has shown a need for
> hadoop RPC integrity validation beyond assurances already provided by 802.3
> link layer and TCP 16-bit CRC.
> During an unusual occurrence on a 4k node cluster, we've seen as high as 4
> TCP anomalies per second on a single node, sustained over an hour (14k per
> hour). A TCP anomaly would be an escaped link layer packet that resulted
> in a TCP CRC failure, TCP packet out of sequence
> or TCP packet size error.
> According to this paper[*]: http://tinyurl.com/3aue72r
> TCP's 16-bit CRC has an effective detection rate of 2^10. 1 in 1024 errors
> may escape detection, and in fact what originally alerted us to this issue
> was seeing failures due to bit-errors in hadoop traffic. Extrapolating from
> that paper, one might expect 14 escaped packet errors per hour for that
> single node of a 4k cluster. While the above error rate
> was unusually high due to a broadband aggregate switch issue, hadoop not
> having an integrity check on RPC makes it problematic to discover, and limit
> any potential data damage due to
> acting on a corrupt RPC message.
> ------
> [*] In case this jira outlives that tinyurl, the IEEE paper cited is:
> "Performance of Checksums and CRCs over Real Data" by Jonathan Stone, Michael
> Greenwald, Craig Partridge, Jim Hughes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
