Author: rangadi
Date: Wed Jul 9 10:45:23 2008
New Revision: 675265
URL: http://svn.apache.org/viewvc?rev=675265&view=rev
Log:
HADOOP-3328. When client is writing data to DFS, only the last
datanode in the pipeline needs to verify the checksum. Saves around
30% CPU on intermediate datanodes. (rangadi)
Modified:
hadoop/core/trunk/CHANGES.txt
hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
Modified: hadoop/core/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=675265&r1=675264&r2=675265&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Wed Jul 9 10:45:23 2008
@@ -72,6 +72,10 @@
singleton MessageDigester by an instance per Thread using
ThreadLocal. (Iv?n de Prado via omalley)
+ HADOOP-3328. When client is writing data to DFS, only the last
+ datanode in the pipeline needs to verify the checksum. Saves around
+ 30% CPU on intermediate datanodes. (rangadi)
+
BUG FIXES
HADOOP-3563. Refactor the distributed upgrade code so that it is
Modified:
hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
URL:
http://svn.apache.org/viewvc/hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java?rev=675265&r1=675264&r2=675265&view=diff
==============================================================================
---
hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
(original)
+++
hadoop/core/trunk/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
Wed Jul 9 10:45:23 2008
@@ -2652,7 +2652,16 @@
buf.position(buf.limit()); // move to the end of the data.
- verifyChunks(pktBuf, dataOff, len, pktBuf, checksumOff);
+ /* skip verifying checksum iff this is not the last one in the
+ * pipeline and clientName is non-null. i.e. Checksum is verified
+ * on all the datanodes when the data is being written by a
+ * datanode rather than a client. Whe client is writing the data,
+ * protocol includes acks and only the last datanode needs to verify
+ * checksum.
+ */
+ if (mirrorOut == null || clientName.length() == 0) {
+ verifyChunks(pktBuf, dataOff, len, pktBuf, checksumOff);
+ }
try {
if (!finalized) {
