Shashikant Banerjee created HDFS-12594:
------------------------------------------
Summary: SnapshotDiff - snapshotDiff fails if the snapshotDiff .
report exceeds the RPC response limit
Key: HDFS-12594
URL: https://issues.apache.org/jira/browse/HDFS-12594
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Reporter: Shashikant Banerjee
Assignee: Shashikant Banerjee
The snapshotDiff command fails if the snapshotDiff report size is larger than
the configuration value of ipc.maximum.response.length which is by default 128
MB.
Worst case, with all Renames ops in sanpshots each with source and target name
equal to MAX_PATH_LEN which is 8k characters, this would result in at 8192
renames.
SnapshotDiff is currently used by distcp to optimize copy operations and in
case of the the diff report exceeding the limit , it fails with the below
exception:
Test set: org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.095 sec <<<
FAILURE! - in
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport
testDiffReportWithMillionFiles(org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport)
Time elapsed: 111.906 sec <<< ERROR!
java.io.IOException: Failed on local exception:
org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length;
Host Details : local host is: "hw15685.local/10.200.5.230"; destination host
is: "localhost":59808;
Attached is the proposal for the changes required.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
