This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new e599e3b HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source
webhdfs url has special characters in hdfs file path. Contributed by Srinivasu
Majeti.
e599e3b is described below
commit e599e3b54617d29ed4536397481487225e5faa66
Author: Srinivasu Majeti <[email protected]>
AuthorDate: Fri May 17 19:19:12 2019 +0200
HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has
special characters in hdfs file path. Contributed by Srinivasu Majeti.
Signed-off-by: Wei-Chiu Chuang <[email protected]>
(cherry picked from commit 3e5e5b028ad7e199d08e524fe7cddeee5db51a6d)
(cherry picked from commit 85f3664bb84f393f8006d3ab781d8ce5ebb1ab62)
---
.../main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
index 6310d3e..35c49a7 100644
---
a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
+++
b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
@@ -609,7 +609,13 @@ public class WebHdfsFileSystem extends FileSystem
boolean pathAlreadyEncoded = false;
try {
fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
- pathAlreadyEncoded = true;
+ //below condition check added as part of fixing HDFS-14323 to make
+ //sure pathAlreadyEncoded is not set in the case the input url does
+ //not have any encoded sequence already.This will help pulling data
+ //from 2.x hadoop cluster to 3.x using 3.x distcp client operation
+ if(!fspathUri.getPath().equals(fspathUriDecoded)) {
+ pathAlreadyEncoded = true;
+ }
} catch (IllegalArgumentException ex) {
LOG.trace("Cannot decode URL encoded file", ex);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
