[
https://issues.apache.org/jira/browse/HDFS-9670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15243389#comment-15243389
]
Yongjun Zhang commented on HDFS-9670:
-------------------------------------
Hi [~jzhuge],
Thanks for working on this issue. The solution looks good to me. One comment
about the test code here. The cost of starting Mini cluster is expensive,
ideally we could try to think about using the same cluster for the set of
tests. In this case, can we at least try to create a single cluster and do
distcp within the same cluster?
Thanks.
> DistCp throws NPE when source is root
> -------------------------------------
>
> Key: HDFS-9670
> URL: https://issues.apache.org/jira/browse/HDFS-9670
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
> Affects Versions: 2.6.0
> Reporter: Yongjun Zhang
> Assignee: John Zhuge
> Labels: supportability
> Fix For: 2.8.0
>
> Attachments: HDFS-9670.001.patch
>
>
> Symptom:
> {quote}
> [root@vb0724 ~]# hadoop distcp hdfs://X:8020/ hdfs://Y:8020/
> 16/01/20 11:33:33 INFO tools.DistCp: Input Options:
> DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
> ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
> copyStrategy='uniformsize', sourceFileListing=null,
> sourcePaths=[hdfs://X:8020/], targetPath=hdfs://Y:8020/,
> targetPathExists=true, preserveRawXattrs=false, filtersFile='null'}
> 16/01/20 11:33:33 INFO client.RMProxy: Connecting to ResourceManager at Z:8032
> 16/01/20 11:33:33 ERROR tools.DistCp: Exception encountered
> java.lang.NullPointerException
> at
> org.apache.hadoop.tools.util.DistCpUtils.getRelativePath(DistCpUtils.java:144)
> at
> org.apache.hadoop.tools.SimpleCopyListing.writeToFileListing(SimpleCopyListing.java:598)
> at
> org.apache.hadoop.tools.SimpleCopyListing.writeToFileListingRoot(SimpleCopyListing.java:583)
> at
> org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:313)
> at
> org.apache.hadoop.tools.SimpleCopyListing.doBuildListing(SimpleCopyListing.java:174)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86)
> at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:365)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:171)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:122)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:429)
> {quote}
> Relevant code:
> {code}
> private Path computeSourceRootPath(FileStatus sourceStatus,
> DistCpOptions options) throws
> IOException {
> Path target = options.getTargetPath();
> FileSystem targetFS = target.getFileSystem(getConf());
> final boolean targetPathExists = options.getTargetPathExists();
> boolean solitaryFile = options.getSourcePaths().size() == 1
> &&
> !sourceStatus.isDirectory();
> if (solitaryFile) {
> if (targetFS.isFile(target) || !targetPathExists) {
> return sourceStatus.getPath();
> } else {
> return sourceStatus.getPath().getParent();
> }
> } else {
> boolean specialHandling = (options.getSourcePaths().size() == 1 &&
> !targetPathExists) ||
> options.shouldSyncFolder() || options.shouldOverwrite();
> return specialHandling && sourceStatus.isDirectory() ?
> sourceStatus.getPath() :
> sourceStatus.getPath().getParent();
> }
> }
> {code}
> We can see that it could return NULL at the end when doing
> {{sourceStatus.getPath().getParent()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
