[
https://issues.apache.org/jira/browse/HDFS-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154356#comment-14154356
]
Yongjun Zhang commented on HDFS-7036:
-------------------------------------
Hi [~wheat9],
Although I still think the right solution is to fix webhdfs because the root
cause is that webhdfs is broken, since we really had an disagreement here, I
looked into how to fix in distcp you proposed. Below is what I found. I would
appreciate if you can tell if my analysis below makes sense to you.
As a refresher, below is the interesting part of the stack trace (reported in
HDFS-6776) relevant to the discussion here.
{code}
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:440)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1018)
at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
{code}
To fix in fistcp, we should look into the boundary portion between distcp and
file system in the above stack trace:
{code
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
{code}
The boundary code in {{GlobbedCopyListing.doBuildListing}} is below
{code}
for (Path p : options.getSourcePaths()) {
FileSystem fs = p.getFileSystem(getConf());
FileStatus[] inputs = fs.globStatus(p); <================this is where
the exception is thrown
if(inputs != null && inputs.length > 0) {
for (FileStatus onePath: inputs) {
globbedPaths.add(onePath.getPath());
}
} else {
throw new InvalidInputException(p + " doesn't exist");
}
}
{code}
I assume your proposed solution is to catch the exception here and do a hacked
file listing using the {{p}} in the above code. However, this hack won't work
for *wildcard* case:, e,g,:
{code}
hadoop distcp webhdfs://<insecureCluster>/path/* webhdfs://<secureCluster>
{code}
We won't be able to find what the wildcard will expand to, which is supposed to
be got by {{FileStatus[] inputs = fs.globStatus(p);}}. Unless we hack into the
file system.
Did I misunderstand your approach? I'd appreciate if you could comment. Thanks
in advance.
BTW, You asked what applications other than distcp could apply in your last
comment, I answered you with the "hadoop fs -lsr" example. Would you please
confirm whether it addressed your question? Do you think we should fix it too?
Thanks.
> HDFS-6776 fix requires to upgrade insecure cluster, which means quite some
> user pain
> ------------------------------------------------------------------------------------
>
> Key: HDFS-7036
> URL: https://issues.apache.org/jira/browse/HDFS-7036
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.5.1
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
> Attachments: HDFS-7036.001.patch
>
>
> Issuing command
> {code}
> hadoop fs -lsr webhdfs://<insecureCluster>
> {code}
> at a secure cluster side fails with message "Failed to get the token ...",
> similar symptom as reported in HDFS-6776.
> If the fix of HDFS-6776 is applied to only the secure cluster, doing
> {code}
> distcp webhdfs://<insecureCluster> <secureCluster>
> {code}
> would fail same way.
> Basically running any application in secure cluster to access insecure
> cluster via webhdfs would fail the same way, if the HDFS-6776 fix is not
> applied to the insecure cluster.
> This could be quite some user pain. Filing this jira for a solution to make
> user's life easier.
> One proposed solution was to add a msg-parsing mechanism in webhdfs, which is
> a bit hacky. The other proposed solution is to do the same kind of hack at
> application side, which means the same hack need to be applied in each
> application.
> Thanks [~daryn], [~wheat9], [~jingzhao], [~tucu00] and [~atm] for the
> discussion in HDFS-6776.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
