[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111814#comment-15111814
]
Aaron T. Myers commented on HDFS-7037:
--
[~wheat9] - reviving this very old thread... can you please respond to my
points above? As I've said previously, your only objection seems to be that
this introduces a security vulnerability, but as I've pointed out several times
already, we as a project have chosen not to treat this issue as a security
vulnerability in other areas, and thus I think we should go ahead and check in
this change.
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111818#comment-15111818
]
Hadoop QA commented on HDFS-7037:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HDFS-7037 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12668640/HDFS-7037.001.patch |
| JIRA Issue | HDFS-7037 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/14199/console |
This message was automatically generated.
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804594#comment-14804594
]
Haohui Mai commented on HDFS-7037:
--
bq. adding this capability to HFTP does not change the security semantics of
Hadoop at all, since RPC and other interfaces used for remote access already
support allowing configurable insecure fallback
Please correct me if I misunderstood. (1) The current behavior of RPC / WebHDFS
is less than ideal and it is vulnerable to attack. (2) You argue that the
proposed changes makes HFTP vulnerable for the fallback, but it is no worse
than what we have in RPC / WebHDFS today.
As an analogy, it seems to me that the argument is that it's okay to have a
broken window given that we have many broken windows already?
My question is that is there a need to create yet another workaround, given
that we know that it is prone for security vulnerability? I'd like to
understand your use cases better? Can you please elaborate why you'll need
another workaround in HFTP, given that you guys have put the workaround in
WebHDFS already?
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804611#comment-14804611
]
Aaron T. Myers commented on HDFS-7037:
--
bq. Please correct me if I misunderstood. (1) The current behavior of RPC /
WebHDFS is less than ideal and it is vulnerable to attack. (2) You argue that
the proposed changes makes HFTP vulnerable for the fallback, but it is no worse
than what we have in RPC / WebHDFS today.
Correct.
bq. As an analogy, it seems to me that the argument is that it's okay to have a
broken window given that we have many broken windows already?
I don't think that's a reasonable analogy. The point you were making is that
this change introduces a possible security vulnerability. I'm saying that this
is demonstrably not a security vulnerability, since we consciously chose to add
this capability to other interfaces. HADOOP-11701 will make things configurably
more secure for all interfaces, but that's a separate discussion.
bq. My question is that is there a need to create yet another workaround, given
that we know that it is prone for security vulnerability?
Like I said above, this should not be considered a security vulnerability. If
it is, then we should have never added this capability to WebHDFS/RPC, and we
should be reverting it from WebHDFS/RPC right now.
bq. I'd like to understand your use cases better? Can you please elaborate why
you'll need another workaround in HFTP, given that you guys have put the
workaround in WebHDFS already?
Simple: because some users use HFTP and not WebHDFS, specifically for distcp
from older clusters.
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746406#comment-14746406
]
Aaron T. Myers commented on HDFS-7037:
--
[~wheat9] - with regard to your comment that "the security concerns remain
unaddressed," could you please respond to this point specifically:
bq. adding this capability to HFTP does not change the security semantics of
Hadoop at all, since RPC and other interfaces used for remote access already
support allowing configurable insecure fallback. This is not a security
vulnerability. If it were, we should be removing the ability to configure
insecure fallback at all in Hadoop. We're not doing that, because it was a
deliberate choice to add that feature.
i.e., this change _is not changing the security level of Hadoop_, so I don't
understand what security concerns you have with this change. This change is
proposing to expand the fallback capability that already exists in other RPC
interfaces to the HFTP interface.
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746325#comment-14746325
]
Aaron T. Myers commented on HDFS-7037:
--
[~wheat9] - it's been 5 months and I've received no response from you on this
matter, and there's been no progress made on HADOOP-11701. As I said back in
April, I don't think that fixing this bug in HFTP should not be gated on
implementing that new feature. Would you please consider changing your -1 to a
-0, so that we can fix this issue for users who are encountering this problem?
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746344#comment-14746344
]
Hadoop QA commented on HDFS-7037:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12668640/HDFS-7037.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 34ef1a0 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12459/console |
This message was automatically generated.
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14746350#comment-14746350
]
Haohui Mai commented on HDFS-7037:
--
It looks like nothing has changed so far. The security concerns remain
unaddressed, thus I think my -1 still holds. Just to echo my previous comments
I'm willing to change it to -0 if there are solutions like HADOOP-11701 to
limit the impact of such a configuration. I suggest doing something that is
along the line with HADOOP-11701 in this patch to wrap up this jira.
> Using distcp to copy data from insecure to secure cluster via hftp doesn't
> work (branch-2 only)
>
>
> Key: HDFS-7037
> URL: https://issues.apache.org/jira/browse/HDFS-7037
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security, tools
>Affects Versions: 2.6.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Labels: BB2015-05-TBR
> Attachments: HDFS-7037.001.patch
>
>
> This is a branch-2 only issue since hftp is only supported there.
> Issuing "distcp hftp:// hdfs://" gave the
> following failure exception:
> {code}
> 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
> remote token:
> java.io.IOException: Error when dealing remote token: Internal Server Error
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
> at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
> at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
> at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
> at
> org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
> at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
> at
> org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
> at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
> 14/09/13 22:07:40 WARN security.UserGroupInformation:
> PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
> cause:java.io.IOException: Unable to obtain remote token
> 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
> java.io.IOException: Unable to obtain remote token
> at
> org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
> at
> org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
> at
> org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
> at
>
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484301#comment-14484301
]
Aaron T. Myers commented on HDFS-7037:
--
Thanks for the reply, [~wheat9].
As I've said previously, adding this capability to HFTP does not change the
security semantics of Hadoop at all, since RPC and other interfaces used for
remote access already support allowing configurable insecure fallback. This is
_not_ a security vulnerability. If it were, we should be removing the ability
to configure insecure fallback at all in Hadoop. We're not doing that, because
it was a deliberate choice to add that feature. Given that, I still don't
understand why you'd be unwilling to fix this issue in HFTP. HFTP, like WebHDFS
and RPC, is supposed to be able to work with either secure or insecure
clusters, when configured to do so. It should be viewed as a bug that HFTP
doesn't currently work, whereas the others do. Implementing HADOOP-11701 is a
good idea in general, but fixing this bug in HFTP should not be gated on
implementing that new feature.
So, I'll ask again, would you please consider changing your -1 to a -0?
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484317#comment-14484317
]
Hadoop QA commented on HDFS-7037:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668640/HDFS-7037.001.patch
against trunk revision 5b8a3ae.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HDFS-Build/10197//console
This message is automatically generated.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14376733#comment-14376733
]
Haohui Mai commented on HDFS-7037:
--
[~atm], sorry for the delay as I'm busy with 2.7 blockers.
bq. Note that in the latest patch allowing connections to fall back to an
insecure cluster is configurable, and disabled by default.
Yes you can disable it through configuration but as this is a global
configuration that affects every HFTP connections misconfiguration is still a
concern from a practical point of view (which I raised in HDFS-6776). I think
[~cnauroth] has an excellent articulation on the issue in
https://issues.apache.org/jira/browse/HADOOP-11321?focusedCommentId=14225238page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14225238:
{quote}
...
This is pretty standard Hadoop code review feedback. As a result, Hadoop now
has 762 configuration properties. That's from a grep -c of core-default.xml,
hdfs-default.xml, yarn-default.xml and mapred-default.xml, so the count doesn't
include undocumented properties.
...
{quote}
Also, the fallback behavior is problematic from a security point of view. Chris
has also proposed HADOOP-11701 to limit the impacts of potential configuration.
Indeed it is not an ideal solution but it is a practical one given the
constraints on backward compatibility. Maybe we can do something similar in
this jira.
To summarize:
* -1 on putting fallback logics in FileSystem in general due to potential
security vulnerabilities.
* Given the fact that HFTP is deprecated and it is used in limited use cases,
I'm willing to change it to -0 if there are solutions like HADOOP-11701 to
limit the impact of such a configuration.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException:
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372012#comment-14372012
]
Aaron T. Myers commented on HDFS-7037:
--
Hey [~wheat9], have you had any chance to think about my last comment? Absent
this being a security vulnerability (which I don't think this is, for the
reason stated) I don't see any reason not to fix this in HFTP. We can certainly
work on more general solutions in HADOOP-11726, but I'd still really like to
get this issue fixed in HFTP in the meantime.
Thanks very much.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367627#comment-14367627
]
Haohui Mai commented on HDFS-7037:
--
Thanks for the ping. My position is still the same -- falling back to insecure
mode at the filesystem layer unanimously opens up subtle security
vulnerabilities. Unfortunately I have both hit the issue and misconfiguration
in practice.
I have strong preferences not to do so where my reasonings can be found in
relevant jiras. As I pointed out in HDFS-6776, you'll need to fix this issue
for every single filesystem. I appreciate if you can continue to investigate on
doing it in distcp. I'll comment on HDFS-7036 later today.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367665#comment-14367665
]
Aaron T. Myers commented on HDFS-7037:
--
[~wheat9] thanks for the response.
bq. I have strong preferences not to do so where my reasonings can be found in
relevant jiras. As I pointed out in HDFS-6776, you'll need to fix this issue
for every single filesystem. I appreciate if you can continue to investigate on
doing it in distcp.
Of course, but if we do a fix only in distcp, then other relevant tools that
use the various file systems (e.g. even simple ones like `hadoop fs ...') still
won't work. So the question is: do we fix all the tools that use FileSystem? Or
do we fix all the FileSystem implementations? The right answer seems to me to
quite clearly be that we should fix the FileSystem implementations, as we
should not require this workaround to be implemented by anyone coding against
FileSystem.
To be clear, are you -1 on doing this fix for HFTP? Or just -0?
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367803#comment-14367803
]
Aaron T. Myers commented on HDFS-7037:
--
bq. My question is how to fix all FileSystem implementations, given that there
are multiple HCFS implementations (e.g., MapRFs, Ceph) that inherit the public
FileSystem APIs, all of which sit outside of the repository of hadoop? Should
we ask them to take care of this issue on their own?
That's up to them, but it still seems obvious to me that we should fix the
FileSystem implementations that are in our repository. The alternative you've
proposed, as I mentioned previously, is fixing all _users of FileSystem
implementations_, of which there are obviously many outside of the Hadoop
repository.
bq. -1 given the concern on security vulnerability.
Note that in the latest patch allowing connections to fall back to an insecure
cluster is configurable, and disabled by default. So given that, making this
change in HFTP is no different than how Hadoop RPC currently works, and thus
there is no vulnerability being introduced here. This proposed change really
only amounts to addressing a bug in HFTP that even when client fallback is
enabled, HFTP still can't connect to insecure clusters, since the client can't
handle it when a DT can't be fetched.
If the reasoning behind your -1 is really only predicated on this being a
security vulnerability, then I'd ask you to please consider withdrawing it.
I'd really like to get this fixed in HFTP. It's been burning plenty of users
for a long time.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14367784#comment-14367784
]
Haohui Mai commented on HDFS-7037:
--
bq. do we fix all the tools that use FileSystem? Or do we fix all the
FileSystem implementations? The right answer seems to me to quite clearly be
that we should fix the FileSystem implementations
My question is how to fix all FileSystem implementations, given that there are
multiple HCFS implementations (e.g., MapRFs, Ceph) that inherit the public
FileSystem APIs, all of which sit outside of the repository of hadoop? Should
we ask them to take care of this issue on their own?
bq. To be clear, are you -1 on doing this fix for HFTP? Or just -0?
As I mentioned earlier I was hit by this issue as well thus I would appreciate
if it is fixed, but -1 as given the concern on security vulnerability. We can
discuss potential fixes in HADOOP-11726 and HADOOP-11701.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14366004#comment-14366004
]
Aaron T. Myers commented on HDFS-7037:
--
I agree with Yongjun - this fix is basically equivalent to the fix done in
HDFS-6776, but this time for HFTP instead of WebHDFS. The fix for this issue
should not be implemented in distcp, as this issue affects all users of HFTP,
including just directly using it from the FS shell.
+1, the latest patch looks good to me.
[~wheat9] - haven't heard from you on this JIRA in a while, despite Yongjun's
questions. Are you OK with the patch? If I don't hear back from you in the next
day or so I'm going to go ahead and commit it.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314980#comment-14314980
]
Yongjun Zhang commented on HDFS-7037:
-
HI [~wheat9],
Would you please clarify your earlier comment at
https://issues.apache.org/jira/browse/HDFS-7037?focusedCommentId=14286085page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14286085
Did you mean you disagree with HDFS-6776 solution?
I'm confused because I thought the result of the extensive discussion in
HDFS-6776 is:
* HDFS-6776 solution is agreed
* my patch for HDFS-7036 is not yet agreed
BTW, I have some long standing questions for you in HDFS-7036. I'd appreciate
if you could help answering them.
Thanks.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14310388#comment-14310388
]
Yongjun Zhang commented on HDFS-7037:
-
Hi [~wheat9],
Thanks a lot for your comment, sorry I did not see your comment until today. My
bad!
About HDFS-6776, we did fix webhdfs, to handle null token from insecure
cluster.
About HDFS-7037, I referenced HDFS-3905, and did a similar fix here. In some
sense, HDFS-7037 fix is similar to HDFS-6776 fix.
My understanding is that both HDFS-6776 and HDFS-3905 approaches were agreed
and committed. Particularly the final committed fix of HDFS-6776 is very
similar to one patch version you submitted to HDFS-6776 (where we did the fix
in webhdfs); And my understanding is, the only thing that was not agreed is
HDFS-7036, where I had some open questions for you expecting answer.
Please correct me if I'm wrong.
Thanks.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286085#comment-14286085
]
Haohui Mai commented on HDFS-7037:
--
Note that distcp over webhdfs has the same issue as it has been discussed
extensively in HDFS-6776. This should be fixed in distcp, not in
{{HftpFileSystem}}.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285856#comment-14285856
]
Yongjun Zhang commented on HDFS-7037:
-
Thanks [~qwertymaniac], indeed what you described is a very good use case here.
Hi [~atm] and [~daryn], you guys are not pushed by me because I thought we had
webhdfs as an alternative solution:-) For the scenario Harsh described (distcp
from old pre-security release that doesn't have webhdfs support yet), we don't
have alternative, would you please help reviewing the patch? Thanks much!
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285840#comment-14285840
]
Harsh J commented on HDFS-7037:
---
FWIW, this patch is still required in order to get basically any post-security
releases to copy from pre-security releases running today.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285848#comment-14285848
]
Hadoop QA commented on HDFS-7037:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668640/HDFS-7037.001.patch
against trunk revision 6b17eb9.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9294//console
This message is automatically generated.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145601#comment-14145601
]
Yongjun Zhang commented on HDFS-7037:
-
Hi [~atm] and [~daryn], the patch I posted here was tested against real
cluster, would you please help taking a look? thanks a lot.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133106#comment-14133106
]
Yongjun Zhang commented on HDFS-7037:
-
I referenced HDFS-3905 when working on the patch of this jira. Thanks [~daryn]
for the work in HDFS-3905.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133108#comment-14133108
]
Hadoop QA commented on HDFS-7037:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12668640/HDFS-7037.001.patch
against trunk revision 14e2639.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8018//console
This message is automatically generated.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
[jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only)
[
https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133109#comment-14133109
]
Yongjun Zhang commented on HDFS-7037:
-
Since it's branch-2 only, it's expected that the patch won't apply at trunk.
Using distcp to copy data from insecure to secure cluster via hftp doesn't
work (branch-2 only)
Key: HDFS-7037
URL: https://issues.apache.org/jira/browse/HDFS-7037
Project: Hadoop HDFS
Issue Type: Bug
Components: security, tools
Affects Versions: 2.6.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7037.001.patch
This is a branch-2 only issue since hftp is only supported there.
Issuing distcp hftp://insecureCluster hdfs://secureCluster gave the
following failure exception:
{code}
14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing
remote token:
java.io.IOException: Error when dealing remote token: Internal Server Error
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375)
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
at
org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:121)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:390)
14/09/13 22:07:40 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Unable to obtain remote token
14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered
java.io.IOException: Unable to obtain remote token
at
org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247)
at
org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457)
at
org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472)
at
org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501)
at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
at
