[
https://issues.apache.org/jira/browse/HADOOP-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483594#comment-15483594
]
Steve Loughran commented on HADOOP-13593:
-----------------------------------------
# please can you stick the full stack trace of the exception in as a comment;
good for future searches of the same lines, & IDEs that navigate up the stack.
Leaving out the bug description —as you did— is great, because it keeps out of
all the emails, but having that stack in the comments is always useful
# anything checking hostnames is going to be there for a reason. Presumably so
that people can use URLs like hdfs://server:4040/ and
hdfs://server.example.org:4040/ and have atomic operations. Changing this
fairly fundamental behaviour is not going to happen, because it affects so much
more than swift For that reason, it's going to have to be a -1 there, sorry.
# none of the object stores support atomic renames, so an atomic distcp isn't
going to work. in fact, maybe they should all reject the option outright to
stop people thinking of it.
If there were to be a patch on this, it'd need tests. Here I'd recommend that
(a) swift adds an implementation of {{AbstractContractDistCpTest}}, and that
(b), that base test added a check for the atomic flag —one that every fs
contract XML would have to declare whether or not it supported. The option.
> `hadoop distcp -atomic` invokes improper host check while copying data from
> HDFS to Swift
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-13593
> URL: https://issues.apache.org/jira/browse/HADOOP-13593
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Yuanbo Liu
> Attachments: HADOOP-13593.001.patch, HADOOP-13593.002.patch
>
>
> While copying data from HDFS to Swift by using `hadoop distcp -atomic`, for
> example:
> {code}
> hadoop distcp -atomic /tmp/100M swift://testhadoop.softlayer//tmp
> {code}
> it throws
> {code}
> java.lang.IllegalArgumentException: Work path
> swift://testhadoop.softlayer/._WIP_tmp546958075 and target path
> swift://testhadoop.softlayer/tmp are in different file system
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:351)
> .....
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
