[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391525#comment-16391525 ] Steve Loughran commented on HADOOP-13622: - having been playing with distcp, I consider HADOOP-15281 a priority item as it means that every upload forces a rename of data, even without the -atomic operator > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu >Priority: Minor > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524783#comment-15524783 ] Yuanbo Liu commented on HADOOP-13622: - [~ste...@apache.org] Sure, thanks for your effort. > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu >Priority: Minor > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522968#comment-15522968 ] Steve Loughran commented on HADOOP-13622: - Actually, I did the writeup this weekend anyway; I was testing some of the latest S3a changes and distcp/FS shell, so went through and updated the docs. Have a look at the pull request and review the text: your comments there will help get the docs into a shape for everyone to find useful, after which we can commit it to Hadoop 2.7.4+ > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu >Priority: Minor > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15521843#comment-15521843 ] Yuanbo Liu commented on HADOOP-13622: - [~ste...@apache.org] Sorry for the late response, I'm quite busy last week. Yes, I'm glad to do the job. I will focus on the items you mentioned in comment 3 and update HADOOP-13655. Thanks for your comment. > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu >Priority: Minor > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15521009#comment-15521009 ] Steve Loughran commented on HADOOP-13622: - HADOOP-13655 covers the JIRA for documenting the issue; this one looks at the software side of things > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[
https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512833#comment-15512833
]
Steve Loughran commented on HADOOP-13622:
-
no, no timetable.
What would be good would be some special section in
{{hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md}} on
"fs shell commands and object stores"
* why you can get not found exceptions on an {{-fs ls s3a://bucket }} URL if
you leave off the trailing / (it's because there's no homedir, and the shell
assumes its a relative path)
* distcp atomic not working
* why you should use {{fs -put -d}} to upload files without the rename
* why you should use {{-skipTrash}} when deleting things
* why you shouldn't put secrets in the URLs, e.g. s3n://awsId:secret@mybucket/
Looks also like the document doesn't even mention put's -d and -f options, so
it could do with a quick comparison of what usage info gets printed in a
{{hadoop fs}} message and what the docs say, updating the docs.
Would you be willing to do this?
> `-atomic` should not be supported while using `distcp` command in object file
> system
>
>
> Key: HADOOP-13622
> URL: https://issues.apache.org/jira/browse/HADOOP-13622
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yuanbo Liu
>Assignee: Yuanbo Liu
>
> After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point
> that none of the object stores support atomic renames. So I file a new jira
> and ready to provide a patch to disable `distcp -atomic` in object file
> system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511815#comment-15511815 ] Yuanbo Liu commented on HADOOP-13622: - [~ste...@apache.org] Thanks for your comment. Yes you're right, it would be difficult to distinguish object store unless HADOOP-9565 is committed. Since the jira is halted for a long time, do you have any idea when the jira can move on so that I can fix this issue? > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13622) `-atomic` should not be supported while using `distcp` command in object file system
[ https://issues.apache.org/jira/browse/HADOOP-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503029#comment-15503029 ] Steve Loughran commented on HADOOP-13622: - One challenge here is that we aren't going to easily tell if an FS is an object store, not unless HADOOP-9565 or equivalent gets in > `-atomic` should not be supported while using `distcp` command in object file > system > > > Key: HADOOP-13622 > URL: https://issues.apache.org/jira/browse/HADOOP-13622 > Project: Hadoop Common > Issue Type: Bug >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > > After discussing with [~ste...@apache.org] in HADOOP-13593, I get the point > that none of the object stores support atomic renames. So I file a new jira > and ready to provide a patch to disable `distcp -atomic` in object file > system. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
