[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773509#comment-17773509
]
ASF GitHub Bot commented on HADOOP-15430:
-
whenamanlies commented on PR #1646:
URL: https://github.com/apache/hadoop/pull/1646#issuecomment-1754221384
Is it possible that Path.toUri() is missing .getPath() as it seem to have
broken listing of prexies containing spaces?
Space becomes **%20** and **%20** becomes **%2520**
DEBUG o.apache.hadoop.fs.s3a.S3AFileSystem - Stripping trailing '/' from
s3://my.bucket/my folder/
DEBUG o.apache.hadoop.fs.s3a.S3AFileSystem -
listFiles(s3://my.bucket/my%20folder, true)
DEBUG o.apache.hadoop.fs.s3a.S3AFileSystem - Recursive list of all entries
under my%20folder/
DEBUG o.apache.hadoop.fs.s3a.S3AFileSystem - LIST List
my.bucket:/my%20folder/ delimiter=null keys=5000 requester pays=false
DEBUG com.amazonaws.auth.AWS4Signer - AWS4 Canonical Request: '"GET
/
fetch-owner=false=2=5000=my%2520folder%2F
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057991#comment-17057991
]
Hudson commented on HADOOP-15430:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18046 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18046/])
HADOOP-15430. hadoop fs -mkdir -p path-ending-with-slash/ fails with (github:
rev 0a9b3c98b160f2cf825ea6e8422ce093a8ae7cb3)
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
* (edit)
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/PathMetadataDynamoDBTranslation.java
* (add)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardFsShell.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AMiscOperations.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/AbstractS3GuardToolTestBase.java
* (edit)
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/TestPathMetadataDynamoDBTranslation.java
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596532#comment-16596532
]
Sunil Govindan commented on HADOOP-15430:
-
Thanks [~ste...@apache.org].
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596520#comment-16596520
]
Steve Loughran commented on HADOOP-15430:
-
Lets downgrade this one; I can live without it. I do have some other blockers
with patches which I do need in
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595902#comment-16595902
]
Sunil Govindan commented on HADOOP-15430:
-
Hi [~ste...@apache.org]
As this jira is marked for 3.2 as a blocker, cud u pls help to take this
forward. 3.2 code freeze date is nearby in a weeks. Kindly help to check the
same.
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478998#comment-16478998
]
Steve Loughran commented on HADOOP-15430:
-
* I'll look at the comment and cleanup
bq. This part, I'm assuming is for the trailing double slash case, right?
I worry now that this is superfluous with the previous code. And you need to
handle "", don't you? Stripping all is better.
I think the part you are asking about is from my first fix attempt; then I just
copied in the stuff from the Path code.
Let me review this and see if those tests failures are actually independent of
this code by switching branches & seeing how isolated they are.
I'll consider your +1 on hold...
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478404#comment-16478404
]
Aaron Fabbri commented on HADOOP-15430:
---
Great work here. Test code is very nice.
{quote}+ // here the problem: the URI constructor doesn't strip trailing "/"
chars
{quote}
Sigh. Unfortunate.
{noformat}
+ @Override
+ public Path makeQualified(final Path path) {
+Path q = super.makeQualified(path);
+if (!q.isRoot()) {
+ String urlString = q.toUri().toString();
+ if (urlString.endsWith(Path.SEPARATOR)) {
+// this is a path which needs root stripping off to avoid
+// confusion, See HADOOP-15430
+LOG.debug("Stripping trailing '/' from {}", q);
+// deal with an empty "/" at the end by mapping to the parent and
+// creating a new path from it
{noformat}
super minor nit.. you could omit that last comment i think, seems like you
changed your approach?
{noformat}
+q = new Path(urlString.substring(0, urlString.length() - 1));
+ }
+}
+if (!q.isRoot() && q.getName().isEmpty()) {
+ q = q.getParent();
+}
{noformat}
This part, I'm assuming is for the trailing double slash case, right?
Anyways +1, LGTM pending paying the checkstyle tax.
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478210#comment-16478210
]
genericqa commented on HADOOP-15430:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 11s{color} | {color:orange} root: The patch generated 7 new + 47 unchanged -
0 fixed = 54 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
59s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
45s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15430 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12923764/HADOOP-15430-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 323851ae9a45 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e3b7d7a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16478007#comment-16478007
]
Steve Loughran commented on HADOOP-15430:
-
Patch 003:
* move qualify logic to S3AFileSystem.makeQualified()
* make S3AFileSystem.qualify() a private redirect to that, moving other uses
where needed. We could cut it altogether.
* add direct tests of path qualification
* uncomment bits of (new) ITestS3GuardFsShell turned off to debug things.
Tested: s3a ireland w/ DDB. Some of the ITestS3GuardTool testPrune tests
failing (HADOOP-14946)...running the same tests on trunk to see if its a
regression or the test is just playing up...it's been very unreliable for me
recently
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch,
> HADOOP-15430-003.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462814#comment-16462814
]
Steve Loughran commented on HADOOP-15430:
-
Got this
# If you create a path with {{new Path(String)}}, then {{Path.normalizePath()}}
with strip trailing slashes off the path.
# If you create a path with {{new Path(URI)}} then the URI goes in as is. So if
it is not normalized, it stays un-normalized. And as downstream things creating
new paths always use that {{new Path(otherpath.getURI())}}, that propagates
through things like qualify().
This is a fairly fundamental issue, and there's no way I'd want to go near path
or FsShell. Also explains why we didn't pick this up in the contract tests,
we've been constructing paths with the string ctor, not via a URI. Who knew?
Proposed, we do that stripping, and fix the tests where they fail.
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462657#comment-16462657
]
genericqa commented on HADOOP-15430:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 15s{color} | {color:orange} root: The patch generated 2 new + 39 unchanged -
0 fixed = 41 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
55s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
49s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15430 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921765/HADOOP-15430-002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 79c606c05cf3 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / ee2ce92 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462640#comment-16462640
]
Steve Loughran commented on HADOOP-15430:
-
removing trailing / would mean that "touch a/" would match "touch a", which
certainly isn't true on the linux CLI. But at the same time, if a dir a/
exists, then "touch a" is required to fail.
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch, HADOOP-15430-002.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459354#comment-16459354
]
genericqa commented on HADOOP-15430:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
4m 21s{color} | {color:orange} root: The patch generated 1 new + 39 unchanged -
0 fixed = 40 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
45s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15430 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12921309/HADOOP-15430-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e895a4959d83 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fc074a3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459177#comment-16459177
]
Steve Loughran commented on HADOOP-15430:
-
Patch submitted
test run: nowhere
this patch doesn't try to fix things, it adds validation with the built up
query item that all keys != "". I can't trigger that in an IT test, but on the
CLI it produces the stack above.
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15430-001.patch
>
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459095#comment-16459095
]
Steve Loughran commented on HADOOP-15430:
-
yeah, that test shows we've been close to this. I'm about to submit a patch,
but I can't replicate it with a unit tests which doesn't go near fsshell, and I
haven't put one together yet
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458970#comment-16458970
]
Aaron Fabbri commented on HADOOP-15430:
---
I guess we need a test case for the CLI mkdir, as a similar case had a
regression before (HADOOP-14428).
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458951#comment-16458951
]
Steve Loughran commented on HADOOP-15430:
-
And with some checking of the query before it is launched
{code}
java.lang.IllegalStateException: Empty string value for attribute child
at
com.google.common.base.Preconditions.checkState(Preconditions.java:172)
at
org.apache.hadoop.fs.s3a.s3guard.PathMetadataDynamoDBTranslation.pathToKey(PathMetadataDynamoDBTranslation.java:284)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerGet(DynamoDBMetadataStore.java:459)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$get$2(DynamoDBMetadataStore.java:439)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:438)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2110)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2037)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2007)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326)
at
org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:77)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:288)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
{code}
I can't seem to recreate it in a test case though, only on the CLI
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458861#comment-16458861
]
Steve Loughran commented on HADOOP-15430:
-
And with a patch to fs.shell to log stack traces, you get the stack
{code}
AttributeValue may not contain an empty string (Service: AmazonDynamoDBv2;
Status Code: 400; Error Code: ValidationException; Request ID:
0NGJDAHOTK2A51KRR5MVVVTHPFVV4KQNSO5AEMVJF66Q9ASUAAJG): One or more parameter
values were invalid: An AttributeValue may not contain an empty string
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;
Request ID: 0NGJDAHOTK2A51KRR5MVVVTHPFVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:389)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:181)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.get(DynamoDBMetadataStore.java:438)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2110)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2088)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2037)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2007)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2326)
at
org.apache.hadoop.fs.shell.Mkdir.processNonexistentPath(Mkdir.java:77)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:288)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One
or more parameter values were invalid: An AttributeValue may not contain an
empty string (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ValidationException; Request ID:
0NGJDAHOTK2A51KRR5MVVVTHPFVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2925)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2901)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeGetItem(AmazonDynamoDBClient.java:1640)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.getItem(AmazonDynamoDBClient.java:1616)
at
com.amazonaws.services.dynamodbv2.document.internal.GetItemImpl.doLoadItem(GetItemImpl.java:77)
at
com.amazonaws.services.dynamodbv2.document.internal.GetItemImpl.getItem(GetItemImpl.java:66)
at
com.amazonaws.services.dynamodbv2.document.Table.getItem(Table.java:608)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.getConsistentItem(DynamoDBMetadataStore.java:423)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerGet(DynamoDBMetadataStore.java:459)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.lambda$get$2(DynamoDBMetadataStore.java:439)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
... 15 more
mkdir: get on s3a://bucket/output/5/:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One or more
parameter values were invalid: An AttributeValue may not contain an empty
string (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ValidationException; Request ID:
0NGJDAHOTK2A51KRR5MVVVTHPFVV4KQNSO5AEMVJF66Q9ASUAAJG): One or more parameter
values were invalid: An
[jira] [Commented] (HADOOP-15430) hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
[
https://issues.apache.org/jira/browse/HADOOP-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458834#comment-16458834
]
Steve Loughran commented on HADOOP-15430:
-
log (but not full stack). We have some tests for this at the FS level; maybe
the shell is doing something
{code}
18/04/30 18:16:56 DEBUG s3a.S3AFileSystem: Getting path status for
s3a://bucket/terasort/output/3 (qe/terasort/output/3)
18/04/30 18:16:56 DEBUG s3guard.DynamoDBMetadataStore: Get from table
hwdev-steve-new in region us-west-1: s3a://bucket/terasort/output/3
18/04/30 18:16:56 DEBUG s3guard.DynamoDBMetadataStore: Get from table
hwdev-steve-new in region us-west-1 returning for
s3a://bucket/terasort/output/3: null
18/04/30 18:16:56 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1
-> 1
18/04/30 18:16:56 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1
-> 2
18/04/30 18:16:56 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 ->
1
18/04/30 18:16:56 DEBUG s3a.S3AFileSystem: Not Found:
s3a://bucket/terasort/output/3
18/04/30 18:16:56 DEBUG s3a.S3AStorageStatistics: op_exists += 1 -> 1
18/04/30 18:16:56 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2
18/04/30 18:16:56 DEBUG s3a.S3AFileSystem: Getting path status for
s3a://bucket/terasort/output (qe/terasort/output)
18/04/30 18:16:56 DEBUG s3guard.DynamoDBMetadataStore: Get from table
hwdev-steve-new in region us-west-1: s3a://bucket/terasort/output
18/04/30 18:16:57 DEBUG s3guard.DynamoDBMetadataStore: Get from table
hwdev-steve-new in region us-west-1 returning for s3a://bucket/terasort/output:
PathMetadata{fileStatus=FileStatus{path=s3a://bucket/terasort/output;
isDirectory=true; modification_time=1525112217001; access_time=0; owner=hrt_qa;
group=hrt_qa; permission=rwxrwxrwx; isSymlink=false; hasAcl=false;
isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN;
isDeleted=false}
18/04/30 18:16:57 DEBUG s3a.S3AFileSystem: Making directory:
s3a://bucket/terasort/output/3/
18/04/30 18:16:57 DEBUG s3a.S3AStorageStatistics: op_mkdirs += 1 -> 1
18/04/30 18:16:57 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 3
18/04/30 18:16:57 DEBUG s3a.S3AFileSystem: Getting path status for
s3a://bucket/terasort/output/3/ (qe/terasort/output/3/)
18/04/30 18:16:57 DEBUG s3guard.DynamoDBMetadataStore: Get from table
hwdev-steve-new in region us-west-1: s3a://bucket/terasort/output/3/
mkdir: get on s3a://bucket/terasort/output/3/:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: One or more
parameter values were invalid: An AttributeValue may not contain an empty
string (Service: AmazonDynamoDBv2; Status Code: 400; Error Code:
ValidationException; Request ID:
6M9S1ALKKSJ5AJV5A9EKDPROFBVV4KQNSO5AEMVJF66Q9ASUAAJG): One or more parameter
values were invalid: An AttributeValue may not contain an empty string
(Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;
Request ID: 6M9S1ALKKSJ5AJV5A9EKDPROFBVV4KQNSO5AEMVJF66Q9ASUAAJG)
18/04/30 18:16:57 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-new is
closed
{code}
> hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard
> --
>
> Key: HADOOP-15430
> URL: https://issues.apache.org/jira/browse/HADOOP-15430
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.1.0
>Reporter: Steve Loughran
>Priority: Blocker
>
> if you call {{hadoop fs -mkdir -p path/}} on the command line with a path
> ending in "/:. you get a DDB error "An AttributeValue may not contain an
> empty string"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
