[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126207#comment-16126207 ] Aaron Fabbri commented on HADOOP-14749: --- Awesome, thanks for doing this [~ste...@apache.org] > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: HADOOP-13345 > > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch, > HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch, > HADOOP-14749-HADOOP-13345-006.patch, HADOOP-14749-HADOOP-13345-007.patch, > HADOOP-14749-HADOOP-13345-008.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124654#comment-16124654
]
Hadoop QA commented on HADOOP-14749:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-13345 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
13s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} HADOOP-13345 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 1
new + 43 unchanged - 16 fixed = 44 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14749 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881621/HADOOP-14749-HADOOP-13345-008.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 189b4e1fb9be 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b114f24 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13021/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13021/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13021/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13021/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> review s3guard docs & code prior to merge
>
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124632#comment-16124632
]
Hadoop QA commented on HADOOP-14749:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-13345 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
17s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} HADOOP-13345 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 2
new + 54 unchanged - 5 fixed = 56 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14749 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881616/HADOOP-14749-HADOOP-13345-007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 93db725ee00d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b114f24 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13019/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13019/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13019/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13019/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> review s3guard docs & code prior to merge
>
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124605#comment-16124605 ] Steve Loughran commented on HADOOP-14749: - Note: the throttle events are coming from the failing client. You won't need to bind to a different client for different failure modes, just different options set We could rename it *now* to be the FailingAWSClient and make it a bit more generic, > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch, > HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124573#comment-16124573 ] Steve Loughran commented on HADOOP-14749: - # We haven't any tests of IAM role with r/o bucket and r/w of DDB. I did accidentally try to use s3guard and landsat, and don't remember it being a particularly good experience. That's really something we need to test before making any statement about its use. Hence my caution. I will state "this has been undertested and may not work" # I already have throttling in the fault injection in HADOOP-13786., [here|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/InconsistentAmazonS3Client.java]. Shows how we don't handle it at all well today. It's not an optimistic promise then, more a "forward looking on a future merge of existing code" statement. The hard part isn't this injection, it's reworking every since direct call of the AWS SDK (other than via xfer manager) to handle the error. Hence [S3ALambda|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ALambda.java], which I use for all the committer and upload calls in [WriteOperationHelper|https://github.com/steveloughran/hadoop/blob/s3guard/HADOOP-13786-committer/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/WriteOperationHelper.java]. We will need to go through every single API GET/HEAD call the same way for resilience. > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch, > HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124000#comment-16124000
]
Aaron Fabbri commented on HADOOP-14749:
---
Quick review of the v5 patch:
{noformat}
+### Security
+
+All users of the DynamoDB table must have write access to it. This
+effectively means they must have write access to the entire object store.
+
{noformat}
Couldn't you have an IAM role with write access to DynamoDB but only read
access for the bucket?
{noformat}
**Warning do not enable any type of failure injection in production. The
-following settings are for test development only.**
-
-## Inconsistency Injection
{noformat}
No change needed to your patch here.. but my hope for the sub heading was that
we will introduce other types of failure injection in the future. We can
re-add the main heading when we do.
{noformat}
-// TODO
-// 1. Add properties query to MetadataStore interface
-// supportsAuthoritativeDirectories() or something.
-// 2. Add "isNew" flag to MetadataStore.put(DirListingMetadata)
-// 3. If #1 is true, assert that directory is still fully cached here.
-// assertTrue("Created dir is fully cached", dirMeta.isAuthoritative());
-
+// TODO HADOOP-1475 instrument MetadataStore for asserting & testing
{noformat}
Wrong JIRA # here. Should be HADOOP-14756
I tweaked that JIRA a bit to capture my original intent here (things have
changed a little).
So +1 after you fix the JIRA #, and consider clarifying bit about requiring
write access to buckets.
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch,
> HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123831#comment-16123831
]
Hadoop QA commented on HADOOP-14749:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-13345 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
46s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 25
new + 54 unchanged - 5 fixed = 79 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14749 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881513/HADOOP-14749-HADOOP-13345-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 38f6cb70d3a6 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b114f24 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13009/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13009/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13009/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13009/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> review s3guard docs & code prior to merge
>
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123792#comment-16123792
]
Aaron Fabbri commented on HADOOP-14749:
---
{quote}
If we added a field for each entry as to when the record itself was created,
then we could have AWS TTL do the pruning automatically.
{quote}
I think we will want a "entry last written" mod time field in DDB, but I don't
think we can use S3's TTL feature without breaking the "all ancestors of any
path P in DDB must be present" invariant. I chatted with my friend that works
on the DynamoDB team and he did not believe that their TTL deletion feature was
strongly ordered enough to guarantee it, even if we could ensure we always
wrote ancestors before children. Maybe there is another algorithm I'm not
thinking of though.
I do think we want a v2 prune implementation for dynamo which works better
(i.e. actually expires directories properly). I think that the authoritative
mode support for dynamodb will be a big motivator for this, as if you are
relying on DDB as source of truth for listings, then reliable expiry of stale
data becomes more important. I've also been thinking about the online
algorithm variant of prune (doing it on demand in client, probabilistically /
randomized perhaps, or on access).
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch,
> HADOOP-14749-HADOOP-13345-004.patch, HADOOP-14749-HADOOP-13345-005.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123635#comment-16123635
]
Steve Loughran commented on HADOOP-14749:
-
I know the answer to that now. I also see that you can configure DDB with an
explicit TTL. If we added a field for each entry as to when the record itself
was created, then we could have AWS TTL do the pruning automatically.
Also, on the CLI I'm getting told off for an encryption key I don' think I've
set
{code}
./hadoop s3guard import s3a://hwdev-steve-ireland-new
2017-08-11 17:56:50,139 INFO Configuration.deprecation:
fs.s3a.server-side-encryption-key is deprecated. Instead, use
fs.s3a.server-side-encryption.key
2017-08-11 17:56:50,651 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new} is
initialized.
Inserted 3 items into Metadata Store
{code}
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch,
> HADOOP-14749-HADOOP-13345-004.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123392#comment-16123392
]
Hadoop QA commented on HADOOP-14749:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 16 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-13345 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} HADOOP-13345 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 22
new + 55 unchanged - 4 fixed = 77 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14749 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881459/HADOOP-14749-HADOOP-13345-004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux c900fd7fa4dd 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b114f24 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13007/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13007/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13007/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/13007/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> review s3guard docs & code prior to merge
>
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122901#comment-16122901
]
Mingliang Liu commented on HADOOP-14749:
{quote}
provision burden?
{quote}
I was thinking that, suppose a user has dozens of buckets in a region, if each
bucket has a dedicated DDB table, then the user will have to provision dozens
of tables according to each table's peak/idle load. Instead, if she shares the
metadata in a single DDB table for all the buckets in that region, she will
need to only provision one table capacity according to overall usage. This
amortizes the provision burden.
{quote}
how to handle s3guard as a read-only client.
{quote}
I know IAM role can have fine granularity about READ access to S3 bucket (e.g.
"s3:GetObject") and DDB table (e.g. "dynamodb:Query", "dynamodb:Get" etc). This
might be considered/operated by the user. But in code, we should not populate
the metadata from S3 to DDB in the read-only case.
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122184#comment-16122184 ] Steve Loughran commented on HADOOP-14749: - thx, I'll go through Mingliang's comments and do a final revision. * yes. should use DynamoDB everywhere. * we do generate javadocs, though I'm not sure the publishing is code. So I'm trying to be consistent. * provision burden? You mean allow us to overspecify for peak loads. Makes sense BTW, one situation I'm thinking about now: how to handle s3guard as a read-only client. That is, you can get the auth/hon-auth data from others, but you can't write back yourself (indeed, can't write back to the FS) > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16121069#comment-16121069
]
Mingliang Liu commented on HADOOP-14749:
+1. Nice work. Thanks [~ste...@apache.org].
Nits:
# I saw a few TODOs that do not have associated JIRA numbers. Should we file
and point to them?
# According to my experience at Amazon, {{DynamoDB}} and {{Dynamo}} are two
different systems though they share lots of core principles and design. Should
we replace all {{dynamo}} in doc/comment as {{DynamoDB}}?
# In {{S3GuardTool}} L1130, {{code System.exit() on all exeuction paths.}}
should be {{@code System.exit() on all exeuction paths.}} This has a broader
question: we currently don't use javadoc to generate HTML doc anymore (don't
we?), so perhaps we don't need those HTML tags in javadoc which most serves as
comment. I saw some usage of {{}} for e.g.
# In doc, should we also mention sharing DDB table amortizes the provision
burden besides cost-effective?
# In doc, there is duplicate "uses" in sentence {{+service uses uses the same
authentication mechanisms as S3. S3Guard}}
# {{+### Delete a table: `s3guard destroy`}} has double spaces before destroy
# In the testing doc,
{quote}
... launch the server if it is not yet started; creating the table if it does
not exist.
{quote}
{{DynamoDBLocalClientFactory}} is starting a new in-memory local server whose
instance or data is not shared among tests. So it always starts a new server,
and create new table. Need to confirm.
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120562#comment-16120562 ] Aaron Fabbri commented on HADOOP-14749: --- +1 on v3 patch. > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch, HADOOP-14749-HADOOP-13345-003.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120298#comment-16120298
]
Hadoop QA commented on HADOOP-14749:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
|| || || || {color:brown} HADOOP-13345 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
25s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} HADOOP-13345 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} HADOOP-13345 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 13s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 22
new + 55 unchanged - 4 fixed = 77 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HADOOP-14749 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12881045/HADOOP-14749-HADOOP-13345-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux dd0430918c9b 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HADOOP-13345 / b4c2ab2 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12995/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-aws.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12995/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12995/testReport/ |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12995/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> review s3guard docs & code prior to merge
>
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120177#comment-16120177
]
Steve Loughran commented on HADOOP-14749:
-
+ feedback from [~ehiggs]
{code}
+ // with a metadata store, the object entries need tup be updated,
Grammar/spelling.
+ * This will always be non-null, but may be bound to the
If something will be not null, maybe use @NotNull. I don’t see any uses of it
yet in the Hadoop codebase, so maybe someone decided against using it.
+ if (status == DirectoryStatus.DOES_NOT_EXIST
+ || status == DirectoryStatus.EXISTS_AND_IS_DIRECTORY_ON_S3_ONLY) {
I think this indents the || one too many. checkstyle should pick it up.
+ // TODO s3guard: retry on file not found exception
Other places you are normalizing spelling to use capital S and capital G (even
in comments) and the nature of this patch is nit fixes... :)
+ * Generally, callers should use {@link #initialize(FileSystem)}
+ * with an initialized S3 file system.
A wise man once said “Object Stores are not File Systems”. So do we want “with
an initialized {@link S3AFileSystem} ? or “initialized S3 FileSystem” so it
includes S3 and S3N (which will will be removed soon).
+ * Without a filesystem to act as a reference point, the configuration itself
file system or filesystem. cf previous comment.
+ Errpr `"DynamoDB table TABLE does not exist in region REGION;
auto-creation is turned off"`
Error (spelling).
+
+### Warning About Concurrent Tests
+
+You must not run S3A and S3N tests in parallel on the same bucket. This is
+especially true when S3Guard is enabled. S3Guard requires that all clients
+that are modifying the bucket have S3Guard enabled, so having S3N
+integration tests running in parallel with S3A tests will cause strange
+failures.
So if someone adds to the bucket using s3cmd in production what will happen?
This seems like a severe limitation that can effect of ephemeral mounts for
Provided Storage where a purpose is to async repl between s3 and hdfs.
+The two S3Guard scale testse are `ITestDynamoDBMetadataStoreScale` and
tests (spelling)
{code}
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch,
> HADOOP-14749-HADOOP-13345-002.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120172#comment-16120172 ] Steve Loughran commented on HADOOP-14749: - Aaron: just seen your comments. Yes, the patch was out of date. And I have moved all s3guard testing into the "testing" doc as everyone testing s3a needs to know about it, while general s3guard users don't. I'll do a revised patch > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch, > HADOOP-14749-HADOOP-13345-002.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119955#comment-16119955
]
Aaron Fabbri commented on HADOOP-14749:
---
Thanks for the patch [~ste...@apache.org]. This is good stuff.
{noformat}
/**
- * Should not be called by clients. Only used so {@link org.apache.hadoop
- * .fs.s3a.s3guard.MetadataStore} can maintain this flag when caching
- * FileStatuses on behalf of s3a.
+ * Should not be called by clients. Only used so {@code MetadataStore}
+ * can maintain this flag when caching FileStatuses on behalf of s3a.
* @param value for directories: TRUE / FALSE if known empty/not-empty,
* UNKNOWN otherwise
*/
{noformat}
Actually, can we remove {{setIsEmptyDirectory()}} now? IIRC this is not used
since I reworked the empty directory handling logic.
{noformat}
+ // with a metadata store, the object entries need tup be updated,
+ // including, potentially, the ancestors
{noformat}
/tup/to/
{noformat}
+ /**
+ * Determine the directory status of a path, going via any
+ * MetadataStore before checking S3.
+ * @param path path to check
+ * @return the determined status
+ * @throws IOException IO failure other than FileNotFoundException
+ */
private DirectoryStatus checkPathForDirectory(Path path) throws
IOException {
{noformat}
I thought HADOOP-14505 eliminated checkPathForDirectory()? I had suggested
just using getFileStatus() would be more efficient and less code.
{noformat}
+// metadata listing is authoritative, so return it directory
{noformat}
/directory/directly/ ?
{noformat}
-// If FileStatus' path is missing host, but should have one, add it.
+// If FileStatus's path is missing host, but should have one, add it.
{noformat}
Either is correct, BTW.
{noformat}
-assertQualified(srcRoot);
-assertQualified(srcPath);
-assertQualified(dstPath);
+assertQualified(srcRoot, srcPath, dstPath);
{noformat}
Nice.
{noformat}
+ Errpr `"DynamoDB table TABLE does not exist in region REGION;
auto-creation is turned off"`
{noformat}
/Errpr/Error/
The docs changes look good, but the diff became a bit hard to follow. Looks
like you moved some stuff to testing doc, which is fine.
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119772#comment-16119772
]
Hadoop QA commented on HADOOP-14749:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HADOOP-14749 does not apply to HADOOP-13345. Rebase required?
Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HADOOP-14749 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880921/HADOOP-14749-HADOOP-13345-001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12990/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119127#comment-16119127 ] Steve Loughran commented on HADOOP-14749: - Testing All well apart from existing failures HADOOP-14735 and HADOOP-14733 (patches available); one run failed with HADOOP-14750 stack trace > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14749-HADOOP-13345-001.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[
https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119123#comment-16119123
]
Steve Loughran commented on HADOOP-14749:
-
{{S3Guard.assertQualified}} added a vargs version to make things shorter...not
something I'm too opinionated about
{{DirectoryStatus checkPathForDirectory}} seems to always go to S3 if the path
maps to a file, even if the store has a record in s3guard. Have I misread it?
h3. site docs
* should we use the term {{MetadataStore}} in the docs, or {{Metadata Store}}?
* what architecture doc should go in? There's a lot in the javadocs...we could
just say "look there", but its nice to have a good online doc we can point
people at.
> review s3guard docs & code prior to merge
> -
>
> Key: HADOOP-14749
> URL: https://issues.apache.org/jira/browse/HADOOP-14749
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: documentation, fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14749-HADOOP-13345-001.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Pre-merge cleanup while it's still easy to do
> * Read through all the docs, tune
> * Diff the trunk/branch files to see if we can reduce the delta (and hence
> the changes)
> * Review the new tests
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118432#comment-16118432 ] Steve Loughran commented on HADOOP-14749: - + * review javadocs * arranging imports in roughly the same order as our style requirements. * review tests > review s3guard docs & code prior to merge > - > > Key: HADOOP-14749 > URL: https://issues.apache.org/jira/browse/HADOOP-14749 > Project: Hadoop Common > Issue Type: Sub-task > Components: documentation, fs/s3 >Affects Versions: HADOOP-13345 >Reporter: Steve Loughran >Assignee: Steve Loughran > Original Estimate: 24h > Remaining Estimate: 24h > > Pre-merge cleanup while it's still easy to do > * Read through all the docs, tune > * Diff the trunk/branch files to see if we can reduce the delta (and hence > the changes) > * Review the new tests -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
