[jira] [Commented] (HADOOP-15176) Enhance IAM Assumed Role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365843#comment-16365843 ] Hudson commented on HADOOP-15176: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13663 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13663/]) HADOOP-15176. Enhance IAM Assumed Role support in S3A client. (stevel: rev 9a013b255f301c557c3868dc1ad657202e9e7a67) * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/RoleModel.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java * (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumedRoleCommitOperations.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/TestLambdaTestUtils.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JsonSerialization.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/RoleTestUtils.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/RolePolicies.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractCommitITest.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/AssumedRoleCredentialProvider.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/package-info.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/CommitOperations.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestAssumeRole.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/contract/s3a/ITestS3AContractDistCpAssumedRole.java > Enhance IAM Assumed Role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365642#comment-16365642
]
Steve Loughran commented on HADOOP-15176:
-
changing to
{code}
LOG.warn("Cannot create directory marker at {}: {}",
f.getParent(), e.toString());
{code}
> Enhance IAM assumed role support in S3A client
> --
>
> Key: HADOOP-15176
> URL: https://issues.apache.org/jira/browse/HADOOP-15176
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.1.0
> Environment:
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch,
> HADOOP-15176-003.patch, HADOOP-15176-004.patch
>
>
> Followup HADOOP-15141 with
> * Code to generate basic AWS json policies somewhat declaratively (no hand
> coded strings)
> * Tests to simulate users with different permissions down the path of a
> single bucket
> * test-driven changes to S3A client to handle user without full write up the
> FS tree
> * move the new authenticator into the s3a sub-package "auth", where we can
> put more auth stuff (that base s3a package is getting way too big)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365630#comment-16365630 ] Steve Loughran commented on HADOOP-15176: - bq. For me? Michael Jackson, when I was really young and Thriller came out. I digress. you know, I didn't even see that when I wrote it. I'll leave it in for your amusement: you can practise your moves during test runs bq, Minor compatibility issue here to call out for existing configs Never shipped. I just think that the root package is getting too big, so I want to put all new auth stuff into s3.auth, and misc extensions under s3a.ext. And if you haven't noticed, I've claimed org.apache.fs.store in hadoop-common for object store stuff to be shared cross connectors, which is a goal I have. There's been too much historical copy and paste, and we are suffering now that there are 6 connectors in our own codebase. bq. This is where we catch AccessDeniedException in delete(). Shouldn't the error message say "access denied" so folks know why, without having to use debug level logging? will do. I'll apply the change, do a full retest, then commit. thanks for the revie. > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365136#comment-16365136
]
Aaron Fabbri commented on HADOOP-15176:
---
{noformat}
public static final String NAME
- = "org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider";
+ = "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider";
{noformat}
Minor compatibility issue here to call out for existing configs. +1 from me
though, this was still stabilizing before now.
{noformat}
+ * Jackson Role Model
{noformat}
For me? Michael Jackson, when I was really young and Thriller came out. I
digress.
Documentation looks great. I really appreciate the work here, especially the
particulars around directory markers. Formatting looks good in my IDE's
markdown rendering.
{noformat}
+ LOG.warn("Cannot create directory marker at {}}",
+ f.getParent());
{noformat}
This is where we catch {{AccessDeniedException}} in delete(). Shouldn't the
error message say "access denied" so folks know why, without having to use
debug level logging?
I would just change it to {{LOG.warn("Cannot create directory marker (access
denied) at {}}"}}
Other than that, +1 LGTM.
Tested in us-west-2 w/ and w/o S3Guard.
> Enhance IAM assumed role support in S3A client
> --
>
> Key: HADOOP-15176
> URL: https://issues.apache.org/jira/browse/HADOOP-15176
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.1.0
> Environment:
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Blocker
> Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch,
> HADOOP-15176-003.patch, HADOOP-15176-004.patch
>
>
> Followup HADOOP-15141 with
> * Code to generate basic AWS json policies somewhat declaratively (no hand
> coded strings)
> * Tests to simulate users with different permissions down the path of a
> single bucket
> * test-driven changes to S3A client to handle user without full write up the
> FS tree
> * move the new authenticator into the s3a sub-package "auth", where we can
> put more auth stuff (that base s3a package is getting way too big)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365111#comment-16365111 ] Wangda Tan commented on HADOOP-15176: - Thanks [~ste...@apache.org], [~liuml07], could you help with patch review? > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364188#comment-16364188 ] Steve Loughran commented on HADOOP-15176: - javac are about use of deprecated AWS SDK methods; same issue as before: SDK moving too fast. checkstyle are about public fields in a structure which is marshalled to/from JSON I don't have any more to add here > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch, HADOOP-15176-004.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363904#comment-16363904
]
genericqa commented on HADOOP-15176:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 14s{color}
| {color:red} root generated 2 new + 1232 unchanged - 2 fixed = 1234 total (was
1234) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 9s{color} | {color:orange} root: The patch generated 8 new + 16 unchanged -
0 fixed = 24 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 22s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 26s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
47s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.http.TestHttpServerWithSpengo |
| | hadoop.security.token.delegation.web.TestWebDelegationToken |
| | hadoop.log.TestLogLevel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15176 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910566/HADOOP-15176-004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363149#comment-16363149
]
Steve Loughran commented on HADOOP-15176:
-
Test failure: looks like the specific test needs credentials, and so is an IT
Test. Will move
{code}
com.amazonaws.AmazonClientException: No AWS Credentials provided by
SimpleAWSCredentialsProvider :
org.apache.hadoop.fs.s3a.CredentialInitializationException: Access key, secret
key or session token is unset
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:139)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1163)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:762)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:724)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)
at
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)
at
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)
at
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)
at
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)
at
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)
at
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)
at
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)
at
com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)
at
com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)
at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)
at
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)
at
org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:143)
at
org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:132)
at
org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.lambda$testAssumedInvalidRole$0(TestS3AAWSCredentialsProvider.java:286)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.lambda$interceptClosing$0(S3ATestUtils.java:486)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:491)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:377)
at
org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:446)
at
org.apache.hadoop.fs.s3a.S3ATestUtils.interceptClosing(S3ATestUtils.java:484)
at
org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider.testAssumedInvalidRole(TestS3AAWSCredentialsProvider.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362855#comment-16362855
]
genericqa commented on HADOOP-15176:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 32s{color}
| {color:red} root generated 2 new + 1230 unchanged - 2 fixed = 1232 total (was
1232) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 6s{color} | {color:orange} root: The patch generated 28 new + 16 unchanged
- 0 fixed = 44 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged
- 0 fixed = 2 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
38s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 45s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.s3a.TestS3AAWSCredentialsProvider |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15176 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910418/HADOOP-15176-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux c4c9adff97ac
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362664#comment-16362664 ] Steve Loughran commented on HADOOP-15176: - Testing: S3 ireland. This is ready for review now > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16362660#comment-16362660 ] Steve Loughran commented on HADOOP-15176: - * skip tests of rename in assumed role which fail with s3guard enabled * some javadoc and line length fixup in the test suites. This patch sets things up for HADOOP-15183. whose success will be measurable by "can the assume() statement on the rename test be removed?" > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch, > HADOOP-15176-003.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332797#comment-16332797
]
genericqa commented on HADOOP-15176:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 5s{color}
| {color:red} root generated 2 new + 1239 unchanged - 2 fixed = 1241 total (was
1241) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 21s{color} | {color:orange} root: The patch generated 33 new + 16 unchanged
- 0 fixed = 49 total (was 16) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
27s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged
- 0 fixed = 2 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
6s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 36s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-tools/hadoop-aws |
| | Format string should use %n rather than n in
org.apache.hadoop.fs.s3a.S3AUtils.translateMultiObjectDeleteException(String,
MultiObjectDeleteException) At S3AUtils.java:rather than n in
org.apache.hadoop.fs.s3a.S3AUtils.translateMultiObjectDeleteException(String,
MultiObjectDeleteException) At S3AUtils.java:[line 409] |
| Failed junit tests |
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332577#comment-16332577 ] Steve Loughran commented on HADOOP-15176: - Regarding the current patch, after yetus has bashed through its style issues, it's ready for review, even though S3Guard is failing. I'd propose committing even though the renames fail so that we know s3guard is broken there > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15176-001.patch, HADOOP-15176-002.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332572#comment-16332572
]
Steve Loughran commented on HADOOP-15176:
-
Patch 002
Fails with S3guard + DDB enabled, because of HADOOP-15183: when the delete
operation after/during a rename() raises an exception, DDB isn't updated with
the current state of the store, and, if there were tombstone markers in the
dest directory whose filenames match the newly created ones, well, you don't
get the new files in the listing.
* {{S3AUtils.translateMultiObjectDeleteException()}} can look inside a multi
object delete response (200 + list of failed deletes) and
extract details. If the any of the failures was AccessDenied, the ex becomes an
AccessDeniedException. Otherwise its an AWSS3IOException with
a full list of the failed paths and error codes
* With {{translateMultiObjectDeleteException}} working, permission failures in
delete calls in delete() and rename() correctly raise an
{{AccessDeniedException}}
* {{S3AFileSystem.delete()}} downgrades failure to mkdir the parent marker to a
warn;
* {{S3AFileSystem.deleteObjects}} to log the details of a multi object delete
at debug only.
* test for various operations being correctly denied with both single and multi
deletes enabled: renames, deletes, commit calls
* Found, Fixed bug with error reporting in
{{CommitOperations.abortAllSinglePendingCommits}} (i.e. it wasn't).
* LambdaTestUtils has a new method, {{eval(Callable)}} which wraps any
raised checked exceptions with an AssertionError. This makes it straighforward
to use FS API calls in Java 8 streams, especially the parallel streams, which
significantly speedup things like
creation of 10 test file. + tests, obviously.
* ITestAssumedRoleCommitOperations subclasses ITestaCommitOperations and runs
under an assumed role with a policy of RW only permitted under
the test directory. Ensures that we are choosing the right permissions and
nothing is being written to other paths.
* remove duplicate properties in core-default.xml, review text.
* assumed_role.md has a section on policies: what's required for read and write
* Special section there on "why mixing permissions on different paths will
complicate your life"
Testing, S3 Ireland. Without S3Guard, All good.
With S3Guard, renames of read only file tests fail, for single delete and
multiple delete HTTP calls
{code}
java.lang.AssertionError:
files copied to the destination: expected 11 files in
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest
but got 10
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-1
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-10
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-2
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-3
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-4
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-5
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-6
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-7
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-8
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlySingleDelete/renameDest/file-9
at
org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.assertFileCount(ITestAssumeRole.java:766)
at
org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.executeRenameReadOnlyData(ITestAssumeRole.java:559)
at
org.apache.hadoop.fs.s3a.auth.ITestAssumeRole.testRestrictedRenameReadOnlySingleDelete(ITestAssumeRole.java:484)
[ERROR]
testRestrictedRenameReadOnlyData(org.apache.hadoop.fs.s3a.auth.ITestAssumeRole)
Time elapsed: 5.036 s <<< FAILURE!
java.lang.AssertionError:
files copied to the destination: expected 11 files in
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest
but got 10
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-1
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-10
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-2
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-3
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-4
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-5
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-6
s3a://hwdev-steve-ireland-new/test/testRestrictedRenameReadOnlyData/renameDest/file-7
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329495#comment-16329495
]
genericqa commented on HADOOP-15176:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 17s{color}
| {color:red} root generated 2 new + 1239 unchanged - 2 fixed = 1241 total (was
1241) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 58s{color} | {color:orange} root: The patch generated 16 new + 15 unchanged
- 0 fixed = 31 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 46s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged
- 0 fixed = 2 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
51s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 37s{color}
| {color:red} hadoop-aws in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.s3a.TestS3AAWSCredentialsProvider |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-15176 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906454/HADOOP-15176-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 6cfa560cdd23 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[
https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329310#comment-16329310
]
Aaron Fabbri commented on HADOOP-15176:
---
{quote}
Patch 001, still WiP
{quote}
Ok, I won't commit it yet. ;) Shout when you feel like it is ready and I'll
give it a good review.
> Enhance IAM assumed role support in S3A client
> --
>
> Key: HADOOP-15176
> URL: https://issues.apache.org/jira/browse/HADOOP-15176
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.1.0
> Environment:
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-15176-001.patch
>
>
> Followup HADOOP-15141 with
> * Code to generate basic AWS json policies somewhat declaratively (no hand
> coded strings)
> * Tests to simulate users with different permissions down the path of a
> single bucket
> * test-driven changes to S3A client to handle user without full write up the
> FS tree
> * move the new authenticator into the s3a sub-package "auth", where we can
> put more auth stuff (that base s3a package is getting way too big)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329275#comment-16329275 ] Steve Loughran commented on HADOOP-15176: - Patch 001, still WiP Changes as noted in the description, with tests which give the assumed role restricted access the bucket (just some subdirs) and then attempt mkdir/delete/touch/rename operations underneath to see what happens. Patches S3A FS to not overreact if you can't create an empty directory marker because you don't have the permission to, even though the outcome isn't quite what a normal FS would deliver. The "role model" class is in the hadoop-aws JAR tagged as unstable and for testing only...useful to be able to dynamically create a policy without making a mess of the JSON. I want it in the production JAR so that I can use it in some downstream testing too, seeing what other apps fail when you restrict access. Not because I expect people to use assumed roles in production much, but because the fact that you can do it in tests makes it straightforward to simulate restricted permissions. Fun little thing there I'm still to try: what if you don't have s3guard write access & try using it in auth mode? I expect it gets inconsistent fast > Enhance IAM assumed role support in S3A client > -- > > Key: HADOOP-15176 > URL: https://issues.apache.org/jira/browse/HADOOP-15176 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 3.1.0 > Environment: >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-15176-001.patch > > > Followup HADOOP-15141 with > * Code to generate basic AWS json policies somewhat declaratively (no hand > coded strings) > * Tests to simulate users with different permissions down the path of a > single bucket > * test-driven changes to S3A client to handle user without full write up the > FS tree > * move the new authenticator into the s3a sub-package "auth", where we can > put more auth stuff (that base s3a package is getting way too big) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
