[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748223#comment-16748223 ] Steve Loughran commented on HADOOP-14556: - For people watching this, I've stuck up a video showing distcp collecting DTs and using it to do a cross-bucket copy in a test cluster which doesn't have any credentials: https://www.youtube.com/watch?v=rpyLkDEzIxI > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556-029.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742384#comment-16742384 ] Hudson commented on HADOOP-14556: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15770 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15770/]) HADOOP-14556. S3A to support Delegation Tokens. (stevel: rev 6d0bffe17eadedd60d4599427248b0db4a7c5502) * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/package-info.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/IAMInstanceCredentialsProvider.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationIT.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenIdentifier.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/NanoTimerStats.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitMRJob.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageStatistics.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AWSPolicyProvider.java * (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_tokens.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/RoleTestUtils.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentials.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DirListingMetadata.java * (add) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_token_architecture.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/TestStagingPartitionedFileListing.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/fileContext/ITestS3AFileContextStatistics.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestDelegatedMRJob.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AbstractAWSCredentialProvider.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/NoAwsCredentialsException.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/TestS3ADelegationTokenSupport.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/RoleModel.java * (edit) hadoop-project/pom.xml * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AbstractSessionCredentialsProvider.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/RolePolicies.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestUtils.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenBinding.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3native/S3xLoginHelper.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/TestStagingCommitter.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/DurationInfo.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADelegationTokens.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDTService.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestRoleDelegationInFileystem.java * (add)
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742356#comment-16742356 ] Steve Loughran commented on HADOOP-14556: - OK, taking larry's previous +1 & akira's happiness as binding, Gabor not seeing any failures. recommitting. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556-029.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739913#comment-16739913 ] Akira Ajisaka commented on HADOOP-14556: 'mvn javadoc:javadoc' passed on my local. Thanks. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556-029.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739488#comment-16739488 ] Gabor Bota commented on HADOOP-14556: - All tests are passing now for me, and the docs is looking better with the {{fs.s3a.assumed.role.arn}} described. +1 (non-binding) > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556-029.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738645#comment-16738645 ] Steve Loughran commented on HADOOP-14556: - javadoc happy, junit happy other than the (independent) SSL failure, and the checkstyles are either existing issues flagged as new, some line length == 81 and some URLs-in-javadocs are too long errors. Is everyone happy with this updated patch? > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556-029.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737486#comment-16737486 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 40 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 37s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 17s{color} | {color:green} root generated 0 new + 1489 unchanged - 1 fixed = 1489 total (was 1490) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 31s{color} | {color:orange} root: The patch generated 14 new + 171 unchanged - 13 fixed = 185 total (was 184) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 2s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 156 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 22s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 53s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 20s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 35s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737348#comment-16737348 ] Steve Loughran commented on HADOOP-14556: - next patch will move from fails to skips {code} org.junit.AssumptionViolatedException: No ARN for role tests at org.junit.Assume.assumeTrue(Assume.java:59) at org.apache.hadoop.fs.s3a.S3ATestUtils.assume(S3ATestUtils.java:1030) at org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationIT.assumeRoleTests(AbstractDelegationIT.java:211) at org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens.setup(ITestRoleDelegationTokens.java:61) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:745) {code} > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556-029.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737288#comment-16737288 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m 41s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 40 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 57s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 40s{color} | {color:green} root generated 0 new + 1489 unchanged - 1 fixed = 1489 total (was 1490) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 56s{color} | {color:orange} root: The patch generated 22 new + 172 unchanged - 12 fixed = 194 total (was 184) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 41s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 156 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 7s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 15s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 19s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 25s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737266#comment-16737266 ] Steve Loughran commented on HADOOP-14556: - * whitespace is unimportant; I always fix on final apply * checkstyle, I've tried to address all the fixable ones, when they complain about line length in javadocs when the line is a URL, noting I can do about it. This test failure is from not having a role ARN defined for assumed roles, *and the test not downgrading to skip in this situation*. # I'll harden the test against that # and make sure the docs are clear about what you need For the tests to work, # you need the ARN of a role which your account call STS.assumeRole() on; # role needs rights to work with S3, DynamoDB, AWS KMS # set this in your auth-keys in the property fs.s3a.assumed.role.arn > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737215#comment-16737215 ] Gabor Bota commented on HADOOP-14556: - Thanks for working on this [~ste...@apache.org]! Tested the newest patch against eu-west-1 with {{mvn verify -Dparallel-tests -DtestsThreadCount=8 -Ds3guard -Ddynamo -Dauth}} (I usually run tests with these params). I had the following error: {noformat} [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 20.79 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens [ERROR] testCreateAndUseDT(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens) Time elapsed: 3.484 s <<< ERROR! java.lang.IllegalStateException: No role ARN defined in fs.s3a.assumed.role.arn at com.google.common.base.Preconditions.checkState(Preconditions.java:145) at org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:134) at org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:50) at org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:140) at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:422) at org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testCreateAndUseDT(ITestSessionDelegationTokens.java:176) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testSaveLoadTokens(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens) Time elapsed: 2.145 s <<< ERROR! java.lang.IllegalStateException: No role ARN defined in fs.s3a.assumed.role.arn at com.google.common.base.Preconditions.checkState(Preconditions.java:145) at org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:134) at org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:50) at org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:140) at org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:422) at org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testSaveLoadTokens(ITestSessionDelegationTokens.java:121) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737121#comment-16737121 ] Steve Loughran commented on HADOOP-14556: - Tested: S3 Ireland > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556-028.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737050#comment-16737050 ] Steve Loughran commented on HADOOP-14556: - thanks for fixing the build [~ajisakaa], rolling this fast > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737042#comment-16737042 ] Steve Loughran commented on HADOOP-14556: - OK, two little details to follow on from this # I somehow managed to include the patch for HADOOP-16018 in this commit. I'm going to revert just that bit of the patch as a new patch and reapply separately. # HADOOP-16033: hamcrest-library is declared as compile-time dependency, not test-time. One-line POM fix to come in. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736777#comment-16736777 ] Hudson commented on HADOOP-14556: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15741 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15741/]) Revert "HADOOP-14556. S3A to support Delegation Tokens." (aajisaka: rev 7f783970364930cc461d1a73833bc58cdd10553e) * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSECBlockOutputStream.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/NoAwsCredentialsException.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/RoleTestUtils.java * (delete) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_token_architecture.md * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestRoleDelegationTokens.java * (delete) hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.security.token.DtFetcher * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageStatistics.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestSSEConfiguration.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AbstractSessionCredentialsProvider.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/EncryptionSecrets.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/Csvout.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/AbstractITCommitMRJob.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/SessionTokenBinding.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/TemporaryAWSCredentialsProvider.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/TestStagingCommitter.java * (edit) hadoop-tools/hadoop-aws/pom.xml * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/DurationInfo.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/RoleModel.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenIdentifier.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ILoadTestSessionCredentials.java * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenIdentifier.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumedRoleCommitOperations.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentials.java * (edit) hadoop-project/pom.xml * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/DelegationTokenIOException.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/staging/TestStagingPartitionedJobCommit.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DirListingMetadata.java * (delete) hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/yarn/ITestS3AMiniYarnCluster.java * (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java * (delete) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/FullCredentialsTokenBinding.java * (edit)
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736752#comment-16736752 ] Akira Ajisaka commented on HADOOP-14556: In addition to Kai's comment, this commit caused javadoc error. {noformat:title=AbstractS3ATokenIdentifier.java} * Kind => class, which is then looked up to deserialize token {noformat} {noformat:title=AbstractS3ATokenIdentifier.java} * catch & downgrade. RuntimeExceptions (e.g. Preconditions checks) are {noformat} {noformat:title=AbstractDelegationTokenBinding.java} * This is logged during after service start & binding: {noformat} {{&}} and {{>}} must be escaped in javadoc. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736638#comment-16736638 ] Kai Xie commented on HADOOP-14556: -- Hi [~ste...@apache.org] It seems the commit contains the patch for distcp from HADOOP-16018, could you help to revert that? thanks > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735856#comment-16735856 ] Hudson commented on HADOOP-14556: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15725 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15725/]) HADOOP-14556. S3A to support Delegation Tokens. (stevel: rev d7152332b32a575c3a92e3f4c44b95e58462528d) * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/TestS3AAWSCredentialsProvider.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSES3BlockOutputStream.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/SimpleAWSCredentialsProvider.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenBinding.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/RolePolicies.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/Csvout.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/NoAuthWithAWSException.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/IAMInstanceCredentialsProvider.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentialProvider.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/STSClientFactory.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/AbstractS3ATestBase.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractGetFileStatusTest.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/MarshalledCredentials.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/RoleTokenIdentifier.java * (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AWSPolicyProvider.java * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADelegationTokens.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/S3GuardTool.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationTokenBinding.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStore.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java * (add) hadoop-tools/hadoop-aws/src/main/resources/META-INF/services/org.apache.hadoop.security.token.DtFetcher * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/NanoTimerStats.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/NoAwsCredentialsException.java * (delete) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionSSECBlockOutputStream.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDelegationIT.java * (edit) hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractDTService.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/AbstractFileSystem.java * (edit) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/RoleTestUtils.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/DefaultS3ClientFactory.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Invoker.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Statistic.java * (edit) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DirListingMetadata.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestRoleDelegationInFileystem.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/AbstractS3ATokenIdentifier.java * (add) hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/delegation/ITestSessionDelegationInFileystem.java * (add) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/delegation/S3ADtFetcher.java
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735247#comment-16735247 ] Larry McCay commented on HADOOP-14556: -- Thanks for the insights and changes, [~ste...@apache.org]. +1 > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732350#comment-16732350 ] Steve Loughran commented on HADOOP-14556: - Patch 027; tested s3 ireland +dynamodb. All happy. h3. Larry's comments {quote}If delegation tokens are enabled it seems that there is no fallback to other providers. This seems appropriate to me but just want to make sure that interpretation is correct. {quote} Its up to the DT binding to provide the fallback chain of its choice. This lets it choose "fallback to some other property", "fail dramatically". {quote}While there are specific implementations provided in the patch it also seems to be extensible by 3rd parties that which to provide their own DT and binding code. {quote} that's the goal. {quote}Can you provide any details on renewal of DT's in the provided implementations? {quote} There is no renewal of DTs. It doesn't make sense. {quote}AWSCredentialProviderList + name {quote} it's there to provide slightly more informative errors. I didn't bother enforcing it or being clever. change: I patch things up in setName(). {quote}ASSUMED_ROLE_STS_ENDPOINT_REGION and ASSUMED_ROLE_STS_ENDPOINT_REGION {quote} this is really complex. Have a look at {{STSClientFactory.builder()}}. If you don't set a region then it defaults to the central endpoint, but you can't declare that as the endpoint without declaring a signing region in your EndpointConfiguration class. Which you don't need to do with the empty string. Really, *you do not want to begin trying to understand the logic here as there is no logic to understand* {quote}S3A.java: it possible for the toString() to have credentials from the URI? {quote} not since we took away the ability to put credentials in URIs, no. {quote}DurationInfo logAtInfo - should this be javadoc'd to explain how and when it is set to debug? {quote} done {quote}S3AEncryptionMethods. Can the following javadoc be better explained? {quote} done. They just round off the enum with all supported methods. Added tests to verify the matching works. {quote}S3AEncryptionMethods. should we consider making encryption algorithms such as AES256 configurable rather than hardcoded? {quote} nope. These are the exact strings which AWS requires in its headers. h3. Other Changes * add Optional expiry time accessor of MarshalledCredentials; always returns a UTC-zoned value or empty() when no expiry is known. . Diving into the AWS SDKs, they seem to expect it to be UTC, though it's not ever spelt out in the docs properly. This is used in tests and toString. The wire format is still seconds in the Unix epoch with no TZ marker. It MUST be UTC. * Tests working in this probably work even if the person running the tests isn't in/near UTC. I think up until now, some of the assertions about expiry only worked when the offset between UTC and local time was close enough. that (UTC-expiry-time - local-clock-time) was positive * EncryptionSecrets don't have a ref to S3AUtils in, so no indirect ref to any AWS classes. Harder to isolate tokens than you'd think. I'm almost tempted to move the token identifiers into hadoop-common to guarantee that (a) they get everywhere and (b) no accidental extra dependencies. But that'd probably create other problems. Token providers have the ability to provide a UA suffix for the client, which gets passed all the way down to the S3 client. Why so? Makes it possible to debug what's going wrong from S3A access logs, as the DT UUID can make it into the logs. Once you start playing with assumed role, user IDs get lost, and of course, that's when things start to go wrong. This propagates all the way into the S3 Client factories, I'm afraid. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731063#comment-16731063 ] Larry McCay commented on HADOOP-14556: -- Hi [~ste...@apache.org] - I have finally gotten through the majority of the patch. This looks like a great contribution! A few observations that I would like to verify: # If delegation tokens are enabled it seems that there is no fallback to other providers. This seems appropriate to me but just want to make sure that interpretation is correct. # While there are specific implementations provided in the patch it also seems to be extensible by 3rd parties that which to provide their own DT and binding code. Can you provide any details on renewal of DT's in the provided implementations? A number of nits and questions/comments from the review: core-default.xml Typos: s/provide/to provide/ s/disnabled/disabled/ {code} + The name of a class provide delegation tokens support in S3A. + If unset: delegation token support is disnabled. {code} AWSCredentialProviderList Probably want a space after name in {code} + String message = name + "No AWS Credentials provided by " Actually the javadoc seems to indicate that ": " will always be there. + /** + * The name, with a ": " suffix. + */ + private String name = ""; + {code} How is this policed and does it really make sense in a name? Can you explain what the name is for here? Are we providing the ability to have multiple named provider chains in a config? Constants Is the following supposed to be an empty string and the ASSUMED_ROLE_STS_ENDPOINT_REGION would be set if DEFAULT_ASSUMED_ROLE_STS_ENDPOINT isn't an empty string? {code} + * Default endpoint for session tokens: \{@value}. + * This is the central STS endpoint which, for v3 signing, can + * issue STS tokens for any region. + */ + public static final String DEFAULT_ASSUMED_ROLE_STS_ENDPOINT = ""; + + /** + * Region for the STS endpoint; needed if the endpoint + * is set to anything other then the central one.: \{@value}. */ public static final String ASSUMED_ROLE_STS_ENDPOINT_REGION = "fs.s3a.assumed.role.sts.endpoint.region"; {code} and again {code} + public static final String ASSUMED_ROLE_STS_ENDPOINT_REGION_DEFAULT = ""; {code} S3A.java Is it possible for the toString() to have credentials from the URI? {code} + @Override + public String toString() { + final StringBuilder sb = new StringBuilder("S3A{"); + sb.append("URI =").append(fsImpl.getUri()); + sb.append("; fsImpl=").append(fsImpl); + sb.append('}'); + return sb.toString(); } {code} DurationInfo.java logAtInfo - should this be javadoc'd to explain how and when it is set to debug? S3AEncryptionMethods.java Can the following javadoc be better explained? {code} * There's scope in here for client encryption options, even while not * currently supported in S3A. {code} Also, should we consider making encryption algorithms such as AES256 configurable rather than hardcoded? > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726979#comment-16726979 ] Steve Loughran commented on HADOOP-14556: - HADOOP-14556 patch 026 by popular request: [an architecture document|https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/delegation_token_architecture.md] * DT bindings have the right to say "we are not serving tokens", without actually failing. * (Marshalled) EncryptionSecrets remove their references of AWS SDK classes. This ensures that S3ADT Identifiers will load with the AWS SDK on the CP. (stops the RM needing it, though it will need hadoop-aws). * minor test tuning > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726248#comment-16726248 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 56s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 52s{color} | {color:green} root generated 0 new + 1489 unchanged - 1 fixed = 1489 total (was 1490) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 51s{color} | {color:orange} root: The patch generated 15 new + 171 unchanged - 11 fixed = 186 total (was 182) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 26s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 104 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 15s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 36s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 46s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 33s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 43s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726168#comment-16726168 ] Steve Loughran commented on HADOOP-14556: - Aaron: yes, I can do this. What I Can't do this week (i.e. this year) is give a real live demo...do something early in jan and I can show it working on a test cluster where I can submit a query on my laptop (with my secrets) and have it executed in a cluster with no creds ... and have the only secrets being passed down being some role secrets only valid for the target bucket/DDB store and for the next 12 h only. BTW, I've been wondering what we could do here with jenkins runs: the restricted role stuff would in theory let a secure site create restricted tokens which it'd feed to jenkins on a run-by-run basis > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725483#comment-16725483 ] Aaron Fabbri commented on HADOOP-14556: --- Hi [~ste...@apache.org]. Looks like you are needing reviews on this but it is a lot to digest for reviewers. Any interest in doing a show-and-tell (conf call) to walk through the patch and answer questions? Totally up to you, and just an idea, but if you do this, I'm in. [~gabor.bota] might be interested as well. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722956#comment-16722956 ] Steve Loughran commented on HADOOP-14556: - Tested on S3 Ireland BTW There are 10 people watching this. I need 1 or 2 people to actually look at the code and comment. Yes, it's a big piece of work, yes, its complex -but that's because unlike the DT plugin points of the other object stores (wasb, abfs) I'm actually implementing the token support, with simple options (session) and advanced (generating restricted roles after determining exact requirements of the user). If anyone watching this JIRA has any intention of using this feature, then they should really review it. Thanks. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556-024.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718334#comment-16718334 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 12s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 5s{color} | {color:green} root generated 0 new + 1489 unchanged - 1 fixed = 1489 total (was 1490) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 24s{color} | {color:orange} root: The patch generated 15 new + 171 unchanged - 11 fixed = 186 total (was 182) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 11s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 104 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 58s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 29s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 47s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718032#comment-16718032 ] Steve Loughran commented on HADOOP-14556: - For people wanting to play with this stuff, note that [cloudstore}https://github.com/steveloughran/cloudstore/releases/tag/release_2018_12_11] now does everything you can imagine with tokens: > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718031#comment-16718031 ] Steve Loughran commented on HADOOP-14556: - +make sure that nothing marshalled in the token identifiers have any dependency on the AWS SDK. I think I was too clever in the MarshalledCredentials code. Makes it too easy to damage the RM > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710352#comment-16710352 ] Steve Loughran commented on HADOOP-14556: - Something to add on that reporting of auth chain problems: {{AWSCredentialProviderList}} to have a constructor taking a string, and use that as part of the generated exception. Why so? Helps, when tracking down remote DT problems and so differentiate * binding is enabled, no DT supplied * binding isn't enabled, DT not picked up Because its the exception which makes it way back to the client (spark-submit ), the more which can be done without going near the logs is useful > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16703063#comment-16703063 ] Steve Loughran commented on HADOOP-14556: - One other comment on [~elgoiri]'s feedback > IAMInstanceCredentialsProvider#getCredentials leaves comments behind. I actually switched to the commented one, which wraps the exception raised by the IAM provider. why so? The default error message you get in the absence of any credentials is the "cannot connect to 169.xx.xx.xx" error from the IAM provider which cannot talk to the IAM server. Because we have that on the default chain, and unless you are in an EC2 deployment (were it will never fail as you always get the VM's credentials), it is guaranteed to fail. So I'm wrapping that as an {{NoAwsCredentialsException}} as that's what it means. The error raising in {{AWSCredentialProviderList}} is tweaked to move from throwing the last exception to "the most recent exception which isn't just a {{NoAwsCredentialsException}}. That means if you have an auth chain where your DT plugin is failing for a complex reason, that failure gets thrown, even if you have a fallback chain of other things afterwards (env vars,etc) > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701881#comment-16701881 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 1s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 47s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 31s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 2s{color} | {color:green} root generated 0 new + 1487 unchanged - 1 fixed = 1487 total (was 1488) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 3s{color} | {color:orange} root: The patch generated 13 new + 170 unchanged - 11 fixed = 183 total (was 181) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 98 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 41s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 35s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 2s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 31s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 45s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s{color} | {color:green} The patch does not generate ASF License warnings. {color}
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701742#comment-16701742 ] Steve Loughran commented on HADOOP-14556: - patch 023 : patch 022 with the sole viable checkstyle issue (indentation) fixed w.r.t the other style errors, they split into * complaints about existing code (wont-fix-here) * links in javadocs too long (wontfix) * some lines being just past the 80 char barrier, where splitting has no tangible benefit. I really need reviews here from anyone who can do this. We also depend on HADOOP-15808 to harden token load, so ensure that nothing added here breaks token support for everything if the classpath is incomplete. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701269#comment-16701269 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 21m 30s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 46s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 21s{color} | {color:green} root generated 0 new + 1487 unchanged - 1 fixed = 1487 total (was 1488) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 39s{color} | {color:orange} root: The patch generated 14 new + 171 unchanged - 11 fixed = 185 total (was 182) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 29s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 98 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 4s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 13s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 16s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 41s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700677#comment-16700677 ] Larry McCay commented on HADOOP-14556: -- [~ste...@apache.org] - thanks for considering that! In addition to a demo - it may also be helpful to provide a high level walkthrough of the patch itself. Just some landmarks and some indications of areas for which you would like more detailed review - if that makes sense. The demo would actually provide a bit of that as well as long as you include details of configuration and discussion of the marshalling/unmarshalling requirements, etc. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700653#comment-16700653 ] Steve Loughran commented on HADOOP-14556: - Hi [~elgoiri]: thanks for this review; not had a chance to reply until now. bq. The unit tests cover the basic cases well. I'd have liked to have a real mini-yarn cluster with distcp, but couldn't get kerberos to work with miniyarn and minihdfs to the extent the cluster would come up. If/when someone can do that. I'd revisit it. bq. Very long patch and even though there are a bunch of interfaces which are pretty verbose, there is a lot here. I'm not sure if there are ways to split it. For example the utilities to fetch the DT. I know, and I always worry about adding more complexity for the following reason: other people have to maintain it, and if they can't either the code is neglected or I'm expected to be the maintainer indefinitely. I've tried to keep all DT support out in its own home, with not that much in the S3A FS -but as I changed the encryption stuff there may be too much of a diff there. I could perhaps revert some of that. Less elegant but a smaller diff for that file, and so less risk of merge conflict. And because I was going near session credential management, I also tried to coalesce stuff that the credential providers were doing. Again, I could look to pull that for now Otherwise: I've needed to do all 3 including the role stuff, to make sure I hadn't blocked out those. I even believe that I've done enough to support more advanced bindings. We could strip out the full credentials as it doesn't reduce risk, and so only support session and role secrets? that'd work well for locking down AWS, but I would also like to support third party stores which don't have sessions regarding the docs, [~lmccay] has suggested I could actually do a video of this at work. Would people be interested? That'd be a real demo of role-base-DT => live cluster for distcp. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693721#comment-16693721 ] Íñigo Goiri commented on HADOOP-14556: -- Thanks [~ste...@apache.org] for [^HADOOP-14556-021.patch]. * I think we can solve most of the checkstyle issues left here )specially left spaces). * {{to directly authenticate with S3 and DynamoDB services..}}, you have a double dot here. * The indices in core-default.xml are dupe, I'm not sure if there's a point on having a number; anyway, if we keep it we should change the numbers. * The {{assertEquals}} message in {{AbstractContractGetFileStatusTest}} should have a space or something in between. * {{AWSCredentialProviderList}} now does LOG and throw, I think we should keep it as just throw. * Constants#DEFAULT_ASSUMED_ROLE_STS_ENDPOINT should fit in one line. * S3AUtils#getCanonicalServiceURI has comments left behind. * IAMInstanceCredentialsProvider#getCredentials leaves comments behind. * In MarshalledCredentialProvider#NAME can use the class.getName() or one of the variations? * MarshalledCredentials#equals can use EqualsBuilder (same for EncryptionSecrets). * We use the {{Collections.unmodifiableList(Arrays.asList(}} a lot, it might be worth adding a method to define constant lists. In general I can provide a very high level review but somebody else should review deeper. In any case, general comments: * The documentation seems to cover the added cases well; it might be worth doing a full pass to these documents once everything is finished up. * The unit tests cover the basic cases well. * Very long patch and even though there are a bunch of interfaces which are pretty verbose, there is a lot here. I'm not sure if there are ways to split it. For example the utilities to fetch the DT. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch, > HADOOP-14556-021.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693267#comment-16693267 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 3s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 27s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 13s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 43s{color} | {color:green} root generated 0 new + 1448 unchanged - 1 fixed = 1448 total (was 1449) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 21s{color} | {color:orange} root: The patch generated 14 new + 167 unchanged - 11 fixed = 181 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 97 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 23s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 17s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 16s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 41s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692228#comment-16692228 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 13s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 12s{color} | {color:green} root generated 0 new + 1448 unchanged - 1 fixed = 1448 total (was 1449) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 48s{color} | {color:orange} root: The patch generated 31 new + 167 unchanged - 11 fixed = 198 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 98 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 34s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 38s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 33s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 8s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 25s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685177#comment-16685177 ] Steve Loughran commented on HADOOP-14556: - Patch 019: logging, resilience and debugging, mostly * All S3A tokens have a (string) UUID, this is the sole field used for equality, and it is printed. Makes it easy to verify propagation. * reverted constructor of Optional; instead tag as @Nullable, and make clear everywhere this is true...adding tests where appropriate to catch regressions. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683789#comment-16683789 ] Steve Loughran commented on HADOOP-14556: - +aw, see comment on https://issues.apache.org/jira/browse/HADOOP-12563?focusedCommentId=16635508=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16635508 I think we should harden dtclient, I'm just trying to avoid JAR-spanning-code changes here or adding too many dependencies on other patches. This is big/complex enough as it is > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683651#comment-16683651 ] Steve Loughran commented on HADOOP-14556: - Allen, good q. # Dtutil only fetches DTs if UGI is in secure mode, whereas fetchdt asks the FS irrespective of the local security state. Therefore it can issue DTs without Kerberos. You can't use them for job submission as MR's token fetching (also used by Distcp) requires Kerberos, as does the spark token collection. But you can use the tokens collected by fetchdt in other apps, as the [latest relase of cloudstore does|https://github.com/steveloughran/cloudstore/releases/tag/tag_2018_11_09b] # Because the probe for "Are tokens available" doesn't take the FS URI , the impl has to say "yes" without knowing if the FS actually does. # Dtutil expects that when a token is requested, the impl always returns 1+ token. Because s3a token issuing is optional (as it is on azure, abfs), if you ask the FS for a token and it doesn't issue one, you get a stack trace (Array out of bounds or something similar) For fetch DT to work in this world, it needs * service loading to be resilient to classpath problems (FWIW, so does whole token mechanism: HADOOP-15808) * FS (or at least s3a FS) code to say "true" whenever probed to see if tokens are available * dtutil to be ready to handle the case where "no tokens actually get issued" (at the very least make it an option) that means: changes in DTutil, and the fs binding > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682565#comment-16682565 ] Allen Wittenauer commented on HADOOP-14556: --- Why does the documentation, etc, use 'hdfs fetchdt' when 3.x has the generic 'hadoop dtutil'? > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch, > HADOOP-14556-018a.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674068#comment-16674068 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 26s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 25m 3s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 23m 33s{color} | {color:green} root generated 0 new + 1448 unchanged - 1 fixed = 1448 total (was 1449) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 40s{color} | {color:orange} root: The patch generated 17 new + 170 unchanged - 8 fixed = 187 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 37s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 78 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 59s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 3s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 16s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 58s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 57s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672474#comment-16672474 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 22m 55s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 39s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 24s{color} | {color:green} root generated 0 new + 1448 unchanged - 1 fixed = 1448 total (was 1449) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 27s{color} | {color:orange} root: The patch generated 22 new + 170 unchanged - 8 fixed = 192 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 69 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 12m 4s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s{color} | {color:red} hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 6s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 12s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 39s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672084#comment-16672084 ] Steve Loughran commented on HADOOP-14556: - patch 017 * Move {{S3AFileSystem.dtIntegration}} field in S3A Filesystem to being an optional type; change name & do a best effort at java-8 code. * declaring a binding is enough to turn DTs on. Simplifies configs and docs, reduces misconfig risks. You can't ever have DT enabled without a binding class; or have a binding class declared but for some reason not have tokens picked up just because the enabled bit was false. * marshalled credentials have validity checks, it should be impossible to have null values there; meaningful messages when they don't meet the requirements of the caller (Full tokens, session, either...) * validation methods let callers declare excactly what they want to validate the creds with * test improvements * move from Preconditions.checkNotNull to Objects.requireNonNull, use lambda-for errors when it seems worthwhile The property for DT binding auth providers switched to fs.s3a.aws.credentials.provider, that is, the normal list for credential providers when DT bindings is not enabled (i.e. not the assumed role one.) The list of default credential providers now includes Temporary/Session credentials as the first entry in the list. This is a change from before. where people had to explicitly turn it on. In contrast, the Env var plugin looked for session creds first, and of course IaM roles is always temp. This gives the following sequence for finding credentials # fs.s3a.account.key + fs.s3a.secret.key + fs.s3a.session.token => session credentials # fs.s3a.account.key + fs.s3a.secret.key => full credentials. (because of the ordering, this will only be reached if #1 is unsatisifed) # Env vars, first session env vars, then full # IAM Role. I've also set things up in future to move to async IAM credential refresh with a new (not yet documented, still private) credential provider there, which internally shares a reference to the single IAM instance credentials provider. Testing: s3a ireland, all well, but (and this was a scale run) MPU tests still failing; WiP by Ewan there {code} [ERROR] testMultipartUploadEmptyPart(org.apache.hadoop.fs.contract.s3a.ITestS3AContractMultipartUploader) Time elapsed: 0.749 s <<< ERROR! java.lang.IllegalArgumentException: partNumber must be between 1 and 1 inclusive, but is 0 at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115) at org.apache.hadoop.fs.s3a.WriteOperationHelper.newUploadPartRequest(WriteOperationHelper.java:377) at org.apache.hadoop.fs.s3a.S3AMultipartUploader.putPart(S3AMultipartUploader.java:97) {code} Bad request on the SSEC huge file upload {code} [ERROR] test_040_PositionedReadHugeFile(org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks) Time elapsed: 0.322 s <<< ERROR! org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on test/: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 670ACD7E81475D64; S3 Extended Request ID: 22WuQmeTqICfvd+9bgfSnwiptwCNA80ZlQqoF1hDBJJ0wlfPYTkmlO+r4g0tHBILG5l2NYIHVb8=), S3 Extended Request ID: 22WuQmeTqICfvd+9bgfSnwiptwCNA80ZlQqoF1hDBJJ0wlfPYTkmlO+r4g0tHBILG5l2NYIHVb8=:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 670ACD7E81475D64; S3 Extended Request ID: 22WuQmeTqICfvd+9bgfSnwiptwCNA80ZlQqoF1hDBJJ0wlfPYTkmlO+r4g0tHBILG5l2NYIHVb8=) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:227) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2382) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2320) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2258) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2207) at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2177) at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2274) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338) at org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193) at org.apache.hadoop.fs.s3a.scale.S3AScaleTestBase.setup(S3AScaleTestBase.java:90) at org.apache.hadoop.fs.s3a.scale.AbstractSTestS3AHugeFiles.setup(AbstractSTestS3AHugeFiles.java:78) at org.apache.hadoop.fs.s3a.scale.ITestS3AHugeFilesSSECDiskBlocks.setup(ITestS3AHugeFilesSSECDiskBlocks.java:41) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16670050#comment-16670050 ] Steve Loughran commented on HADOOP-14556: - I'm thinking: if we do want more independent upload of fs data in job submission, should we include all bucket-specific options in the store * signing type * endpoint * etc Pro: lets me submit work to a cluster which can include a whole new endpoint, auth mech, etc con: it gets complicated fast. What I might do is add to the s3a token identifier the map of k->v options for this, but not collect or use them yet, just read and write. I know, I could just give up and embrace protobuf rather than try and do versioning in my own code, but, well, its no like protoc likes maps either > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665812#comment-16665812 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 39 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 56s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 41s{color} | {color:green} root generated 0 new + 1448 unchanged - 1 fixed = 1448 total (was 1449) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 25s{color} | {color:orange} root: The patch generated 19 new + 185 unchanged - 8 fixed = 204 total (was 193) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 38 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 11s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 14s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 11s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 40s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s{color} |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16665595#comment-16665595 ] Steve Loughran commented on HADOOP-14556: - Patch 016 # improve subclassing, scope of methods # lifecycle change: serviceStart triggers bonded/unbonded immediately. This has forced me to add a package-scoped way to reset/rebind a delegation token for ease of testing binding without going near UGI; that's a special codepath # the DT Binding always provides a list of credential providers, even when deployed without a DT. # move to Optional<> over nullables for fields, embrace java8 #* document how to subclass # Use of origin diagnostics string through identifiers is consistent # empty AWS credentials can be marshalled The fact that credential setup is always controlled when you turn DTs on is signficant but needed once you start doing really complex stuff with the bindings: your DT provider needs to be able to bootstrap your login directly. While things like wasb/abfs force you declare a consistent pair of (issue, auth) entries, having the dt binding do everything lets it instantiate instances all glued together Tests, s3 ireland. {{ITestS3ATemporaryCredentials.}} tests are failing because somehow a dt binding is being enabled (shared fs instance?) and this dynamic binding isnt' switching to the session creds, only key & secret are being handed in, which is rejected "UnrecognizedClientException" I'm going to fix that, and in the process tune the default auth chain to be that which the DTs will also use by default, to, in order * Temp/session credentials (fs.s3a.{access, secret, session) * long-lived credentials (fs.s3a.{access, secret) * env vars * IAM ref I'm also considering adding an IAMInstance cred provider which does async refresh, but for that it needs to be ref counted, so I'm pushing it out of this JIRA > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656069#comment-16656069 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 39 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 45s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 25s{color} | {color:green} root generated 0 new + 1316 unchanged - 1 fixed = 1316 total (was 1317) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 26s{color} | {color:orange} root: The patch generated 28 new + 185 unchanged - 8 fixed = 213 total (was 193) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 159 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 1s{color} | {color:red} hadoop-tools/hadoop-aws generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 31s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 7s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 13s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green}
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655935#comment-16655935 ] Steve Loughran commented on HADOOP-14556: - HADOOP-14556 patch 014 * Session & role tokens postpone reading of client side config options, building STS client until token creation, so server-side deployments without the relevant options all work. (+tests) * Writable/Serializable encryption methods class adds enum for client side too, version uid & checks in writable to verify it hasn't changed. Why so? Placeholder for client side. I know that's controversial but I don't want to box it out. * using Optional over null in a few places. As usual, mixed feelings: we can't use map or foreach much because all our code throws IOEs. * Tests: more, a base class with common methods for them * Fixup bouncy castle classpath after latest yarn changes. testing? Not right now. It's late. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch, > HADOOP-14556-015.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652427#comment-16652427 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 37 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 5s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 15s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 22s{color} | {color:green} root generated 0 new + 1326 unchanged - 1 fixed = 1326 total (was 1327) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 29s{color} | {color:orange} root: The patch generated 20 new + 185 unchanged - 8 fixed = 205 total (was 193) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 57s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 136 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 34s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 28s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 13s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 38s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650982#comment-16650982 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 38 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 2s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 11s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 11s{color} | {color:red} root generated 1 new + 1326 unchanged - 1 fixed = 1327 total (was 1327) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 27s{color} | {color:orange} root: The patch generated 20 new + 168 unchanged - 6 fixed = 188 total (was 174) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 5s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 128 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 3s{color} | {color:red} The patch 3 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 45s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 31s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 58s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 26s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:red}-1{color} |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650850#comment-16650850 ] Steve Loughran commented on HADOOP-14556: - Note that the partially {{ITestDelegatedMRJob}} test does show that S3A tokens are picked up for MR job submit; tested for full, session and role tokens. One fun detail: if your fs.s3a.secret.key attributes are set in the job conf you launch with, they end up at the far end, even though you are using DTs. Why? well, because they are config options, aren't they? To get the lockdown to work, you need to be serving up the secrets inside a hadoop credential provider file such as localjceks file. That way, the job conf will not contain the secrets. There's no > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650831#comment-16650831 ] Steve Loughran commented on HADOOP-14556: - HADOOP-14556 patch 013 * ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn cluster to verify that MR job submission collects DTs for source and destination paths. To do this the MockJob class had to go into hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java and job.connect() made an override point (so it can be skipped) * default assumed role duration returned to 1h; it had been extended to 6h but that only works if your role has been explicitly extended to > 1h duration. * and docs on increasing it (plus error messages you get if you don't) improved/extended in assumed_roles.md as well as delegation_tokens.md. All AWS error messages related to STS/session and role requests are now in assumed_roles.md to avoid duplication & inconsistencies. * ITestS3ADelegationTokenSupport tests that the Session DT binding will forward any session creds it gets from its own auth chain, rather than ask for new ones (which it can't do with session creds) * Also: I'm using a Hadoop cred provider for storing secrets; this broke the AssumeRole and delegation tests which were clearing or overwriting the fs.s3a.{auth, secret, session} options, as those in the creds file were still being picked up. Fix: explicitly reset hadoop.security.credential.provider.path for all the tests which were now failing. * minor checkstyle fixup tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of {{ITestS3GuardToolLocal\#testDestroyNoBucket }} *even when I was running with dynamodb*. I think that test suite is running when it shouldn't. More research needed there > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch, > HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650124#comment-16650124 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 14s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m 20s{color} | {color:green} root generated 0 new + 1326 unchanged - 1 fixed = 1326 total (was 1327) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 26s{color} | {color:orange} root: The patch generated 15 new + 112 unchanged - 6 fixed = 127 total (was 118) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 110 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 3s{color} | {color:red} The patch 3 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 42s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 10s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 35s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 39s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643754#comment-16643754 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 47s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 24s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 46s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 50s{color} | {color:green} root generated 0 new + 1326 unchanged - 1 fixed = 1326 total (was 1327) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 58s{color} | {color:orange} root: The patch generated 33 new + 119 unchanged - 7 fixed = 152 total (was 126) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 14s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 105 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 12s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 51s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 49s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 36s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 47s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 49s{color} | {color:green} The patch does not
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643389#comment-16643389 ] Steve Loughran commented on HADOOP-14556: - patch 011 * disable secure minicluster test but retain it with warnings for now * fix checkstyle warnings by not exporting mutable arrays; move to serving up immutable lists instead. This code is now at a state where it needs people to play with & see how well it works in real-world use, especially given that I can't bring up a miniKDC, mini-yarn cluster. I did think about writing some new yarn example class, "touch" h3. test results tested s3a ireland with ddb at auth. Failures (which I consider unrelated but will need to fix elsewhere) {code} [INFO] Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 260.274 s - in org.apache.hadoop.fs.s3a.ITestS3AContractGetFileStatusV1List [ERROR] Tests run: 43, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 524.063 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract [ERROR] testListStatus(org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract) Time elapsed: 11.539 s <<< FAILURE! java.lang.AssertionError: expected:<1> but was:<2> at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:542) at org.apache.hadoop.fs.FileSystemContractBaseTest.testListStatus(FileSystemContractBaseTest.java:309) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) [WARNING] Tests run: 68, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 1,073.11 s - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextMainOperations [ERROR] Tests run: 11, Failures: 0, Errors: 2, Skipped: 1, Time elapsed: 929.822 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB [ERROR] testPruneCommandCLI(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB) Time elapsed: 600.03 s <<< ERROR! java.lang.Exception: test timed out after 60 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.retryBackoffOnBatchWrite(DynamoDBMetadataStore.java:813) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.processBatchWriteRequest(DynamoDBMetadataStore.java:765) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPut(DynamoDBMetadataStore.java:851) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.removeAuthoritativeDirFlag(DynamoDBMetadataStore.java:1080) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:1033) at org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:993) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommand(AbstractS3GuardToolTestBase.java:271) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommandCLI(AbstractS3GuardToolTestBase.java:286) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640059#comment-16640059 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 48s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 20m 56s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 57s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 19s{color} | {color:green} root generated 0 new + 1326 unchanged - 1 fixed = 1326 total (was 1327) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 41s{color} | {color:orange} root: The patch generated 30 new + 118 unchanged - 7 fixed = 148 total (was 125) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 54s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 100 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 8s{color} | {color:red} hadoop-tools/hadoop-aws generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 57s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 24s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 34s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 50s{color} | {color:green} The patch does not
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639776#comment-16639776 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color} | {color:red} HADOOP-14556 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 | | JIRA Issue | HADOOP-14556 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12942151/HADOOP-14556-009.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/15299/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch, > HADOOP-14556-010.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16639773#comment-16639773 ] Steve Loughran commented on HADOOP-14556: - Patch 010. Contains a failed attempt to do a full e2e MR job with secure tokens. I made the mistake of trying to get a clone of the (Working) {{ITestS3AMiniYarnCluster}} test to now fetch delegation tokens, where the following sequence of problems were encountered, in roughly this order. # YARN doesn't fetch DTs unless security is enabled. Action: enable security. # MR reduce doesn't work with the local FS as the cluster FS when security is enabled, because LocalFetcher demands the native libs at this point for secure Posix operations, and test suites can't depend on that. Action: switch to HDFS # HDFS is a PITA to bring up securely, Action, create new class {{MiniKerberizeHadoopCluster}} tries to encapsulate the actions and config changes from some HDFS tests to bring up a secure HDFS cluster. # YARN is a PITA to bring up securely, and none of the YARN tests seem to do a full secure MiniKDC + HDFS + YARN. Action. Try to get {{MiniKerberizeHadoopCluster}} to do this, too, with a bit more guesswork on setttings # Job submit fails as the NM can't auth with the DN, even though the junit thread can. DN keeps breaking connections. Possibly some localhost/external IP address problem, which I can't debug because I can't turn the firewall off. Action: give up on secure HDFS # Switch to trying to use S3A itself as the cluster FS; lifting bits of code from (insecure) MR atop HDFS test cases to get the relevant MR JAR up there. # Test case fails {{testWordCount}} fails to submit job (server-side SASL problems where principal doesn't have a full user/name@realm structure ("Empty nameString not allowed"). # That problem goes away if you run all three tests in the test suite (word count, FS access and probe for the RM port being open). Action: run all three tests from the IDE. (note, they set up individual YARN clusters each) # Then you can submit work. But it doesn't run. And there's no logs other than this: {code} [2018-10-05 13:34:13.217]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Oct 05, 2018 1:34:00 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class Oct 05, 2018 1:34:00 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class Oct 05, 2018 1:34:00 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class Oct 05, 2018 1:34:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25 AM' Oct 05, 2018 1:34:00 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" Oct 05, 2018 1:34:00 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" Oct 05, 2018 1:34:00 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest" log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. {code} I've given up on the test at this point. I'm submitting the patch, and I'm going to keep it but set to skip, "broken", always for now. OR just tag & delete in my git repo. * I now understand why there aren't any end-to-end tests of MR across a secure mini-(kdc, hdfs, yarn) cluster. * the only way I can see this working is to not attempt wordcount but instead some trivial AM which just checks for FS access and then exits. That way, all the MR pain is gone. Still doesn't deal with secure yarn launch though. Tested: s3 ireland. All tests working except {code} [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 149.206 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob [ERROR] testWordCount(org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob) Time elapsed: 99.301 s <<< FAILURE! java.lang.AssertionError:
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636117#comment-16636117 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 47s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 31 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 11s{color} | {color:red} root in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 12s{color} | {color:red} root in trunk failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 1s{color} | {color:orange} The patch fails to run checkstyle in root {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 43s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 5m 10s{color} | {color:red} branch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 12s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 12s{color} | {color:red} root in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 4s{color} | {color:orange} The patch fails to run checkstyle in root {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 82 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m 15s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 21s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 11s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 39s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}120m 15s{color} |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635348#comment-16635348 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 31 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 54s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 44s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 9s{color} | {color:green} root generated 0 new + 1326 unchanged - 1 fixed = 1326 total (was 1327) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 55s{color} | {color:orange} root: The patch generated 28 new + 113 unchanged - 6 fixed = 141 total (was 119) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 44s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 73 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 41s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 59s{color} | {color:red} hadoop-tools/hadoop-aws generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s{color} | {color:red} hadoop-tools_hadoop-aws generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 7s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 29s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 44s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634404#comment-16634404 ] Steve Loughran commented on HADOOP-14556: - HADOOP-14556 patch 008 * coalesce config options so use assumed.role config options instead of adding new ones * role policies setting things up properly, including the listBucket command (which is per bucket, not bucket path) * tests for role tokens verify that you can't access other S3 resources * AWS credential provider subclassing now does on-demand init, so that you can have some cred providers (temporary) which only raise an AWS exception on call for creds, others can fail fast in the constructor * discussing use in hadoop-aws docs * clarifying implementation/subclassing details in javadocs * Deprecate test.fs.s3a.sts.endpoint in favour of fs.s3a.assumed.role.sts.endpoint; register it as a Configuration deprecation. * Logging @ debug (correct?) time to acquire/refresh DTs; useful given there are potentially long-haul calls taking place. * s3guard bucket-info prints token info, including a warning if delegation is set but you aren't logged in. + fix up the findbugs/checkstyle warnings from the previous run Testing: S3 london; got one error (HADOOP-15807) without S3Guard: Consistency? Or some race condition in tests? The `s3guard bucket-info` command now prints status {code} $ bin/hadoop s3guard bucket-info s3a://landsat-pds/ Filesystem s3a://landsat-pds Location: us-west-2 Filesystem s3a://landsat-pds is not using S3Guard The "magic" committer is supported S3A Client Endpoint: fs.s3a.endpoint=s3.amazonaws.com Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal Delegation token support is disabled {code} And when you enable it to the default value (session tokens); it prints the kind & then warns that security is off {code} $ bin/hadoop s3guard -D fs.s3a.delegation.tokens.enabled=true bucket-info s3a://landsat-pds/ Filesystem s3a://landsat-pds Location: us-west-2 Filesystem s3a://landsat-pds is not using S3Guard The "magic" committer is supported S3A Client Endpoint: fs.s3a.endpoint=s3.amazonaws.com Encryption: fs.s3a.server-side-encryption-algorithm=none Input seek policy: fs.s3a.experimental.input.fadvise=normal Delegation Support enabled; token kind = S3ADelegationToken/Session Warning: Hadoop security is disabled; delegation tokens will not be generated. {code} It might be good to make this an option where you can declare the token value, e.g -dtkind S3ADelegationToken/Role" which would fail if the token mapping wasn't right (type, flags, kinit, etc). That way: no ambiguity. Indeed, it could actually ask for a DT, though I'd rather that dttuil worked there: if it doesn't right now, that's something to fix. One issue here which I've made "go away" is to declare the FS default port is 0; the canonical URI == the normal s3 URI (no port, no user prefix). Putting the port #into URIs broke many things (FileContext, mock staging committer tests, S3Guard's existing URIs). I hope everyone is happy with that. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630349#comment-16630349 ] Steve Loughran commented on HADOOP-14556: - bq. I did this so long ago, w/o understanding as much about s3 as you, that I need to refresh my memory on whether it was intentional or not. it certainly makes sense if you allow >1 login for a specific bucket, e.g {code} fs1 = Filesystem.get("s3a://user1:keyt@bucket1") fs2 = Filesystem.get("s3a://user2:key2@bucket1") {code} then submit some job which reads from both filesystem instances. the FS1 DT would have the session info of user1; that of the FS2 DT would be that of the session info of user2. Having the user in the URI would be needed to ensure that the right DT was retrieved for the different filesystems, so the restricted permissions picked up One other thing your patch does well is ask the existing auth chain for session tokens and uses them too. I want to make sure I can do that too, as a use case I'm thinking of is "yubikey authenticated user submitting credentials with a job". I don't think such a user can call assumeRole() (as they get session credentials with the 2FA login), but their original session tokens can be propagated bq. Some of the design consideration your highlight, such always getting a STS token, were purposeful shortcuts (hence not submitted to community) since internally it is a solid requirement. I understand. But I also know that you understand a lot more about how DTs work than me, so if there is something I don't understand, my first assumption is generally "I need to know more" > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628944#comment-16628944 ] Steve Loughran commented on HADOOP-14556: - +want some CLI tools to go with this, for testing as well as basic use * ability to check encryption status/KMS key of a file (via getObjectMetadata) * ability to use a persisted DT as the login for hadoop fs options +cloudstore/storediag to look for DT support in all the filesystems > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628926#comment-16628926 ] Daryn Sharp commented on HADOOP-14556: -- bq, irrespective of the user:secret issue, I don't think I fully understand why there's support for multiple sessions here I did this so long ago, w/o understanding as much about s3 as you, that I need to refresh my memory on whether it was intentional or not. Some of the design consideration your highlight, such always getting a STS token, were purposeful shortcuts (hence not submitted to community) since internally it is a solid requirement. Thanks for carrying this forward, I will try to review when I'm back from travel! > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16628733#comment-16628733 ] Steve Loughran commented on HADOOP-14556: - +also: because s3guard stores the URI including port, explicitly including 443 as the default port causes chaos and confusion. the default port of the FS must be "0" so it doesn't get inserted or compared > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627805#comment-16627805 ] Steve Loughran commented on HADOOP-14556: - bq. TestStagingPartitionedTaskCommit this test is failing in the mock because the tests are saying "pretend a s3a://bucket/path" exists, but with canonicalization the path being committed is now s3a://bucket:443/path". Canonicalization is trouble. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621172#comment-16621172 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m 38s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 3s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 54s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 54s{color} | {color:red} root generated 3 new + 1335 unchanged - 0 fixed = 1338 total (was 1335) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 54s{color} | {color:orange} root: The patch generated 58 new + 103 unchanged - 6 fixed = 161 total (was 109) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 43s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 0s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s{color} | {color:red} hadoop-tools/hadoop-aws generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-tools_hadoop-aws generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 29s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 43s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 43s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color}
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621018#comment-16621018 ] Steve Loughran commented on HADOOP-14556: - Patch 007. still a WiP * For people who want to review this, the code is also [on github|https://github.com/steveloughran/hadoop/tree/s3/HADOOP-14556-delegation-token]. This fairly complex design is intended to * support different back-end token bindings * and leave it open for anyone who ever does Kerberos binding (as Wasb permits) to do so. Supported bindings * Full: your normal AWS secrets. Should work with non-AWS S3 services. * Session: session tokens are requested off STS * Role. This is the complex one, but the most significant. Ask for a restricted role with a configured role ARN and a dynamically created role policy restricted purely to the bucket & DDB table used by the FS (there's some interfaces there to let them tell the token binding what those policies are). Example: {code} 2018-09-19 19:15:10,324 [JUnit-testDTFileSystem] DEBUG auth.STSClientFactory (STSClientFactory.java:requestRole(181)) - Requesting role arn:aws:iam::111:role/stevel-s3guard with duration 21600; policy = { "Version" : "2012-10-17", "Statement" : [ { "Sid" : "7", "Effect" : "Allow", "Action" : [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource" : "arn:aws:s3:::hwdev-steve-ireland-new" }, { "Sid" : "8", "Effect" : "Allow", "Action" : [ "s3:Get*", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts", "s3:ListBucket*" ], "Resource" : "arn:aws:s3:::hwdev-steve-ireland-new/*" }, { "Sid" : "1", "Effect" : "Allow", "Action" : [ "kms:Decrypt", "kms:GenerateDataKey" ], "Resource" : "arn:aws:kms:*" }, { "Sid" : "9", "Effect" : "Allow", "Action" : [ "dynamodb:BatchGetItem", "dynamodb:BatchWriteItem", "dynamodb:DeleteItem", "dynamodb:DescribeTable", "dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:Query", "dynamodb:UpdateItem" ], "Resource" : "arn:aws:dynamodb:eu-west-1:000:table/hwdev-steve-ireland-new" } ] } {code} This token can be passed on to a shared hive/spark cluster, knowing that the maximum access anything with that token can have will be full R/W access to the destination bucket and any S3Guard table h3. Scale There's some ILoad* tests to see what the sustainable rate of issuing STS session and role tokens is. The TSV datasets [are available for download|https://github.com/steveloughran/datasets/releases/tag/tag_2018-09-17-aws] and analysis in your favourite notebook. Any analysis + different results from different locations would be great! Key points: # you can get about 500-1000 requests/second before calls get rejected. # Calls to STS do need to catch & retry on throttle events in the case this does occure. For anyone planning those tests, you need to invoke them by name and set -Dscale. Others users in your AWS account using the same STS endpoint may have calls rejected for throttling too, which may be "observable". Test carefully by selecting an explicit location and/or doing it in quiet periods. h3. TODO * if that token really does contain user info (i.e someone ever did kerberos support), it should somehow be preserved. What to do? * docs, obviously. * I now know more about role permissions; improve our docs there too. * FileContext tests are failing due to port mismatches in "canonical" paths. hence the improved detail on the failing exception being raised ... issue is still outstanding. * S3a FS to pick up encryption settings from DT; will permit SSE-C to propagate from client to shared service, in particular * Some downstream tests in Hive & Spark. These only seem look for DTs if the user has kerberos enabled. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556-007.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595643#comment-16595643 ] Steve Loughran commented on HADOOP-14556: - Findbugs errors all look valid; will need to fix. [~daryn]: is the reason for the s3a://user@bucket feature so that you can go from s3a://user:secret/ to a session token triple for that user? And so when you get to the far end the right secrets are picked up? as if so, I have another solution: remove support for embedding AWS credentials in URIs; HADOOP-14833 . We've been telling people this feature will go away since hadoop 2.8 & per-bucket access; S3Guard doesn't play well with it either. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.2.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch, > HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583341#comment-16583341 ] genericqa commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 9 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 8s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 7s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 28m 1s{color} | {color:red} root generated 2 new + 1458 unchanged - 0 fixed = 1460 total (was 1458) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 19s{color} | {color:orange} root: The patch generated 35 new + 32 unchanged - 0 fixed = 67 total (was 32) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 19s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 58s{color} | {color:red} hadoop-tools/hadoop-aws generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 34s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 45s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 35s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}146m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-aws | | | Found reliance on default encoding in org.apache.hadoop.fs.s3a.S3SessionToken.getCredentials(Token):in org.apache.hadoop.fs.s3a.S3SessionToken.getCredentials(Token): new String(byte[]) At S3SessionToken.java:[line 59] | | | Found reliance on default encoding in org.apache.hadoop.fs.s3a.S3SessionToken.newInstance(String, AWSSessionCredentials, String):in org.apache.hadoop.fs.s3a.S3SessionToken.newInstance(String, AWSSessionCredentials, String): String.getBytes() At S3SessionToken.java:[line 47] | | | Unwritten field:AbstractSessionCredentialsProvider.java:[line 53] | | |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583137#comment-16583137 ] Steve Loughran commented on HADOOP-14556: - Tests of TestS3AAWSCredentialsProvider fail as there's now something being forcibly injected into the chain. Cancelling the patch; I just wanted to show that it all built OK. Pulling some more aspects of the code into to my patch One aspect of the policy is that the user of a token identifier {{org.apache.hadoop.fs.s3a.S3SessionToken.Identifier}} is not defined as a kerberos user, but as the user {{UserGroupInformation.createUserForTesting(getBucket(), new String[0]);}}. That is: a remote user with no kerberos credentials, and the name of the dest bucket. . What I think it means is that when you send a DT over the wire, what comes back is not a token for the current user but for that bucket; which presumably gets identified in the token stuff differently. While I don't understand it, I'm going to go with it on the basis that I didn't really understand the code I'd had to write before, and as Daryn's understands his code and says it works, it can only be better > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581862#comment-16581862 ] genericqa commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 38s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s{color} | {color:red} hadoop-tools_hadoop-aws generated 2 new + 7 unchanged - 1 fixed = 9 total (was 8) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 19s{color} | {color:orange} hadoop-tools/hadoop-aws: The patch generated 10 new + 10 unchanged - 0 fixed = 20 total (was 10) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 6s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 56s{color} | {color:red} hadoop-tools/hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 4m 45s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 69m 55s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-aws | | | Found reliance on default encoding in org.apache.hadoop.fs.s3a.S3SessionToken.getCredentials(Token):in org.apache.hadoop.fs.s3a.S3SessionToken.getCredentials(Token): new String(byte[]) At S3SessionToken.java:[line 59] | | | Found reliance on default encoding in org.apache.hadoop.fs.s3a.S3SessionToken.newInstance(String, AWSSessionCredentials, String):in org.apache.hadoop.fs.s3a.S3SessionToken.newInstance(String, AWSSessionCredentials, String): String.getBytes() At S3SessionToken.java:[line 47] | | Failed junit tests | hadoop.fs.s3a.TestS3AAWSCredentialsProvider | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14556 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12935787/HADOOP-14556.oath-002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581834#comment-16581834 ] Steve Loughran commented on HADOOP-14556: - Attached: HADOOP-14556.oath-002.patch This is the oath patch merged into trunk. Compiles locally; not tested. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556.oath-002.patch, > HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581777#comment-16581777 ] Steve Loughran commented on HADOOP-14556: - h2. Review of the Oauth S3A delegation token design. h3. Token representation {{org.apache.hadoop.fs.s3a.S3SessionToken}} is the token. The token identifier {{S3SessionToken.Identifier}} uses a URI to represent the bucket as {code} s3a://accesskey@bucket#sessionId {code} The session secret is marshalled in the base token in its password field; h3. Credential retrieval: {{S3SessionToken.CredentialsProvider}} This class implements {{AWSSessionCredentialsProvider}} and can be directly used for authenticating AWS Calls. It gets the token for the canonical service of the FS URI passed in its constructor, and if it is the right kind, extracts the (access key, session secret, session ID) values. These are used to build the {{BasicSessionCredentials}} which is returned from to {{AWSSessionCredentialsProvider.getCredentials}}. It implements the refresh() method to get the credentials, if (somehow) that UGI token were updated, it would update the credentials. h3. Canonical service URI, {{S3AUtils.getCanonicalServiceURI}} This is used to create a URI for mapping tokens: {code} public static URI getCanonicalServiceURI(URI uri) { String sessionKey = uri.getUserInfo(); if (sessionKey != null) { sessionKey = sessionKey.split(":")[0]; } if (sessionKey == null || sessionKey.isEmpty()) { sessionKey = "default"; } return URI.create("s3://" + sessionKey + "@" + uri.getHost()); } {code} That is: if the URI comes in with a sessionKey as its user info, that is included in the canonicalisation; if not, it goes to default. so {{getCanonicalServiceURI("s3a://bucket")}} is "s3://default:bucket", while "s3a://sessionId:bucket" would map to "s3://sessionId:bucket". # Issue: why s3 over s3a? Assume they're using the s3:// prefix to ease migration from EMR s3 to ASF s3a; URLs can be shared. (Maybe we should think about easing that, or at least test that you can do it). # Issue: when would you put a sessionId in a URL? It'd allow you to have different bindings to the same bucket from the same user. This seems like a complication. h2. Job-submission time binding {{S3AFileSystem.getCanonicalServiceName()}} returns that canonical URI. {code} public String getCanonicalServiceName() { return S3AUtils.getCanonicalServiceURI(uri).toString(); } {code} {{S3AFilesystem.getDelegationToken()}} looks at the current S3 client (the {{s3}} field) and gets its credentials. These are then returned as a new {{S3SessionToken}} {code} public Token getDelegationToken(String renewer) throws IOException { Token token = null; if (s3 instanceof AWSSessionCredentialsProvider) { AWSSessionCredentials sessionCreds = ((AWSSessionCredentialsProvider)s3).getCredentials(); token = S3SessionToken.newInstance( getBucket(), sessionCreds, getCanonicalServiceName()); } return token; } {code} h2. Session-aware AWS client: {{AmazonS3ClientWithSTS}} This is used as the S3 client by the filesystem; it is the one called in {{getDelegationToken()}} to get those session tokens for marshalling. {code} AmazonS3ClientWithSTS extends AmazonS3Client implements AWSSessionCredentialsProvider { ... public AWSSessionCredentials getCredentials() { // fetch session credentials if the current credentials are not // session credentials. AWSCredentials creds = awsCredentialsProvider.getCredentials(); AWSSessionCredentials sessionCredentials; if (creds instanceof AWSSessionCredentials) { sessionCredentials = (AWSSessionCredentials)creds; } else { sessionCredentials = getSessionCredentials(lifetime); } return sessionCredentials; } ... } {code} That is: if the first credentials returned from the provider list are session credentials, they are returned for propagation. If not, the current credentials are used to create a connection to STS and request those session credentials {code} // AmazonS3ClientWithSTS private AWSSessionCredentials getSessionCredentials(int duration) { AWSSecurityTokenService stsClient = new AWSSecurityTokenServiceClient( awsCredentialsProvider, clientConfiguration); GetSessionTokenRequest tokenRequest = new GetSessionTokenRequest().withDurationSeconds(duration); Credentials stsCredentials = stsClient.getSessionToken(tokenRequest).getCredentials(); return new BasicSessionCredentials( stsCredentials.getAccessKeyId(), stsCredentials.getSecretAccessKey(), stsCredentials.getSessionToken()); } {code} Note also, {{S3AUtils.createAWSCredentialProviderSet}} always inserts an instance of {{S3SessionToken.CredentialsProvider()}} at the top of the provider list, so, *irrespective of what your providers listed in "fs.s3a.aws.credentials.provider" do*,
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580370#comment-16580370 ] Steve Loughran commented on HADOOP-14556: - OK, looking at your patch, the key thing is that the canonical name includes the session key, so that you can have >1 FS instance for a bucket & get unique keys. But now I'm confused about how references to s3a URLs are managed across apps. e.g spark-submit with dest = s3a://bucket # client: get DT for bucket containing session or assumed role secrets # YARN: marshall DTs from launch request to launched AM # Spark AM: given a destination of s3a://bucket, locate the credentials for that specific bucket. action #3 is what confuses me: if we're inserting session IDs into buckets, how to know which to look up? Or will it be that you can just invent a session id to refer to a specific FS instantiation at both ends? I think we also need to take the knife to user:secret in s3a URLs HADOOP-14833. That's fine, I've been looking forward to deleting that code for a while. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580051#comment-16580051 ] Steve Loughran commented on HADOOP-14556: - [~daryn] : thanks. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579870#comment-16579870 ] Daryn Sharp commented on HADOOP-14556: -- Sorry for delay. Attached patch we've been using for over 2 years for your consideration. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556.oath.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579129#comment-16579129 ] Steve Loughran commented on HADOOP-14556: - HADOOP-14556 patch 004; rebased onto trunk & with the new tests working (US west); not rerun the rest though. This is not yet ready to go, as it doesn't support my goal of making the DT provider pluggable; I know a lot more about assumed roles now too. But it is the start, and its current with trunk. Plan decouple DT provider from binding in S3A, with support for: * none..the obvious one :) * simple: pass in keys & encryption details. Simplest & may work with third-party stores. * session: as here, use session token * role: use assumed role and request restricted rights. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch, > HADOOP-14556-003.patch, HADOOP-14556-004.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571310#comment-16571310 ] genericqa commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 10 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 21s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 18s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 26m 9s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 26m 9s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 32s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 30s{color} | {color:red} hadoop-tools_hadoop-aws generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 19s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}128m 56s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HADOOP-14556 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12934600/HADOOP-14556-003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 49d3ce97abb5 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2e4e02b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_171 | | findbugs | v3.1.0-RC1 | | mvninstall |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16540157#comment-16540157 ] Steve Loughran commented on HADOOP-14556: - AWS extended Assume role lifespans from 1h to 12h in march. So starts to become possible to do something clever with a bucket issuing role credentials locked down to just that bucket and the matching DDB table, then we could have the FS issue that as some token. Once HADOOP-15583 is in I'll revisit this, see if I can make it a plugin point and allow for something to generate restricted assumed role secrets. So if I try to access stevel ireland, it'd create some role with R/W/D access to that, the matching S3Guard DDB table and full KMS access, marshall that along with any encryption options and keys & so let you spark-submit with the credentials passed in *only* for the buckets you are working with. This can guarantee that queries sent to a cluster can't access any of the other AWS services to which a user has access to. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347887#comment-16347887 ] Steve Loughran commented on HADOOP-14556: - [~daryn] any chance I could see your patch before I start trying to get mine to work properly? > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Priority: Major > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293834#comment-16293834 ] genericqa commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 0s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 8 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 12s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 38s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 38s{color} | {color:red} root generated 2 new + 1230 unchanged - 2 fixed = 1232 total (was 1232) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 4s{color} | {color:orange} root: The patch generated 16 new + 30 unchanged - 0 fixed = 46 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 53s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s{color} | {color:red} hadoop-tools/hadoop-aws generated 6 new + 0 unchanged - 0 fixed = 6 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 14s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 37s{color} | {color:green} hadoop-aws in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}105m 4s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-tools/hadoop-aws | | | Unused field:DelegationTokenCredentialProvider.java | | | Unused field:DelegationTokenCredentialProvider.java | | | Dead store to user in org.apache.hadoop.fs.s3a.S3ADelegationTokens.createDelegationToken() At S3ADelegationTokens.java:org.apache.hadoop.fs.s3a.S3ADelegationTokens.createDelegationToken() At S3ADelegationTokens.java:[line 101] | | | Dead store to simpleLogin in org.apache.hadoop.fs.s3a.S3ADelegationTokens.createTokenIdentifier(UserGroupInformation, Configuration, String, int) At
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293802#comment-16293802 ] Steve Loughran commented on HADOOP-14556: - Other thought. If a client is getting > 1 DT, so that it could talk to >1 bucket (mandatory, as DTs are restricted to the single FS), then you don't want to talk to the STS for new tokens for every FS, more once per credential -> assumed role. Pooling would be extra work, and would need some secureness to ensure its only per user. Simplest to avoid until there are some scale/throttling issues observed > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran > Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270747#comment-16270747 ] Steve Loughran commented on HADOOP-14556: - Oh, as this doesn't work with session tokens as the current auth mech, it'd be nice just to pick up pass on those session credentials as is, adding in any encryption secrets. That way, if I am logged in with some session credentials, I can still pass those down to submitted jobs > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran > Attachments: HADOOP-14556-001.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270744#comment-16270744 ] Steve Loughran commented on HADOOP-14556: - Daryn, got that patch for this yet? > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran > Attachments: HADOOP-14556-001.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153277#comment-16153277 ] Steve Loughran commented on HADOOP-14556: - ooh, nice work Daryn, if you've got code I'd like to see it & will think about how to test. We already have s3a tests for session credentials, so more tests won't be impossible I'll take your word for my patch being broken :) > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14556-001.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141904#comment-16141904 ] Daryn Sharp commented on HADOOP-14556: -- We've been running with s3 session tokens as delegation tokens for about a year. Would have pushed it back to community except I didn't have time to figure out how to write tests. There are a number of blockers with the currently posted patch: I do notice the semantics of getDelegationToken are completely broken. It must unconditionally fetch a token, regardless of whether the UGI contains one. The client morphs based on the current user. This violates the client requirement to always remain as the user when it was instantiated. Security flaw. This also doesn't consider that proxy users need special treatment to avoid being tricked into using the wrong credentials. Security flaw. I'll toss up my patch today or monday for your consideration. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.1 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14556-001.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115159#comment-16115159 ] Hadoop QA commented on HADOOP-14556: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 51s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 24s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 24s{color} | {color:red} root generated 2 new + 1416 unchanged - 2 fixed = 1418 total (was 1418) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 5s{color} | {color:orange} root: The patch generated 15 new + 28 unchanged - 0 fixed = 43 total (was 28) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 38s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 23s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 23s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 25s{color} | {color:red} hadoop-tools_hadoop-aws generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 23s{color} | {color:red} hadoop-aws in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 33s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 78m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HADOOP-14556 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880448/HADOOP-14556-001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 0c830ba819aa 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f44b349 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | javac |
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114871#comment-16114871 ] Steve Loughran commented on HADOOP-14556: - Tests All the new tests work, but the filecontext ones fail in equality/contains checks for paths. The port number (443) is getting into the result of FC.makeQualified(), which is then breaking assertions. {code} Tests run: 17, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 67.173 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI testListStatus(org.apache.hadoop.fs.s3a.fileContext.ITestS3AFileContextURI) Time elapsed: 4.097 sec <<< FAILURE! java.lang.AssertionError: expected: but was: at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.junit.Assert.assertEquals(Assert.java:144) at org.apache.hadoop.fs.FileContextURIBase.testListStatus(FileContextURIBase.java:534) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Running org.apache.hadoop.fs.s3a.yarn.ITestS3A Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 4.271 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADeleteManyFiles Running org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.874 sec - in org.apache.hadoop.fs.s3a.ITestS3ATemporaryCredentials Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.344 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3A Running org.apache.hadoop.fs.s3native.ITestInMemoryNativeS3FileSystemContract Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 7.643 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3ADirectoryPerformance Running org.apache.hadoop.fs.s3native.ITestJets3tNativeFileSystemStore Tests run: 52, Failures: 0, Errors: 0, Skipped: 52, Time elapsed: 0.818 sec - in org.apache.hadoop.fs.s3native.ITestInMemoryNativeS3FileSystemContract Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.525 sec - in org.apache.hadoop.fs.s3native.ITestJets3tNativeFileSystemStore Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 10.345 sec - in org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.526 sec - in org.apache.hadoop.fs.s3a.yarn.ITestS3AMiniYarnCluster Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 113.496 sec - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractDistCp Tests run: 43, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.174 sec - in org.apache.hadoop.fs.s3a.ITestS3AFileSystemContract Tests run: 62, Failures: 3, Errors: 0, Skipped: 3, Time elapsed: 169.85 sec <<< FAILURE! - in
[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16114868#comment-16114868 ] Steve Loughran commented on HADOOP-14556: - Patch 001, as a PoC rather than anything ready for line by line review This adds the option of enabling delegation tokens. If set, when logged in with full credentials, then DTs can be obtained for a limited (configurable) lifespan...these are just session credentials. The DT can be marshalled over the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}} provider will find them and use them for auth. The encryption settings are also passed in, so a DT can contain the key for These tokens cannot be renewed, nor can they be revoked. But they will always have limited life. * Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}} apart from the init logic and binding to the {{ getCanonicalServiceName() }}, and {{getDelegationToken()}} methods. As long as the tokens also work with DDB then s3guard integration should be straightforward. * bits of refactoring about how session credentials are extracted and validated in our AWS credential providers. * There's no support for propagating encryption options & key secrets; that's the obvious next step. A user should be able to provide their own key for file access, which would then be marshalled over to the workers. > S3A to support Delegation Tokens > > > Key: HADOOP-14556 > URL: https://issues.apache.org/jira/browse/HADOOP-14556 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-14556-001.patch > > > S3A to support delegation tokens where > * an authenticated client can request a token via > {{FileSystem.getDelegationToken()}} > * Amazon's token service is used to request short-lived session secret & id; > these will be saved in the token and marshalled with jobs > * A new authentication provider will look for a token for the current user > and authenticate the user if found > This will not support renewals; the lifespan of a token will be limited to > the initial duration. Also, as you can't request an STS token from a > temporary session, IAM instances won't be able to issue tokens. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org