[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737215#comment-16737215
]
Gabor Bota edited comment on HADOOP-14556 at 1/8/19 5:25 PM:
-
Thanks for working on this [~ste...@apache.org]!
Tested the newest patch against eu-west-1 with {{mvn verify -Dparallel-tests
-DtestsThreadCount=8 -Ds3guard -Ddynamo -Dauth}} (I usually run tests with
these params).
I had the following error:
{noformat}
[ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 20.79 s
<<< FAILURE! - in
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[ERROR]
testCreateAndUseDT(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens)
Time elapsed: 3.484 s <<< ERROR!
java.lang.IllegalStateException: No role ARN defined in fs.s3a.assumed.role.arn
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:134)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:50)
at
org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:140)
at
org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:422)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testCreateAndUseDT(ITestSessionDelegationTokens.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR]
testSaveLoadTokens(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens)
Time elapsed: 2.145 s <<< ERROR!
java.lang.IllegalStateException: No role ARN defined in fs.s3a.assumed.role.arn
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:134)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:50)
at
org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:140)
at
org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:422)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testSaveLoadTokens(ITestSessionDelegationTokens.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737215#comment-16737215
]
Gabor Bota edited comment on HADOOP-14556 at 1/8/19 3:31 PM:
-
Thanks for working on this [~ste...@apache.org]!
Tested the newest patch against eu-west-1 with {{mvn verify -Dparallel-tests
-DtestsThreadCount=8 -Ds3guard -Ddynamo -Dauth}} (I usually run tests with
these params).
I had the following error:
{noformat}
[ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 20.79 s
<<< FAILURE! - in
org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
[ERROR]
testCreateAndUseDT(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens)
Time elapsed: 3.484 s <<< ERROR!
java.lang.IllegalStateException: No role ARN defined in fs.s3a.assumed.role.arn
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:134)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:50)
at
org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:140)
at
org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:422)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testCreateAndUseDT(ITestSessionDelegationTokens.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[ERROR]
testSaveLoadTokens(org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens)
Time elapsed: 2.145 s <<< ERROR!
java.lang.IllegalStateException: No role ARN defined in fs.s3a.assumed.role.arn
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:134)
at
org.apache.hadoop.fs.s3a.auth.delegation.RoleTokenBinding.createTokenIdentifier(RoleTokenBinding.java:50)
at
org.apache.hadoop.fs.s3a.auth.delegation.AbstractDelegationTokenBinding.createDelegationToken(AbstractDelegationTokenBinding.java:140)
at
org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens.createDelegationToken(S3ADelegationTokens.java:422)
at
org.apache.hadoop.fs.s3a.auth.delegation.ITestSessionDelegationTokens.testSaveLoadTokens(ITestSessionDelegationTokens.java:121)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718032#comment-16718032
]
Steve Loughran edited comment on HADOOP-14556 at 12/11/18 9:33 PM:
---
For people wanting to play with this stuff, note that
[cloudstore|https://github.com/steveloughran/cloudstore/releases/tag/release_2018_12_11]
now does everything you can imagine with tokens:
was (Author: ste...@apache.org):
For people wanting to play with this stuff, note that
[cloudstore}https://github.com/steveloughran/cloudstore/releases/tag/release_2018_12_11]
now does everything you can imagine with tokens:
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch,
> HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650850#comment-16650850
]
Steve Loughran edited comment on HADOOP-14556 at 10/15/18 9:51 PM:
---
Note that the partially {{ITestDelegatedMRJob}} test does show that S3A tokens
are picked up for MR job submit; tested for full, session and role tokens.
One fun detail: if your fs.s3a.secret.key &c attributes are set in the job conf
you launch with, they end up at the far end, even though you are using DTs.
Why? well, because they are config options, aren't they?
To get the lockdown to work, you need to be serving up the secrets inside a
hadoop credential provider file such as localjceks file. That way, the job
conf will not contain the secrets.
There's no obvious way to patch the options, so that's going to have to go down
as what to do. Setting the AWS env vars would also work, though as spark
automatically picks up those values and patches the fs config (without any
check for the properties first), they may get in.
was (Author: ste...@apache.org):
Note that the partially {{ITestDelegatedMRJob}} test does show that S3A tokens
are picked up for MR job submit; tested for full, session and role tokens.
One fun detail: if your fs.s3a.secret.key &c attributes are set in the job conf
you launch with, they end up at the far end, even though you are using DTs.
Why? well, because they are config options, aren't they?
To get the lockdown to work, you need to be serving up the secrets inside a
hadoop credential provider file such as localjceks file. That way, the job
conf will not contain the secrets.
There's no
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch,
> HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16650831#comment-16650831
]
Steve Loughran edited comment on HADOOP-14556 at 10/15/18 9:45 PM:
---
HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn
cluster to verify that MR job submission collects DTs for source and
destination paths.
To do this the MockJob class had to go into
hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java and
job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but
that only works if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't)
improved/extended in assumed_roles.md as well as delegation_tokens.md.
All AWS error messages related to STS/session and role requests are now in
assumed_roles.md to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward
any session creds it gets from its own auth chain, rather than ask for new ones
(which it can't do with session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the
AssumeRole and delegation tests which were clearing or overwriting the
fs.s3a.{auth, secret, session} options, as those in the creds file were still
being picked up. Fix: explicitly reset hadoop.security.credential.provider.path
for all the tests which were now failing.
* minor checkstyle fixup
tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of
{{ITestS3GuardToolLocal\#testDestroyNoBucket}} *even when I was running with
dynamodb*. I think that test suite is running when it shouldn't. More research
needed there
was (Author: ste...@apache.org):
HADOOP-14556 patch 013
* ITestDelegatedMRJob mixes a mock job submission API with a real miniYarn
cluster to verify that MR job submission collects DTs for source and
destination paths.
To do this the MockJob class had to go into
hadoop-aws/src/test/java/org/apache/hadoop/mapreduce/MockJob.java and
job.connect() made an override point (so it can be skipped)
* default assumed role duration returned to 1h; it had been extended to 6h but
that only works if your role has been explicitly extended to > 1h duration.
* and docs on increasing it (plus error messages you get if you don't)
improved/extended in assumed_roles.md as well as delegation_tokens.md.
All AWS error messages related to STS/session and role requests are now in
assumed_roles.md to avoid duplication & inconsistencies.
* ITestS3ADelegationTokenSupport tests that the Session DT binding will forward
any session creds it gets from its own auth chain, rather than ask for new ones
(which it can't do with session creds)
* Also: I'm using a Hadoop cred provider for storing secrets; this broke the
AssumeRole and delegation tests which were clearing or overwriting the
fs.s3a.{auth, secret, session} options, as those in the creds file were still
being picked up. Fix: explicitly reset hadoop.security.credential.provider.path
for all the tests which were now failing.
* minor checkstyle fixup
tested, S3A ireland. Apart from the cred problem (fixed), I got a failure of
{{ITestS3GuardToolLocal\#testDestroyNoBucket }} *even when I was running with
dynamodb*. I think that test suite is running when it shouldn't. More research
needed there
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556.oath-002.patch,
> HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114868#comment-16114868
]
Steve Loughran edited comment on HADOOP-14556 at 8/4/17 7:32 PM:
-
Patch 001, as a PoC rather than anything ready for line by line review
This adds the option of enabling delegation tokens. If set, when logged in with
full credentials, then DTs can be obtained for a limited (configurable)
lifespan...these are just session credentials. The DT can be marshalled over
the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}}
provider will find them and use them for auth. The encryption settings are also
passed in, so a DT can contain the key for
These tokens cannot be renewed, nor can they be revoked. But they will always
have limited life.
* Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}}
apart from the init logic and binding to the {{getCanonicalServiceName()}} and
{{getDelegationToken()}} methods. As long as the tokens also work with DDB then
s3guard integration should be straightforward.
* bits of refactoring about how session credentials are extracted and validated
in our AWS credential providers.
* There's no support for propagating encryption options & key secrets; that's
the obvious next step. A user should be able to provide their own key for file
access, which would then be marshalled over to the workers.
+ does some reinstatement of the URI, HADOOP-14723, but that should really be
isolated
was (Author: ste...@apache.org):
Patch 001, as a PoC rather than anything ready for line by line review
This adds the option of enabling delegation tokens. If set, when logged in with
full credentials, then DTs can be obtained for a limited (configurable)
lifespan...these are just session credentials. The DT can be marshalled over
the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}}
provider will find them and use them for auth. The encryption settings are also
passed in, so a DT can contain the key for
These tokens cannot be renewed, nor can they be revoked. But they will always
have limited life.
* Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}}
apart from the init logic and binding to the {{getCanonicalServiceName()}},
and {{getDelegationToken()}} methods. As long as the tokens also work with DDB
then s3guard integration should be straightforward.
* bits of refactoring about how session credentials are extracted and validated
in our AWS credential providers.
* There's no support for propagating encryption options & key secrets; that's
the obvious next step. A user should be able to provide their own key for file
access, which would then be marshalled over to the workers.
+ does some reinstatement of the URI, HADOOP-14723, but that should really be
isolated
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14556-001.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114868#comment-16114868
]
Steve Loughran edited comment on HADOOP-14556 at 8/4/17 7:31 PM:
-
Patch 001, as a PoC rather than anything ready for line by line review
This adds the option of enabling delegation tokens. If set, when logged in with
full credentials, then DTs can be obtained for a limited (configurable)
lifespan...these are just session credentials. The DT can be marshalled over
the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}}
provider will find them and use them for auth. The encryption settings are also
passed in, so a DT can contain the key for
These tokens cannot be renewed, nor can they be revoked. But they will always
have limited life.
* Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}}
apart from the init logic and binding to the {{getCanonicalServiceName()}},
and {{getDelegationToken()}} methods. As long as the tokens also work with DDB
then s3guard integration should be straightforward.
* bits of refactoring about how session credentials are extracted and validated
in our AWS credential providers.
* There's no support for propagating encryption options & key secrets; that's
the obvious next step. A user should be able to provide their own key for file
access, which would then be marshalled over to the workers.
+ does some reinstatement of the URI, HADOOP-14723, but that should really be
isolated
was (Author: ste...@apache.org):
Patch 001, as a PoC rather than anything ready for line by line review
This adds the option of enabling delegation tokens. If set, when logged in with
full credentials, then DTs can be obtained for a limited (configurable)
lifespan...these are just session credentials. The DT can be marshalled over
the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}}
provider will find them and use them for auth. The encryption settings are also
passed in, so a DT can contain the key for
These tokens cannot be renewed, nor can they be revoked. But they will always
have limited life.
* Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}}
apart from the init logic and binding to the {{ getCanonicalServiceName() }},
and {{getDelegationToken()}} methods. As long as the tokens also work with DDB
then s3guard integration should be straightforward.
* bits of refactoring about how session credentials are extracted and validated
in our AWS credential providers.
* There's no support for propagating encryption options & key secrets; that's
the obvious next step. A user should be able to provide their own key for file
access, which would then be marshalled over to the workers.
+ does some reinstatement of the URI, HADOOP-14723, but that should really be
isolated
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14556-001.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14556) S3A to support Delegation Tokens
[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16114868#comment-16114868
]
Steve Loughran edited comment on HADOOP-14556 at 8/4/17 7:28 PM:
-
Patch 001, as a PoC rather than anything ready for line by line review
This adds the option of enabling delegation tokens. If set, when logged in with
full credentials, then DTs can be obtained for a limited (configurable)
lifespan...these are just session credentials. The DT can be marshalled over
the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}}
provider will find them and use them for auth. The encryption settings are also
passed in, so a DT can contain the key for
These tokens cannot be renewed, nor can they be revoked. But they will always
have limited life.
* Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}}
apart from the init logic and binding to the {{ getCanonicalServiceName() }},
and {{getDelegationToken()}} methods. As long as the tokens also work with DDB
then s3guard integration should be straightforward.
* bits of refactoring about how session credentials are extracted and validated
in our AWS credential providers.
* There's no support for propagating encryption options & key secrets; that's
the obvious next step. A user should be able to provide their own key for file
access, which would then be marshalled over to the workers.
+ does some reinstatement of the URI, HADOOP-14723, but that should really be
isolated
was (Author: ste...@apache.org):
Patch 001, as a PoC rather than anything ready for line by line review
This adds the option of enabling delegation tokens. If set, when logged in with
full credentials, then DTs can be obtained for a limited (configurable)
lifespan...these are just session credentials. The DT can be marshalled over
the wire then unmarshalled, where the new {{DelegationTokenCredentialProvider}}
provider will find them and use them for auth. The encryption settings are also
passed in, so a DT can contain the key for
These tokens cannot be renewed, nor can they be revoked. But they will always
have limited life.
* Minimal changes to S3AFS, as the new work is all in {{S3ADelegationTokens}}
apart from the init logic and binding to the {{ getCanonicalServiceName() }},
and {{getDelegationToken()}} methods. As long as the tokens also work with DDB
then s3guard integration should be straightforward.
* bits of refactoring about how session credentials are extracted and validated
in our AWS credential providers.
* There's no support for propagating encryption options & key secrets; that's
the obvious next step. A user should be able to provide their own key for file
access, which would then be marshalled over to the workers.
> S3A to support Delegation Tokens
>
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-14556-001.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
