[
https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16732350#comment-16732350
]
Steve Loughran commented on HADOOP-14556:
-----------------------------------------
Patch 027; tested s3 ireland +dynamodb. All happy.
h3. Larry's comments
{quote}If delegation tokens are enabled it seems that there is no fallback to
other providers. This seems appropriate to me but just want to make sure that
interpretation is correct.
{quote}
Its up to the DT binding to provide the fallback chain of its choice. This lets
it choose "fallback to some other property", "fail dramatically".
{quote}While there are specific implementations provided in the patch it also
seems to be extensible by 3rd parties that which to provide their own DT and
binding code.
{quote}
that's the goal.
{quote}Can you provide any details on renewal of DT's in the provided
implementations?
{quote}
There is no renewal of DTs. It doesn't make sense.
{quote}AWSCredentialProviderList + name
{quote}
it's there to provide slightly more informative errors. I didn't bother
enforcing it or being clever.
change: I patch things up in setName().
{quote}ASSUMED_ROLE_STS_ENDPOINT_REGION and ASSUMED_ROLE_STS_ENDPOINT_REGION
{quote}
this is really complex. Have a look at {{STSClientFactory.builder()}}. If you
don't set a region then it defaults to the central endpoint, but you can't
declare that as the endpoint without declaring a signing region in your
EndpointConfiguration class. Which you don't need to do with the empty string.
Really, *you do not want to begin trying to understand the logic here as there
is no logic to understand*
{quote}S3A.java: it possible for the toString() to have credentials from the
URI?
{quote}
not since we took away the ability to put credentials in URIs, no.
{quote}DurationInfo logAtInfo - should this be javadoc'd to explain how and
when it is set to debug?
{quote}
done
{quote}S3AEncryptionMethods. Can the following javadoc be better explained?
{quote}
done. They just round off the enum with all supported methods. Added tests to
verify the matching works.
{quote}S3AEncryptionMethods. should we consider making encryption algorithms
such as AES256 configurable rather than hardcoded?
{quote}
nope. These are the exact strings which AWS requires in its headers.
h3. Other Changes
* add Optional<OffsetDateTime> expiry time accessor of MarshalledCredentials;
always returns a UTC-zoned value or empty() when no expiry is known.
. Diving into the AWS SDKs, they seem to expect it to be UTC, though it's not
ever spelt out in the docs properly. This is used in tests and toString. The
wire format is still seconds in the Unix epoch with no TZ marker. It MUST be
UTC.
* Tests working in this probably work even if the person running the tests
isn't in/near UTC. I think up until now, some of the assertions about expiry
only worked when the offset between UTC and local time was close enough. that
(UTC-expiry-time - local-clock-time) was positive
* EncryptionSecrets don't have a ref to S3AUtils in, so no indirect ref to any
AWS classes. Harder to isolate tokens than you'd think. I'm almost tempted to
move the token identifiers into hadoop-common to guarantee that (a) they get
everywhere and (b) no accidental extra dependencies. But that'd probably create
other problems.
Token providers have the ability to provide a UA suffix for the client, which
gets passed all the way down to the S3 client. Why so? Makes it possible to
debug what's going wrong from S3A access logs, as the DT UUID can make it into
the logs. Once you start playing with assumed role, user IDs get lost, and of
course, that's when things start to go wrong. This propagates all the way into
the S3 Client factories, I'm afraid.
> S3A to support Delegation Tokens
> --------------------------------
>
> Key: HADOOP-14556
> URL: https://issues.apache.org/jira/browse/HADOOP-14556
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-14556-001.patch, HADOOP-14556-002.patch,
> HADOOP-14556-003.patch, HADOOP-14556-004.patch, HADOOP-14556-005.patch,
> HADOOP-14556-007.patch, HADOOP-14556-008.patch, HADOOP-14556-009.patch,
> HADOOP-14556-010.patch, HADOOP-14556-010.patch, HADOOP-14556-011.patch,
> HADOOP-14556-012.patch, HADOOP-14556-013.patch, HADOOP-14556-014.patch,
> HADOOP-14556-015.patch, HADOOP-14556-016.patch, HADOOP-14556-017.patch,
> HADOOP-14556-018a.patch, HADOOP-14556-019.patch, HADOOP-14556-020.patch,
> HADOOP-14556-021.patch, HADOOP-14556-022.patch, HADOOP-14556-023.patch,
> HADOOP-14556-024.patch, HADOOP-14556-025.patch, HADOOP-14556-026.patch,
> HADOOP-14556-027.patch, HADOOP-14556.oath-002.patch, HADOOP-14556.oath.patch
>
>
> S3A to support delegation tokens where
> * an authenticated client can request a token via
> {{FileSystem.getDelegationToken()}}
> * Amazon's token service is used to request short-lived session secret & id;
> these will be saved in the token and marshalled with jobs
> * A new authentication provider will look for a token for the current user
> and authenticate the user if found
> This will not support renewals; the lifespan of a token will be limited to
> the initial duration. Also, as you can't request an STS token from a
> temporary session, IAM instances won't be able to issue tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
