On Wed, 6 May 2020 at 23:32, Eric Yang <eric...@gmail.com> wrote: > Hi all, > > > 4. Passing different form of tokens does not work well with cloud provider > security mechanism. For example, passing AWS sts token for S3 bucket. > There is no renewal mechanism, nor good way to identify when the token > would expire. > > well, HADOOP-14556 does it fairly well, supporting session and role tokens. We even know when they expire because we ask for a duration when we request the session/role creds. See org.apache.hadoop.fs.s3a.auth.delegation.AbstractS3ATokenIdentifier for the core of what we marshall, including encryption secrets.
The main issue there is that Yarn can't refresh those tokens because a new triple of session credentials are required; currently token renewal assumes the token is unchanged and a request is made to the service to update their table of issued tokens. But even if the RM could get back a new token from a refresh call, we are left with the problem of "how to get an updated set of creds to each process"
