[
https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929243#comment-16929243
]
Steve Loughran commented on HADOOP-16547:
-----------------------------------------
with the test plan, I can verify that prune, set-capacity, destroy and init
fail to work with Delegation Token auth. with the patch, all of these *except
init* work. Init is special as because its initing the bucket, it doesn't want
a filesystem. I'm not worrying about this, as its a major admin command which
you wouldn't normally use through DTs.
Old code
{code}
~/P/R/fsck bin/hadoop s3guard prune -seconds 0 -tombstone
s3a://hwdev-steve-ireland-new/
java.nio.file.AccessDeniedException: hwdev-steve-ireland-new:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:200)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:1840)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:521)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.initMetadataStore(S3GuardTool.java:318)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:1072)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:402)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1767)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1776)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS
Credentials provided by TemporaryAWSCredentialsProvider
SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider
IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to
load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or
AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:216)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1225)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:801)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:751)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4279)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4246)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeDescribeTable(AmazonDynamoDBClient.java:1905)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.describeTable(AmazonDynamoDBClient.java:1871)
at
com.amazonaws.services.dynamodbv2.document.Table.describe(Table.java:137)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:1775)
... 7 more
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials
from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and
AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
at
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:177)
... 22 more
2019-09-13 14:19:23,373 [main] INFO util.ExitUtil
(ExitUtil.java:terminate(210)) - Exiting with status -1:
java.nio.file.AccessDeniedException: hwdev-steve-ireland-new:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider :
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
~/P/R/fsck
{code}
new code, with DTs set up
{code}
bin/hadoop s3guard prune -seconds 0 -tombstone s3a://hwdev-steve-ireland-new/
2019-09-13 14:32:05,178 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:login(260)) - hadoop login
2019-09-13 14:32:05,181 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(193)) - hadoop login commit
2019-09-13 14:32:05,185 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(221)) - using local user:UnixPrincipal: stevel
2019-09-13 14:32:05,185 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(227)) - Using user: "UnixPrincipal: stevel"
with name stevel
2019-09-13 14:32:05,185 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:commit(241)) - User entry: "stevel"
2019-09-13 14:32:05,187 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:createLoginUser(768)) - Reading credentials from
location /Users/stevel/Projects/Releases/secrets.bin
2019-09-13 14:32:05,237 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:createLoginUser(773)) - Loaded 1 tokens from
/Users/stevel/Projects/Releases/secrets.bin
2019-09-13 14:32:05,237 [main] DEBUG security.UserGroupInformation
(UserGroupInformation.java:createLoginUser(815)) - UGI loginUser:stevel
(auth:SIMPLE)
2019-09-13 14:32:05,589 [main] DEBUG delegation.S3ADelegationTokens
(S3ADelegationTokens.java:serviceInit(185)) - Filesystem
s3a://hwdev-steve-ireland-new is using delegation tokens of kind
S3ADelegationToken/Session
2019-09-13 14:32:05,755 [main] DEBUG delegation.S3ADelegationTokens
(S3ADelegationTokens.java:lookupToken(606)) - Looking for token for service
s3a://hwdev-steve-ireland-new in credentials
2019-09-13 14:32:05,758 [main] DEBUG delegation.S3ADelegationTokens
(S3ADelegationTokens.java:lookupToken(610)) - Found token of kind
S3ADelegationToken/Session
2019-09-13 14:32:05,829 [main] INFO delegation.S3ADelegationTokens
(S3ADelegationTokens.java:bindToDelegationToken(327)) - Using delegation token
S3ATokenIdentifier{S3ADelegationToken/Session;
uri=s3a://hwdev-steve-ireland-new; timestamp=1568381494900; encryption=(no
encryption); 33e1aa81-d84c-4073-b934-21249b1e2a8f; Created on
HW13176-2.local/192.168.1.139 at time 2019-09-13T13:31:34.793Z.}; session
credentials, expiry 2019-09-14T01:31:34Z; (valid)
2019-09-13 14:32:05,830 [main] INFO delegation.S3ADelegationTokens
(DurationInfo.java:<init>(72)) - Starting: Creating Delegation Token
2019-09-13 14:32:05,831 [main] INFO delegation.S3ADelegationTokens
(DurationInfo.java:close(87)) - Creating Delegation Token: duration 0:00.001s
2019-09-13 14:32:05,832 [main] DEBUG delegation.S3ADelegationTokens
(S3ADelegationTokens.java:serviceStart(200)) - S3A Delegation support token
S3ATokenIdentifier{S3ADelegationToken/Session;
uri=s3a://hwdev-steve-ireland-new; timestamp=1568381494900; encryption=(no
encryption); 33e1aa81-d84c-4073-b934-21249b1e2a8f; Created on
HW13176-2.local/192.168.1.139 at time 2019-09-13T13:31:34.793Z.}; session
credentials, expiry 2019-09-14T01:31:34Z; (valid) with Session token binding
for user stevel, with STS endpoint "sts.eu-west-2.amazonaws.com", region
"eu-west-2" and token duration 720:00
2019-09-13 14:32:07,511 [main] INFO s3guard.S3GuardTool
(S3GuardTool.java:initMetadataStore(323)) - Metadata store
DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new,
tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new}
is initialized.
2019-09-13 14:32:07,531 [main] INFO s3guard.DynamoDBMetadataStore
(DurationInfo.java:<init>(72)) - Starting: Pruning DynamoDB Store
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel/target/test/data/MJ40cykvme
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel/target/test/data/iN22YWbgT1
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel/target/test/data/lRU1a4xwTj
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel/target/test/data
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel/target/test
2019-09-13 14:32:07,616 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user/stevel/target
2019-09-13 14:32:07,617 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/test
2019-09-13 14:32:07,617 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/test_400_rm_root_recursive-01
2019-09-13 14:32:07,617 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/test_400_rm_root_recursive-02
2019-09-13 14:32:07,617 [main] DEBUG s3guard.Operations
(DynamoDBMetadataStore.java:logDelete(2466)) - #(Prune-0001) DELETE
s3a:///hwdev-steve-ireland-new/user
2019-09-13 14:32:07,737 [main] INFO s3guard.DynamoDBMetadataStore
(DurationInfo.java:close(87)) - Pruning DynamoDB Store: duration 0:00.206s
2019-09-13 14:32:07,737 [main] INFO s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:innerPrune(1605)) - Finished pruning 11 items in
batches of 25
2019-09-13 14:32:07,744 [shutdown-hook-0] DEBUG delegation.S3ADelegationTokens
(S3ADelegationTokens.java:serviceStop(221)) - Stopping delegation tokens
~/P/R/prune
{code}
> s3guard prune command doesn't get AWS auth chain from FS
> --------------------------------------------------------
>
> Key: HADOOP-16547
> URL: https://issues.apache.org/jira/browse/HADOOP-16547
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> s3guard prune command doesn't get AWS auth chain from any FS, so it just
> drives the DDB store from the conf settings. If S3A is set up to use
> Delegation tokens then the DTs/custom AWS auth sequence is not picked up, so
> you get an auth failure.
> Fix:
> # instantiate the FS before calling initMetadataStore
> # review other commands to make sure problem isn't replicated
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
