[
https://issues.apache.org/jira/browse/HADOOP-16573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929192#comment-16929192
]
Steve Loughran commented on HADOOP-16573:
-----------------------------------------
{code}
ard"
2019-09-13 14:22:38,687 [main] INFO s3guard.S3GuardTool
(S3GuardTool.java:initMetadataStore(323)) - Metadata store
DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new,
tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new}
is initialized.
2019-09-13 14:22:38,708 [main] INFO s3guard.DynamoDBMetadataStore
(DurationInfo.java:<init>(72)) - Starting: Pruning DynamoDB Store
2019-09-13 14:22:38,766 [main] INFO s3guard.DynamoDBMetadataStore
(DurationInfo.java:close(87)) - Pruning DynamoDB Store: duration 0:00.058s
java.nio.file.AccessDeniedException: /hwdev-steve-ireland-new:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User:
arn:aws:sts::980678866538:assumed-role/stevel-s3guard/89db9060-6066-4f84-af7c-a40babaacb2e
is not authorized to perform: dynamodb:Scan on resource:
arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request
ID: 6I1ACO9K5DRGJK70M9BDPF834VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:437)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPrune(DynamoDBMetadataStore.java:1602)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:1534)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:1133)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:425)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1700)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1709)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
User:
arn:aws:sts::980678866538:assumed-role/stevel-s3guard/89db9060-6066-4f84-af7c-a40babaacb2e
is not authorized to perform: dynamodb:Scan on resource:
arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request
ID: 6I1ACO9K5DRGJK70M9BDPF834VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4279)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4246)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeScan(AmazonDynamoDBClient.java:3040)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.scan(AmazonDynamoDBClient.java:3006)
at
com.amazonaws.services.dynamodbv2.document.internal.ScanCollection.firstPage(ScanCollection.java:53)
at
com.amazonaws.services.dynamodbv2.document.internal.PageIterator.next(PageIterator.java:45)
at
com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.nextResource(IteratorSupport.java:87)
at
com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.hasNext(IteratorSupport.java:55)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPrune(DynamoDBMetadataStore.java:1552)
... 6 more
{code}
> IAM role created by S3A DT doesn't include DynamoDB scan
> --------------------------------------------------------
>
> Key: HADOOP-16573
> URL: https://issues.apache.org/jira/browse/HADOOP-16573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Priority: Minor
>
> You can't run {{s3guard prune}} with role DTs as we don't create it with
> permissons to do so.
> I think it may actually be useful to have an option where we don't restrict
> the role. This doesn't just help with debugging, it would let things like SQS
> integration pick up the creds from S3A.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
