[jira] [Commented] (HADOOP-16573) IAM role created by S3A DT doesn't include DynamoDB scan
[
https://issues.apache.org/jira/browse/HADOOP-16573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943569#comment-16943569
]
Steve Loughran commented on HADOOP-16573:
-
or just ask for tags and scan permissions. Tag so that on dynamic setting of
tag from version works. Not very important as we should have tagged the table
on the client already
> IAM role created by S3A DT doesn't include DynamoDB scan
>
>
> Key: HADOOP-16573
> URL: https://issues.apache.org/jira/browse/HADOOP-16573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> You can't run {{s3guard prune}} with role DTs as we don't create it with
> permissons to do so.
> I think it may actually be useful to have an option where we don't restrict
> the role. This doesn't just help with debugging, it would let things like SQS
> integration pick up the creds from S3A.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16573) IAM role created by S3A DT doesn't include DynamoDB scan
[
https://issues.apache.org/jira/browse/HADOOP-16573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929220#comment-16929220
]
Steve Loughran commented on HADOOP-16573:
-
we're not actually asking for all admin permissions in the role.
I'm going to propose we do ask for all permissons, and rely on role-level
restrictions instead, which is the way to be 100% confident that everyone in
that role lacks the rights
> IAM role created by S3A DT doesn't include DynamoDB scan
>
>
> Key: HADOOP-16573
> URL: https://issues.apache.org/jira/browse/HADOOP-16573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> You can't run {{s3guard prune}} with role DTs as we don't create it with
> permissons to do so.
> I think it may actually be useful to have an option where we don't restrict
> the role. This doesn't just help with debugging, it would let things like SQS
> integration pick up the creds from S3A.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16573) IAM role created by S3A DT doesn't include DynamoDB scan
[
https://issues.apache.org/jira/browse/HADOOP-16573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929192#comment-16929192
]
Steve Loughran commented on HADOOP-16573:
-
{code}
ard"
2019-09-13 14:22:38,687 [main] INFO s3guard.S3GuardTool
(S3GuardTool.java:initMetadataStore(323)) - Metadata store
DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new,
tableArn=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new}
is initialized.
2019-09-13 14:22:38,708 [main] INFO s3guard.DynamoDBMetadataStore
(DurationInfo.java:(72)) - Starting: Pruning DynamoDB Store
2019-09-13 14:22:38,766 [main] INFO s3guard.DynamoDBMetadataStore
(DurationInfo.java:close(87)) - Pruning DynamoDB Store: duration 0:00.058s
java.nio.file.AccessDeniedException: /hwdev-steve-ireland-new:
com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: User:
arn:aws:sts::980678866538:assumed-role/stevel-s3guard/89db9060-6066-4f84-af7c-a40babaacb2e
is not authorized to perform: dynamodb:Scan on resource:
arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request
ID: 6I1ACO9K5DRGJK70M9BDPF834VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateDynamoDBException(S3AUtils.java:437)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPrune(DynamoDBMetadataStore.java:1602)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.prune(DynamoDBMetadataStore.java:1534)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool$Prune.run(S3GuardTool.java:1133)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:425)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.run(S3GuardTool.java:1700)
at
org.apache.hadoop.fs.s3a.s3guard.S3GuardTool.main(S3GuardTool.java:1709)
Caused by: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
User:
arn:aws:sts::980678866538:assumed-role/stevel-s3guard/89db9060-6066-4f84-af7c-a40babaacb2e
is not authorized to perform: dynamodb:Scan on resource:
arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new (Service:
AmazonDynamoDBv2; Status Code: 400; Error Code: AccessDeniedException; Request
ID: 6I1ACO9K5DRGJK70M9BDPF834VVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:4279)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:4246)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.executeScan(AmazonDynamoDBClient.java:3040)
at
com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.scan(AmazonDynamoDBClient.java:3006)
at
com.amazonaws.services.dynamodbv2.document.internal.ScanCollection.firstPage(ScanCollection.java:53)
at
com.amazonaws.services.dynamodbv2.document.internal.PageIterator.next(PageIterator.java:45)
at
com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.nextResource(IteratorSupport.java:87)
at
com.amazonaws.services.dynamodbv2.document.internal.IteratorSupport.hasNext(IteratorSupport.java:55)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.innerPrune(DynamoDBMetadataStore.java:1552)
... 6 more
{code}
> IAM role created by S3A DT doesn't include DynamoDB scan
>
>
> Key: HADOOP-16573
> URL: https://issues.apache.org/jira/browse/HADOOP-16573
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>Priority: Minor
>
> You can't run {{s3guard pr
