[
https://issues.apache.org/jira/browse/HADOOP-15572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16527483#comment-16527483
]
Steve Loughran commented on HADOOP-15572:
-----------------------------------------
Log of a create with an assumed role meant to be restricted to bucket & ddb
r/w, but not ddb create
{code}
bin/hadoop s3guard $ARN init $IRL
2018-06-29 12:19:59,827 [main] DEBUG s3guard.S3GuardTool
(S3GuardTool.java:run(1458)) - Executing command init
2018-06-29 12:19:59,845 [main] DEBUG s3guard.S3GuardTool
(S3GuardTool.java:initS3AFileSystem(301)) - updated bucket store option
org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore
2018-06-29 12:20:00,591 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(237)) - Initializing S3AFileSystem for
hwdev-steve-ireland-new
2018-06-29 12:20:00,593 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1001)) - Propagating entries under
fs.s3a.bucket.hwdev-steve-ireland-new.
2018-06-29 12:20:00,676 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1022)) - Updating fs.s3a.endpoint from
[core-site.xml]
2018-06-29 12:20:00,677 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1022)) - Updating
fs.s3a.committer.magic.enabled from [core-site.xml]
2018-06-29 12:20:00,677 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:propagateBucketOptions(1022)) - Updating
fs.s3a.metadatastore.impl from [S3AUtils]
2018-06-29 12:20:00,881 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(628)) - Credential provider class is
org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
2018-06-29 12:20:00,882 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(628)) - Credential provider class is
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2018-06-29 12:20:00,890 [main] DEBUG auth.AssumedRoleCredentialProvider
(AssumedRoleCredentialProvider.java:<init>(117)) -
AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3guard',
session'stevel', duration=1800}
2018-06-29 12:20:00,892 [main] DEBUG auth.AssumedRoleCredentialProvider
(AssumedRoleCredentialProvider.java:<init>(130)) - Credentials to obtain role
credentials: AWSCredentialProviderList: SimpleAWSCredentialsProvider
2018-06-29 12:20:01,402 [main] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(122)) - Using credentials from
SimpleAWSCredentialsProvider
2018-06-29 12:20:02,070 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProviderSet(572)) - For URI
s3a://hwdev-steve-ireland-new//, using credentials AWSCredentialProviderList:
AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3guard',
session'stevel', duration=1800}
2018-06-29 12:20:02,070 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.maximum is 15
2018-06-29 12:20:02,071 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.attempts.maximum is 20
2018-06-29 12:20:02,071 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.establish.timeout
is 5000
2018-06-29 12:20:02,072 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.timeout is 5000
2018-06-29 12:20:02,072 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.socket.send.buffer is 65536
2018-06-29 12:20:02,072 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.socket.recv.buffer is 32678
2018-06-29 12:20:02,074 [main] DEBUG s3a.S3AFileSystem
(DefaultS3ClientFactory.java:initUserAgent(183)) - Using User-Agent: Hadoop
3.2.0-SNAPSHOT
2018-06-29 12:20:02,128 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.paging.maximum is 5000
2018-06-29 12:20:02,130 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:longBytesOption(887)) - Value of fs.s3a.block.size is 33554432
2018-06-29 12:20:02,130 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:longBytesOption(887)) - Value of fs.s3a.readahead.range is 524288
2018-06-29 12:20:02,130 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.max.total.tasks is 5
2018-06-29 12:20:02,130 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:longOption(865)) - Value of fs.s3a.threads.keepalivetime is 60
2018-06-29 12:20:02,142 [main] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(122)) - Using credentials from
AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3guard',
session'stevel', duration=1800}
2018-06-29 12:20:02,323 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:getEncryptionAlgorithm(1253)) - Data is unencrypted
2018-06-29 12:20:02,323 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:getEncryptionAlgorithm(1256)) - Using SSE-C with empty key
2018-06-29 12:20:02,324 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(313)) - Input fadvise policy = normal
2018-06-29 12:20:02,324 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(317)) - Filesystem support for magic committers
is enabled
2018-06-29 12:20:02,328 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.fast.upload.active.blocks is 4
2018-06-29 12:20:02,328 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:initialize(333)) - Using S3ABlockOutputStream with buffer =
disk; block=8388608; queue limit=4
2018-06-29 12:20:02,330 [main] DEBUG s3guard.S3Guard
(S3Guard.java:getMetadataStoreClass(125)) - Metastore option source
fs.s3a.bucket.hwdev-steve-ireland-new.metadatastore.impl via [S3AUtils]
2018-06-29 12:20:02,331 [main] DEBUG s3guard.S3Guard
(S3Guard.java:getMetadataStore(97)) - Using NullMetadataStore metadata store
for s3a filesystem
2018-06-29 12:20:02,331 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:longOption(865)) - Value of fs.s3a.multipart.purge.age is 3600000
2018-06-29 12:20:02,411 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:initialize(277)) - Inferring DynamoDB region from
S3 bucket: eu-west-1
2018-06-29 12:20:02,411 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:createDynamoDB(255)) - Creating DynamoDB client
class
org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory
with S3 region eu-west-1
2018-06-29 12:20:02,412 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(628)) - Credential provider class is
org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider
2018-06-29 12:20:02,412 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProvider(628)) - Credential provider class is
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
2018-06-29 12:20:02,412 [main] DEBUG auth.AssumedRoleCredentialProvider
(AssumedRoleCredentialProvider.java:<init>(117)) -
AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3guard',
session'stevel', duration=1800}
2018-06-29 12:20:02,412 [main] DEBUG auth.AssumedRoleCredentialProvider
(AssumedRoleCredentialProvider.java:<init>(130)) - Credentials to obtain role
credentials: AWSCredentialProviderList: SimpleAWSCredentialsProvider
2018-06-29 12:20:02,415 [main] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(122)) - Using credentials from
SimpleAWSCredentialsProvider
2018-06-29 12:20:02,836 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:createAWSCredentialProviderSet(572)) - For URI (null URI), using
credentials AWSCredentialProviderList:
AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3guard',
session'stevel', duration=1800}
2018-06-29 12:20:02,836 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.maximum is 15
2018-06-29 12:20:02,836 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.attempts.maximum is 20
2018-06-29 12:20:02,837 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.establish.timeout
is 5000
2018-06-29 12:20:02,837 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.connection.timeout is 5000
2018-06-29 12:20:02,837 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.socket.send.buffer is 65536
2018-06-29 12:20:02,838 [main] DEBUG s3a.S3AUtils
(S3AUtils.java:intOption(844)) - Value of fs.s3a.socket.recv.buffer is 32678
2018-06-29 12:20:02,838 [main] DEBUG s3a.S3AFileSystem
(DefaultS3ClientFactory.java:initUserAgent(183)) - Using User-Agent: Hadoop
3.2.0-SNAPSHOT
2018-06-29 12:20:02,839 [main] DEBUG s3guard.DynamoDBClientFactory
(DynamoDBClientFactory.java:createDynamoDBClient(84)) - Creating DynamoDB
client in region eu-west-1
2018-06-29 12:20:02,932 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:initTable(893)) - Binding to table
hwdev-steve-ireland-new
2018-06-29 12:20:02,979 [main] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(122)) - Using credentials from
AssumedRoleCredentialProvider{role='arn:aws:iam::980678866538:role/stevel-s3guard',
session'stevel', duration=1800}
2018-06-29 12:20:03,198 [main] INFO s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:createTable(1035)) - Creating non-existent DynamoDB
table hwdev-steve-ireland-new in region eu-west-1
2018-06-29 12:20:03,273 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:createTable(1042)) - Awaiting table becoming active
2018-06-29 12:20:13,490 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:putItem(1062)) - Putting item { Item:
{parent=../VERSION, child=../VERSION, table_version=100,
table_created=1530271213483} }
2018-06-29 12:20:13,552 [main] INFO s3guard.S3GuardTool
(S3GuardTool.java:initMetadataStore(270)) - Metadata store
DynamoDBMetadataStore{region=eu-west-1, tableName=hwdev-steve-ireland-new} is
initialized.
Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new
description=S3Guard metadata store in DynamoDB
name=hwdev-steve-ireland-new
persist.authoritative.bit=false
read-capacity=500
region=eu-west-1
retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100
MILLISECONDS)
size=0
status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
{AttributeName: parent,AttributeType: S}],TableName:
hwdev-steve-ireland-new,KeySchema: [{AttributeName: parent,KeyType: HASH},
{AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime:
Fri Jun 29 12:20:03 BST 2018,ProvisionedThroughput: {NumberOfDecreasesToday:
0,ReadCapacityUnits: 500,WriteCapacityUnits: 100},TableSizeBytes: 0,ItemCount:
0,TableArn:
arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new,TableId:
99dfc220-a142-4816-9b3b-bfed91de2d30,}
write-capacity=100
2018-06-29 12:20:13,598 [pool-2-thread-1] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:close(2465)) - Filesystem s3a://hwdev-steve-ireland-new is
closed
{code}
> Test S3Guard ops with assumed roles & verify required permissions
> -----------------------------------------------------------------
>
> Key: HADOOP-15572
> URL: https://issues.apache.org/jira/browse/HADOOP-15572
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0
> Reporter: Steve Loughran
> Priority: Major
>
> We haven't documented permissions for S3Guard (WiP of mine); when I try to
> test using the AssumedRoleCredentialProvider & a role nominally restricted to
> R/W of S3guard *but not create/delete*, I can still create and destroy buckets
> Either I've got my list wrong, or how S3Guard sets up its auth isn't right &
> somehow falling back to the full role
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
