[ https://issues.apache.org/jira/browse/HADOOP-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16456625#comment-16456625 ]
Steve Loughran edited comment on HADOOP-15422 at 4/27/18 3:41 PM: ------------------------------------------------------------------ The first stack was a spurious failure, but with the corrected credentials, an ls of the root directory fails saying the child path must be a child of the path-with-secrets-inline {code} Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ -ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of s3a://A..A:X..X@hwdev-steve-2/ {code} . Listing of child paths which exist/don't exist pass Other commands (get, diff) work and don't include secrets in their listing This is probably related to our work to strip out secrets from URIs. We don't want them in the database, or in logs {code} > hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/" ... 18/04/27 15:21:49 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2 18/04/27 15:21:49 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,} 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false 18/04/27 15:21:49 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/ () 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: List status for path: s3a://A..A:X..X@hwdev-steve-2/ 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_list_status += 1 -> 1 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/ () 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ -ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of s3a://A..A:X..X@hwdev-steve-2/ .. Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>] [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] ? echo $? 255 {code} Works for a child entry which exists {code} [root@ctr-e138-1518143905142-264443-01-000006 ~]# hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/cloud-integration" 18/04/27 15:24:29 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for hwdev-steve-2 18/04/27 15:24:29 DEBUG s3a.S3AUtils: Propagating entries under fs.s3a.bucket.hwdev-steve-2. 18/04/27 15:24:29 WARN s3native.S3xLoginHelper: The Filesystem URI contains login details. This is insecure and may be unsupported in future. 18/04/27 15:24:29 DEBUG s3a.S3AUtils: For URI s3a://hwdev-steve-2//cloud-integration, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3 ... 18/04/27 15:24:30 DEBUG s3guard.S3Guard: Using DynamoDBMetadataStore metadata store for s3a filesystem 18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Inferring DynamoDB region from S3 bucket: us-west-1 18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Creating DynamoDB client class org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory with S3 region us-west-1 18/04/27 15:24:30 DEBUG s3a.S3AUtils: For URI (null URI), using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.maximum is 15 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.attempts.maximum is 20 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.establish.timeout is 5000 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.timeout is 200000 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.socket.send.buffer is 8192 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.socket.recv.buffer is 8192 18/04/27 15:24:30 DEBUG s3a.S3AFileSystem: Using User-Agent: User-Agent: APN/1.0 Hortonworks/1.0 HDP/3.0.0.0-1252, Hadoop 3.0.0.3.0.0.0-1252 18/04/27 15:24:30 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1 18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2 18/04/27 15:24:30 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,} 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false 18/04/27 15:24:31 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/cloud-integration (cloud-integration) 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://A..A:X..X@hwdev-steve-2/cloud-integration: PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-2/cloud-integration; isDirectory=true; modification_time=1524842671173; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false} 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: List status for path: s3a://hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_list_status += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Getting path status for s3a://hwdev-steve-2/cloud-integration (cloud-integration) 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://hwdev-steve-2/cloud-integration: PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-2/cloud-integration; isDirectory=true; modification_time=1524842671185; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false} 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Listing table hwdev-steve-2 in region us-west-1: s3a://hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory cloud-integration/ 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Adding directory: S3AFileStatus{path=s3a://hwdev-steve-2/cloud-integration/DELAY_LISTING_ME; isDirectory=true; modification_time=0; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} isEmptyDirectory=FALSE 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Added 1 entries; ignored 0; hasNext=true; hasMoreObjects=false 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Start iterating the provided status. Found 1 items drwxrwxrwx - root root 0 2018-04-27 15:24 s3a://hwdev-steve-2/cloud-integration/DELAY_LISTING_ME 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-2 is closed 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Shutting down DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2} {code} And for one which doesn;t {code} # hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/missing" 18/04/27 15:25:21 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for hwdev-steve-2 ... 18/04/27 15:25:23 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2 18/04/27 15:25:23 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,} 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false 18/04/27 15:25:23 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/missing (missing) 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/missing 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://A..A:X..X@hwdev-steve-2/missing: null 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1 -> 2 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Not Found: s3a://A..A:X..X@hwdev-steve-2/missing ls: `s3a://A..A:X..X@hwdev-steve-2/missing': No such file or directory 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-2 is closed 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Shutting down DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2} {code} was (Author: ste...@apache.org): The first stack was a spurious failure, but with the corrected credentials, an ls of the root directory fails saying the child path must be a child of the path-with-secrets-inline {code} Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ -ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of s3a://A..A:X..X@hwdev-steve-2/ {code} . Listing of child paths which exist/don't exist pass Other commands (get, diff) work and don't include secrets in their listing This is probably related to our work to strip out secrets from URIs. We don't want them in the database, or in logs {code} > hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/" ... {code} 18/04/27 15:21:49 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2 18/04/27 15:21:49 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,} 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false 18/04/27 15:21:49 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/ () 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: List status for path: s3a://A..A:X..X@hwdev-steve-2/ 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_list_status += 1 -> 1 18/04/27 15:21:49 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2 18/04/27 15:21:49 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/ () 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ 18/04/27 15:21:49 DEBUG s3guard.DynamoDBMetadataStore: Listing table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/ -ls: childPath s3a://hwdev-steve-2/cloud-integration must be a child of s3a://A..A:X..X@hwdev-steve-2/ .. Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>] [-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] ? echo $? 255 {code} Works for a child entry which exists {code} [root@ctr-e138-1518143905142-264443-01-000006 ~]# hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/cloud-integration" 18/04/27 15:24:29 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for hwdev-steve-2 18/04/27 15:24:29 DEBUG s3a.S3AUtils: Propagating entries under fs.s3a.bucket.hwdev-steve-2. 18/04/27 15:24:29 WARN s3native.S3xLoginHelper: The Filesystem URI contains login details. This is insecure and may be unsupported in future. 18/04/27 15:24:29 DEBUG s3a.S3AUtils: For URI s3a://hwdev-steve-2//cloud-integration, using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3 ... 18/04/27 15:24:30 DEBUG s3guard.S3Guard: Using DynamoDBMetadataStore metadata store for s3a filesystem 18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Inferring DynamoDB region from S3 bucket: us-west-1 18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Creating DynamoDB client class org.apache.hadoop.fs.s3a.s3guard.DynamoDBClientFactory$DefaultDynamoDBClientFactory with S3 region us-west-1 18/04/27 15:24:30 DEBUG s3a.S3AUtils: For URI (null URI), using credentials AWSCredentialProviderList: BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider com.amazonaws.auth.InstanceProfileCredentialsProvider@14008db3 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.maximum is 15 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.attempts.maximum is 20 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.establish.timeout is 5000 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.connection.timeout is 200000 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.socket.send.buffer is 8192 18/04/27 15:24:30 DEBUG s3a.S3AUtils: Value of fs.s3a.socket.recv.buffer is 8192 18/04/27 15:24:30 DEBUG s3a.S3AFileSystem: Using User-Agent: User-Agent: APN/1.0 Hortonworks/1.0 HDP/3.0.0.0-1252, Hadoop 3.0.0.3.0.0.0-1252 18/04/27 15:24:30 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1 18/04/27 15:24:30 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2 18/04/27 15:24:30 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,} 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false 18/04/27 15:24:31 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/cloud-integration (cloud-integration) 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://A..A:X..X@hwdev-steve-2/cloud-integration: PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-2/cloud-integration; isDirectory=true; modification_time=1524842671173; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false} 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: List status for path: s3a://hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_list_status += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 2 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Getting path status for s3a://hwdev-steve-2/cloud-integration (cloud-integration) 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://hwdev-steve-2/cloud-integration: PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-2/cloud-integration; isDirectory=true; modification_time=1524842671185; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false} 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Listing table hwdev-steve-2 in region us-west-1: s3a://hwdev-steve-2/cloud-integration 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: listStatus: doing listObjects for directory cloud-integration/ 18/04/27 15:24:31 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 -> 1 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Adding directory: S3AFileStatus{path=s3a://hwdev-steve-2/cloud-integration/DELAY_LISTING_ME; isDirectory=true; modification_time=0; access_time=0; owner=root; group=root; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false} isEmptyDirectory=FALSE 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Added 1 entries; ignored 0; hasNext=true; hasMoreObjects=false 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Start iterating the provided status. Found 1 items drwxrwxrwx - root root 0 2018-04-27 15:24 s3a://hwdev-steve-2/cloud-integration/DELAY_LISTING_ME 18/04/27 15:24:31 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-2 is closed 18/04/27 15:24:31 DEBUG s3guard.DynamoDBMetadataStore: Shutting down DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2} {code} And for one which doesn;t {code} # hadoop fs -ls "s3a://A..A:X..X@hwdev-steve-2/missing" 18/04/27 15:25:21 DEBUG s3a.S3AFileSystem: Initializing S3AFileSystem for hwdev-steve-2 ... 18/04/27 15:25:23 DEBUG s3guard.DynamoDBClientFactory: Creating DynamoDB client in region us-west-1 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Binding to table hwdev-steve-2 18/04/27 15:25:23 DEBUG s3a.AWSCredentialProviderList: Using credentials from BasicAWSCredentialsProvider 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Table state: {AttributeDefinitions: [{AttributeName: child,AttributeType: S}, {AttributeName: parent,AttributeType: S}],TableName: hwdev-steve-2,KeySchema: [{AttributeName: parent,KeyType: HASH}, {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime: Wed Dec 06 14:25:57 UTC 2017,ProvisionedThroughput: {LastIncreaseDateTime: Thu Apr 05 13:27:34 UTC 2018,LastDecreaseDateTime: Wed Feb 28 14:11:15 UTC 2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 30,WriteCapacityUnits: 30},TableSizeBytes: 59156,ItemCount: 432,TableArn: arn:aws:dynamodb:us-west-1:980678866538:table/hwdev-steve-2,TableId: 11eb665e-746c-4255-8cc1-6890b0bbea24,} 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Using existing DynamoDB table hwdev-steve-2 in region us-west-1 created Wed Dec 06 14:25:39 UTC 2017 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Using metadata store DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2}, authoritative=false 18/04/27 15:25:23 DEBUG s3a.S3AUtils: Value of fs.s3a.multipart.purge.age is 86400 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: op_glob_status += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: op_get_file_status += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Getting path status for s3a://A..A:X..X@hwdev-steve-2/missing (missing) 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1: s3a://A..A:X..X@hwdev-steve-2/missing 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Get from table hwdev-steve-2 in region us-west-1 returning for s3a://A..A:X..X@hwdev-steve-2/missing: null 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_metadata_requests += 1 -> 2 18/04/27 15:25:23 DEBUG s3a.S3AStorageStatistics: object_list_requests += 1 -> 1 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Not Found: s3a://A..A:X..X@hwdev-steve-2/missing ls: `s3a://A..A:X..X@hwdev-steve-2/missing': No such file or directory 18/04/27 15:25:23 DEBUG s3a.S3AFileSystem: Filesystem s3a://hwdev-steve-2 is closed 18/04/27 15:25:23 DEBUG s3guard.DynamoDBMetadataStore: Shutting down DynamoDBMetadataStore{region=us-west-1, tableName=hwdev-steve-2} {code} > s3guard doesn't list root dir when the secrets are in the s3a URI > ----------------------------------------------------------------- > > Key: HADOOP-15422 > URL: https://issues.apache.org/jira/browse/HADOOP-15422 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 > Affects Versions: 3.1.0 > Reporter: Steve Loughran > Priority: Minor > > If the AWS secrets are in the login, S3guard doesn't list the root dir -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org