[
https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16378903#comment-16378903
]
Steve Loughran commented on HADOOP-15269:
-----------------------------------------
Possibly related. Running cloudstore diags against the path
s3a://hwdev-stevel-ireland-new/example4 works (it lists, mkdirs if needed, adds
children), but against /test it fails in DDB/S3Guard
{code}
ard_metadatastore_throttle_rate99thPercentileFrequency (Hz)=0} }}
2018-02-27 16:37:33,753 [main] INFO diag.StoreDiag
(DurationInfo.java:<init>(50)) - Starting: GetFileStatus
s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,753 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
1
2018-02-27 16:37:33,753 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for
s3a://hwdev-steve-ireland-new/ ()
2018-02-27 16:37:33,754 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:get(437)) - Get from table hwdev-steve-ireland-new
in region eu-west-1: s3a://hwdev-steve-ireland-new/
root entry S3AFileStatus{path=s3a://hwdev-steve-ireland-new/; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=UNKNOWN
2018-02-27 16:37:33,760 [main] INFO diag.StoreDiag
(DurationInfo.java:close(83)) - GetFileStatus s3a://hwdev-steve-ireland-new/:
duration 0:00:007
2018-02-27 16:37:33,760 [main] INFO diag.StoreDiag
(DurationInfo.java:<init>(50)) - Starting: Listing
s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerListStatus(1876)) - List status for path:
s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_list_status += 1 -> 1
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
2
2018-02-27 16:37:33,761 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for
s3a://hwdev-steve-ireland-new/ ()
2018-02-27 16:37:33,761 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:get(437)) - Get from table hwdev-steve-ireland-new
in region eu-west-1: s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,761 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:listChildren(505)) - Listing table
hwdev-steve-ireland-new in region eu-west-1: s3a://hwdev-steve-ireland-new/
2018-02-27 16:37:33,810 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerListStatus(1893)) - listStatus: doing listObjects for
directory
2018-02-27 16:37:33,812 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_list_requests += 1
-> 1
2018-02-27 16:37:33,867 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(452)) -
s3a://hwdev-steve-ireland-new/file.txt: file.txt size=0
2018-02-27 16:37:33,867 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(458)) - Adding:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/file.txt; isDirectory=false;
length=0; replication=1; blocksize=33554432; modification_time=1519744033000;
access_time=0; owner=stevel; group=stevel; permission=rw-rw-rw-;
isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}
isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/Users; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/example4; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0001; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,868 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0002; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,869 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0003; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,869 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0004; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,869 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0005; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,869 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0006; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,869 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0007; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,870 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/fork-0008; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,870 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/path; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,870 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(473)) - Adding directory:
S3AFileStatus{path=s3a://hwdev-steve-ireland-new/test; isDirectory=true;
modification_time=0; access_time=0; owner=stevel; group=stevel;
permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false} isEmptyDirectory=FALSE
2018-02-27 16:37:33,871 [main] DEBUG s3a.S3AFileSystem
(Listing.java:buildNextStatusBatch(486)) - Added 13 entries; ignored 0;
hasNext=true; hasMoreObjects=false
2018-02-27 16:37:33,871 [main] DEBUG s3a.S3AFileSystem
(Listing.java:sourceHasNext(378)) - Start iterating the provided status.
s3a://hwdev-steve-ireland-new/ root entry count: 17
2018-02-27 16:37:33,873 [main] INFO diag.StoreDiag
(DurationInfo.java:close(83)) - Listing s3a://hwdev-steve-ireland-new/:
duration 0:00:112
2018-02-27 16:37:33,873 [main] INFO diag.StoreDiag
(DurationInfo.java:<init>(50)) - Starting: Reading file
s3a://hwdev-steve-ireland-new/file.txt
2018-02-27 16:37:33,873 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_open += 1 -> 1
2018-02-27 16:37:33,873 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:open(692)) - Opening
's3a://hwdev-steve-ireland-new/file.txt' for reading; input policy = normal
2018-02-27 16:37:33,873 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
3
2018-02-27 16:37:33,873 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for
s3a://hwdev-steve-ireland-new/file.txt (file.txt)
2018-02-27 16:37:33,873 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:get(437)) - Get from table hwdev-steve-ireland-new
in region eu-west-1: s3a://hwdev-steve-ireland-new/file.txt
2018-02-27 16:37:33,908 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:innerGet(461)) - Get from table
hwdev-steve-ireland-new in region eu-west-1 returning for
s3a://hwdev-steve-ireland-new/file.txt:
PathMetadata{fileStatus=FileStatus{path=s3a://hwdev-steve-ireland-new/file.txt;
isDirectory=false; length=0; replication=1; blocksize=33554432;
modification_time=1519744033000; access_time=0; owner=stevel; group=stevel;
permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false;
isErasureCoded=false}; isEmptyDirectory=UNKNOWN; isDeleted=false}
First character of file s3a://hwdev-steve-ireland-new/file.txt is 0xffffffff:
'(n/a)'
2018-02-27 16:37:33,912 [main] DEBUG s3a.S3AInputStream
(S3AInputStream.java:close(463)) - Statistics of stream file.txt
StreamStatistics{OpenOperations=0, CloseOperations=0, Closed=0, Aborted=0,
SeekOperations=0, ReadExceptions=0, ForwardSeekOperations=0,
BackwardSeekOperations=0, BytesSkippedOnSeek=0, BytesBackwardsOnSeek=0,
BytesRead=0, BytesRead excluding skipped=0, ReadOperations=0,
ReadFullyOperations=0, ReadsIncomplete=0, BytesReadInClose=0,
BytesDiscardedInAbort=0, InputPolicy=0, InputPolicySetCount=1}
2018-02-27 16:37:33,913 [main] INFO diag.StoreDiag
(DurationInfo.java:close(83)) - Reading file
s3a://hwdev-steve-ireland-new/file.txt: duration 0:00:040
2018-02-27 16:37:33,913 [main] INFO diag.StoreDiag
(DurationInfo.java:<init>(50)) - Starting: Listing directory
s3a://hwdev-steve-ireland-new/test
2018-02-27 16:37:33,913 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - op_get_file_status += 1 ->
4
2018-02-27 16:37:33,913 [main] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:innerGetFileStatus(2098)) - Getting path status for
s3a://hwdev-steve-ireland-new/test (test)
2018-02-27 16:37:33,913 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:get(437)) - Get from table hwdev-steve-ireland-new
in region eu-west-1: s3a://hwdev-steve-ireland-new/test
2018-02-27 16:37:33,947 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:innerGet(461)) - Get from table
hwdev-steve-ireland-new in region eu-west-1 returning for
s3a://hwdev-steve-ireland-new/test: null
2018-02-27 16:37:33,949 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 1
2018-02-27 16:37:33,991 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 2
2018-02-27 16:37:34,055 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 1
2018-02-27 16:37:34,558 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 3
2018-02-27 16:37:34,715 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 2
2018-02-27 16:37:35,215 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 4
2018-02-27 16:37:35,393 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 3
2018-02-27 16:37:35,897 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 5
2018-02-27 16:37:36,072 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 4
2018-02-27 16:37:36,575 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 6
2018-02-27 16:37:37,048 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 5
2018-02-27 16:37:37,552 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 7
2018-02-27 16:37:37,687 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 6
2018-02-27 16:37:38,189 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 8
2018-02-27 16:37:38,335 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 7
2018-02-27 16:37:38,836 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 9
2018-02-27 16:37:39,094 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 8
2018-02-27 16:37:39,598 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 10
2018-02-27 16:37:39,782 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 9
2018-02-27 16:37:40,283 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 11
2018-02-27 16:37:40,447 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 10
2018-02-27 16:37:40,952 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 12
2018-02-27 16:37:41,090 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 11
2018-02-27 16:37:41,592 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 13
2018-02-27 16:37:41,756 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 12
2018-02-27 16:37:42,256 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 14
2018-02-27 16:37:42,410 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 13
2018-02-27 16:37:42,914 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 15
2018-02-27 16:37:43,053 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 14
2018-02-27 16:37:43,558 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 16
2018-02-27 16:37:43,715 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 15
2018-02-27 16:37:44,217 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 17
2018-02-27 16:37:44,375 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 16
2018-02-27 16:37:44,879 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 18
2018-02-27 16:37:45,021 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 17
2018-02-27 16:37:45,525 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 19
2018-02-27 16:37:45,665 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 18
2018-02-27 16:37:46,168 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 20
2018-02-27 16:37:46,327 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 19
2018-02-27 16:37:46,829 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 21
2018-02-27 16:37:47,013 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - ignored_errors += 1 -> 20
2018-02-27 16:37:47,514 [main] DEBUG s3a.S3AStorageStatistics
(S3AStorageStatistics.java:incrementCounter(63)) - object_metadata_requests +=
1 -> 22
2018-02-27 16:37:47,673 [main] INFO diag.StoreDiag
(DurationInfo.java:close(83)) - Listing directory
s3a://hwdev-steve-ireland-new/test: duration 0:13:760
org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on test/:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon
S3; Status Code: 400; Error Code: 400 Bad Request; Request ID:
E08905C9F092936F; S3 Extended Request ID:
DBP8Sk9n93bJfC1OYc4SP02OkCEe5Cma80zN75YCqp1v6S37sG8v39cmEIvIUkYWRFhNH2xgRs0=),
S3 Extended Request ID:
DBP8Sk9n93bJfC1OYc4SP02OkCEe5Cma80zN75YCqp1v6S37sG8v39cmEIvIUkYWRFhNH2xgRs0=:400
Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code:
400 Bad Request; Request ID: E08905C9F092936F; S3 Extended Request ID:
DBP8Sk9n93bJfC1OYc4SP02OkCEe5Cma80zN75YCqp1v6S37sG8v39cmEIvIUkYWRFhNH2xgRs0=)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:212)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2202)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2140)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2079)
at
org.apache.hadoop.fs.store.diag.StoreDiag.executeFileSystemOperations(StoreDiag.java:419)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:274)
at org.apache.hadoop.fs.store.diag.StoreDiag.run(StoreDiag.java:170)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.store.diag.StoreDiag.exec(StoreDiag.java:535)
at org.apache.hadoop.fs.store.diag.StoreDiag.main(StoreDiag.java:545)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:308)
at org.apache.hadoop.util.RunJar.main(RunJar.java:222)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID:
E08905C9F092936F; S3 Extended Request ID:
DBP8Sk9n93bJfC1OYc4SP02OkCEe5Cma80zN75YCqp1v6S37sG8v39cmEIvIUkYWRFhNH2xgRs0=),
S3 Extended Request ID:
DBP8Sk9n93bJfC1OYc4SP02OkCEe5Cma80zN75YCqp1v6S37sG8v39cmEIvIUkYWRFhNH2xgRs0=
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4325)
at
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4272)
at
com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1264)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$4(S3AFileSystem.java:1227)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1224)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2185)
{code}
This is hitting DDB, not S3A, yet its failing with bad request too.
I have no idea WTF is going on, and can only hypothesise that something has
happened to the path /test/ which has confused both S3Guard and S3 itself.
> S3 returning 400 on the directory /test/ GET of getFileStatus
> -------------------------------------------------------------
>
> Key: HADOOP-15269
> URL: https://issues.apache.org/jira/browse/HADOOP-15269
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.1.0, 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Critical
>
> Since Monday Feb 26, I'm getting intermittent failures of getFileStatus on a
> directory
> # file path: {{/test}} is returning 404, as expected
> # directory path {{//test/}} is returning 400, so failing the entire operation
> S3 Ireland.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
