Steve Loughran created HADOOP-19057:
---------------------------------------
Summary: S3 public test bucket landsat-pds unreadable -needs
replacement
Key: HADOOP-19057
URL: https://issues.apache.org/jira/browse/HADOOP-19057
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.6, 3.2.4, 3.4.0, 3.3.9, 3.5.0
Reporter: Steve Loughran
The s3 test bucket used in hadoop-aws tests of S3 select and large file reads
is no longer publicly accessible
{code}
java.nio.file.AccessDeniedException: landsat-pds: getBucketMetadata() on
landsat-pds: software.amazon.awssdk.services.s3.model.S3Exception: null
(Service: S3, Status Code: 403, Request ID: 06QNYQ9GND5STQ2S, Extended Request
ID:
O+u2Y1MrCQuuSYGKRAWHj/5LcDLuaFS8owNuXXWSJ0zFXYfuCaTVLEP351S/umti558eKlUqV6U=):null
{code}
* Because HADOOP-18830 has cut s3 select, all we need in 3.4.1+ is a large file
for some reading tests
* changing the default value disables s3 select tests on older releases
* if fs.s3a.scale.test.csvfile is set to " " then other tests which need it
will be skipped
Proposed
* we locate a new large file under the (requester pays) s3a://usgs-landsat/
bucket . All releases with HADOOP-18168 can use this
* update 3.4.1 source to use this; document it
* do something similar for 3.3.9 + maybe even cut s3 select there too.
* document how to use it on older releases with requester-pays support
* document how to completely disable it on older releases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
