[
https://issues.apache.org/jira/browse/HADOOP-16202?focusedWorklogId=756325&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756325
]
ASF GitHub Bot logged work on HADOOP-16202:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Apr/22 11:26
Start Date: 13/Apr/22 11:26
Worklog Time Spent: 10m
Work Description: mehakmeet commented on PR #2584:
URL: https://github.com/apache/hadoop/pull/2584#issuecomment-1097937077
Ran the aws test suite on CSE. Everything ran fine, did see some region
errors in `ITestS3ARequesterPays`
```
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
11.433 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.ITestS3ARequesterPays
[ERROR]
testRequesterPaysDisabledFails(org.apache.hadoop.fs.s3a.ITestS3ARequesterPays)
Time elapsed: 9.323 s <<< ERROR!
org.apache.hadoop.fs.s3a.AWSRedirectException: getFileStatus on
s3a://usgs-landsat/collection02/catalog.json:
com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this
region: us-west-2. Please use this region to retry the request (Service: Amazon
S3; Status Code: 301; Error Code: 301 Moved Permanently; Request ID:
Z09V8PMEEN5PHDRZ; S3 Extended Request ID:
B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=;
Proxy: null), S3 Extended Request ID:
B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=:301
Moved Permanently: The bucket is in this region: us-west-2. Please use this
region to retry the request (Service: Amazon S3; Status Code: 301; Error Code:
301 Moved Permanently; Request ID: Z09V8PMEEN5PHDRZ; S3 Extended Request ID:
B7KDQntCuVmLJAyXvuY4UNXjdUrgn3xd26n8u7ThueNNxvKas6g3RsXo7oxBcvHrpcous2L+Lbk=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:233)
```
Looking at the test, it seems like we should be removing base and bucket
overrides for endpoint property too.
Issue Time Tracking
-------------------
Worklog Id: (was: 756325)
Time Spent: 19h (was: 18h 50m)
> Enhance openFile() for better read performance against object stores
> ---------------------------------------------------------------------
>
> Key: HADOOP-16202
> URL: https://issues.apache.org/jira/browse/HADOOP-16202
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs, fs/s3, tools/distcp
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Time Spent: 19h
> Remaining Estimate: 0h
>
> The {{openFile()}} builder API lets us add new options when reading a file
> Add an option {{"fs.s3a.open.option.length"}} which takes a long and allows
> the length of the file to be declared. If set, *no check for the existence of
> the file is issued when opening the file*
> Also: withFileStatus() to take any FileStatus implementation, rather than
> only S3AFileStatus -and not check that the path matches the path being
> opened. Needed to support viewFS-style wrapping and mounting.
> and Adopt where appropriate to stop clusters with S3A reads switched to
> random IO from killing download/localization
> * fs shell copyToLocal
> * distcp
> * IOUtils.copy
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
