steveloughran opened a new pull request, #6141:
URL: https://github.com/apache/hadoop/pull/6141
1. handle getBucketLocation() unsupported
2. append / to marker tool searches if the first call fails.
3. downgrade change control errors a bit better. TODO: useful error text/docs
4. handle unknown storage classes.
5. Docs on how to bind to and test third party stores, including google gcs.
Not doing the region logic; this is other test failures.
Exception translation/resilience
* new AWSUnsupportedFeatureException ex for unsupported/unavailable errors
* handle 501 method unimplemented the same
* all ex > 500 mapped to the AWSStatus500Exception
* precondition errors handled a bit better
* GCS throttle exception also recognised.
* New class ErrorHandling to move our error handling logic too, initially
somewhere for new stuff
* ErrorHandling.maybeTranslateNetworkException() scans down an exception
chain and if the inner one is of a known type
- UnknownHostException
- NoRouteToHostException
- ConnectException
Then a new exception of that specific type is created, with the top level ex
its cause.
This is done to retain the whole stack chain.
* reduce the number of retries within the AWS SDK
* and those of our S3ARetryPolicy
The reduction in retries is because its clear when you try to create a
bucket
which doesn't resolve that the time for even an UnknownHostException to
eventually fail over 90s, which then hit the s3a retry code.
- Reducing the sdk retries means these escalate to our code better.
- Cutting back on our own retries makes it a bit more responsive for most
real deployments.
- maybeTranslateNetworkException() and s3a retry policy means that
UnknownHostException is recognised and fails fast.
Lots of changes to the test code:
* more switches to turn off test suites (acls, v1list, ...)
* more resilience to different test failures
* pull up skip() checks into config create phase, rather than waiting for fs
to be instantiated (faster; keeps the fs cache clean)
* tuning for new exceptions
### How was this patch tested?
* AWS S3 london: no regressions observed.
* a third party endpoint: showed troublespots, especially how handling of
UnknownHostException needed to be radically improved.
Gave me the stack traces for the docs too
Docs cover google gcs binding; I've not tested that yet, but the settings
came from
a colleague who has connected the v1 sdk to it.
A full test run against GCS should be something we require here as it
supports
a different subset of the s3 apis (no bulk delete, v1 list only...)
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [X] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
