steveloughran commented on PR #44834:
URL: https://github.com/apache/spark/pull/44834#issuecomment-1905892184
fips support is in 3.4.1; i've just cherrypicked a chain of commits from the
last week into branch-3.4, but not pushing for the 3.4.0 RC to be blocked on
them. if the rc fails for other reasons I will cherrypick there, but otherwise
wait for a 3.4.1
| Hash | Date | Commit message
|
|--------------|---------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| 19b9e6a97b8f | 2023-12-12 15:15:32 +0000 | HADOOP-19008. S3A: update
aws-sdk version to 2.21.41 (#6334)
|
| 2f1e1558b6fc | 2024-01-11 17:13:31 +0000 | HADOOP-19004. S3A: Support
Authentication through HttpSigner API (#6324)
|
| 36198b5edf5b | 2024-01-16 14:14:03 +0000 | HADOOP-19027. S3A:
S3AInputStream doesn't recover from HTTP/channel exceptions (#6425)
|
| d37885379009 | 2024-01-16 14:16:12 +0000 | HADOOP-18975 S3A: Add option
fs.s3a.endpoint.fips to use AWS FIPS endpoints (#6277)
|
| 7b1570e2f15d | 2024-01-16 17:06:28 -0600 | HADOOP-19015. Increase
fs.s3a.connection.maximum to 500 to minimize risk of Timeout waiting for
connection from pool. (#6372) |
| eeb657e85f3f | 2024-01-17 18:34:14 +0000 | HADOOP-19033. S3A: disable
checksums when fs.s3a.checksum.validation = false (#6441)
|
HADOOP-19033 is a performance regression, but HADOOP-19027 and input stream
resilience worries me. We will need more stack traces from the wild to be able
to complete the resilience there as the new sdk stack is raising different
failures and we need to see them.
I also want to get deeper into the sdk internals as it looks like rather
than a blind "retry on IOE" class we could be a bit more specific and have some
things failfast (UnknownHostException etc). But I'm not sure if the SDK lets us
be that sophisticated policy-wise. And we cannot turn off its retries
unless/until we move off the sdk transfer manager for multipart copy
operations. Implement that ourselves and we can tell the sdk to never retry -we
can take over that. Tempting
anyway, lets get 3.4.0 out and see how complains about what.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
