moomindani commented on PR #17177: URL: https://github.com/apache/iceberg/pull/17177#issuecomment-5465448594
@singhpk234 I mentioned you here on 7 August along with two others, so apologies for coming back — this time I have a more specific reason to ask you in particular. #15242, which you merged in April, is the same shape as this PR: an `S3FileIOProperties` knob that lets users opt out of an AWS SDK default that some S3-compatible stores reject. You have already ruled on that pattern once, which is why I am asking you rather than casting wider. What this does: since SDK 2.30.0 the default `RequestChecksumCalculation.WHEN_SUPPORTED` attaches a CRC32 trailer to every PutObject even when `s3.checksum-enabled=false`, and Dell ECS and the GCS S3-compatible API reject it (#14439). The PR adds `s3.request-checksum-calculation` and `s3.response-checksum-validation` (`when_supported` / `when_required`), both leaving the SDK default untouched when unset, wired through all four client factories. It revives #15391 by @rcjverhoef, whose commit is preserved as the first one here, at his suggestion. Rebased onto current `main`, CI green. The one question I would rather not settle myself is the default: keep the SDK default, as this PR does, so there is no behavior change and `when_required` is strictly opt-in — or flip it so S3-compatible stores work out of the box. I put the same question to @danielcweeks a week ago. Either answer works for me; I can change it here, or take the flip to dev@ as its own discussion. A correction to my earlier comments here: I wrote that #17179 stacks on this PR and should wait for it. That is wrong. #17179 branches from `main`, shares no commits with this branch, and can be reviewed on its own, in either order. Sorry for the misdirection. @steveloughran you pointed at the S3A settings on #14439 back in April; this is the Iceberg-side equivalent, and your view on which default is right would be valuable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
