FrankChen021 commented on code in PR #20055:
URL: https://github.com/apache/druid/pull/20055#discussion_r3804006198
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java:
##########
@@ -125,6 +127,26 @@ public boolean apply(Throwable e)
}
};
+ /**
+ * Restores {@code Content-MD5} request checksums on every given builder,
for S3-compatible stores that reject the
+ * CRC32 checksums the SDK sends by default since 2.30.0.
+ * <p>
+ * Takes all the builders for one client set rather than one builder per
call, so the sync and async clients cannot
+ * end up disagreeing about checksum behavior, and so the switch is logged
once per client set.
+ */
+ public static void configureLegacyMd5(
+ final AWSClientConfig clientConfig,
+ final S3BaseClientBuilder<?, ?>... s3ClientBuilders
+ )
+ {
+ if (clientConfig.isEnableLegacyMd5()) {
+ log.info("Legacy MD5 compatibility mode is enabled for the S3 client.");
+ for (S3BaseClientBuilder<?, ?> s3ClientBuilder : s3ClientBuilders) {
+ s3ClientBuilder.addPlugin(LegacyMd5Plugin.create());
Review Comment:
[P1] Legacy mode still emits CRC32 on uploads
This only adds LegacyMd5Plugin; the builders retain the SDK default request
checksum mode (WHEN_SUPPORTED). Optional streaming PutObject and UploadPart
requests therefore continue sending CRC32 checksums, including through the
default async Transfer Manager, instead of Content-MD5. S3-compatible stores
rejecting CRC32 headers can still fail normal segment uploads despite
druid.s3.enableLegacyMd5=true. Configure checksum calculation on both builders
to suppress optional SDK checksums and verify required operations emit the
intended MD5 header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
