[
https://issues.apache.org/jira/browse/FLINK-38350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095631#comment-18095631
]
Michael Johnson commented on FLINK-38350:
-----------------------------------------
Hi,
I think this is becoming important due to changes in how SSE-C works for S3!
MultiPartUploads to S3 are in 3 parts:
* CreateMultipartUpload:
[https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html]
* UploadPart:
[https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html]
* CompleteMultipartUpload:
[https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html]
When using SSE-C, I believe this Bug was corrected for only the first 2
parts...but not the 3rd!
Meaning, when calling the CompleteMultipartUpload function, it does not pass
the required Encryption Headers!
If you look at the S3 Docs for CompleteMultipartUpload:
[https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestParameters]
These Headers are still required when SSE-C is used:
*
*[x-amz-server-side-encryption-customer-algorithm|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax]*
*
{*}[x-amz-server-side-encryption-customer-key|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax]{*}{*}{{*}}
*
{*}[x-amz-server-side-encryption-customer-key-MD5|https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax]{*}{*}{{*}}
Hadoop library is used internally by Apache Flink for Snapshotting when SSE-C
is required...as such, it will cause Snapshotting to fail on the 3rd stage!
Upon deeper investigation:
* Flink 2.2.0 and Flink 2.3.0 are using Hadoop 3.3.4:
[https://github.com/apache/flink/blob/1baacaaade29ecc9dbc13b66c390b646a45d9544/flink-filesystems/pom.xml#L37]
* Hadoop 3.3.4 uses AWS SDK Java 1.12.262:
[https://github.com/apache/hadoop/blob/a585a73c3e02ac62350c136643a5e7f6095a3dbb/hadoop-project/pom.xml#L187]
* AWS SDK Java 1.12.262 does NOT implement SSE-C for CompleteMultipartUpload:
[https://github.com/aws/aws-sdk-java/blob/1.12.262/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/CompleteMultipartUploadRequest.java#L41]
However:
* Next highest version of Hadoop is 3.4.0, which is using AWS SDK Java
1.12.599:
[https://github.com/apache/hadoop/blob/branch-3.4.0/hadoop-project/pom.xml#L189]
* AWS SDK Java 1.12.599 DOES implement SSE-C for CompleteMultipartUpload:
[https://github.com/aws/aws-sdk-java/blob/1.12.599/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/CompleteMultipartUploadRequest.java#L41]
So I think the necessary thing is that Flink will need to upgrade Hadoop to at
least 3.4.0!
> Update Hadoop Library to 3.4.2
> ------------------------------
>
> Key: FLINK-38350
> URL: https://issues.apache.org/jira/browse/FLINK-38350
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
> Affects Versions: 2.0-preview, 1.20.2
> Reporter: Paul Ashley
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> Hadoop 3.4.2 which includes support for AWS conditional writes in
> https://issues.apache.org/jira/browse/HADOOP-19256
>
> It would be valuable for Flink to include this, as it would allow S3 based
> table sinks to be able to guarantee write integrity without relying on other
> concurrency controls (or accepting data loss).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
