[
https://issues.apache.org/jira/browse/FLINK-35536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juliusz Nadberezny updated FLINK-35536:
---------------------------------------
Summary: FileSystem sink on S3 produces invalid Avros when compaction is
turned off (was: FileSystem sink on S3 produces invalid Avros when compaction
is disabled)
> FileSystem sink on S3 produces invalid Avros when compaction is turned off
> --------------------------------------------------------------------------
>
> Key: FLINK-35536
> URL: https://issues.apache.org/jira/browse/FLINK-35536
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.19.0
> Reporter: Juliusz Nadberezny
> Priority: Major
>
> Compaction on FileSystem sink on S3 uses multipart upload process.
> When compaction is disabled after being enabled, the files that where being
> kept by multipart upload and then are "released" with CompleteMultipartUpload
> will be broken.
> Broken Avro files seem to have Avro schema duplicated at the beginning of the
> file.
>
> Steps to reproduce:
> 1. Deploy job with FileSystem sink with compaction enabled writing to
> S3/MinIO.
> 2. Wait for job to produce some output.
> 3. Redeploy job with compaction disabled.
> 4. Wait for multipart upload complete and verify released files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
