[
https://issues.apache.org/jira/browse/FLINK-29678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17619423#comment-17619423
]
Martijn Visser commented on FLINK-29678:
----------------------------------------
[~luoyuxia] Could you elaborate a bit on this? Is this a mistake in the file
system implementation, in the Sink interfaces, etc?
> Data may loss when sink bounded stream into filesystem with auto compact
> enabled in streaming mode
> ---------------------------------------------------------------------------------------------------
>
> Key: FLINK-29678
> URL: https://issues.apache.org/jira/browse/FLINK-29678
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.15.0
> Reporter: luoyuxia
> Priority: Major
>
> In stream mode, when writing bounded data stream into filesystem with auto
> compactation enabel, the data may loss.
> We can reproduce it by adding one code line `'auto-compaction'='true' ` in
> `FileSystemITCaseBase#open` to enable auto compact.
> {code:java}
> tableEnv.executeSql(
> s"""
> |create table partitionedTable (
> | x string,
> | y int,
> | a int,
> | b bigint,
> | c as b + 1
> |) partitioned by (a, b) with (
> | 'connector' = 'filesystem',
> | 'auto-compaction'='true', // added line to enable auto compaction.
> | 'path' = '$getScheme://$resultPath',
> | ${formatProperties().mkString(",\n")}
> |)
> """.stripMargin
> ) {code}
> Then the test `StreamFileSystemTestCsvITCase#testPartialDynamicPartition`
> will fail with the assert failure:
> {code:java}
> java.lang.AssertionError:
> Expected :List(x18,18)
> Actual :List() {code}
> There is no data has been written into the table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
