[
https://issues.apache.org/jira/browse/FLINK-18303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136241#comment-17136241
]
Jingsong Lee commented on FLINK-18303:
--------------------------------------
First, I don't think this is a technical bug, but it can be a user oriented bug.
The options control the "see the part file" for csv,json are:
# rolling-policy.time-interval
# bucket check interval
# checkpoint interval
As long as one of the three options is not set small, it will not achieve the
desired effect.
> Filesystem connector doesn't flush part files after rolling interval
> --------------------------------------------------------------------
>
> Key: FLINK-18303
> URL: https://issues.apache.org/jira/browse/FLINK-18303
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Table SQL / Ecosystem
> Reporter: Jark Wu
> Assignee: Jark Wu
> Priority: Major
> Fix For: 1.11.0
>
>
> I have set "execution.checkpointing.interval" to "5s" and
> "sink.rolling-policy.time-interval" to "2s". However, it still take 60
> seconds to see the first part file.
> This can be reproduced by the following code in SQL CLI:
> {code:sql}
> CREATE TABLE CsvTable (
> event_timestamp STRING,
> `user` STRING,
> message STRING,
> duplicate_count BIGINT,
> constant STRING
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = '$RESULT',
> 'format' = 'csv',
> 'sink.rolling-policy.time-interval' = '2s'
> );
> INSERT INTO CsvTable -- read from Kafka Avro, and write into Filesystem Csv
> SELECT AvroTable.*, RegReplace('Test constant folding.', 'Test', 'Success')
> AS constant
> FROM AvroTable;
> {code}
> This is found when I migrate SQLClientKafkaITCase to use DDL (FLINK-18086).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
