Ying Xu created FLINK-13027:
-------------------------------
Summary: StreamingFileSink bulk-encoded writer supports file
rolling upon customized events
Key: FLINK-13027
URL: https://issues.apache.org/jira/browse/FLINK-13027
Project: Flink
Issue Type: New Feature
Components: API / DataStream
Reporter: Ying Xu
When writing in bulk-encoded format such as Parquet, StreamingFileSink only
supports OnCheckpointRollingPolicy, which rolls file at checkpointing time.
In many scenarios, it is beneficial that the sink can roll file upon certain
events, for example, when the file size reaches a limit. Such a rolling policy
can also potentially alleviate some of the side effects of
OnCheckpointRollingPolicy, e.g.,, most of the heavy liftings including file
uploading all happen at the checkpoint time.
Specifically, this Jira calls for a new rolling policy that rolls file:
# whenever a customized event happens, e.g., the file size reaches certain
limit.
# whenever a checkpoint happens. This is needed for providing exactly-once
guarantees when writing bulk-encoded files.
Users of this rolling policy need to be aware that the customized event and the
next checkpoint epoch may be close to each other, thus may yield a tiny file
per checkpoint at the worst.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
