[
https://issues.apache.org/jira/browse/FLINK-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060015#comment-17060015
]
Piotr Nowojski commented on FLINK-11499:
----------------------------------------
Just writing back some offline discussion about ideas how to implement this
feature.
We could have two output streams. The main output stream, committed & rolled
over per policy, for example Parquet file rolled every 1h, independently of
checkpointing. That’s going to be our official result. We also would have write
ahead log (WAL) stream, committed and probably closed & reopened new one on
every checkpoint, used only for recovery.
On happy path, we don’t care about WAL stream. We just periodically have to
clean it up. Without failures, main output stream rolled every 1h is everything
that we care about.
In case of failures/recovery:
# we need to have a point in the past were both of the streams align, so
probably if we want to roll main output stream at most once per hour, we still
need to roll it on checkpoint.
# we discard latest WIP main output stream (it could span multiple checkpoints)
# we reach out to WAL stream for the discarded main output stream's period of
time and we recreate the discarded part by reading matching WAL files
# we resume processing as before the failure
> Extend StreamingFileSink BulkFormats to support arbitrary roll policies
> -----------------------------------------------------------------------
>
> Key: FLINK-11499
> URL: https://issues.apache.org/jira/browse/FLINK-11499
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: Seth Wiesman
> Priority: Major
> Labels: usability
> Fix For: 1.11.0
>
>
> Currently when using the StreamingFilleSink Bulk-encoding formats can only be
> combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress
> part file on every checkpoint.
> However, many bulk formats such as parquet are most efficient when written as
> large files; this is not possible when frequent checkpointing is enabled.
> Currently the only work-around is to have long checkpoint intervals which is
> not ideal.
>
> The StreamingFileSink should be enhanced to support arbitrary roll policy's
> so users may write large bulk files while retaining frequent checkpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
