[
https://issues.apache.org/jira/browse/FLINK-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062688#comment-17062688
]
Stephan Ewen commented on FLINK-11499:
--------------------------------------
[~kkl0u] I think the specialized solution for every format is infeasible at
this point, and it is not even clear to me whether it is possible with every
format.
The common solution is also far from defeating the purpose. We store the WAL
for a rather brief time only, compared to the longer-term storage of the final
file. And because price is a factor of {{storage x time}}, the short time we
store the WAL is not adding that much. It is also a fundamental issue - if you
put it in state, it ends up in storage (S3, etc.) as well. If you store it via
Kafka/Kinesis, you actually pay much more. S3 and the likes are the cheapest in
$/byte.
In terms of raw bandwidth to storage, let us see. For systems like S3, GCS, you
have to go to quite a high number of records/single-operator before you hit
that limit. Most users report hitting request rate SLAs before hitting the
bandwidth limit.
> Extend StreamingFileSink BulkFormats to support arbitrary roll policies
> -----------------------------------------------------------------------
>
> Key: FLINK-11499
> URL: https://issues.apache.org/jira/browse/FLINK-11499
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: Seth Wiesman
> Priority: Major
> Labels: usability
> Fix For: 1.11.0
>
>
> Currently when using the StreamingFilleSink Bulk-encoding formats can only be
> combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress
> part file on every checkpoint.
> However, many bulk formats such as parquet are most efficient when written as
> large files; this is not possible when frequent checkpointing is enabled.
> Currently the only work-around is to have long checkpoint intervals which is
> not ideal.
>
> The StreamingFileSink should be enhanced to support arbitrary roll policy's
> so users may write large bulk files while retaining frequent checkpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
