[
https://issues.apache.org/jira/browse/FLINK-11499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061543#comment-17061543
]
Kostas Kloudas commented on FLINK-11499:
----------------------------------------
Thanks for moving the discussion forward [~pnowojski]!
I think that this solution seems to have the nice property of providing a
"one-size-fits-all" mechanism that can be applied to any format and "transform
it" to exactly-once. But I would say that it has a pretty high price-tag.
I think that the proposed solution defeats the purpose of using compression for
example. In addition, it can be incompatible with usecases where a bulk format
is used to encrypt data for security. In this case, we will be writing the
plaintext alongside the encrypted data?
To elaborate a bit on the "defeating the purpose" statement, and focusing on
bulk formats only used for compression, I believe that users use them to save
space and bandwidth. If we agree on that, then I think that the solution
proposed has the following drawbacks:
1) we write to a DFS twice (main output and WAL), so more bandwidth
requirements and more storage, which are the two metrics that usually public
cloud providers charge you for
2) the above also potentially means increased latency for the job itself as we
send more data to the DFS
3) upon recovery, we will have to re-read the WAL and re-compress it, which
leads to slow recovery times and load on the DFS especially for high
parallelism jobs
I think that the solution to the above would require the specialised bulk
writers per format to work together with our checkpointing mechanism to
guarantee exactly-once. For example in Parquet, we may need to go deeper and
not use the hadoop writers that hide the internal parquet metadata of the
"in-progress" file, but implement our own writers.
Actually a related open issue for Orc is here
https://issues.apache.org/jira/browse/FLINK-10114.
What do you think?
> Extend StreamingFileSink BulkFormats to support arbitrary roll policies
> -----------------------------------------------------------------------
>
> Key: FLINK-11499
> URL: https://issues.apache.org/jira/browse/FLINK-11499
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: Seth Wiesman
> Priority: Major
> Labels: usability
> Fix For: 1.11.0
>
>
> Currently when using the StreamingFilleSink Bulk-encoding formats can only be
> combined with the `OnCheckpointRollingPolicy`, which rolls the in-progress
> part file on every checkpoint.
> However, many bulk formats such as parquet are most efficient when written as
> large files; this is not possible when frequent checkpointing is enabled.
> Currently the only work-around is to have long checkpoint intervals which is
> not ideal.
>
> The StreamingFileSink should be enhanced to support arbitrary roll policy's
> so users may write large bulk files while retaining frequent checkpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
