[
https://issues.apache.org/jira/browse/FLINK-19345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201349#comment-17201349
]
Jingsong Lee commented on FLINK-19345:
--------------------------------------
Hi [~kkl0u], I think you mean FLINK-17505. I think the reason why I create a
new JIRA is that I think FLINK-17505 provides a more general and perfect set of
merging solutions. I don't mean that I want to split table and DataStream. The
solutions of table layer are built on DataStream StreamingFileSink.
However, there are some advanced things in the table layer. Such as FileWriter
is an operator rather than a sink, such as Hive's partition committer, such as
small file compaction. Table goes a little bit further, but I believe these
requirements are reasonable.
[~maguowei], [~gaoyunhaii], [~aljoscha] and community partners are doing a lot
of great work to provide more through-depth abstractions and solutions,
including [1], including unified sink. I believe that positive communication
between us can make things more smooth.
[1]https://docs.google.com/document/d/1or7V024ptedwFzsmHbSzoJapq9Ah5L03SYPnnHTfoEg/edit#
> Introduce File streaming sink compaction
> ----------------------------------------
>
> Key: FLINK-19345
> URL: https://issues.apache.org/jira/browse/FLINK-19345
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
> Reporter: Jingsong Lee
> Assignee: Jingsong Lee
> Priority: Major
> Fix For: 1.12.0
>
>
> Users often complain that many small files are written out. Small files will
> affect the performance of file reading and the DFS system, and even the
> stability of the DFS system.
> Target:
> * Compact all files generated by this job in a single checkpoint.
> * With compaction, Users can have smaller checkpoint interval, even to
> seconds.
> Document:
> https://docs.google.com/document/d/1cdlyoqgBq9yJEiHFBziimIoKHapQiEY2-0Tn8IF6G-c/edit?usp=sharing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
