Hi Chen,
Table Filesystem/Hive sink file compaction has been merged into master,
detail in [1]. It is included in Flink 1.12.
Hope you can have a try and test.
[1]https://issues.apache.org/jira/browse/FLINK-19345
Best,
Jingsong
On Thu, Nov 19, 2020 at 2:31 PM Chen Qin wrote:
> Hi there,
>
>
We just added this feature to 1.12 [1][2], it would be great that you can
download the 1.12 RC to test
it out, and give us some feedback.
In case you will wonder why I linked 2 jiras, it's because both FileSystem
& Hive connector share
the same option options and also the implementations.
[1]
Hi there,
We are testing out writing Kafka to hive table as parquet format.
Currently, we have seen user has to choose to create lots of small files in
min level folder to gain latency benefits. I recall FF2020 Global folks
mentioned implement compaction logic during the checkpointing time.
