[
https://issues.apache.org/jira/browse/FLINK-30204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng closed FLINK-30204.
-------------------------------
Resolution: Fixed
> Table Store support separated compact jobs
> ------------------------------------------
>
> Key: FLINK-30204
> URL: https://issues.apache.org/jira/browse/FLINK-30204
> Project: Flink
> Issue Type: New Feature
> Components: Table Store
> Affects Versions: table-store-0.3.0
> Reporter: Caizhi Weng
> Assignee: Caizhi Weng
> Priority: Major
> Fix For: table-store-0.3.0
>
>
> Currently table store sinks will write and compact data files from the same
> job. While this implementation is enough and more economical for most users,
> some user may expect higher or more steady write throughput.
> We decided to support creating separated compact jobs for Table Store. This
> will bring us the following advantages:
> * Write jobs can concentrate only on writing files. Their throughput will be
> higher and more steady.
> * By creating only one compact job for each table, no commit conflicts will
> occur.
> The structure of a separated compact job is sketched out as follows:
> * There should be three vertices in a compact job. One source vertex, one
> sink (compactor) vertex and one commit vertex.
> * The source vertex is responsible for generating records containing
> partitions and buckets to be compacted.
> * The sink vertex accepts records containing partitions and buckets, and
> compact these buckets.
> * The commit vertex commit the changes from the sink vertex. It is possible
> that the user mistakenly creates other compact jobs so commit conflicts may
> still occur. However as compact changes are optional, this commit vertex will
> commit changes in an at-most-once style.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
