Yue Zhang created HUDI-8780:
-------------------------------
Summary: RFC-83 Incremental Table Service
Key: HUDI-8780
URL: https://issues.apache.org/jira/browse/HUDI-8780
Project: Apache Hudi
Issue Type: New Feature
Reporter: Yue Zhang
In Hudi, when scheduling Compaction and Clustering, the default behavior is to
scan all partitions under the current table. When there are many historical
partitions, such as 640,000 in our production environment, this scanning and
planning operation becomes very inefficient. For Flink, it often leads to
checkpoint timeouts, resulting in data delays.
As for cleaning, we already have the ability to do cleaning for incremental
partitions.
This RFC will draw on the design of Incremental Clean to generalize the
capability of processing incremental partitions to all table services, such as
Clustering and Compaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
