Davis Zhang created HUDI-9163:
---------------------------------
Summary: RLI & secondary index column file group counts can scale
horizontally
Key: HUDI-9163
URL: https://issues.apache.org/jira/browse/HUDI-9163
Project: Apache Hudi
Issue Type: Bug
Reporter: Davis Zhang
Fix For: 1.1.0
h3. Motivation
As of today for RLI we only set a fix number of file groups at all times.
Individual file group size will keep expanding as the size of the record count
increase. The size of individual file group will go unbounded which does not
scale.
To address this issue, we need to figure out a way to dynamically repartition
the existing file groups to a larger / smaller amount of file groups as needed.
* Minimum downtime / exclusive access blocking all other operations while we
scaling\
* basic correctness
* trigger of the scale up / down
* Integration with other operations on MDT
This should work for both RLI and secondary index on data columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
