prathik-onehouse opened a new pull request, #20604:
URL: https://github.com/apache/hudi/pull/20604

   ### Describe the issue this Pull Request addresses
   
   closes #20603
   
   A record index initialized on a fresh table is sized from zero records, so 
its file group count stays at the configured minimum, and new partitions of the 
partitioned record index always get the minimum.
   
   ### Summary and Changelog
   
   Adds `hoodie.metadata.record.level.index.defer.init.min.records` (default 0, 
used with `hoodie.metadata.record.level.index.defer.init`). The record index is 
deferred until the latest base files hold that many records, and is then built 
and sized from them. Until then, lookups use the existing fallback index.
   
   - The deferral is decided where the metadata writer picks its enabled 
indexes (`IndexerFactory`), not only around the build, so a deferred index gets 
no updates or file groups.
   - Once over the threshold, the build waits while another data table instant 
is pending, and only a non-streaming metadata writer builds the index. A writer 
that started while the index was deferred therefore never misses it.
   - The threshold is rejected with multi-writer concurrency modes, the Flink 
engine (which has no fallback index) and negative values, and is ignored with 
async metadata indexing.
   - In Spark, a new data partition of the partitioned record index gets the 
larger of the p90 file group count of the existing partitions and the count 
estimated from the commit's inserts. This applies to all partitioned record 
indexes on Spark, not only with the threshold.
   - While deferred, the global record index fallback moves records across 
partitions as `hoodie.record.index.update.partition.path` says, so the behavior 
does not change when the index is built.
   
   ### Impact
   
   New opt-in config, no change with the default. The commit that builds the 
index takes the build time. Tables that do not populate meta fields and 
rollbacks of commits included in the build are not addressed here.
   
   ### Risk Level
   
   low. Tests cover deferral, sizing and the transition with no duplicates 
through the datasource, Spark SQL, structured streaming and HoodieStreamer 
writers, on table versions 6 and 10.
   
   ### Documentation Update
   
   Config descriptions updated. A website note for the new config can follow.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to