prathik-onehouse opened a new pull request, #20604: URL: https://github.com/apache/hudi/pull/20604
### Describe the issue this Pull Request addresses closes #20603 A record index initialized on a fresh table is sized from zero records, so its file group count stays at the configured minimum, and new partitions of the partitioned record index always get the minimum. ### Summary and Changelog Adds `hoodie.metadata.record.level.index.defer.init.min.records` (default 0, used with `hoodie.metadata.record.level.index.defer.init`). The record index is deferred until the latest base files hold that many records, and is then built and sized from them. Until then, lookups use the existing fallback index. - The deferral is decided where the metadata writer picks its enabled indexes (`IndexerFactory`), not only around the build, so a deferred index gets no updates or file groups. - Once over the threshold, the build waits while another data table instant is pending, and only a non-streaming metadata writer builds the index. A writer that started while the index was deferred therefore never misses it. - The threshold is rejected with multi-writer concurrency modes, the Flink engine (which has no fallback index) and negative values, and is ignored with async metadata indexing. - In Spark, a new data partition of the partitioned record index gets the larger of the p90 file group count of the existing partitions and the count estimated from the commit's inserts. This applies to all partitioned record indexes on Spark, not only with the threshold. - While deferred, the global record index fallback moves records across partitions as `hoodie.record.index.update.partition.path` says, so the behavior does not change when the index is built. ### Impact New opt-in config, no change with the default. The commit that builds the index takes the build time. Tables that do not populate meta fields and rollbacks of commits included in the build are not addressed here. ### Risk Level low. Tests cover deferral, sizing and the transition with no duplicates through the datasource, Spark SQL, structured streaming and HoodieStreamer writers, on table versions 6 and 10. ### Documentation Update Config descriptions updated. A website note for the new config can follow. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
