prathik-onehouse opened a new issue, #20603:
URL: https://github.com/apache/hudi/issues/20603

   On a fresh table, the metadata table and the record index are initialized at 
the start of the first write, when the table holds no data. The record count 
estimate is 0, so the record index file group count is clamped to the 
configured minimum and stays there for the life of the table. The partitioned 
record index gets the minimum for every data partition, including partitions 
added later. `hoodie.metadata.record.level.index.defer.init` only waits for the 
second commit, which for a streaming writer is usually another small 
micro-batch.
   
   Proposal: defer the record index until the latest base files hold a 
configurable number of records, then build it sized from them, and size new 
partitions of the partitioned record index from the existing partitions and the 
records the commit inserts. Writes before that use the existing fallback index 
(GLOBAL_SIMPLE or SIMPLE), as they do today while the index is missing.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to