codope opened a new pull request, #20558:
URL: https://github.com/apache/hudi/pull/20558

   ### Describe the issue this Pull Request addresses
   
   Closes #20557
   
   A Spark writer with `hoodie.metadata.index.async=true` and the record index 
enabled fails its first commit with:
   
   ```
   java.lang.IllegalStateException: Partition record_index should be part of 
inflight metadata partitions here []
   ```
   
   With async indexing on, `initializeIfNeeded` leaves the record index to 
`HoodieIndexer`, so `record_index` is neither completed nor inflight in 
`hoodie.properties`. The non-streaming write path takes the partitions to 
update from the table config and is fine. The streaming path (the Spark 
default) takes `record_index` from the write config and then fails when it 
finds no file groups for it.
   
   ### Summary and Changelog
   
   The first commit now succeeds and writes only the metadata partitions that 
exist. Once `HoodieIndexer` has scheduled and built `record_index`, the writer 
updates it, as it already does with 
`hoodie.metadata.streaming.write.enabled=false`.
   
   Both partition selectors used by a streaming commit in 
`HoodieBackedTableMetadataWriter` now apply the table-config rule from 
`getMetadataPartitionsToUpdate` (completed or inflight partitions):
   
   - `getStreamingMetadataPartitionsToUpdate`: stream to `record_index` only 
when the table config lists it.
   - `getNonStreamingMetadataPartitionsToUpdate`: send to the batch path only 
enabled partitions that the table config lists.
   
   The second change is needed too. With only the first, `record_index` falls 
through to the batch path in `completeStreamingCommit`, and 
`tagRecordsWithLocation` fails the same check. The second change also covers 
other enabled partitions that the async indexer hasn't built yet, such as 
column stats.
   
   Test: 
`TestMetadataWriterCommit#testStreamingWriteSkipsRecordIndexLeftToAsyncIndexer` 
runs a streaming commit with async indexing and the global record index 
enabled. It asserts that only `files` is bootstrapped and only `files` is 
written. It fails on master with the exception above.
   
   ### Impact
   
   Fixes the async record-index workflow (ingestion keeps running while 
`HoodieIndexer` builds the index) with streaming metadata writes, which Spark 
uses by default. When async indexing is off, every enabled partition is already 
in the table config after initialization, so nothing changes.
   
   Flink is out of scope. Flink streams to the metadata table only when 
`index.type` is a record-level index. The writer then needs the record index to 
tag records, and it has no fallback. So the async workflow can't work there. 
With this change, a Flink job configured that way fails on its first 
record-index lookup with "Record index is not initialized in MDT", instead of 
failing at the first commit. A fail-fast config check for Flink can follow 
separately.
   
   ### Risk Level
   
   low. Partitions are only ever dropped from the update set, never added. The 
ones dropped are not completed or inflight in the table config, and writing to 
them would have failed the same check. Ran `TestMetadataWriterCommit`, 
`TestCreateHandle`, `TestMergeHandle`, `TestAppendHandle` and 
`TestHoodieIndexer` locally (Spark 3.5).
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to