Davis-Zhang-Onehouse opened a new pull request, #19717: URL: https://github.com/apache/hudi/pull/19717
### Describe the issue this Pull Request addresses closes #19716 An indexing action for one of several uninitialized secondary (or expression) indexes built nothing, yet completed and marked the requested partition complete in `hoodie.table.metadata.partitions`: `buildMetadataPartitions` reduced the plan's partition path to a type, and the indexer inferred which partition to build from every uninitialized definition of that type, skipping the type when that was not exactly one. ### Summary and Changelog - `IndexInitializationContext` carries the partition an indexing action asked for (empty for a regular write; the five-argument `of` is kept). - `BaseIndexer.resolvePartitionsToInit` resolves it once for both `SecondaryIndexer` and `ExpressionIndexer`: the requested partition when its definition exists; otherwise the single uninitialized definition, which is also where a first-time index mints its definition from the write config; and when a requested partition cannot be resolved that way the action fails instead of completing with nothing built. - `buildMetadataPartitions` records one requested path per type (all a plan can carry, since each initialization needs its own instant) and threads it through `initializeFromFilesystem` and `initializeMetadataPartition`. - `TestHoodieIndexer.testIndexerBuildsEachRequestedIndexAmongSeveralUninitialized` registers two secondary-index and two expression-index definitions and builds all four with consecutive indexing actions, asserting after each action that only the requested partition appeared (in the table config and on storage) and at the end that every built partition has file slices. Without the fix the first action fails on `metadataPartitionExists` (the table config lists the index, storage has no partition). `TestSecondaryIndexer` unit-tests the resolver branches an indexer run cannot reach directly: the requested partition among several uninitialized definitions, the already-initialized skip, and the unresolvable-request failure. - Added after the original review pass: a single inference candidate answers a request only when it is the requested partition, so an indexer run naming an index with no definition no longer builds a different one; the definition lookup is a supplier evaluated only when inference is needed, so a skipped partition leaves no definition behind; `RunIndexActionExecutor` catches `HoodieMetadataException` so a failed initialization aborts instead of stranding the partition inflight, and re-reads the table config before adding to it. - The requested partition bootstraps under a solo-family instant, not the indexing action's own. With async indexing off, the next indexing action's metadata writer initializes any remaining index at construction time, keyed off the last completed data instant, which is the previous action's; `generateUniqueInstantTime` hands an indexing instant back unchanged, and committing there again rolls the earlier bootstrap back first, leaving that index listed complete with no records. While an indexing action is pending, solo-family metadata commits get the same eager-rollback exemption in `BaseHoodieTableServiceClient` that the action's own instant gets. The index executor's abort also deletes the requested instant file, not only the inflight one, so a failed build leaves nothing pending on the data timeline. generateUniqueInstantTime reuses an indexing instant only while that action is pending, so a bootstrap keyed off a completed indexing instant takes a solo-family instant too instea d of aliasing another action's commit. A bootstrap left inflight by an indexer that died mid-build is no longer rolled back by the next writer once the indexer's heartbeat expires, since a solo-family commit matches neither that path nor eager rollback while the action is pending. It stays inflight in the table config, pending instants are excluded from reads, and the next run of the index picks a fresh offset and clears the partition first. ### Impact No public API change. Indexing actions on tables with several uninitialized definition-driven indexes now build the requested index. Regular writes, `HoodieIndexer` and `CREATE INDEX` for a first-time index behave as before. ### Risk Level low: the regular-write path passes an empty request and is unchanged. Validation: - The requested-partition guard is covered end to end by `TestHoodieIndexer.testIndexerFailsWhenTheOnlyUninitializedDefinitionIsNotTheRequestedIndex` and at resolver level by `TestSecondaryIndexer`; both were confirmed to fail with the guard reverted. - `TestHoodieIndexer.testIndexActionsBuiltOneAfterAnotherKeepEveryIndexPopulated` builds two secondary indexes through consecutive indexing actions with a synchronous-metadata write client and asserts the metadata timeline carries no rollback, each index bootstrapped under a solo-family instant, and each index holds one record per data record. With the bootstrap under the indexing instant it fails with the first index at zero records and one rollback on the metadata timeline. - `TestHoodieIndexer.testAnIndexBuildThatFailsMidwayIsAbortedAndCanBeRetried` defines a secondary index on a column the table does not have, runs the indexing action through a synchronous-metadata write client, asserts the abort leaves the index neither inflight nor complete, no indexing instant pending and no partition on storage, then fixes the definition and builds the index. It fails without the requested-instant deletion, as does the request-mismatch test's new assertion that the pending index timeline is empty. - `TestHoodieIndexer.testEagerWriterDoesNotRollBackAnInFlightRequestedIndexBootstrap` covers the exemption: a concurrent writer on the EAGER cleaning policy leaves a solo-family bootstrap alone while its indexing action is pending, and the same commit becomes rollback-eligible again once no indexing action is pending. - The abort path is covered by the request mismatch test, which now also asserts that nothing is left inflight in the table config or on the timeline after the failure, and was validated live in spark-shell: a genuine indexing action failed mid-catchup and left no inflight indexing instant and nothing inflight in the table config, where previously the partition would have stranded inflight. - The table-config re-read before the executor records its partitions has no automated test: it matters only when another writer completes a partition while the action runs, and the indexer refuses to schedule a partition before the files partition exists, so the shape cannot be produced through the indexer alone. - The skip-before-lookup change is covered by `TestSecondaryIndexer.testSkippingAnInitializedPartitionDoesNotConsultTheDefinitionLookup`. - `TestSecondaryIndexer` and `TestIndexActionExecutors` pass in full locally, as do the `TestHoodieIndexer` cases above. ### Documentation Update None. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
