chrevanthreddy commented on PR #20107: URL: https://github.com/apache/hudi/pull/20107#issuecomment-6431609632
Thanks @yihua, @vinishjail97 and the bot for the reviews. The changes and follow-up scope are summarized below; per-thread replies are included below. **What changed since the last review** - Partitioned RLI finalists are batched per RLI file group (yihua). - Bootstrap frontier follows the async-indexer cutoff (in-flight writers excluded). - Exact search supports all column types, reads MOR log rows on executors, and returns current MOR column values. - Query option validation, catalog-qualified names, NULL TVF args, bounded caches, plus dead code / duplication cleanup. **Scope / stack** 1. This PR: schema, RaBitQ, bootstrap, read path. 2. https://github.com/chrevanthreddy/hudi/pull/4: metadata-key routing for vector lookups (1B: first query 734s → 28.5s; repeat-query p50 17.3s → 7.0s approx, 24.5s → 14.8s exact; recall unchanged). 3. https://github.com/chrevanthreddy/hudi/pull/5: synchronous index maintenance (`buildUpdate`), with the lifecycle validation; depends on #19801. The shared-code changes include vector schema/payload support, index scheduling/configuration integration, and reuse of the existing Parquet row-read configuration. The payload slow path keeps its positional contract and adds a schema guard. MOR row positions come from the base file using the existing key/position utility, avoiding the proposed shared row-index ordinal fix. The separate routing PR changes `HoodieBackedTableMetadata`, restricted to vector partitions. **Validation** - 1B COW (BigANN, 16384 clusters, 4-bit RaBitQ, nprobes=32, refine=50): approx recall 0.865 / exact 0.95, p50 6.8s / 12.5s with https://github.com/chrevanthreddy/hudi/pull/4 (20 queries × 2 passes). - Lifecycle with exact == brute force before and after upsert/delete/clustering at 10M COW+MOR and 1B COW. - Local Spark 4.0 validation: all three stacked branches passed builds/style and the selected unit/end-to-end suites. Base/routing: 162 common, 12 hadoop-common, 40 client-common, 7 spark-client, 110 Spark tests each. Update: 164 / 12 / 50 / 26 / 114, all zero failures/errors. This run predates only test-output/comment cleanup and an EOF whitespace change; the two cleaned-up residual tests passed separately. - Routing's focused regression suite passed 8/8 separately (the original broad selector omitted it). - Spark 3.5 builds/style passed on all branches. Common tests: 228 / 236 / 238; hadoop-common: 12 each. Base/routing E2E: 110 each; update E2E: 114 with one retry-dependent in-flight-writer flake (passed three subsequent independent runs with retries disabled; root cause not established). Unmodified client selections encountered missing Hadoop shaded runtime classes. Matching-runtime diagnostic reruns passed client-common 45 / 45 / 55 and spark-client 13 / 13 / 35 on base/routing/update respectively; these are classpath-adjusted results, not unmodified-build passes. - Supplemental update Spark 4.0 tests passed: common 79, client 17, streaming 9, Spark-common 16. Flink 2.2 build passed; handler 2 passed and coordinator/event buffers 72 passed with 1 skipped. All results are local, not remote CI. Validation caveats are retained here and in the stacked PR descriptions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
