rangareddy commented on issue #17101: URL: https://github.com/apache/hudi/issues/17101#issuecomment-5391459145
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9613). **Findings: confirmed, the pattern is still there.** `hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java:522`: ```java ValidationUtils.checkState(secondaryKeys instanceof HoodieListData, "readSecondaryIndex only support HoodieListData at the moment"); ``` with a second implementation-driven branch at `:550` (`if (keys instanceof HoodieListData)`). So an API that advertises `HoodieData` still asserts a specific in-memory implementation at runtime. A caller passing an RDD-backed `HoodieData` fails outright rather than scaling - which is the "day 1 the API took a java collection" history the description describes, still visible in the type check. **Three sibling tickets in this same batch are the same underlying problem**, and they are much cheaper to fix as one piece of work than four: - #17052 (HUDI-9515) - index lookup APIs require the lookup set to fit on one machine; needs an RDD-based implementation - #17063 (HUDI-9542) - `HoodieListData` / `HoodieListPairData` need a notion of partitions so they stop diverging from `JavaRDD` - #17064 (HUDI-9544) - the RLI metric does not work for RDD because of lazy evaluation All four are the `HoodieData` abstraction not being honoured end to end. Recommend a single design covering what `HoodieData` guarantees, then removing these `instanceof` assertions as a consequence. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
