rangareddy commented on issue #17063: URL: https://github.com/apache/hudi/issues/17063#issuecomment-5391452801
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9542). **Findings: still open.** Partially addressed. `HoodieListData.mapPartitions(SerializableFunction<Iterator<T>, Iterator<O>> func, boolean preservesPartitioning)` now exists at `hudi-common/src/main/java/org/apache/hudi/common/data/HoodieListData.java:150`, so the map-partitions capability is there. What is not there is the rest of the ask: a partitioner interface, an abstraction over partitioning, and parity with RDD partitioning semantics. Without those, `preservesPartitioning` is a parameter the list implementation cannot meaningfully honour - which is worth flagging, because a caller passing `true` gets silence rather than an error. **This is one of four related tickets**, all the same underlying problem - the `HoodieData` abstraction is not honoured end to end, so callers cannot pass a distributed implementation where the API says they can: - #17052 (HUDI-9515) - index lookup requires the set to fit on one machine; needs an RDD-based implementation - #17063 (HUDI-9542) - `HoodieListData` / `HoodieListPairData` need a notion of partitions - #17064 (HUDI-9544) - the RLI metric does not fire for RDD because of lazy evaluation - #17101 (HUDI-9613) - `HoodieBackedTableMetadata` asserts `instanceof HoodieListData` at runtime Fixing them independently risks four partial answers. Recommend one design covering what `HoodieData` guarantees - partitioning, laziness, and which implementations each API accepts - with these as its consequences. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
