rangareddy commented on issue #17064: URL: https://github.com/apache/hudi/issues/17064#issuecomment-5391453358
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9544). **Findings: still open.** The method named in the description, `HoodieBackedTableMetadata#readRecordIndexWithMapping`, still exists. The reported cause is structural rather than a single defect: metrics recorded inside a lazily-evaluated RDD transformation never fire if the caller only builds the RDD and never forces it. That means the fix is a design choice - record the metric in an action, use an accumulator, or document that the metric is only valid after materialisation - rather than a line to change. **This is one of four related tickets**, all the same underlying problem - the `HoodieData` abstraction is not honoured end to end, so callers cannot pass a distributed implementation where the API says they can: - #17052 (HUDI-9515) - index lookup requires the set to fit on one machine; needs an RDD-based implementation - #17063 (HUDI-9542) - `HoodieListData` / `HoodieListPairData` need a notion of partitions - #17064 (HUDI-9544) - the RLI metric does not fire for RDD because of lazy evaluation - #17101 (HUDI-9613) - `HoodieBackedTableMetadata` asserts `instanceof HoodieListData` at runtime Fixing them independently risks four partial answers. Recommend one design covering what `HoodieData` guarantees - partitioning, laziness, and which implementations each API accepts - with these as its consequences. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
