Re: [PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
wombatu-kun commented on PR #11074: URL: https://github.com/apache/hudi/pull/11074#issuecomment-2072072439 > @wombatu-kun The refactoring done here is useful. We can add pre-CBO rules later. I'm simply asking to breakdown [HUDI-7639](https://issues.apache.org/jira/browse/HUDI-7639) to two

Re: [PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
codope commented on PR #11074: URL: https://github.com/apache/hudi/pull/11074#issuecomment-2072006646 > So I've just created hierarchy of IndexSupport that is usable without if-else branches, is easy to extend with new types of indices and it works with Spark <3.1. And to be honest I don't

Re: [PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
wombatu-kun commented on PR #11074: URL: https://github.com/apache/hudi/pull/11074#issuecomment-2071996001 @codope huh, probably i misunderstood the task. I focused on " All indexes are being used in lookupCandidateFilesInMetadataTable method through if-else branches. This is not only hard

Re: [PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
hudi-bot commented on PR #11074: URL: https://github.com/apache/hudi/pull/11074#issuecomment-2071826728 ## CI report: * 3163b1edc784fd9d9e0cb303fde89db9d328e185 Azure:

Re: [PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
hudi-bot commented on PR #11074: URL: https://github.com/apache/hudi/pull/11074#issuecomment-2071708406 ## CI report: * 3163b1edc784fd9d9e0cb303fde89db9d328e185 Azure:

Re: [PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
hudi-bot commented on PR #11074: URL: https://github.com/apache/hudi/pull/11074#issuecomment-2071630522 ## CI report: * 3163b1edc784fd9d9e0cb303fde89db9d328e185 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run

[PR] [HUDI-7639] Refactor HoodieFileIndex so that different indexes can be used via optimizer rules [hudi]

2024-04-23 Thread via GitHub
wombatu-kun opened a new pull request, #11074: URL: https://github.com/apache/hudi/pull/11074 ### Change Logs Task: https://issues.apache.org/jira/browse/HUDI-7639 Created new abstract class SparkBaseIndexSupport with abstract methods `getIndexName`, `isIndexAvailable`,