bhat-vinay opened a new pull request, #10792: URL: https://github.com/apache/hudi/pull/10792
There are a couple of issues in how functional indexes are managed. 1. HoodieSparkFunctionalIndexClient::create(...) was failing a register a functional index iff a (different) functional index was already created. Fixed this check by looking up the index-name in the FunctionalIndexMetadata 2. HoodieTableConfig `TABLE_METADATA_PARTITIONS` and `TABLE_METADATA_PARTITIONS_INFLIGHT` should actually store the Metadata partition path. While the path is contained in the `MeatadatPartitionType` for most of the indexes, it is not correct for functional-index. MeatadatPartitionType.FUNCTIONAL_INDEX onlt store the prefix (i.e func_index_). The actual partition path needs to be extracted from the index-name. 3. Because of #2, most of the helper methods that operate on metadata-partitions, should take partition-path (and not partition-type) This PR addresses the problem listed above. This fix is required to add SQL support for secondary-indexes (the configs for which will be based on functional-index-config). Note that there are still issues with some functional-index operations (like drop index / delete partition) because of the issues listed here. Those will be fixed in a subsequent PR. ### Change Logs There are a couple of issues in how functional indexes are managed. 1. HoodieSparkFunctionalIndexClient::create(...) was failing a register a functional index iff a (different) functional index was already created. Fixed this check by looking up the index-name in the FunctionalIndexMetadata 2. HoodieTableConfig `TABLE_METADATA_PARTITIONS` and `TABLE_METADATA_PARTITIONS_INFLIGHT` should actually store the Metadata partition path. While the path is contained in the `MeatadatPartitionType` for most of the indexes, it is not correct for functional-index. MeatadatPartitionType.FUNCTIONAL_INDEX onlt store the prefix (i.e func_index_). The actual partition path needs to be extracted from the index-name. 3. Because of #2, most of the helper methods that operate on metadata-partitions, should take partition-path (and not partition-type) This PR addresses the problem listed above. This fix is required to add SQL support for secondary-indexes (the configs for which will be based on functional-index-config). Note that there are still issues with some functional-index operations (like drop index / delete partition) because of the issues listed here. Those will be fixed in a subsequent PR. ### Impact None ### Risk level (write none, low medium or high below) Low ### Documentation Update NA ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
