somandal opened a new pull request, #9333: URL: https://github.com/apache/pinot/pull/9333
This PR adds an option to disable the forward index for a given column via the FieldConfig properties list. This is a PR to solve issue https://github.com/apache/pinot/issues/6473. This PR does not allow the forward index to be disabled yet by adding a check in the `TableConfig`. This feature will be allowed once we address the reload code path to toggle the flag to disable the forward index. To disable the forward index, the following mandates have been added at the moment (depending on the usage of this feature we may decide to relax some of them over time): - The inverted index for the column must be enabled - The column must be a dictionary based column - The column cannot be sorted - The column cannot have a V1 type range index present The above checks have been added in the code in the following places: - TableConfig validator - Segment reload path (check when creating the default column) - Segment creation path The ability to disable the forward index is currently implemented for the following cases: - New immutable segments generated for a column (all refresh use cases and reload use cases for new segments) - Add a default value column with the forward index disabled This PR does **not** add support for disabling the forward index for: - Mutable segments - Enabling / disabling the disable forward index flag for an existing column -> we will add support for this reload case in a follow up PR shortly after this PR Validations with other indices can be broadly classified into the following paths: - Query path - ensure that none of the other indices rely on the forward index - We found the range index v1 resorts to a scan on the forward index, thus disabled this feature if a range index of version 1 is present - Sorted columns rely on the forward index as the inverted index as well. We disabled this feature if the column is sorted - Segment creation path does not rely on the forward index to create other indices. The `indexRow` function creates all the indices needed at the same time for a given row. - Segment reload path relies on the forward index to add new indices such as range index, text index, etc. This path cannot be supported until we address enabling/disabling this feature on the reload path and will be done as part of a future PR along with any additional validations needed. cc @siddharthteotia @Jackie-Jiang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
