CrownChu opened a new pull request, #8547:
URL: https://github.com/apache/paimon/pull/8547
### Purpose
A global index may contain one primary field and multiple extra fields.
However, the current scan paths assume that:
1. a full-text search column must be the primary index field; and
2. a vector-primary index cannot also serve scalar predicates on its extra
fields.
These assumptions prevent multi-field indexes from serving full-text and
scalar queries correctly. They may also cause indexed row ranges to be skipped
when different index definitions cover different ranges of the same column.
This PR makes the generic global-index scan and read paths multi-field
aware:
- Recognize full-text columns in both `indexFieldId` and `extraFieldIds`.
- Select one index definition for each `(column, row range)`.
- Prefer a dedicated full-text index when multiple indexes cover the same
range.
- Preserve ranges that are covered only by an extra-field index.
- Create the global indexer for each split from that split's own metadata.
- Allow a vector-primary multi-field index to serve scalar predicates on
its extra fields.
- Preserve raw vector-search fallback when the vector column is not the
primary field.
- Prune redundant `IS NOT NULL` predicates only when another predicate on
the same field is null-rejecting, preserving the semantics of expressions such
as:
`field IS NULL AND field IS NOT NULL`.
The changes are backend-independent and do not introduce dependencies on
any specific global-index implementation.
### Tests
- `GlobalIndexEvaluatorTest`: 19 tests passed.
- Added and verified full-text search coverage for:
- a text column stored as an extra field;
- preferring a dedicated full-text index;
- ranges covered only by an extra-field index;
- indexes that do not contain the queried text column.
- Relevant `FullTextSearchBuilderTest` cases: 4 tests passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
