JingsongLi opened a new pull request, #11250:
URL: https://github.com/apache/paimon/pull/11250

   ### Purpose
   
   Fix scalar index coverage and residual filtering on row-tracked Data 
Evolution tables. A B-tree and a full-text provider can cover different row 
ranges, including on the same column. Using the presence of any index on that 
column to prune splits can lose matching rows or read rows outside the scalar 
index coverage.
   
   - Resolve coverage for providers that also support ranked search before fast 
mode drops splits. Reuse a provider's scalar matches when it supplies them. 
Full and detail modes include the ranges left uncovered by the participating 
index.
   - Track which predicate branches an index evaluated. A ranked query with 
both `name IS NOT NULL` and `lower(name) = 'beta'` must apply the second 
condition before selecting the nearest row.
   - Document the per-file source row count on `ResultEntry`. Flink already 
counts the rows it reads and logs the shard total. Comparing that total with 
the first output file raises a spurious warning for writers which produce 
multiple files.
   
   The planner can perform a scalar index lookup for a provider that also 
supports full-text or vector search, even with 
`global-index.query-in-reader.enabled=true`. Built-in scalar indexes can still 
run in the readers.
   
   ### Tests
   
   Added regressions using B-tree, full-text and vector test indexes on 
existing table formats. On master, the split-coverage cases failed with 
reader-side lookup both enabled and disabled, and the ranked query returned row 
0 instead of row 1. The new tests also check per-file counts for a two-file 
build.
   
   All 280 focused tests passed across the common, core and Flink modules.
   
   Extracted from #11238 so that the global index behavior can be reviewed and 
merged first.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to