XiaoHongbo-Hope commented on code in PR #9050:
URL: https://github.com/apache/paimon/pull/9050#discussion_r3722021426
##########
paimon-python/pypaimon/globalindex/data_evolution_global_index_scanner.py:
##########
@@ -103,24 +106,33 @@ def _create_evaluator(self, fields, file_io, index_path,
index_files):
options = self._options
def readers_function(field: DataField) ->
Collection[GlobalIndexReader]:
+ groups = []
group = index_metas.get(field.id)
if group is not None:
- return _create_readers(
- file_io, index_path, group.metas, field, executor, options)
+ groups.append(group)
extra_groups = extra_index_metas.get(field.id)
- if not extra_groups:
+ if extra_groups:
+ groups.extend(
+ extra_group
+ for extra_group in extra_groups
+ if extra_group not in groups
+ )
+ if not groups:
return []
+ if len(groups) == 1:
+ return _create_readers(
+ file_io, index_path, groups[0].metas, field, executor,
options)
union_coverage = Range.sort_and_merge_overlap(
[
range_key
- for group in extra_groups
+ for group in groups
for range_key in group.coverage_ranges
],
True,
)
readers = []
- for group in extra_groups:
+ for group in groups:
pad_ranges = _exclude_ranges(union_coverage,
group.coverage_ranges)
readers.extend(
_create_readers(
Review Comment:
> [P1] This merge can let an unsupported alternate index poison an otherwise
usable primary reader. For example, if `c` has a dedicated BTree index and is
also a companion/extra field of a Java-built multi-column `es-index(a, c)`, the
base branch returns the BTree result, but this loop also instantiates the
`es-index` group and `_create_inner_readers` raises `ValueError` because
PyPaimon does not support that index type. `FileScanner` silently loses
pruning, while the indexed and raw vector pre-filter paths propagate the
exception and fail the query. Please make alternate selection capability-aware
while keeping coverage conservative: either exclude unreadable coverage or
represent those ranges as all-hit/fallback padding, and add a regression test
with a supported primary plus a real unsupported extra-field index.
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
