JingsongLi commented on code in PR #8000:
URL: https://github.com/apache/paimon/pull/8000#discussion_r3519653254
##########
paimon-core/src/main/java/org/apache/paimon/table/source/FullTextScanImpl.java:
##########
@@ -102,16 +103,34 @@ public Plan scan() {
.map(IndexManifestEntry::indexFile)
.collect(Collectors.toList());
- // Group full-text index files by column and row range.
+ // A searched text column can be covered by more than one
full-text-capable global index
+ // (e.g. a dedicated full-text index whose primary field is the
column, and a vector index
+ // that carries the column as an extra field). These are different
index definitions and
+ // must not be merged into one reader input. Pick exactly one index
definition per column,
+ // preferring the one where the column is the primary indexFieldId
(dedicated full-text)
+ // over an extra-field match, so every split carries files from a
single index identity.
+ Map<String, IndexIdentity> chosenByColumn =
+ chooseIndexPerColumn(allIndexFiles, textColumnIds, idToColumn);
+
+ // Group full-text index files by column and row range. A multi-column
index serves a text
+ // column through either its primary indexFieldId or its
extraFieldIds, and one file may
+ // cover more than one searched text column.
Map<String, Map<Range, List<IndexFileMeta>>> byColumnAndRange = new
HashMap<>();
for (IndexFileMeta indexFile : allIndexFiles) {
GlobalIndexMeta meta = checkNotNull(indexFile.globalIndexMeta());
- String columnName =
checkNotNull(idToColumn.get(meta.indexFieldId()));
+ IndexIdentity identity = IndexIdentity.of(indexFile);
Range range = new Range(meta.rowRangeStart(), meta.rowRangeEnd());
- byColumnAndRange
- .computeIfAbsent(columnName, k -> new HashMap<>())
- .computeIfAbsent(range, k -> new ArrayList<>())
- .add(indexFile);
+ for (int columnId : matchedTextColumnIds(meta, textColumnIds)) {
+ String columnName = checkNotNull(idToColumn.get(columnId));
+ if (!identity.equals(chosenByColumn.get(columnName))) {
Review Comment:
This selection is global for the whole column, so it can drop valid ranges
from the non-chosen identity. For example, suppose a dedicated full-text index
on `content` covers rows [0, 9], and an ES/vector index with `content` in
`extraFieldIds` covers rows [10, 19]. `chooseIndexPerColumn` picks the
dedicated identity, this branch skips the ES file for [10, 19], and the raw
fallback below still builds `GlobalIndexCoverage` from `allIndexFiles`; because
coverage counts `extraFieldIds`, [10, 19] is considered indexed and is not
added as a raw range. A full-text query then misses matching rows in [10, 19].
Please choose the serving identity per column/range, or compute raw coverage
from the same files that are actually emitted as searchable splits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
