JingsongLi commented on code in PR #8000:
URL: https://github.com/apache/paimon/pull/8000#discussion_r3503442235
##########
paimon-core/src/main/java/org/apache/paimon/table/source/FullTextScanImpl.java:
##########
@@ -101,16 +101,20 @@ public Plan scan() {
.map(IndexManifestEntry::indexFile)
.collect(Collectors.toList());
- // Group full-text index files by column and row range.
+ // Group full-text index files by column and row range. A multi-column
index serves a text
+ // column through either its primary indexFieldId or its
extraFieldIds, and one file may
+ // cover more than one searched text column.
Map<String, Map<Range, List<IndexFileMeta>>> byColumnAndRange = new
HashMap<>();
for (IndexFileMeta indexFile : allIndexFiles) {
GlobalIndexMeta meta = checkNotNull(indexFile.globalIndexMeta());
- String columnName =
checkNotNull(idToColumn.get(meta.indexFieldId()));
Range range = new Range(meta.rowRangeStart(), meta.rowRangeEnd());
- byColumnAndRange
- .computeIfAbsent(columnName, k -> new HashMap<>())
- .computeIfAbsent(range, k -> new ArrayList<>())
- .add(indexFile);
+ for (int columnId : matchedTextColumnIds(meta, textColumnIds)) {
+ String columnName = checkNotNull(idToColumn.get(columnId));
+ byColumnAndRange
+ .computeIfAbsent(columnName, k -> new HashMap<>())
+ .computeIfAbsent(range, k -> new ArrayList<>())
+ .add(indexFile);
Review Comment:
This grouping can put multiple unrelated full-text-capable index definitions
into the same split when they cover the same text column and row range. For
example, a table can have a normal full-text index whose primary field is
`content` and a hybrid/vector ES index whose primary field is another column
but has `content` in `extraFieldIds`; both match `matchedTextColumnIds(...)`
and end up in the same `FullTextSearchSplit`. `FullTextReadImpl` then picks the
first file index type/meta and passes the whole list to one backend reader, but
`ESIndexGlobalIndexReader` still requires `files.size() == 1` and will fail for
this valid layout. Please keep separate splits/readers per index identity (for
example index type + primary field + extra field set) or otherwise avoid
merging files from different global index definitions into one reader input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
