leaves12138 commented on code in PR #7933:
URL: https://github.com/apache/paimon/pull/7933#discussion_r3411023890
##########
paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexScanner.java:
##########
@@ -74,29 +75,86 @@ public GlobalIndexScanner(
GlobalIndexReadThreadPool.getExecutorService(options.get(GLOBAL_INDEX_THREAD_NUM));
this.indexPathFactory = indexPathFactory;
GlobalIndexFileReader indexFileReader = meta ->
fileIO.newInputStream(meta.filePath());
- Map<Integer, Map<String, Map<Range, List<IndexFileMeta>>>> indexMetas
= new HashMap<>();
+ Map<Integer, IndexMetaFileGroup> indexMetas = new HashMap<>();
+ Map<Integer, List<IndexMetaFileGroup>> extraIndexMetas = new
HashMap<>();
for (IndexFileMeta indexFile : indexFiles) {
GlobalIndexMeta meta = checkNotNull(indexFile.globalIndexMeta());
- int fieldId = meta.indexFieldId();
String indexType = indexFile.indexType();
- indexMetas
- .computeIfAbsent(fieldId, k -> new HashMap<>())
- .computeIfAbsent(indexType, k -> new HashMap<>())
- .computeIfAbsent(
- new Range(meta.rowRangeStart(),
meta.rowRangeEnd()),
- k -> new ArrayList<>())
- .add(indexFile);
+ Range range = new Range(meta.rowRangeStart(), meta.rowRangeEnd());
+ int indexFieldId = meta.indexFieldId();
+ List<Integer> fieldIds = meta.getIndexedFieldIds();
+ IndexMetaFileGroup group = indexMetas.get(indexFieldId);
+ if (group == null) {
+ group = new IndexMetaFileGroup(indexFieldId, fieldIds);
+ indexMetas.put(indexFieldId, group);
+ if (meta.extraFieldIds() != null) {
+ for (int extra : meta.extraFieldIds()) {
+ extraIndexMetas.computeIfAbsent(extra, k -> new
ArrayList<>()).add(group);
+ }
+ }
+ } else {
+ checkArgument(
+ group.fieldIds.equals(fieldIds),
+ "Primary field %s owns multiple indexes with different
columns %s and %s; "
+ + "a primary column can own at most one
index.",
+ indexFieldId,
+ group.fieldIds,
+ fieldIds);
+ }
+ group.addFile(indexType, range, indexFile);
}
IntFunction<Collection<GlobalIndexReader>> readersFunction =
- fieldId ->
- createReaders(
- indexFileReader,
- indexMetas.get(fieldId),
- rowType.getField(fieldId));
+ fId -> {
+ IndexMetaFileGroup group = indexMetas.get(fId);
+ if (group != null) {
+ return createReaders(indexFileReader, group, rowType);
+ }
+ List<IndexMetaFileGroup> extraGroups =
extraIndexMetas.get(fId);
+ if (extraGroups == null || extraGroups.isEmpty()) {
+ return Collections.emptyList();
+ }
+ // Union readers from all groups that share this extra
column
+ List<GlobalIndexReader> allReaders = new ArrayList<>();
Review Comment:
This still does not actually union the extra-column groups.
`GlobalIndexEvaluator.visitLeafAsync` combines every reader returned for the
same leaf predicate with `and(...)`, while `UnionGlobalIndexReader` is the
layer that performs the OR/union. So returning one reader per extra group here
can turn `(a, b)` and `(c, b)` into `result(a,b) AND result(c,b)` for a
predicate on `b`, which may produce an empty/partial bitmap and then
`DataEvolutionBatchScan` pushes that bitmap down as row ranges. We should
either wrap the readers from all `extraGroups` into a single
`UnionGlobalIndexReader`, or otherwise make the evaluator combine these
alternative groups with OR instead of AND.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
