Gabriel39 commented on code in PR #68687:
URL: https://github.com/apache/doris/pull/68687#discussion_r4151627770
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScalarIndexPlanner.java:
##########
@@ -131,7 +162,17 @@ private static Plan groupFragments(LanceTableMetadata
metadata, List<LanceIndexS
coveredRows += Math.max(fragment.getPhysicalRows(), 0);
}
if (!fragments.isEmpty()) {
- splits.addIndexSegmentSplit(segment.getUuid(), fragments,
physicalRows);
+ // Complement selectivity is unknown. Preserve fragment-level
scheduling so a
+ // broad NOT cannot funnel all covered data through one BE.
Each task retains
+ // the same segment UUID but owns a disjoint domain, including
on fallback.
+ if (splitComplementByFragment) {
Review Comment:
Fixed in f5b9247b28e. Segment grouping now requires a positive, indexable
necessary condition; a separate complement no longer splits that segment into
repeated searches. Complement-only scans retain one task per fragment with
scalar-index use disabled. Added a two-fragment test for both a combined AND
tree and separate pushed conjuncts. Actual FE plans executed through lance-c
return the expected rows with exactly one segment search for the mixed
predicate, including partial coverage.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScalarIndexPlanner.java:
##########
@@ -61,18 +64,21 @@ static Plan plan(LanceTableMetadata metadata, List<Expr>
pushedConjuncts,
// provides a stable winner when multiple indices cover the same
number of rows.
List<LanceIndexSegmentGroup> indices = new
ArrayList<>(metadata.getIndexes());
indices.sort(java.util.Comparator.comparing(LanceIndexSegmentGroup::getName));
+ boolean splitComplementByFragment = pushedConjuncts.stream()
+ .anyMatch(LanceScalarIndexPlanner::containsComplement);
Plan selected = null;
for (LanceIndexSegmentGroup logicalIndex : indices) {
List<LanceIndexSegmentInfo> segments = logicalIndex.getSegments();
- // PR #79 supports one top-level key in BTree/Bitmap/LabelList
indices. Lance
+ // Segment-scoped scans support one top-level key in
BTree/Bitmap/LabelList indices. Lance
// performs the final typed driver selection and falls back within
the same domain.
LanceIndexSegmentInfo index = segments.get(0);
if ((index.getIndexType() != IndexType.BTREE &&
index.getIndexType() != IndexType.BITMAP
&& index.getIndexType() != IndexType.LABEL_LIST)
|| index.getFieldIds().size() != 1 ||
!filterFields.contains(index.getFieldIds().get(0))) {
continue;
}
- Plan candidate = groupFragments(metadata, segments,
visibleFragments);
+ Plan candidate = groupFragments(metadata, segments,
visibleFragments,
Review Comment:
Fixed in f5b9247b28e. FE now bounds the translated scalar expression against
the pinned native 128-node/32-depth limits before grouping. A 64-label overlap
can use one segment search; 65 labels keep parallel fragment scans with the
full filter pushed and scalar-index use disabled. Added unit tests for the
boundary, enclosing conjuncts, and depth, plus SQL regression assertions for
fragment grouping and runtime counters. Native verification of FE-produced
plans confirmed identical expected rows, one search at 64 labels, and two
fragment scans with zero searches/fallbacks at 65.
##########
thirdparty/vars.sh:
##########
@@ -580,11 +580,11 @@ PUGIXML_SOURCE=pugixml-1.15
PUGIXML_MD5SUM="3b894c29455eb33a40b165c6e2de5895"
# lance-c
-# Complete-segment prefilter fixes are supplied by upstream lance-c, not local
patches.
-LANCE_C_DOWNLOAD="https://codeload.github.com/lance-format/lance-c/tar.gz/9bd730add2ac70316c1d642b8459011e2dd92022";
-LANCE_C_NAME="lance-c-9bd730add2ac70316c1d642b8459011e2dd92022.tar.gz"
-LANCE_C_SOURCE="lance-c-9bd730add2ac70316c1d642b8459011e2dd92022"
-LANCE_C_MD5SUM="63851b09bf1689032579f1a094ff2f37"
+# Includes lance-c #93: scoped boolean scalar-index expressions and Substrait
label filters.
+LANCE_C_DOWNLOAD="https://codeload.github.com/lance-format/lance-c/tar.gz/cd63420bfbe27f6f0a1edcc873b9191af7d52852";
Review Comment:
Fixed in f5b9247b28e and synchronized to master PR #68689 in c2858482cf4.
The build compares the installed source/archive/patch fingerprint with the
current checkout and checks nonempty Lance headers/archive. Installation
invalidates the marker before publishing either file and writes it only after
success; stale external builder output is rejected. The real
build-gate/publication harness passes on both branches, covering
unversioned/stale/matching installs, pin and patch changes, incomplete
artifacts, missing sources, interrupted publication and retry.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScalarIndexPlanner.java:
##########
@@ -81,29 +87,54 @@ static Plan plan(LanceTableMetadata metadata, List<Expr>
pushedConjuncts,
}
private static Set<Integer> collectFilterFields(LanceTableMetadata
metadata, List<Expr> pushedConjuncts) {
+ Set<Integer> fields = new HashSet<>();
+ for (Expr expr : pushedConjuncts) {
+ fields.addAll(collectDriverFields(metadata, expr, false));
+ }
+ return fields;
+ }
+
+ private static Set<Integer> collectDriverFields(LanceTableMetadata
metadata, Expr expr, boolean complete) {
+ if (expr instanceof CompoundPredicate) {
+ CompoundPredicate.Operator op = ((CompoundPredicate) expr).getOp();
+ if (op == CompoundPredicate.Operator.NOT) {
+ // NOT cannot complement a pruned AND: that would discard
valid matches.
+ return collectDriverFields(metadata, expr.getChild(0), true);
+ }
+ Set<Integer> left = collectDriverFields(metadata,
expr.getChild(0), complete);
+ Set<Integer> right = collectDriverFields(metadata,
expr.getChild(1), complete);
+ if (op == CompoundPredicate.Operator.AND && !complete) {
+ left.addAll(right);
+ } else {
+ // One selected index must supply candidates for both OR
branches, and for
+ // every leaf of a negated subtree. lance-c rechecks the
complete predicate.
+ left.retainAll(right);
Review Comment:
Fixed in f5b9247b28e. Driver selection now checks positive leaf indexability
for the selected index type before intersecting OR branches. Suffix/contains
LIKE cannot justify grouping, while simple BTree prefixes remain supported.
Added a multi-fragment unit case for equality OR suffix LIKE and a prefix
control. Executing actual FE plans through lance-c confirmed two fragment scans
and zero searches for the suffix case, versus one segment search for the prefix
case, with expected rows in both cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
