This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 913dab328c1 branch-4.2: [fix](fe) Pick the first matching Variant
field pattern when choosing an index, like the BE #69730 (#69853)
913dab328c1 is described below
commit 913dab328c112072e774fa721dffcb1602a60747
Author: Jack <[email protected]>
AuthorDate: Fri Oct 30 09:40:37 2026 +0800
branch-4.2: [fix](fe) Pick the first matching Variant field pattern when
choosing an index, like the BE #69730 (#69853)
### What problem does this PR solve?
Issue Number: None
Related PR: #69730
Problem Summary: Backport #69730 to `branch-4.2`.
A VARIANT column can declare predefined fields whose patterns overlap,
for example the glob 'a.*' and the name 'a.b'. The BE writer
(variant_util.cpp) routes a subcolumn to the first predefined field
whose pattern matches its path and builds the index of that field.
`OlapTable.getInvertedIndex` looped over all the fields without stopping
and kept the last match, so a query on the same path chose the index of
a different field than the one the data was written with, and fell back
to another index or to a scan.
The loop now stops at the first match, the same order the BE uses, so a
write and a query on one path land on the same index.
#### Branch adaptations
On `branch-4.2` the predefined field's pattern type is the thrift
`TPatternType` (master has `catalog.PatternType`), and `Index` takes
`IndexDef.IndexType`. The `OlapTable.java` change applies unchanged
around those lines. The test is master's test with two adaptations: it
uses `TPatternType` and keeps the branch's own `IndexType` import, and
it builds its lists and maps with Guava's `ImmutableList.of` /
`ImmutableMap.of`, like the other tests of the file, because this
branch's fe-core tests cannot use `List.of` / `Map.of`.
The BE of `branch-4.2` also routes a subcolumn to the first matching
predefined field: `generate_sub_column_info` in `variant_util.cpp`
returns on the first match. So the FE change has the same effect on this
branch. The commit is the cherry-pick of master `7832338fba`.
### Release note
Fix the index chosen for a Variant subpath when the column declares
overlapping field patterns.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- The test of #69730:
InvertedIndexPropertiesTest.testOverlappingFieldPatternsPickTheFirstPredefinedFieldLikeTheBe.
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. With overlapping field patterns, the first declared match
decides the index instead of the last.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Validation
- `InvertedIndexPropertiesTest` on this branch, locally: 28 tests pass,
including the new
`testOverlappingFieldPatternsPickTheFirstPredefinedFieldLikeTheBe`; FE
Checkstyle passes in the same build.
- With `branch-4.2`'s own `OlapTable.java` put back, the new test fails
with "expected idx_glob but was idx_exact", which is the last-match
behavior this change removes.
- CI on `branch-4.2` (`run buildall`) passed: compile, FE unit tests
(12000 passed, 65 ignored, including the new test), P0, vault P0,
external and non-concurrent regression, and the FE coverage check (100%
of the 2 changed lines).
- The first cloud P0 run failed 18 tests because the CI agent ran out of
system memory: queries were cancelled with `MEM_LIMIT_EXCEEDED` ("sys
available memory less than low water mark") and allocations failed. The
rerun (`run cloud_p0`) passed with 4975 tests.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../java/org/apache/doris/catalog/OlapTable.java | 7 +++++-
.../analysis/InvertedIndexPropertiesTest.java | 25 ++++++++++++++++++++++
2 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 6b2dd7fa1d3..85d27de8c40 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -3897,7 +3897,10 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf, GsonPostProc
:
filteredInvertedIndexes.stream().filter(Index::isAnalyzedInvertedIndex).findFirst().orElse(null);
}
- // subPath is not empty, means it is a variant column, find the field
pattern from children
+ // subPath is not empty, means it is a variant column, find the field
pattern from children.
+ // The first match wins, the same as the BE writer that picks the
physical index of a subcolumn
+ // (variant_util.cpp), so a write and a query on one path land on the
same index even when
+ // the predefined patterns overlap.
String subPathString = String.join(".", subPath);
String fieldPattern = "";
if (column.getChildren() != null) {
@@ -3909,6 +3912,7 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf, GsonPostProc
.getPathMatcher("glob:" + childName);
if
(matcher.matches(java.nio.file.Paths.get(subPathString))) {
fieldPattern = childName;
+ break;
}
} catch (Exception e) {
continue;
@@ -3916,6 +3920,7 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf, GsonPostProc
} else if (child.getFieldPatternType() ==
TPatternType.MATCH_NAME) {
if (childName.equals(subPathString)) {
fieldPattern = childName;
+ break;
}
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java
b/fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java
index 7ad2b49f704..f9ec838b43b 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexPropertiesTest.java
@@ -25,6 +25,8 @@ import org.apache.doris.catalog.Index;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Type;
+import org.apache.doris.catalog.VariantField;
+import org.apache.doris.catalog.VariantType;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.indexpolicy.IndexPolicy;
import org.apache.doris.indexpolicy.IndexPolicyMgr;
@@ -35,6 +37,7 @@ import
org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
import org.apache.doris.nereids.types.StringType;
import org.apache.doris.thrift.TExprNode;
import org.apache.doris.thrift.TInvertedIndexFileStorageFormat;
+import org.apache.doris.thrift.TPatternType;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
@@ -683,6 +686,28 @@ public class InvertedIndexPropertiesTest {
ImmutableMap.of("analyzer", "ik", "lower_case", "false"),
"ik"));
}
+ @Test
+ public void
testOverlappingFieldPatternsPickTheFirstPredefinedFieldLikeTheBe() {
+ // The BE writer routes a subcolumn to the first predefined field
whose pattern matches its path,
+ // so the query side has to pick the index of that same field when the
patterns overlap.
+ Index globIndex = new Index(30, "idx_glob", ImmutableList.of("v"),
IndexType.INVERTED,
+ ImmutableMap.of("field_pattern", "a.*"), "");
+ Index exactIndex = new Index(31, "idx_exact", ImmutableList.of("v"),
IndexType.INVERTED,
+ ImmutableMap.of("field_pattern", "a.b"), "");
+ VariantField glob = new VariantField("a.*", Type.STRING, "",
TPatternType.MATCH_NAME_GLOB);
+ VariantField exact = new VariantField("a.b", Type.STRING, "",
TPatternType.MATCH_NAME);
+ OlapTable table = new OlapTable();
+ table.setIndexes(ImmutableList.of(globIndex, exactIndex));
+
+ Column globFirst = new Column("v", new VariantType(new
ArrayList<>(ImmutableList.of(glob, exact))));
+ Assertions.assertSame(globIndex, table.getInvertedIndex(globFirst,
ImmutableList.of("a", "b")));
+
+ Column exactFirst = new Column("v", new VariantType(new
ArrayList<>(ImmutableList.of(exact, glob))));
+ Assertions.assertSame(exactIndex, table.getInvertedIndex(exactFirst,
ImmutableList.of("a", "b")));
+ // A path only the glob covers still resolves to the glob's index.
+ Assertions.assertSame(globIndex, table.getInvertedIndex(exactFirst,
ImmutableList.of("a", "c")));
+ }
+
@Test
public void testExplicitBuiltinIkFallsBackToTheOnlyLegacyIndex() {
Column column = new Column("content", PrimitiveType.STRING);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
