This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new da1eb726de9 [improvement](parser) Left factor primary expression
grammar (#66951)
da1eb726de9 is described below
commit da1eb726de9d0524df45678f73ff6d545c706829
Author: morrySnow <[email protected]>
AuthorDate: Wed Sep 2 16:31:30 2026 +0800
[improvement](parser) Left factor primary expression grammar (#66951)
### What problem does this PR solve?
Problem Summary: The `primaryExpression` rule encoded CASE and CONVERT
branches with repeated prefixes, split query and expression parentheses
into competing alternatives, and represented array access, dereference,
and COLLATE through direct left recursion. This change factors the rule
into a primary base plus ordered postfix suffixes, consolidates the
shared prefixes, and folds suffixes left-to-right in the FE visitor. It
preserves SQL AST semantics, invalid-input positions, Create View source
intervals, and the lexical placeholder order of simple CASE expressions
while allowing the independent parser CST to change.
### Benchmark
This is a targeted JMH measurement of the standalone SQL parser. The
table reports the mean of two complete rounds run in `B1-C1-C2-B2`
order; lower latency is better.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, 1 GB heap
- JMH: 1 thread, AverageTime in us/op, 3 forks, 4 x 300 ms warmup, 7 x
400 ms measurement, `-prof gc`
- Harness:
`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java`;
the build and run command is documented in `fe/fe-sql-parser/README.md`
- Baseline: `0fb9b450865`; parser jar SHA-256
`923ed2a22142ee9b5dcfefbba5766b9696a653a4218e8208e42a61270e9d986f`
- Candidate grammar: `479e33ad4c8`; parser jar SHA-256
`7ab3247fc905dcb519f80fd7a95f3a12e038bdc9d3b25a0825eeda2fb7df321d`
- The follow-up simple CASE visitor-order fix does not change the
standalone parser grammar or these parser measurements.
#### Parser facade (lexing and parsing)
| Workload | Baseline (μs/op) | Candidate (μs/op) | Latency
Improvement | Allocation Change |
| :------------------ | ---------------: | ----------------: |
------------------: | ----------------: |
| Control | 2.7298 | 2.1890 | **19.81%
faster** | -0.48% |
| Typical expressions | 14.3809 | 9.8846 | **31.27%
faster** | +6.76% |
| CASE / CONVERT | 19.8491 | 12.9693 | **34.66%
faster** | +9.02% |
| Postfix chain | 15.3586 | 9.1795 | **40.23%
faster** | +3.54% |
| Wide projection | 131.1269 | 81.4518 | **37.88%
faster** | +9.60% |
#### Parser only (pre-tokenized input)
| Workload | Baseline (us/op) | Candidate (us/op) | Latency
Improvement | Allocation Change |
| :------------------ | ---------------: | ----------------: |
------------------: | ---------------------: |
| Control | 2.0580 | 1.6313 | **20.73%
faster** | +72 B/op (+1.86%) |
| Typical expressions | 13.0225 | 8.3392 | **35.96%
faster** | +1,120 B/op (+7.92%) |
| CASE / CONVERT | 18.6694 | 11.7539 | **37.04%
faster** | +2,087 B/op (+11.17%) |
| Postfix chain | 13.5557 | 7.5474 | **44.32%
faster** | +789 B/op (+5.77%) |
| Wide projection | 123.3741 | 79.0044 | **35.96%
faster** | +18,108 B/op (+13.36%) |
The complex-expression workloads reduced measured parser latency by
about 31%-40% through the public facade and 36%-44% in parser-only
measurements. The trade-off is additional short-lived CST allocation:
about 4%-10% through the facade and 6%-13% parser-only for those
workloads. `gc.alloc.rate.norm` measures total transient allocation, not
retained heap; the CST normally becomes collectible after AST
construction.
---
.../doris/nereids/parser/LogicalPlanBuilder.java | 105 +++++++-------
.../parser/LogicalPlanBuilderForCreateView.java | 9 +-
.../nereids/parser/PrimaryExpressionAstTest.java | 128 +++++++++++++++++
.../benchmark/PrimaryExpressionBenchmark.java | 114 +++++++++++++++
fe/fe-sql-parser/README.md | 4 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 31 +++--
.../doris/sqlparser/PrimaryExpressionTest.java | 153 +++++++++++++++++++++
7 files changed, 474 insertions(+), 70 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index a958d878fd2..55e92d7c0d8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -127,8 +127,8 @@ import
org.apache.doris.nereids.DorisParser.AlterWorkloadGroupContext;
import org.apache.doris.nereids.DorisParser.AlterWorkloadPolicyContext;
import org.apache.doris.nereids.DorisParser.ArithmeticBinaryContext;
import org.apache.doris.nereids.DorisParser.ArithmeticUnaryContext;
+import org.apache.doris.nereids.DorisParser.ArrayAccessContext;
import org.apache.doris.nereids.DorisParser.ArrayLiteralContext;
-import org.apache.doris.nereids.DorisParser.ArraySliceContext;
import org.apache.doris.nereids.DorisParser.BaseTableRefContext;
import org.apache.doris.nereids.DorisParser.BooleanExpressionContext;
import org.apache.doris.nereids.DorisParser.BooleanLiteralContext;
@@ -216,7 +216,6 @@ import
org.apache.doris.nereids.DorisParser.DropTableContext;
import org.apache.doris.nereids.DorisParser.DropUserContext;
import org.apache.doris.nereids.DorisParser.DropWorkloadGroupContext;
import org.apache.doris.nereids.DorisParser.DropWorkloadPolicyContext;
-import org.apache.doris.nereids.DorisParser.ElementAtContext;
import org.apache.doris.nereids.DorisParser.EnableFeatureClauseContext;
import org.apache.doris.nereids.DorisParser.ExceptContext;
import org.apache.doris.nereids.DorisParser.ExceptOrReplaceContext;
@@ -300,6 +299,8 @@ import org.apache.doris.nereids.DorisParser.PlanTypeContext;
import org.apache.doris.nereids.DorisParser.PositionContext;
import org.apache.doris.nereids.DorisParser.PredicateContext;
import org.apache.doris.nereids.DorisParser.PredicatedContext;
+import org.apache.doris.nereids.DorisParser.PrimaryExpressionContext;
+import org.apache.doris.nereids.DorisParser.PrimaryExpressionSuffixContext;
import org.apache.doris.nereids.DorisParser.PrimitiveDataTypeContext;
import org.apache.doris.nereids.DorisParser.PropertyClauseContext;
import org.apache.doris.nereids.DorisParser.PropertyItemContext;
@@ -463,7 +464,6 @@ import
org.apache.doris.nereids.DorisParser.StepPartitionDefContext;
import org.apache.doris.nereids.DorisParser.StringLiteralContext;
import org.apache.doris.nereids.DorisParser.StructLiteralContext;
import org.apache.doris.nereids.DorisParser.SubqueryContext;
-import org.apache.doris.nereids.DorisParser.SubqueryExpressionContext;
import org.apache.doris.nereids.DorisParser.SubstringContext;
import org.apache.doris.nereids.DorisParser.SwitchCatalogContext;
import org.apache.doris.nereids.DorisParser.SyncContext;
@@ -3304,7 +3304,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
}
/**
- * Create a value based [[CaseWhen]] expression. This has the following
SQL form:
+ * Create a condition or value based [[CaseWhen]] expression. This has the
following SQL form:
* {{{
* CASE [expression]
* WHEN [value] THEN [expression]
@@ -3313,30 +3313,23 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
* END
* }}}
*/
+ @Override
+ public Expression
visitCaseExpressionBase(DorisParser.CaseExpressionBaseContext context) {
+ return typedVisit(context.caseExpression());
+ }
+
@Override
public Expression visitSimpleCase(DorisParser.SimpleCaseContext context) {
- Expression e = getExpression(context.value);
+ Expression value = getExpression(context.value);
List<WhenClause> whenClauses = context.whenClause().stream()
.map(w -> new WhenClause(getExpression(w.condition),
getExpression(w.result)))
.collect(ImmutableList.toImmutableList());
if (context.elseExpression == null) {
- return new CaseWhen(e, whenClauses);
+ return new CaseWhen(value, whenClauses);
}
- return new CaseWhen(e, whenClauses,
getExpression(context.elseExpression));
+ return new CaseWhen(value, whenClauses,
getExpression(context.elseExpression));
}
- /**
- * Create a condition based [[CaseWhen]] expression. This has the
following SQL syntax:
- * {{{
- * CASE
- * WHEN [predicate] THEN [expression]
- * ...
- * ELSE [expression]
- * END
- * }}}
- *
- * @param context the parse tree
- */
@Override
public Expression visitSearchedCase(DorisParser.SearchedCaseContext
context) {
List<WhenClause> whenClauses = context.whenClause().stream()
@@ -3400,13 +3393,11 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
}
@Override
- public Expression visitConvertCharSet(DorisParser.ConvertCharSetContext
ctx) {
- return ParserUtils.withOrigin(ctx,
- () -> new ConvertTo(getExpression(ctx.argument), new
StringLiteral(ctx.charSet.getText())));
- }
-
- @Override
- public Expression visitConvertType(DorisParser.ConvertTypeContext ctx) {
+ public Expression
visitConvertExpression(DorisParser.ConvertExpressionContext ctx) {
+ if (ctx.charSet != null) {
+ return ParserUtils.withOrigin(ctx,
+ () -> new ConvertTo(getExpression(ctx.argument), new
StringLiteral(ctx.charSet.getText())));
+ }
return ParserUtils.withOrigin(ctx, () ->
processCast(getExpression(ctx.argument), ctx.castDataType()));
}
@@ -3691,35 +3682,44 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
}
@Override
- public Expression visitDereference(DereferenceContext ctx) {
- return ParserUtils.withOrigin(ctx, () -> {
- Expression e = getExpression(ctx.base);
- if (e instanceof UnboundSlot) {
- UnboundSlot unboundAttribute = (UnboundSlot) e;
+ public Expression visitPrimaryExpression(PrimaryExpressionContext ctx) {
+ Expression expression = typedVisit(ctx.primaryExpressionBase());
+ for (PrimaryExpressionSuffixContext suffix :
ctx.primaryExpressionSuffix()) {
+ if (suffix instanceof ArrayAccessContext) {
+ ArrayAccessContext arrayAccess = (ArrayAccessContext) suffix;
+ if (arrayAccess.COLON() == null) {
+ expression = new ElementAt(expression,
typedVisit(arrayAccess.begin));
+ } else if (arrayAccess.end == null) {
+ expression = new ArraySlice(expression,
typedVisit(arrayAccess.begin));
+ } else {
+ expression = new ArraySlice(
+ expression, typedVisit(arrayAccess.begin),
typedVisit(arrayAccess.end));
+ }
+ } else if (suffix instanceof DereferenceContext) {
+ expression = buildDereference(expression, (DereferenceContext)
suffix, ctx);
+ } else {
+ // COLLATE is accepted syntactically but does not change the
Nereids expression.
+ Preconditions.checkState(suffix instanceof CollateContext);
+ }
+ }
+ return expression;
+ }
+
+ protected Expression buildDereference(
+ Expression base, DereferenceContext ctx, PrimaryExpressionContext
originContext) {
+ return ParserUtils.withOrigin(originContext, () -> {
+ if (base instanceof UnboundSlot) {
+ UnboundSlot unboundAttribute = (UnboundSlot) base;
List<String> nameParts =
Lists.newArrayList(unboundAttribute.getNameParts());
nameParts.add(ctx.fieldName.getText());
UnboundSlot slot = new UnboundSlot(nameParts,
Optional.empty());
return slot;
} else {
- return new DereferenceExpression(e, new
StringLiteral(ctx.identifier().getText()));
+ return new DereferenceExpression(base, new
StringLiteral(ctx.identifier().getText()));
}
});
}
- @Override
- public Expression visitElementAt(ElementAtContext ctx) {
- return new ElementAt(typedVisit(ctx.value), typedVisit(ctx.index));
- }
-
- @Override
- public Expression visitArraySlice(ArraySliceContext ctx) {
- if (ctx.end != null) {
- return new ArraySlice(typedVisit(ctx.value),
typedVisit(ctx.begin), typedVisit(ctx.end));
- } else {
- return new ArraySlice(typedVisit(ctx.value),
typedVisit(ctx.begin));
- }
- }
-
@Override
public Expression visitColumnReference(ColumnReferenceContext ctx) {
// todo: handle quoted and unquoted
@@ -3845,6 +3845,9 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
@Override
public Expression
visitParenthesizedExpression(ParenthesizedExpressionContext ctx) {
+ if (ctx.query() != null) {
+ return ParserUtils.withOrigin(ctx, () -> new
ScalarSubquery(typedVisit(ctx.query())));
+ }
return getExpression(ctx.expression());
}
@@ -5190,11 +5193,6 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
return ParserUtils.withOrigin(namedCtx, () ->
visit(namedCtx.namedExpression(), NamedExpression.class));
}
- @Override
- public Expression visitSubqueryExpression(SubqueryExpressionContext
subqueryExprCtx) {
- return ParserUtils.withOrigin(subqueryExprCtx, () -> new
ScalarSubquery(typedVisit(subqueryExprCtx.query())));
- }
-
@Override
public Expression visitExist(ExistContext context) {
return ParserUtils.withOrigin(context, () -> new
Exists(typedVisit(context.query()), false));
@@ -5495,11 +5493,6 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
return context.getText();
}
- @Override
- public Object visitCollate(CollateContext ctx) {
- return visit(ctx.primaryExpression());
- }
-
@Override
public Object visitSample(SampleContext ctx) {
long seek = ctx.seed == null ? -1L :
Long.parseLong(ctx.seed.getText());
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForCreateView.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForCreateView.java
index e8c0bc77064..6fb1a1f8ba6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForCreateView.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForCreateView.java
@@ -26,6 +26,7 @@ import org.apache.doris.nereids.DorisParser.IdentifierContext;
import org.apache.doris.nereids.DorisParser.LateralViewContext;
import org.apache.doris.nereids.DorisParser.MultipartIdentifierContext;
import org.apache.doris.nereids.DorisParser.NamedExpressionContext;
+import org.apache.doris.nereids.DorisParser.PrimaryExpressionContext;
import org.apache.doris.nereids.DorisParser.StarContext;
import org.apache.doris.nereids.DorisParser.TableAliasContext;
import org.apache.doris.nereids.DorisParser.TableNameContext;
@@ -145,9 +146,11 @@ public class LogicalPlanBuilderForCreateView extends
LogicalPlanBuilder {
}
@Override
- public Expression visitDereference(DereferenceContext ctx) {
- UnboundSlot slot = (UnboundSlot) super.visitDereference(ctx);
- return slot.withIndexInSql(Pair.of(ctx.start.getStartIndex(),
ctx.stop.getStopIndex()));
+ protected Expression buildDereference(
+ Expression base, DereferenceContext ctx, PrimaryExpressionContext
originContext) {
+ UnboundSlot slot = (UnboundSlot) super.buildDereference(base, ctx,
originContext);
+ return slot.withIndexInSql(Pair.of(
+ originContext.start.getStartIndex(), ctx.stop.getStopIndex()));
}
@Override
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/PrimaryExpressionAstTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/PrimaryExpressionAstTest.java
new file mode 100644
index 00000000000..0cab9700b1d
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/PrimaryExpressionAstTest.java
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.analyzer.UnboundFunction;
+import org.apache.doris.nereids.analyzer.UnboundSlot;
+import org.apache.doris.nereids.trees.expressions.Add;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.CaseWhen;
+import org.apache.doris.nereids.trees.expressions.Cast;
+import org.apache.doris.nereids.trees.expressions.DereferenceExpression;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Multiply;
+import org.apache.doris.nereids.trees.expressions.Not;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Subtract;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ArraySlice;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ConvertTo;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.ElementAt;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.List;
+import java.util.stream.Stream;
+
+class PrimaryExpressionAstTest extends ParserTestBase {
+ private final NereidsParser parser = new NereidsParser();
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("representativeExpressions")
+ void preservesRepresentativeExpressionRoots(
+ String description, String sql, Class<? extends Expression>
expectedClass) {
+ Expression expression = parser.parseExpression(sql);
+ Assertions.assertInstanceOf(expectedClass, expression);
+ }
+
+ private static Stream<Arguments> representativeExpressions() {
+ return Stream.of(
+ Arguments.of("searched case", "CASE WHEN a THEN 1 ELSE 2 END",
CaseWhen.class),
+ Arguments.of("simple case", "CASE a WHEN 1 THEN 2 ELSE 3 END",
CaseWhen.class),
+ Arguments.of("convert charset", "CONVERT(IF(a, b, c) USING
utf8)", ConvertTo.class),
+ Arguments.of("convert type", "CONVERT(IF(a, b, c), BIGINT)",
Cast.class),
+ Arguments.of("generic convert function", "CONVERT(a)",
UnboundFunction.class),
+ Arguments.of("keyword column", "CAST", UnboundSlot.class),
+ Arguments.of("function call", "db.fn(a)",
UnboundFunction.class),
+ Arguments.of("element at", "a[1]", ElementAt.class),
+ Arguments.of("array slice", "a[1:2]", ArraySlice.class),
+ Arguments.of("slot dereference", "a.b.c", UnboundSlot.class),
+ Arguments.of("expression dereference", "fn(a).field",
DereferenceExpression.class));
+ }
+
+ @Test
+ void preservesPostfixChainOrder() {
+ Expression expression = parser.parseExpression("fn(a)[1:2][3].field
COLLATE utf8_general_ci");
+
+ Assertions.assertInstanceOf(DereferenceExpression.class, expression);
+ Assertions.assertInstanceOf(ElementAt.class, expression.child(0));
+ Assertions.assertInstanceOf(ArraySlice.class,
expression.child(0).child(0));
+ Assertions.assertInstanceOf(UnboundFunction.class,
expression.child(0).child(0).child(0));
+ }
+
+ @Test
+ void preservesArithmeticComparisonAndBooleanPrecedence() {
+ Expression expression = parser.parseExpression("NOT a = b + c * d OR e
AND f");
+
+ Assertions.assertInstanceOf(Or.class, expression);
+ Assertions.assertInstanceOf(Not.class, expression.child(0));
+ Assertions.assertInstanceOf(EqualTo.class,
expression.child(0).child(0));
+ Assertions.assertInstanceOf(Add.class,
expression.child(0).child(0).child(1));
+ Assertions.assertInstanceOf(Multiply.class,
expression.child(0).child(0).child(1).child(1));
+ Assertions.assertInstanceOf(And.class, expression.child(1));
+ }
+
+ @Test
+ void preservesLeftAssociativeArithmetic() {
+ Expression expression = parser.parseExpression("a - b - c");
+
+ Assertions.assertInstanceOf(Subtract.class, expression);
+ Assertions.assertInstanceOf(Subtract.class, expression.child(0));
+ }
+
+ @Test
+ void preservesSimpleCasePlaceholderOrder() {
+ List<Integer> placeholderIds = parser.parseMultiple("SELECT CASE ?
WHEN ? THEN ? ELSE ? END").get(0).second
+ .getPlaceholders().stream()
+ .map(placeholder -> placeholder.getPlaceholderId().asInt())
+ .collect(ImmutableList.toImmutableList());
+
+ Assertions.assertEquals(List.of(0, 1, 2, 3), placeholderIds);
+ }
+
+ @Test
+ void preservesCreateViewDereferenceSourceInterval() {
+ LogicalPlan plan = parser.parseForCreateView("SELECT db.tbl.col FROM
t");
+ UnboundSlot slot = plan.<LogicalPlan>collectToList(ignored ->
true).stream()
+ .flatMap(node -> node.getExpressions().stream())
+ .flatMap(expression -> expression.<UnboundSlot>collectToList(
+ UnboundSlot.class::isInstance).stream())
+ .filter(unboundSlot ->
unboundSlot.getNameParts().equals(List.of("db", "tbl", "col")))
+ .findFirst()
+ .orElseThrow();
+
+ Assertions.assertEquals(Pair.of(7, 16),
slot.getIndexInSqlString().orElseThrow());
+ }
+}
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
new file mode 100644
index 00000000000..fca29e73bc4
--- /dev/null
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
@@ -0,0 +1,114 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser.benchmark;
+
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.parser.ParseErrorListener;
+import org.apache.doris.nereids.parser.PostProcessor;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.ListTokenSource;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Measures the grammar change both in isolation and through the public
parser facade. */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
+@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
+@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+public class PrimaryExpressionBenchmark {
+ @Param({"control", "typical", "specialForms", "postfixChain",
"wideProjection"})
+ public String workload;
+
+ private final DorisSqlParser facade = new DorisSqlParser();
+ private final PostProcessor postProcessor = new PostProcessor();
+ private final ParseErrorListener errorListener = new ParseErrorListener();
+
+ private String sql;
+ private List<Token> tokens;
+
+ @Setup(Level.Trial)
+ public void setUp() {
+ switch (workload) {
+ case "control":
+ sql = "SELECT 1";
+ break;
+ case "typical":
+ sql = "SELECT a + b * c FROM t "
+ + "WHERE (d = 1 OR e[2].f > 3) AND g IS NOT NULL";
+ break;
+ case "specialForms":
+ sql = "SELECT CASE a WHEN 1 THEN CONVERT(b USING utf8) "
+ + "WHEN 2 THEN CAST(c AS BIGINT) ELSE d + 1 END FROM t
"
+ + "WHERE CASE WHEN e > 0 THEN TRUE ELSE FALSE END";
+ break;
+ case "postfixChain":
+ sql = "SELECT fn(a)[1:2][3].field[4].nested[5:6].leaf "
+ + "COLLATE utf8_general_ci FROM t";
+ break;
+ case "wideProjection":
+ sql = "SELECT " + IntStream.range(0, 64)
+ .mapToObj(i -> "c" + i + " + " + i)
+ .collect(Collectors.joining(", "))
+ + " FROM t WHERE key_col[1].field > 0";
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+
+ CommonTokenStream stream = new CommonTokenStream(facade.newLexer(sql));
+ stream.fill();
+ tokens = List.copyOf(stream.getTokens());
+ }
+
+ @Benchmark
+ public Object parseEndToEnd() {
+ return facade.parseStatement(sql);
+ }
+
+ @Benchmark
+ public Object parsePreTokenized() {
+ CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(tokens));
+ DorisParser parser = new DorisParser(stream);
+ parser.addParseListener(postProcessor);
+ parser.removeErrorListeners();
+ parser.addErrorListener(errorListener);
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+ return parser.singleStatement();
+ }
+}
diff --git a/fe/fe-sql-parser/README.md b/fe/fe-sql-parser/README.md
index 3cab071b33f..73a5662d3b9 100644
--- a/fe/fe-sql-parser/README.md
+++ b/fe/fe-sql-parser/README.md
@@ -67,6 +67,10 @@ The optional `benchmark` profile builds a self-contained JMH
jar without adding
mvn -Pbenchmark -pl fe-sql-parser-benchmark -am package -DskipTests
java -jar fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \
'.*StringLiteralBenchmark.*' -prof gc -rf json -rff
/tmp/string-literal-benchmark.json
+
+# Run the primary-expression end-to-end and pre-tokenized parser benchmarks
+java -jar fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar \
+ '.*PrimaryExpressionBenchmark.*' -prof gc -rf json -rff
/tmp/primary-expression-benchmark.json
```
Use the same JDK, corpus parameters, JMH arguments, and machine state for
baseline and candidate runs. Run the same baseline artifact twice before
comparing a change; the raw JSON and artifact hash should be retained with the
result summary.
diff --git
a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index f1eba2e28c0..b2cdefd8901 100644
--- a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -1872,6 +1872,10 @@ valueExpression
;
primaryExpression
+ : primaryExpressionBase primaryExpressionSuffix*
+ ;
+
+primaryExpressionBase
: name=CURRENT_DATE
#currentDate
| name=CURRENT_TIME
#currentTime
| name=CURRENT_TIMESTAMP
#currentTimestamp
@@ -1879,8 +1883,7 @@ primaryExpression
| name=LOCALTIMESTAMP
#localTimestamp
| name=CURRENT_USER
#currentUser
| name=SESSION_USER
#sessionUser
- | CASE whenClause+ (ELSE elseExpression=expression)? END
#searchedCase
- | CASE value=expression whenClause+ (ELSE elseExpression=expression)? END
#simpleCase
+ | CASE caseExpression
#caseExpressionBase
| name=CAST LEFT_PAREN expression AS castDataType RIGHT_PAREN
#cast
| name=TRY_CAST LEFT_PAREN expression AS castDataType RIGHT_PAREN
#tryCast
| DEFAULT LEFT_PAREN qualifiedName RIGHT_PAREN
#defaultValue
@@ -1892,8 +1895,8 @@ primaryExpression
arguments+=expression (COMMA arguments+=expression)*
(USING charSet=identifierOrText)?
RIGHT_PAREN
#charFunction
- | CONVERT LEFT_PAREN argument=expression USING charSet=identifierOrText
RIGHT_PAREN #convertCharSet
- | CONVERT LEFT_PAREN argument=expression COMMA castDataType RIGHT_PAREN
#convertType
+ | CONVERT LEFT_PAREN argument=expression
+ (USING charSet=identifierOrText | COMMA castDataType) RIGHT_PAREN
#convertExpression
| GROUP_CONCAT LEFT_PAREN (DISTINCT|ALL)?
(LEFT_BRACKET identifier RIGHT_BRACKET)?
argument=expression
@@ -1908,19 +1911,25 @@ primaryExpression
| (ISNULL | IS_NULL_PRED) LEFT_PAREN expression RIGHT_PAREN
#isnull
| IS_NOT_NULL_PRED LEFT_PAREN expression RIGHT_PAREN
#is_not_null_pred
| functionCallExpression
#functionCall
- | value=primaryExpression LEFT_BRACKET index=valueExpression RIGHT_BRACKET
#elementAt
- | value=primaryExpression LEFT_BRACKET begin=valueExpression
- COLON (end=valueExpression)? RIGHT_BRACKET
#arraySlice
- | LEFT_PAREN query RIGHT_PAREN
#subqueryExpression
+ | LEFT_PAREN (query | expression) RIGHT_PAREN
#parenthesizedExpression
| ATSIGN identifierOrText
#userVariable
| DOUBLEATSIGN (kind=(GLOBAL | SESSION) DOT)? identifier
#systemVariable
| BINARY? identifier
#columnReference
- | base=primaryExpression DOT fieldName=identifier
#dereference
- | LEFT_PAREN expression RIGHT_PAREN
#parenthesizedExpression
| KEY (dbName=identifier DOT)? keyName=identifier
#encryptKey
| EXTRACT LEFT_PAREN field=unitIdentifier FROM (DATE | TIMESTAMP)?
source=valueExpression RIGHT_PAREN
#extract
- | primaryExpression COLLATE (identifier | STRING_LITERAL | DEFAULT)
#collate
+ ;
+
+primaryExpressionSuffix
+ : LEFT_BRACKET begin=valueExpression
+ (COLON (end=valueExpression)?)? RIGHT_BRACKET
#arrayAccess
+ | DOT fieldName=identifier
#dereference
+ | COLLATE (identifier | STRING_LITERAL | DEFAULT)
#collate
+ ;
+
+caseExpression
+ : whenClause+ (ELSE elseExpression=expression)? END
#searchedCase
+ | value=expression whenClause+ (ELSE elseExpression=expression)? END
#simpleCase
;
exceptOrReplace
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/PrimaryExpressionTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/PrimaryExpressionTest.java
new file mode 100644
index 00000000000..aacedbef795
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/PrimaryExpressionTest.java
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisParser.ExpressionContext;
+import org.apache.doris.nereids.exceptions.ParseException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.stream.Stream;
+
+class PrimaryExpressionTest {
+ private final DorisSqlParser parser = new DorisSqlParser();
+
+ @ParameterizedTest(name = "{0}")
+ @MethodSource("primaryExpressionAlternatives")
+ void parsesEveryPrimaryExpressionAlternative(String description, String
sql) {
+ ExpressionContext context = parser.parseExpression(sql);
+ Assertions.assertNotNull(context);
+ }
+
+ private static Stream<Arguments> primaryExpressionAlternatives() {
+ return Stream.of(
+ Arguments.of("current date", "CURRENT_DATE"),
+ Arguments.of("current time", "CURRENT_TIME"),
+ Arguments.of("current timestamp", "CURRENT_TIMESTAMP"),
+ Arguments.of("local time", "LOCALTIME"),
+ Arguments.of("local timestamp", "LOCALTIMESTAMP"),
+ Arguments.of("current user", "CURRENT_USER"),
+ Arguments.of("session user", "SESSION_USER"),
+ Arguments.of("searched case", "CASE WHEN a > 0 THEN 1 ELSE 2
END"),
+ Arguments.of("simple case", "CASE a WHEN 1 THEN 2 WHEN 3 THEN
4 END"),
+ Arguments.of("cast", "CAST(a + 1 AS BIGINT)"),
+ Arguments.of("try cast", "TRY_CAST(a AS DECIMAL(10, 2))"),
+ Arguments.of("default value", "DEFAULT(t.c)"),
+ Arguments.of("null literal", "NULL"),
+ Arguments.of("typed literal", "DATE '2026-08-19'"),
+ Arguments.of("numeric literal", "123.45"),
+ Arguments.of("boolean literal", "TRUE"),
+ Arguments.of("string literal", "'text'"),
+ Arguments.of("varbinary literal", "X'0A0B'"),
+ Arguments.of("array literal", "[1, 2, 3]"),
+ Arguments.of("map literal", "{'a': 1, 'b': 2}"),
+ Arguments.of("struct literal", "{1, 'a'}"),
+ Arguments.of("placeholder", "?"),
+ Arguments.of("interval", "INTERVAL 1 DAY"),
+ Arguments.of("unqualified star", "* EXCEPT(a)"),
+ Arguments.of("qualified star", "db.t.* REPLACE(1 AS a)"),
+ Arguments.of("char function", "CHAR(65, 66 USING utf8)"),
+ Arguments.of("convert charset", "CONVERT(IF(a, b, c) USING
utf8)"),
+ Arguments.of("convert type", "CONVERT(IF(a, b, c), DECIMAL(10,
2))"),
+ Arguments.of("group concat", "GROUP_CONCAT(DISTINCT a ORDER BY
b SEPARATOR ',')"),
+ Arguments.of("trim", "TRIM(BOTH 'x' FROM a)"),
+ Arguments.of("substring", "SUBSTRING(a FROM 2 FOR 3)"),
+ Arguments.of("position", "POSITION('x' IN a)"),
+ Arguments.of("is null function", "ISNULL(a)"),
+ Arguments.of("is not null function", "IS_NOT_NULL_PRED(a)"),
+ Arguments.of("function call", "db.fn(a, b + 1)"),
+ Arguments.of("element at", "a[1]"),
+ Arguments.of("array slice", "a[1:2]"),
+ Arguments.of("scalar subquery", "(SELECT 1)"),
+ Arguments.of("user variable", "@user_var"),
+ Arguments.of("system variable", "@@SESSION.system_var"),
+ Arguments.of("binary column reference", "BINARY a"),
+ Arguments.of("column reference", "a"),
+ Arguments.of("dereference", "a.b.c"),
+ Arguments.of("parenthesized expression", "((a + 1) * 2)"),
+ Arguments.of("encrypt key", "KEY db.key_name"),
+ Arguments.of("extract", "EXTRACT(YEAR FROM DATE
'2026-08-19')"),
+ Arguments.of("collate", "a COLLATE utf8_general_ci"));
+ }
+
+ @ParameterizedTest(name = "ambiguous prefix: {0}")
+ @MethodSource("ambiguousPrefixExpressions")
+ void preservesIdentifierAndSpecialFormBoundaries(String description,
String sql) {
+ ExpressionContext context = parser.parseExpression(sql);
+ Assertions.assertNotNull(context);
+ }
+
+ private static Stream<Arguments> ambiguousPrefixExpressions() {
+ return Stream.of(
+ Arguments.of("current date special form", "CURRENT_DATE"),
+ Arguments.of("current date function", "CURRENT_DATE()"),
+ Arguments.of("case keyword as identifier", "CASE"),
+ Arguments.of("cast keyword as identifier", "CAST"),
+ Arguments.of("convert keyword as identifier", "CONVERT"),
+ Arguments.of("convert generic function", "CONVERT(a)"),
+ Arguments.of("trim special syntax", "TRIM(a FROM b)"),
+ Arguments.of("trim generic function", "TRIM(a)"),
+ Arguments.of("substring special syntax", "SUBSTRING(a FROM
1)"),
+ Arguments.of("substring generic function", "SUBSTRING(a, 1)"),
+ Arguments.of("position special syntax", "POSITION(a IN b)"),
+ Arguments.of("position generic function", "POSITION(a)"),
+ Arguments.of("date typed literal", "DATE '2026-08-19'"),
+ Arguments.of("date generic function", "DATE(a)"),
+ Arguments.of("date column", "DATE"),
+ Arguments.of("interval literal", "INTERVAL 1 DAY"),
+ Arguments.of("interval generic function", "INTERVAL()"),
+ Arguments.of("qualified function", "db.fn(a)"),
+ Arguments.of("qualified star", "db.t.*"),
+ Arguments.of("qualified column", "db.t.c"),
+ Arguments.of("binary string literal", "BINARY 'abc'"),
+ Arguments.of("binary column", "BINARY a"));
+ }
+
+ @ParameterizedTest(name = "invalid: {0}")
+ @MethodSource("invalidExpressions")
+ void rejectsIncompletePrimaryExpressionsAtTheOriginalPosition(
+ String description, String sql, int errorPosition) {
+ ParseException exception =
Assertions.assertThrows(ParseException.class, () ->
parser.parseExpression(sql));
+ Assertions.assertTrue(exception.getMessage().contains("line 1, pos " +
errorPosition), exception::getMessage);
+ }
+
+ private static Stream<Arguments> invalidExpressions() {
+ return Stream.of(
+ Arguments.of("searched case missing END", "CASE WHEN a THEN
b", 5),
+ invalidAtEnd("simple case missing result", "CASE a WHEN 1
THEN"),
+ Arguments.of("convert charset missing charset", "CONVERT(a
USING)", 15),
+ Arguments.of("convert type missing type", "CONVERT(a,)", 10),
+ invalidAtEnd("subquery missing right parenthesis", "(SELECT
1"),
+ invalidAtEnd("expression missing right parenthesis", "(a + 1"),
+ invalidAtEnd("array access missing index", "a["),
+ invalidAtEnd("array slice missing end bracket", "a[1:"),
+ invalidAtEnd("array slice value missing end bracket", "a[1:2"),
+ invalidAtEnd("dereference missing field", "a."),
+ invalidAtEnd("collate missing collation", "a COLLATE"),
+ Arguments.of("cast missing type", "CAST(a AS)", 9),
+ Arguments.of("trim missing source", "TRIM(a FROM)", 11),
+ Arguments.of("trailing garbage", "a[1] garbage", 5));
+ }
+
+ private static Arguments invalidAtEnd(String description, String sql) {
+ return Arguments.of(description, sql, sql.length());
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
