This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 146d3595566 [improvement](parser) Assign query organization to a
single owner (#67429)
146d3595566 is described below
commit 146d35955664cc86abb87b99130425444b2d18d7
Author: morrySnow <[email protected]>
AuthorDate: Fri Sep 4 14:35:53 2026 +0800
[improvement](parser) Assign query organization to a single owner (#67429)
### What problem does this PR solve?
Problem Summary: The grammar allowed both `querySpecification` and its
outer `query` to consume `ORDER BY` and `LIMIT`, so ANTLR repeatedly
entered `queryOrganization` and reported full-context ambiguities. This
change gives every clause group one owner according to ANSI mode and
query level, makes the rule structurally non-empty, and preserves the
existing accepted/rejected SQL matrix. It intentionally uses ANTLR's
default error reporting; three incomplete `ORDER BY`/`LIMIT` forms now
report the end-of-input position instead of re-anchoring the error to
the clause keyword.
### Benchmark
The original P2 benchmark is reused without rerunning it. After
measurement, the follow-up cleanup only removed error-position
compatibility state and grammar actions; it added no parsing decisions
and removed work from the valid-SQL path. Lower latency is better. The
target input is `SELECT a, b, c FROM t WHERE a > 1 ORDER BY a, b DESC
LIMIT 20 OFFSET 10`; the control input has the same SELECT without
query-organization clauses.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB
heap
- JMH: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement; C1-B1-C2
interleaving
- Measurement baseline: `5e0eadb13e9`; parser jar SHA-256
`530f0ed45c4bac3a096373a1e932aae0025c603a6f905c3d975a2ad53bd0e7ca`;
benchmark jar SHA-256
`7543595b2a87f55b9b53538336cfedf55c13484302ffb487efcfb4fdd66b1c0b`
- Measurement candidate: `773c4ee1027`; parser jar SHA-256
`65730a22a5262370a832bda7a04ae4b8e0de5c107be65c5ae80f95c711842f59`;
benchmark jar SHA-256
`47284ffd6779e9d534a19a682de5a4d468f83a62a98e048982807497f1f665b6`
- Harness:
`fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java`
#### Ordered SELECT target
The candidate change is calculated from the mean of C1 and C2 against
B1. Values are us/op with JMH's 99.9% error estimate.
| Path | Mode | Baseline B1 | Candidate C1
| Candidate C2 | Latency Improvement |
|:------------------------|:-------|------------------:|------------------:|------------------:|--------------------:|
| End-to-end | Legacy | 9.926 ± 0.347 | 9.303 ± 0.333
| 9.122 ± 0.267 | **7.2% faster** |
| End-to-end | ANSI | 9.170 ± 0.437 | 8.709 ± 0.646
| 8.492 ± 0.241 | **6.2% faster** |
| Pre-tokenized parser | Legacy | 7.653 ± 0.266 | 7.246 ± 0.244
| 7.069 ± 0.090 | **6.5% faster** |
| Pre-tokenized parser | ANSI | 7.344 ± 0.105 | 7.181 ± 0.099
| 7.099 ± 0.076 | **2.8% faster** |
#### Plain SELECT control
| Path | Mode | Baseline (us/op) | Candidate (us/op) |
Latency Improvement |
|:---------------------|:-------|-----------------:|------------------:|--------------------:|
| End-to-end | Legacy | 6.580 ± 0.710 | 6.657 ± 0.809 |
1.2% slower |
| End-to-end | ANSI | 6.944 ± 0.648 | 6.448 ± 0.440 |
**7.1% faster** |
| Pre-tokenized parser | Legacy | 5.684 ± 1.205 | 5.084 ± 0.054 |
**10.6% faster** |
| Pre-tokenized parser | ANSI | 5.313 ± 0.125 | 5.193 ± 0.089 |
**2.3% faster** |
No control-path latency regression exceeds the 3% threshold.
#### Allocation
The same artifacts were measured separately with `-prof gc`. Timing
under the profiler was noisy, so this table uses only normalized
allocation.
| Path | Mode | Baseline (B/op) | Candidate (B/op) |
Allocation Change |
|:---------------------|:-------|----------------:|------------------:|------------------:|
| End-to-end | Legacy | 15,473.6 | 15,719.6 |
+246.0 (+1.59%) |
| End-to-end | ANSI | 15,426.8 | 15,536.1 |
+109.3 (+0.71%) |
| Pre-tokenized parser | Legacy | 12,368.1 | 12,381.4 |
+13.3 (+0.11%) |
| Pre-tokenized parser | ANSI | 12,317.4 | 12,421.4 |
+104.0 (+0.84%) |
All allocation changes are below the 3% threshold.
#### Profile attribution
- Before this PR, tracked SQL entered `queryOrganization` 13,196 times
and examined 49,902 lookahead tokens; SSB and Trino profiling reported
40 and 20 related ambiguities.
- After this PR, the 4,275 parseable tracked SQL files enter the target
rule 2,794 times and examine 8,645 lookahead tokens, a reduction of
about 79% and 83% respectively.
- LL_EXACT reports zero fallback and zero ambiguity for both the
`querySpecification` and `queryOrganization` target decisions.
- The gain comes from eliminating duplicate ownership and adaptive
lookahead, not from lexer or token changes.
### Semantic differential
- Original corpus: all 4,610 tracked `*.sql` files; manifest SHA-256
`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`
- Original result: 4,275 parsed and 335 rejected in both artifacts and
in both ANSI modes
- The follow-up cleanup does not change any grammar decision or error
occurrence, so accepted/rejected behavior remains unchanged; the full
corpus was not rerun
- Three deliberately changed first-error positions are covered by unit
tests:
- ANSI `SELECT 1 ORDER BY`: pos 9 -> 17
- ANSI `SELECT 1 LIMIT`: pos 9 -> 14
- Legacy `SELECT 1 LIMIT 1 ORDER BY`: pos 17 -> 25
- Lexer and token behavior are unchanged by construction
### Release note
Malformed `ORDER BY` and `LIMIT` clauses now use ANTLR's default error
positions.
---
.../doris/nereids/parser/LogicalPlanBuilder.java | 14 +-
.../nereids/parser/QueryOrganizationPlanTest.java | 84 ++++++++++++
.../benchmark/QueryOrganizationBenchmark.java | 119 +++++++++++++++++
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 19 ++-
.../sqlparser/QueryOrganizationBehaviorTest.java | 142 +++++++++++++++++++++
5 files changed, 364 insertions(+), 14 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index c296637e70c..22415dff1df 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -1241,7 +1241,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
}
LogicalPlan plan = ParserUtils.withOrigin(
ctx.cteContext != null ? ctx.cteContext : ctx.queryTerm(),
- () -> withCte(buildQuery(ctx.queryTerm(),
ctx.queryOrganization()), ctx.cteContext));
+ () -> withCte(buildQuery(ctx.queryTerm(), ctx.organization),
ctx.cteContext));
if (ctx.outFileClause() != null) {
plan = withOutFile(plan, ctx.outFileClause());
} else {
@@ -2144,7 +2144,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
query = withRelations(query, ((FromRelationsContext)
ctx.fromClause()).relations().relation());
}
query = withFilter(query, Optional.ofNullable(ctx.whereClause()));
- query = withQueryOrganization(query, ctx.queryOrganization());
+ query = withQueryOrganization(query, ctx.organization);
query = convertSortOrdinalsToUnboundSlot(query);
String tableAlias = null;
if (ctx.tableAlias().strictIdentifier() != null) {
@@ -2176,9 +2176,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
tableAlias = ctx.tableAlias().strictIdentifier().getText();
}
- boolean hasQueryOrganization = ctx.queryOrganization() != null
- && (ctx.queryOrganization().sortClause() != null
- || ctx.queryOrganization().limitClause() != null);
+ boolean hasQueryOrganization = ctx.organization != null;
Command deleteCommand;
if (ctx.USING() == null && ctx.cteContext == null &&
!hasQueryOrganization) {
query = withFilter(query, Optional.ofNullable(ctx.whereClause()));
@@ -2190,7 +2188,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
query = withRelations(query, ctx.relations().relation());
}
query = withFilter(query, Optional.ofNullable(ctx.whereClause()));
- query = withQueryOrganization(query, ctx.queryOrganization());
+ query = withQueryOrganization(query, ctx.organization);
query = convertSortOrdinalsToUnboundSlot(query);
Optional<LogicalPlan> cte = Optional.empty();
if (ctx.cteContext != null) {
@@ -2670,7 +2668,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
public LogicalPlan visitQuery(QueryContext ctx) {
return ParserUtils.withOrigin(ctx, () -> {
// TODO: need to add withQueryResultClauses and withCTE
- return withCte(buildQuery(ctx.queryTerm(),
ctx.queryOrganization()), ctx.cte());
+ return withCte(buildQuery(ctx.queryTerm(), ctx.organization),
ctx.cte());
});
}
@@ -2778,7 +2776,7 @@ public class LogicalPlanBuilder extends
DorisParserBaseVisitor<Object> {
Optional.ofNullable(ctx.aggClause()),
Optional.ofNullable(ctx.havingClause()),
Optional.ofNullable(ctx.qualifyClause()));
- selectPlan = withQueryOrganization(selectPlan,
ctx.queryOrganization());
+ selectPlan = withQueryOrganization(selectPlan, ctx.organization);
if ((selectHintMap == null) || selectHintMap.isEmpty()) {
return selectPlan;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/QueryOrganizationPlanTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/QueryOrganizationPlanTest.java
new file mode 100644
index 00000000000..8745088a2a2
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/QueryOrganizationPlanTest.java
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.nereids.exceptions.ParseException;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
+import org.apache.doris.qe.GlobalVariable;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class QueryOrganizationPlanTest extends ParserTestBase {
+ private final NereidsParser parser = new NereidsParser();
+
+ @Test
+ void legacyModeAppliesClausesToSetOperands() {
+ withAnsiMode(false, () -> {
+ LogicalUnion unionWithRightLimit =
Assertions.assertInstanceOf(LogicalUnion.class,
+ parseQuery("SELECT 1 UNION ALL SELECT 2 LIMIT 1"));
+ Assertions.assertInstanceOf(LogicalLimit.class,
unionWithRightLimit.child(1));
+
+ LogicalUnion unionWithLeftSort =
Assertions.assertInstanceOf(LogicalUnion.class,
+ parseQuery("SELECT 1 ORDER BY 1 UNION ALL SELECT 2"));
+ Assertions.assertInstanceOf(LogicalSort.class,
unionWithLeftSort.child(0));
+ });
+ }
+
+ @Test
+ void ansiModeAppliesClausesToWholeSetOperation() {
+ withAnsiMode(true, () -> {
+ LogicalLimit<?> limit =
Assertions.assertInstanceOf(LogicalLimit.class,
+ parseQuery("SELECT 1 UNION ALL SELECT 2 LIMIT 1"));
+ Assertions.assertInstanceOf(LogicalUnion.class, limit.child());
+ Assertions.assertThrows(ParseException.class,
+ () -> parser.parseSingle("SELECT 1 ORDER BY 1 UNION ALL
SELECT 2"));
+ });
+ }
+
+ @Test
+ void preservesParenthesizedAndInlineTableClauses() {
+ for (boolean ansi : new boolean[] {false, true}) {
+ withAnsiMode(ansi, () -> {
+ Assertions.assertInstanceOf(LogicalLimit.class,
+ parseQuery("(SELECT 1 UNION ALL SELECT 2) LIMIT 1"));
+ Assertions.assertInstanceOf(LogicalLimit.class,
+ parseQuery("VALUES (1), (2) ORDER BY 1 LIMIT 1"));
+ });
+ }
+ }
+
+ private Plan parseQuery(String sql) {
+ LogicalPlan statement = parser.parseSingle(sql);
+ return statement.child(0);
+ }
+
+ private void withAnsiMode(boolean enabled, Runnable test) {
+ boolean previous =
GlobalVariable.enable_ansi_query_organization_behavior;
+ try {
+ GlobalVariable.enable_ansi_query_organization_behavior = enabled;
+ test.run();
+ } finally {
+ GlobalVariable.enable_ansi_query_organization_behavior = previous;
+ }
+ }
+}
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java
new file mode 100644
index 00000000000..8a907de7abc
--- /dev/null
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrganizationBenchmark.java
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser.benchmark;
+
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.nereids.parser.ParseErrorListener;
+import org.apache.doris.nereids.parser.PostProcessor;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.ListTokenSource;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Measures query-organization ownership in ANSI and legacy modes. */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
+@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
+@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+public class QueryOrganizationBenchmark {
+ @Param({"false", "true"})
+ public boolean ansi;
+
+ @Param({"plainSelect", "orderedSelect", "unionTail", "parenthesizedUnion",
"inlineValues"})
+ public String workload;
+
+ private final PostProcessor postProcessor = new PostProcessor();
+ private final ParseErrorListener errorListener = new ParseErrorListener();
+
+ private DorisSqlParser facade;
+ private String sql;
+ private List<Token> tokens;
+
+ @Setup(Level.Trial)
+ public void setUp() {
+ facade = new DorisSqlParser(false, ansi);
+ String union = buildUnion(12);
+ switch (workload) {
+ case "plainSelect":
+ sql = "SELECT a, b, c FROM t WHERE a > 1";
+ break;
+ case "orderedSelect":
+ sql = "SELECT a, b, c FROM t WHERE a > 1 ORDER BY a, b DESC
LIMIT 20 OFFSET 10";
+ break;
+ case "unionTail":
+ sql = union + " ORDER BY 1 LIMIT 10";
+ break;
+ case "parenthesizedUnion":
+ sql = "(" + union + ") ORDER BY 1 LIMIT 10";
+ break;
+ case "inlineValues":
+ sql = "VALUES (1, 2), (3, 4), (5, 6), (7, 8) ORDER BY 1 LIMIT
3";
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+
+ CommonTokenStream stream = new CommonTokenStream(facade.newLexer(sql));
+ stream.fill();
+ tokens = List.copyOf(stream.getTokens());
+ }
+
+ @Benchmark
+ public Object parseEndToEnd() {
+ return facade.parseStatement(sql);
+ }
+
+ @Benchmark
+ public Object parsePreTokenized() {
+ CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(tokens));
+ DorisParser parser = new DorisParser(stream);
+ parser.ansiSQLSyntax = ansi;
+ parser.addParseListener(postProcessor);
+ parser.removeErrorListeners();
+ parser.addErrorListener(errorListener);
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+ return parser.singleStatement();
+ }
+
+ private static String buildUnion(int count) {
+ return IntStream.range(0, count)
+ .mapToObj(index -> "SELECT " + index + " AS k")
+ .collect(Collectors.joining(" UNION ALL "));
+ }
+}
diff --git
a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 2686fd330fb..e8d71c7be1e 100644
--- a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -49,6 +49,11 @@ options { tokenVocab = DorisLexer; }
ctx.getParent().getText(), ctx);
}
+ private boolean isQueryOrganizationStart() {
+ int tokenType = _input.LA(1);
+ return tokenType == ORDER || tokenType == LIMIT;
+ }
+
private boolean isTupleLambdaBody() {
if (_input.LA(1) != LEFT_PAREN) {
return false;
@@ -131,7 +136,7 @@ statementBase
queryOrDmlStatement
: explainContext=explain? cteContext=cte?
- (queryTerm queryOrganization outFileClause?
+ (queryTerm organization=queryOrganization? outFileClause?
| dmlStatementBody[$explainContext.ctx, $cteContext.ctx])
#explainableStatement
| nonExplainableDmlStatement #dmlStatementAlias
| describeStatement #describeStatementAlias
@@ -326,12 +331,12 @@ dmlStatementBody[ExplainContext explainContext,
CteContext cteContext]
SET updateAssignmentSeq
fromClause?
whereClause?
- queryOrganization #update
+ organization=queryOrganization? #update
| DELETE FROM tableName=multipartIdentifier
partitionSpec? tableAlias
(USING relations)?
whereClause?
- queryOrganization #delete
+ organization=queryOrganization? #delete
| MERGE INTO targetTable=multipartIdentifier
(AS? identifier)? USING srcRelation=relationPrimary
ON expression
@@ -1496,7 +1501,7 @@ outFileClause
;
query
- : cte? queryTerm queryOrganization
+ : cte? queryTerm organization=queryOrganization?
;
queryTerm
@@ -1524,7 +1529,8 @@ querySpecification
aggClause?
havingClause?
qualifyClause?
- ({!ansiSQLSyntax}? queryOrganization | {ansiSQLSyntax}?)
#regularQuerySpecification
+ ({!ansiSQLSyntax}? organization=queryOrganization
+ | {ansiSQLSyntax || !isQueryOrganizationStart()}?)
#regularQuerySpecification
;
cte
@@ -1660,7 +1666,8 @@ unnest:
)?;
queryOrganization
- : sortClause? limitClause?
+ : sortClause (limitClause | {_input.LA(1) != LIMIT}?)
+ | limitClause
;
sortClause
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/QueryOrganizationBehaviorTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/QueryOrganizationBehaviorTest.java
new file mode 100644
index 00000000000..e81e7853dd9
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/QueryOrganizationBehaviorTest.java
@@ -0,0 +1,142 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisParser.QueryOrganizationContext;
+import org.apache.doris.nereids.DorisParser.SingleStatementContext;
+import org.apache.doris.nereids.exceptions.ParseException;
+
+import org.antlr.v4.runtime.tree.ParseTree;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+import java.util.stream.Stream;
+
+class QueryOrganizationBehaviorTest {
+
+ @ParameterizedTest(name = "owners: {0}, ansi={1}")
+ @MethodSource("organizationOwners")
+ void createsOneNonEmptyOwnerPerClauseGroup(String description, boolean
ansi, String sql, int expectedOwners) {
+ SingleStatementContext statement = new DorisSqlParser(false,
ansi).parseStatement(sql);
+ Assertions.assertEquals(expectedOwners,
countOrganizationOwners(statement));
+ }
+
+ private static Stream<Arguments> organizationOwners() {
+ return Stream.of(
+ Arguments.of("plain select", false, "SELECT 1", 0),
+ Arguments.of("plain select", true, "SELECT 1", 0),
+ Arguments.of("ordered select", false, "SELECT 1 ORDER BY 1",
1),
+ Arguments.of("ordered select", true, "SELECT 1 ORDER BY 1", 1),
+ Arguments.of("set tail", false, "SELECT 1 UNION ALL SELECT 2
LIMIT 1", 1),
+ Arguments.of("set tail", true, "SELECT 1 UNION ALL SELECT 2
LIMIT 1", 1),
+ Arguments.of("nested clauses", false, "(SELECT 1 ORDER BY 1)
LIMIT 1", 2),
+ Arguments.of("nested clauses", true, "(SELECT 1 ORDER BY 1)
LIMIT 1", 2),
+ Arguments.of("legacy split clauses", false, "SELECT 1 LIMIT 1
ORDER BY 1", 2));
+ }
+
+ private static int countOrganizationOwners(ParseTree tree) {
+ int owners = 0;
+ if (tree instanceof QueryOrganizationContext) {
+ QueryOrganizationContext organization = (QueryOrganizationContext)
tree;
+ Assertions.assertTrue(organization.sortClause() != null ||
organization.limitClause() != null);
+ owners++;
+ }
+ for (int index = 0; index < tree.getChildCount(); index++) {
+ owners += countOrganizationOwners(tree.getChild(index));
+ }
+ return owners;
+ }
+
+ @ParameterizedTest(name = "{0}, ansi={1}")
+ @MethodSource("acceptedStatements")
+ void preservesAcceptedQueryOrganizationForms(String description, boolean
ansi, String sql) {
+ Assertions.assertNotNull(new DorisSqlParser(false,
ansi).parseStatement(sql));
+ }
+
+ private static Stream<Arguments> acceptedStatements() {
+ return Stream.of(
+ Arguments.of("ordered select", false, "SELECT 1 ORDER BY 1"),
+ Arguments.of("ordered select", true, "SELECT 1 ORDER BY 1"),
+ Arguments.of("simple limit", false, "SELECT 1 LIMIT 3"),
+ Arguments.of("simple limit", true, "SELECT 1 LIMIT 3"),
+ Arguments.of("limit offset", false, "SELECT 1 LIMIT 3 OFFSET
2"),
+ Arguments.of("limit offset", true, "SELECT 1 LIMIT 3 OFFSET
2"),
+ Arguments.of("comma limit", false, "SELECT 1 LIMIT 2, 3"),
+ Arguments.of("comma limit", true, "SELECT 1 LIMIT 2, 3"),
+ Arguments.of("union tail", false, "SELECT 1 UNION ALL SELECT 2
LIMIT 1"),
+ Arguments.of("union tail", true, "SELECT 1 UNION ALL SELECT 2
LIMIT 1"),
+ Arguments.of("intersect tail", false, "SELECT 1 INTERSECT
SELECT 2 ORDER BY 1"),
+ Arguments.of("intersect tail", true, "SELECT 1 INTERSECT
SELECT 2 ORDER BY 1"),
+ Arguments.of("except tail", false, "SELECT 1 EXCEPT SELECT 2
LIMIT 1"),
+ Arguments.of("except tail", true, "SELECT 1 EXCEPT SELECT 2
LIMIT 1"),
+ Arguments.of("parenthesized set", false,
+ "(SELECT 1 UNION ALL SELECT 2) ORDER BY 1 LIMIT 1"),
+ Arguments.of("parenthesized set", true,
+ "(SELECT 1 UNION ALL SELECT 2) ORDER BY 1 LIMIT 1"),
+ Arguments.of("parenthesized operand", false,
+ "SELECT 1 UNION ALL (SELECT 2 ORDER BY 1 LIMIT 1)"),
+ Arguments.of("parenthesized operand", true,
+ "SELECT 1 UNION ALL (SELECT 2 ORDER BY 1 LIMIT 1)"),
+ Arguments.of("inline values", false,
+ "VALUES (1), (2) ORDER BY 1 LIMIT 1"),
+ Arguments.of("inline values", true,
+ "VALUES (1), (2) ORDER BY 1 LIMIT 1"),
+ Arguments.of("CTE", false,
+ "WITH c AS (SELECT 1 ORDER BY 1 LIMIT 1) SELECT * FROM
c LIMIT 1"),
+ Arguments.of("CTE", true,
+ "WITH c AS (SELECT 1 ORDER BY 1 LIMIT 1) SELECT * FROM
c LIMIT 1"),
+ Arguments.of("derived query", false,
+ "SELECT * FROM (SELECT 1 ORDER BY 1 LIMIT 1) t ORDER
BY 1"),
+ Arguments.of("derived query", true,
+ "SELECT * FROM (SELECT 1 ORDER BY 1 LIMIT 1) t ORDER
BY 1"),
+ Arguments.of("IN subquery", false,
+ "SELECT * FROM t WHERE id IN (SELECT id FROM t ORDER
BY id LIMIT 1)"),
+ Arguments.of("IN subquery", true,
+ "SELECT * FROM t WHERE id IN (SELECT id FROM t ORDER
BY id LIMIT 1)"),
+ Arguments.of("legacy operand clauses", false,
+ "SELECT 1 ORDER BY 1 UNION ALL SELECT 2 LIMIT 1"),
+ Arguments.of("legacy split clauses", false,
+ "SELECT 1 LIMIT 1 ORDER BY 1"));
+ }
+
+ @ParameterizedTest(name = "rejects: {0}, ansi={1}")
+ @MethodSource("rejectedStatements")
+ void preservesRejectedQueryOrganizationForms(String description, boolean
ansi, String sql, int errorPosition) {
+ ParseException exception =
Assertions.assertThrows(ParseException.class,
+ () -> new DorisSqlParser(false, ansi).parseStatement(sql));
+ Assertions.assertTrue(exception.getMessage().contains("line 1, pos " +
errorPosition), exception::getMessage);
+ }
+
+ private static Stream<Arguments> rejectedStatements() {
+ return Stream.of(
+ Arguments.of("ANSI unparenthesized operand order", true,
+ "SELECT 1 ORDER BY 1 UNION ALL SELECT 2", 9),
+ Arguments.of("order after limit", true, "SELECT 1 LIMIT 1
ORDER BY 1", 9),
+ Arguments.of("truncated order", false, "SELECT 1 ORDER BY", 9),
+ Arguments.of("truncated order", true, "SELECT 1 ORDER BY", 17),
+ Arguments.of("truncated limit", false, "SELECT 1 LIMIT", 9),
+ Arguments.of("truncated limit", true, "SELECT 1 LIMIT", 14),
+ Arguments.of("truncated trailing limit", false, "SELECT 1
ORDER BY 1 LIMIT", 20),
+ Arguments.of("truncated trailing limit", true, "SELECT 1 ORDER
BY 1 LIMIT", 20),
+ Arguments.of("truncated VALUES limit", false, "VALUES (1)
ORDER BY 1 LIMIT", 22),
+ Arguments.of("truncated VALUES limit", true, "VALUES (1) ORDER
BY 1 LIMIT", 22),
+ Arguments.of("truncated legacy second clause", false, "SELECT
1 LIMIT 1 ORDER BY", 25));
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
