This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d145a5d4983 [improvement](parser) Skip hidden tokens in internal SQL
parsing (#67434)
d145a5d4983 is described below
commit d145a5d4983420e3a84b59d2ddda952173524cee
Author: morrySnow <[email protected]>
AuthorDate: Thu Sep 3 15:53:17 2026 +0800
[improvement](parser) Skip hidden tokens in internal SQL parsing (#67434)
### What problem does this PR solve?
Problem Summary: Internal Nereids AST parsing allocates whitespace and
ordinary line-comment tokens even though ANTLR does not consume
hidden-channel tokens. This independent PR adds an opt-in lean lexer
mode for direct AST construction and skips only `WS` and
`SIMPLE_COMMENT`. Public lexer, scan, and comment-normalization paths
remain in full-token mode; `BRACKETED_COMMENT` remains on channel 2 for
hints; character offsets are unchanged. Syntax errors are emitted
directly from the lean stream. Because ANTLR builds some diagnostics
from token ranges, the lean stream reconstructs those ranges from the
original character stream, preserving the existing exception type,
position, and exact diagnostic text without reparsing with full tokens.
### Benchmark
Lower allocation is better. Normalized allocation (`gc.alloc.rate.norm`)
is the primary metric because wall-clock results were affected by host
scheduling noise.
- Host: MacBookPro17,1, Apple M1 (8 cores, 16 GB), macOS 15.0.1
- Runtime: OpenJDK 17.0.20.1, ANTLR 4.13.1, JMH 1.37, 1 thread, 1 GB
heap
- JMH: 3 forks, 4 x 300 ms warmup, 7 x 400 ms measurement, `-prof gc`
- Baseline: `049410596f4` (`upstream/master`); parser JAR SHA-256
`923ed2a22142ee9b5dcfefbba5766b9696a653a4218e8208e42a61270e9d986f`;
benchmark JAR SHA-256
`f88c09c16f5872ff1e8cef2368e83622cf9cc6e4217350cc845a862dbaa92f56`
- Candidate: `5ef28695247`; parser JAR SHA-256
`20104fd49c902833898e1132f6fb6b2bf200a7608356546d92829543e581593e`;
benchmark JAR SHA-256
`6ecd1a5566f386729b0a7c8b1c4f3db7e7719975cb06512ed04a5f1fc8e33b64`
- Follow-up `89eb5688fdc` only changes syntax-error handling; the
successful parsing path measured below is unchanged, so the existing
benchmark results are reused.
- Workloads: typical SELECT, 32-column comment-heavy SELECT, 128-column
wide SELECT, and hinted join SELECT
Full and lean values below are measured in the same candidate artifact,
so token mode is the only changed variable. Values are bytes/op.
| Path | Workload | Full | Lean | Latency
Improvement |
| :------------- | :------------ | --------: | --------: |
------------------: |
| Tokenize | Typical | 3,448.0 | 2,541.4 | **26.3%
faster** |
| Tokenize | Comment-heavy | 12,874.8 | 6,544.2 | **49.2%
faster** |
| Tokenize | Wide SELECT | 63,832.1 | 40,754.9 | **36.2%
faster** |
| Tokenize | Hinted | 3,328.0 | 2,565.4 | **22.9%
faster** |
| Parser CST | Typical | 15,319.7 | 14,605.6 | **4.7%
faster** |
| Parser CST | Comment-heavy | 52,838.8 | 46,501.7 | **12.0%
faster** |
| Parser CST | Wide SELECT | 263,463.9 | 240,372.4 | **8.8%
faster** |
| Parser CST | Hinted | 15,098.4 | 14,503.0 | **3.9%
faster** |
| FE LogicalPlan | Typical | 29,080.4 | 28,229.5 | **2.9%
faster** |
| FE LogicalPlan | Comment-heavy | 89,697.9 | 83,261.4 | **7.2%
faster** |
| FE LogicalPlan | Wide SELECT | 485,763.7 | 462,621.3 | **4.8%
faster** |
| FE LogicalPlan | Hinted | 34,862.4 | 34,062.6 | **2.3%
faster** |
The reduction comes from avoiding `CommonToken` allocation for
whitespace and ordinary line comments. Hint and default-channel tokens
are still allocated. A frozen-artifact B/C/C/B control with lean mode
disabled showed no normalized-allocation regression for `SELECT 1`
(-0.8%) or a typical SELECT (-0.6%). Control latency was
scheduling-sensitive, including one severely disturbed baseline fork, so
it is not used as acceptance evidence.
Commands:
```shell
cd fe
mvn -Pbenchmark -pl fe-sql-parser,fe-sql-parser-benchmark \
-Dmaven.build.cache.enabled=false package
java -jar fe-sql-parser-benchmark/target/doris-fe-sql-parser-benchmarks.jar
\
LeanTokenModeBenchmark -prof gc
```
### Semantic differential
- Corpus: all 4,610 tracked `*.sql` files; SHA-256 of `git ls-files -s
-- '*.sql'` is
`567e209d57e5eaf6546ff03bf887437b8d647ed5f7ecb85bc657b987dd04be10`
- Current `master` and candidate both accepted 4,063 files and rejected
547 files in Legacy and ANSI modes
- Baseline/candidate CST and exact-error-message signatures matched
byte-for-byte: Legacy
`8036532e957b06a1bbc106338a1caa08cbc9acbd2d08f6eb4fcd7665f6881b2a`; ANSI
`e27f127f60bcc1ce47bbc4ec4cc83351acc0adceb735488fdcb9853727e2c394`
- Candidate full versus lean: 18,440 combinations across the corpus,
`noBackslashEscapes`, and ANSI modes had identical filtered token tuples
and CST signatures
- Focused FE tests cover hints, full-token public consumers, CREATE VIEW
and sync-MV source intervals, encryption offsets, and exact invalid-SQL
diagnostics without a full-token retry
- An 8-thread test covers deterministic lexing with ANTLR's shared
static DFA
---
.../apache/doris/nereids/parser/NereidsParser.java | 38 +++++-
.../doris/nereids/parser/LeanTokenModeTest.java | 123 ++++++++++++++++++
.../benchmark/LeanTokenModeBenchmark.java | 104 +++++++++++++++
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 11 +-
.../sqlparser/DorisLexerLeanTokenModeTest.java | 143 +++++++++++++++++++++
5 files changed, 412 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
index c7a7bed074e..2c3775dc626 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
@@ -49,6 +49,7 @@ import org.antlr.v4.runtime.Recognizer;
import org.antlr.v4.runtime.Token;
import org.antlr.v4.runtime.TokenSource;
import org.antlr.v4.runtime.atn.PredictionMode;
+import org.antlr.v4.runtime.misc.Interval;
import org.antlr.v4.runtime.misc.ParseCancellationException;
import org.antlr.v4.runtime.tree.TerminalNode;
import org.apache.commons.collections4.CollectionUtils;
@@ -343,7 +344,7 @@ public class NereidsParser {
private <T> T parse(String sql, @Nullable LogicalPlanBuilder
logicalPlanBuilder,
Function<DorisParser, ParserRuleContext>
parseFunction) {
- CommonTokenStream tokenStream = parseAllTokens(sql);
+ CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream, parseFunction);
LogicalPlanBuilder realLogicalPlanBuilder = logicalPlanBuilder == null
? new LogicalPlanBuilder(getHintMap(sql, tokenStream,
DorisParser::selectHint))
@@ -352,7 +353,7 @@ public class NereidsParser {
}
public LogicalPlan parseForCreateView(String sql) {
- CommonTokenStream tokenStream = parseAllTokens(sql);
+ CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream,
DorisParser::singleStatement);
LogicalPlanBuilder realLogicalPlanBuilder = new
LogicalPlanBuilderForCreateView(
getHintMap(sql, tokenStream, DorisParser::selectHint));
@@ -360,7 +361,7 @@ public class NereidsParser {
}
public LogicalPlan parseForEncryption(String sql, Map<Pair<Integer,
Integer>, String> indexInSqlToString) {
- CommonTokenStream tokenStream = parseAllTokens(sql);
+ CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream,
DorisParser::singleStatement);
LogicalPlanBuilder realLogicalPlanBuilder = new
LogicalPlanBuilderForEncryption(
getHintMap(sql, tokenStream, DorisParser::selectHint),
indexInSqlToString);
@@ -369,7 +370,7 @@ public class NereidsParser {
/** parseForSyncMv */
public Optional<String> parseForSyncMv(String sql) {
- CommonTokenStream tokenStream = parseAllTokens(sql);
+ CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream,
DorisParser::singleStatement);
LogicalPlanBuilderForSyncMv logicalPlanBuilderForSyncMv = new
LogicalPlanBuilderForSyncMv(
getHintMap(sql, tokenStream, DorisParser::selectHint));
@@ -466,10 +467,37 @@ public class NereidsParser {
}
private static CommonTokenStream parseAllTokens(String sql) {
+ return parseTokens(sql, false);
+ }
+
+ private static CommonTokenStream parseLeanTokens(String sql) {
+ return parseTokens(sql, true);
+ }
+
+ private static CommonTokenStream parseTokens(String sql, boolean
leanTokenMode) {
DorisLexer lexer = new DorisLexer(new
CaseInsensitiveStream(CharStreams.fromString(sql)));
lexer.isNoBackslashEscapes = SqlModeHelper.hasNoBackSlashEscapes();
- CommonTokenStream tokenStream = new CommonTokenStream(lexer);
+ lexer.isLeanTokenMode = leanTokenMode;
+ CommonTokenStream tokenStream = leanTokenMode
+ ? new LeanTokenStream(lexer)
+ : new CommonTokenStream(lexer);
tokenStream.fill();
return tokenStream;
}
+
+ /** Preserve source text used by ANTLR diagnostics without allocating
hidden tokens. */
+ private static final class LeanTokenStream extends CommonTokenStream {
+ private LeanTokenStream(TokenSource tokenSource) {
+ super(tokenSource);
+ }
+
+ @Override
+ public String getText(Token start, Token stop) {
+ if (start == null || stop == null || start.getType() == Token.EOF)
{
+ return "";
+ }
+ return getTokenSource().getInputStream().getText(
+ Interval.of(start.getStartIndex(), stop.getStopIndex()));
+ }
+ }
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/LeanTokenModeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/LeanTokenModeTest.java
new file mode 100644
index 00000000000..882cb453952
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/LeanTokenModeTest.java
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.DorisLexer;
+import org.apache.doris.nereids.analyzer.UnboundSlot;
+import org.apache.doris.nereids.exceptions.ParseException;
+import org.apache.doris.nereids.properties.SelectHintOrdered;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSelectHint;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.GlobalVariable;
+import org.apache.doris.qe.SqlModeHelper;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.TokenSource;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.List;
+
+class LeanTokenModeTest extends ParserTestBase {
+ private final NereidsParser parser = new NereidsParser();
+
+ @Test
+ void preservesHintsInDirectAstParsing() {
+ LogicalPlan plan = parser.parseSingle("SELECT /*+ ORDERED */ a --
ordinary\nFROM t");
+ List<LogicalSelectHint<Plan>> hintPlans =
plan.collectToList(LogicalSelectHint.class::isInstance);
+
+ Assertions.assertEquals(1, hintPlans.size());
+ Assertions.assertTrue(hintPlans.get(0).getHints().stream()
+ .anyMatch(SelectHintOrdered.class::isInstance));
+ }
+
+ @Test
+ void keepsFullTokensForScanAndCommentNormalization() {
+ String sql = "SELECT /* ordinary */ /*+ ORDERED */ a -- tail\nFROM t";
+ List<Integer> tokenTypes = new ArrayList<>();
+ TokenSource tokenSource = NereidsParser.scan(sql);
+ Token token;
+ do {
+ token = tokenSource.nextToken();
+ tokenTypes.add(token.getType());
+ } while (token.getType() != Token.EOF);
+
+ Assertions.assertTrue(tokenTypes.contains(DorisLexer.WS));
+ Assertions.assertTrue(tokenTypes.contains(DorisLexer.SIMPLE_COMMENT));
+
Assertions.assertTrue(tokenTypes.contains(DorisLexer.BRACKETED_COMMENT));
+ Assertions.assertEquals("SELECT /*+ ORDERED */ a FROM t",
+ NereidsParser.removeCommentAndTrimBlank(sql));
+ }
+
+ @Test
+ void preservesCharacterIntervalsAcrossSkippedTokens() {
+ String sql = "SELECT /* leading */ db -- between\n . tbl . col FROM t";
+ LogicalPlan plan = parser.parseForCreateView(sql);
+ UnboundSlot slot = plan.<LogicalPlan>collectToList(ignored ->
true).stream()
+ .flatMap(node -> node.getExpressions().stream())
+ .flatMap(expression -> expression.<UnboundSlot>collectToList(
+ UnboundSlot.class::isInstance).stream())
+ .filter(unboundSlot ->
unboundSlot.getNameParts().equals(List.of("db", "tbl", "col")))
+ .findFirst()
+ .orElseThrow();
+
+ Assertions.assertEquals(Pair.of(sql.indexOf("db"), sql.indexOf("col")
+ 2),
+ slot.getIndexInSqlString().orElseThrow());
+ Assertions.assertEquals(sql, parser.parseForSyncMv(
+ "CREATE MATERIALIZED VIEW mv AS " + sql).orElseThrow());
+ }
+
+ @Test
+ void producesSameErrorsWithoutHiddenTokens() {
+ boolean previousAnsi =
GlobalVariable.enable_ansi_query_organization_behavior;
+ long previousSqlMode =
ConnectContext.get().getSessionVariable().getSqlMode();
+ try {
+ for (boolean noBackslashEscapes : new boolean[] {false, true}) {
+ long sqlMode = noBackslashEscapes
+ ? previousSqlMode |
SqlModeHelper.MODE_NO_BACKSLASH_ESCAPES
+ : previousSqlMode &
~SqlModeHelper.MODE_NO_BACKSLASH_ESCAPES;
+ ConnectContext.get().getSessionVariable().setSqlMode(sqlMode);
+ for (boolean ansi : new boolean[] {false, true}) {
+ GlobalVariable.enable_ansi_query_organization_behavior =
ansi;
+ for (String sql : List.of(
+ "SELECT a\n-- skipped comment\nFROM t WHERE )",
+ "SELECT 1,\n-- missing expression\nFROM t",
+ "SELECT * FROM -- missing relation\r\n",
+ "SELECT /*+ ORDERED */ FROM t",
+ "CREATE TABLE t ( k1 BOOL )",
+ "SELECT 'a\\' FROM t WHERE )")) {
+ ParseException fullException =
Assertions.assertThrows(ParseException.class,
+ () -> new DorisSqlParser(noBackslashEscapes,
ansi).parseStatement(sql));
+ ParseException leanException =
Assertions.assertThrows(ParseException.class,
+ () -> parser.parseSingle(sql));
+ Assertions.assertEquals(fullException.getClass(),
leanException.getClass());
+ Assertions.assertEquals(fullException.getMessage(),
leanException.getMessage());
+ }
+ }
+ }
+ } finally {
+ GlobalVariable.enable_ansi_query_organization_behavior =
previousAnsi;
+
ConnectContext.get().getSessionVariable().setSqlMode(previousSqlMode);
+ }
+ }
+}
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LeanTokenModeBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LeanTokenModeBenchmark.java
new file mode 100644
index 00000000000..ee964394708
--- /dev/null
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/LeanTokenModeBenchmark.java
@@ -0,0 +1,104 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser.benchmark;
+
+import org.apache.doris.nereids.DorisLexer;
+import org.apache.doris.nereids.DorisParser;
+import org.apache.doris.sqlparser.DorisSqlParser;
+
+import org.antlr.v4.runtime.CommonTokenStream;
+import org.antlr.v4.runtime.atn.PredictionMode;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/** Measures whitespace and ordinary-comment token allocation in full and lean
modes. */
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MICROSECONDS)
+@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
+@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
+@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
+@State(Scope.Thread)
+public class LeanTokenModeBenchmark {
+ @Param({"false", "true"})
+ public boolean leanTokenMode;
+
+ @Param({"typical", "commentHeavy", "wideSelect", "hinted"})
+ public String workload;
+
+ private final DorisSqlParser facade = new DorisSqlParser();
+ private String sql;
+
+ @Setup(Level.Trial)
+ public void setUp() {
+ switch (workload) {
+ case "typical":
+ sql = "SELECT a, b, c FROM t WHERE a > 1 AND b < 10 ORDER BY c
LIMIT 20";
+ break;
+ case "commentHeavy":
+ sql = IntStream.range(0, 32)
+ .mapToObj(index -> "c" + index + " -- column " + index
+ "\n")
+ .collect(Collectors.joining(", ", "SELECT ", "FROM t
WHERE c0 > 0"));
+ break;
+ case "wideSelect":
+ sql = IntStream.range(0, 128)
+ .mapToObj(index -> "c" + index + " AS alias_" + index)
+ .collect(Collectors.joining(", ", "SELECT ", " FROM
wide_table"));
+ break;
+ case "hinted":
+ sql = "SELECT /*+ ORDERED */ a, b FROM t1 JOIN t2 ON t1.id =
t2.id WHERE a > 1";
+ break;
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+ }
+
+ @Benchmark
+ public Object tokenize() {
+ DorisLexer lexer = newLexer();
+ CommonTokenStream tokenStream = new CommonTokenStream(lexer);
+ tokenStream.fill();
+ return tokenStream.getTokens();
+ }
+
+ @Benchmark
+ public Object parseStatement() {
+ DorisParser parser = facade.newParser(newLexer());
+ parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
+ return parser.singleStatement();
+ }
+
+ private DorisLexer newLexer() {
+ DorisLexer lexer = facade.newLexer(sql);
+ lexer.isLeanTokenMode = leanTokenMode;
+ return lexer;
+ }
+}
diff --git
a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
index 7217bdb7f7f..4e7987e7f9e 100644
--- a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
+++ b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
@@ -21,6 +21,13 @@ lexer grammar DorisLexer;
@members {
public boolean isNoBackslashEscapes = false;
+ public boolean isLeanTokenMode = false;
+
+ private void skipInLeanTokenMode() {
+ if (isLeanTokenMode) {
+ skip();
+ }
+ }
/**
* Verify whether current token is a valid decimal token (which contains
dot).
@@ -748,7 +755,7 @@ fragment LETTER
;
SIMPLE_COMMENT
- : '--' ('\\\n' | ~[\r\n])* '\r'? '\n'? -> channel(HIDDEN)
+ : '--' ('\\\n' | ~[\r\n])* '\r'? '\n'? {skipInLeanTokenMode();} ->
channel(HIDDEN)
;
BRACKETED_COMMENT
@@ -757,7 +764,7 @@ BRACKETED_COMMENT
WS
- : [ \r\n\t]+ -> channel(HIDDEN)
+ : [ \r\n\t]+ {skipInLeanTokenMode();} -> channel(HIDDEN)
;
// Catch-all for anything we can't recognize.
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisLexerLeanTokenModeTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisLexerLeanTokenModeTest.java
new file mode 100644
index 00000000000..3b13df8ef9b
--- /dev/null
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisLexerLeanTokenModeTest.java
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.sqlparser;
+
+import org.apache.doris.nereids.DorisLexer;
+
+import org.antlr.v4.runtime.Token;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+class DorisLexerLeanTokenModeTest {
+ private static final String SQL = "SELECT /* ordinary */ /*+ ORDERED */ a
-- tail\r\nFROM t";
+
+ @Test
+ void keepsFullTokenStreamByDefault() {
+ Assertions.assertEquals(Arrays.asList(
+ "SELECT|SELECT|0|0|5|1|0",
+ "WS| |1|6|6|1|6",
+ "BRACKETED_COMMENT|/* ordinary */|2|7|20|1|7",
+ "WS| |1|21|21|1|21",
+ "BRACKETED_COMMENT|/*+ ORDERED */|2|22|35|1|22",
+ "WS| |1|36|36|1|36",
+ "IDENTIFIER|a|0|37|37|1|37",
+ "WS| |1|38|38|1|38",
+ "SIMPLE_COMMENT|-- tail\\r\\n|1|39|47|1|39",
+ "FROM|FROM|0|48|51|2|0",
+ "WS| |1|52|52|2|4",
+ "IDENTIFIER|t|0|53|53|2|5",
+ "EOF|<EOF>|0|54|53|2|6"),
+ snapshot(SQL, false, false));
+ }
+
+ @Test
+ void skipsOnlyWhitespaceAndSimpleComments() {
+ List<String> inputs = Arrays.asList(
+ SQL,
+ "SELECT 'a\\'b' -- comment\n, \"x\\\"y\"",
+ "SELECT 中文😀\tFROM 表 -- 尾注释",
+ "-- first\n/* regular */ /*+ SET_VAR(query_timeout=1) */
SELECT 1");
+
+ for (boolean noBackslashEscapes : new boolean[] {false, true}) {
+ for (String sql : inputs) {
+ List<Token> fullTokens = lex(sql, noBackslashEscapes, false);
+ List<Token> expectedLeanTokens = fullTokens.stream()
+ .filter(token -> token.getType() != DorisLexer.WS
+ && token.getType() !=
DorisLexer.SIMPLE_COMMENT)
+ .collect(Collectors.toList());
+ List<Token> leanTokens = lex(sql, noBackslashEscapes, true);
+
+ Assertions.assertEquals(snapshot(expectedLeanTokens),
snapshot(leanTokens));
+ Assertions.assertTrue(leanTokens.stream()
+ .filter(token -> token.getType() ==
DorisLexer.BRACKETED_COMMENT)
+ .allMatch(token -> token.getChannel() == 2));
+ }
+ }
+ }
+
+ @Test
+ void lexesDeterministicallyWithSharedStaticDfa() throws Exception {
+ List<List<String>> expected = new ArrayList<>();
+ for (boolean noBackslashEscapes : new boolean[] {false, true}) {
+ for (boolean leanTokenMode : new boolean[] {false, true}) {
+ expected.add(snapshot(SQL, noBackslashEscapes, leanTokenMode));
+ }
+ }
+
+ ExecutorService executor = Executors.newFixedThreadPool(8);
+ try {
+ List<Callable<Void>> tasks = new ArrayList<>();
+ for (int thread = 0; thread < 8; thread++) {
+ tasks.add(() -> {
+ for (int repetition = 0; repetition < 100; repetition++) {
+ int caseIndex = 0;
+ for (boolean noBackslashEscapes : new boolean[]
{false, true}) {
+ for (boolean leanTokenMode : new boolean[] {false,
true}) {
+
Assertions.assertEquals(expected.get(caseIndex++),
+ snapshot(SQL, noBackslashEscapes,
leanTokenMode));
+ }
+ }
+ }
+ return null;
+ });
+ }
+ for (Future<Void> result : executor.invokeAll(tasks)) {
+ result.get();
+ }
+ } finally {
+ executor.shutdownNow();
+ Assertions.assertTrue(executor.awaitTermination(10,
TimeUnit.SECONDS));
+ }
+ }
+
+ private static List<String> snapshot(String sql, boolean
noBackslashEscapes, boolean leanTokenMode) {
+ return snapshot(lex(sql, noBackslashEscapes, leanTokenMode));
+ }
+
+ private static List<String> snapshot(List<Token> tokens) {
+ return tokens.stream().map(token -> {
+ String type =
DorisLexer.VOCABULARY.getSymbolicName(token.getType());
+ String text = token.getText().replace("\r", "\\r").replace("\n",
"\\n");
+ return type + "|" + text + "|" + token.getChannel() + "|"
+ + token.getStartIndex() + "|" + token.getStopIndex() + "|"
+ + token.getLine() + "|" + token.getCharPositionInLine();
+ }).collect(Collectors.toList());
+ }
+
+ private static List<Token> lex(String sql, boolean noBackslashEscapes,
boolean leanTokenMode) {
+ DorisLexer lexer = new DorisSqlParser(noBackslashEscapes,
false).newLexer(sql);
+ lexer.isLeanTokenMode = leanTokenMode;
+ List<Token> tokens = new ArrayList<>();
+ Token token;
+ do {
+ token = lexer.nextToken();
+ tokens.add(token);
+ } while (token.getType() != Token.EOF);
+ return tokens;
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
