This is an automated email from the ASF dual-hosted git repository.
morrySnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new cdb05784a44 [improvement](parser) Localize identifier post-processing
(#67457)
cdb05784a44 is described below
commit cdb05784a4410fb73ef447b93e6760ae7bef4803
Author: morrySnow <[email protected]>
AuthorDate: Thu Sep 3 15:56:40 2026 +0800
[improvement](parser) Localize identifier post-processing (#67457)
### What problem does this PR solve?
Problem Summary:
`PostProcessor` was installed as a global ANTLR parse listener, so every
rule exit paid listener-dispatch cost even though only three local
actions were needed: normalizing non-reserved identifiers, normalizing
quoted identifiers, and reporting malformed unquoted identifiers.
This PR moves those actions into the corresponding grammar rules and
removes the listener from both `DorisSqlParser` and `NereidsParser`. It
preserves identifier token text/type/positions, double-backtick
unescaping, malformed-identifier errors, and parsing with
`buildParseTree=false`.
### Benchmark
Environment:
- Baseline: `760b14ee62c` (benchmark-only commit on `f054492cbb9`)
- Candidate: `5b45223bc61`
- JDK 17.0.20.1, JMH 1.37, macOS arm64
- 1 thread, 3 forks, `-Xms1g -Xmx1g`, 4 x 300 ms warmup, 7 x 400 ms
measurement, `-prof gc`
- Baseline is the average of two complete runs (B1/B2). Candidate error
is JMH's 99.9% confidence interval.
- A preliminary candidate run affected by host interference was
discarded (`typical` parser-only was 88.8 us/op instead of the stable
7-9 us/op range); the final artifact was rebuilt and measured in a clean
run.
Build and run:
```shell
LC_ALL=en_US.UTF-8 mvn -Pbenchmark -pl
fe-sql-parser,fe-sql-parser-benchmark -am \
-Dmaven.build.cache.enabled=false package
java -jar
fe/fe-sql-parser-benchmark/target/fe-sql-parser-benchmark-*-benchmark.jar \
'IdentifierPostProcessorBenchmark.*' -prof gc -rf json -rff result.json
```
Time is `us/op`; positive change means faster. The workloads cover a
control query, a typical aggregate query, a 64-column query, a
non-reserved-keyword-heavy query, and a quoted-identifier-heavy query.
| Path / Workload | Baseline (B1 / B2) | Candidate |
Latency Improvement | Allocation (B/op): Baseline → Candidate |
|:--------------------------|-----------------------:|----------------:|--------------------:|----------------------------------------:|
| end-to-end / control | 2.817 / 2.694 | 2.191 ± 0.041 |
**25.8% faster** | 4,933.4 → 4,848.0 (-1.73%) |
| end-to-end / typical | 17.143 / 18.288 | 17.173 ± 1.742 |
**3.2% faster** | 18,754.0 → 18,658.9 (-0.51%) |
| end-to-end / wide | 66.555 / 72.165 | 60.816 ± 0.731 |
**14.0% faster** | 141,501.2 → 141,432.9 (-0.05%) |
| end-to-end / nonReserved | 12.278 / 14.034 | 11.044 ± 0.276 |
**19.1% faster** | 22,844.4 → 22,530.9 (-1.37%) |
| end-to-end / quoted | 13.154 / 11.128 | 8.089 ± 1.121 |
**50.1% faster** | 16,081.5 → 15,984.1 (-0.61%) |
| parser-only / control | 1.703 / 1.841 | 1.550 ± 0.016 |
**14.3% faster** | 3,824.0 → 3,744.0 (-2.09%) |
| parser-only / typical | 8.655 / 8.565 | 7.868 ± 0.096 |
**9.4% faster** | 13,872.1 → 13,792.1 (-0.58%) |
| parser-only / wide | 60.700 / 59.489 | 55.784 ± 0.624 |
**7.7% faster** | 115,632.9 → 115,552.8 (-0.07%) |
| parser-only / nonReserved | 10.127 / 10.148 | 9.863 ± 0.756 |
**2.8% faster** | 19,261.5 → 19,189.5 (-0.37%) |
| parser-only / quoted | 6.985 / 7.116 | 6.658 ± 0.186 |
**5.9% faster** | 13,440.1 → 13,346.8 (-0.69%) |
The direct parser path improves by 2.8%-14.3% and allocates less in
every workload. End-to-end means also improve in every workload; the
larger quoted/control figures have more host-level variance and are not
used as the primary conclusion. The gain comes from removing global
rule-exit listener dispatch; the remaining allocation reduction comes
from avoiding listener bookkeeping.
### Semantic verification
- Parsed all 4,610 tracked `.sql` files in default and ANSI modes.
Acceptance/error class/error position/statement count matched the frozen
baseline in all 9,220 cases.
- Tracked SQL path-list SHA-256:
`c1fdf48a22d311f7164516b512e850bef373126be1d3079148c109e5f80f80c1`.
- Detailed CST/token/error snapshots cover ordinary, non-reserved,
quoted, doubled-backtick and multipart identifiers, expressions, DDL,
both parser entry paths, and `test-table` / `test-tbl` errors. Every
rule node and token field matched the baseline.
- Verified the advanced parser path with `buildParseTree=false`.
---
.../apache/doris/nereids/parser/NereidsParser.java | 2 -
.../generator/PatternDescribableProcessor.java | 4 -
.../doris/nereids/parser/NereidsParserTest.java | 2 +-
....java => IdentifierPostProcessorBenchmark.java} | 60 +++++++--------
.../benchmark/PrimaryExpressionBenchmark.java | 3 -
.../benchmark/QueryOrDmlCommonPrefixBenchmark.java | 3 -
fe/fe-sql-parser/README.md | 6 +-
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 32 ++++++++
.../apache/doris/nereids/parser/PostProcessor.java | 71 ------------------
.../org/apache/doris/sqlparser/DorisSqlParser.java | 3 -
.../apache/doris/sqlparser/DorisSqlParserTest.java | 87 ++++++++++++++++++++++
11 files changed, 150 insertions(+), 123 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
index 2c3775dc626..5e532350df4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
@@ -73,7 +73,6 @@ import javax.annotation.Nullable;
public class NereidsParser {
public static final Logger LOG = LogManager.getLogger(NereidsParser.class);
private static final ParseErrorListener PARSE_ERROR_LISTENER = new
ParseErrorListener();
- private static final PostProcessor POST_PROCESSOR = new PostProcessor();
private static final BitSet EXPLAIN_TOKENS = new BitSet();
@@ -410,7 +409,6 @@ public class NereidsParser {
CommonTokenStream tokenStream, Function<DorisParser,
ParserRuleContext> parseFunction) {
DorisParser parser = new DorisParser(tokenStream);
parser.ansiSQLSyntax =
GlobalVariable.enable_ansi_query_organization_behavior;
- parser.addParseListener(POST_PROCESSOR);
parser.removeErrorListeners();
parser.addErrorListener(PARSE_ERROR_LISTENER);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/PatternDescribableProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/PatternDescribableProcessor.java
index e12ccd44189..1e927040350 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/PatternDescribableProcessor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/pattern/generator/PatternDescribableProcessor.java
@@ -152,10 +152,6 @@ public class PatternDescribableProcessor extends
AbstractProcessor {
}
});
- // parser.addParseListener(PostProcessor)
- // parser.removeErrorListeners()
- // parser.addErrorListener(ParseErrorListener)
-
ParserRuleContext tree;
try {
// first, try parsing with potentially faster SLL mode
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java
index c6e783e8b7c..e92dc04f10b 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java
@@ -300,7 +300,7 @@ public class NereidsParserTest extends ParserTestBase {
}
@Test
- public void testPostProcessor() {
+ public void testQuotedIdentifierNormalization() {
parsePlan("select `AD``D` from t1 where a = 1")
.matches(
logicalProject().when(p ->
"AD`D".equals(p.getProjects().get(0).getName()))
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/IdentifierPostProcessorBenchmark.java
similarity index 71%
copy from
fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
copy to
fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/IdentifierPostProcessorBenchmark.java
index fca29e73bc4..27f6f9ad87d 100644
---
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/IdentifierPostProcessorBenchmark.java
@@ -19,7 +19,6 @@ package org.apache.doris.sqlparser.benchmark;
import org.apache.doris.nereids.DorisParser;
import org.apache.doris.nereids.parser.ParseErrorListener;
-import org.apache.doris.nereids.parser.PostProcessor;
import org.apache.doris.sqlparser.DorisSqlParser;
import org.antlr.v4.runtime.CommonTokenStream;
@@ -44,19 +43,18 @@ import java.util.concurrent.TimeUnit;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
-/** Measures the grammar change both in isolation and through the public
parser facade. */
+/** Measures identifier post-processing through the public facade and with
pre-tokenized input. */
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@Fork(value = 3, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
@Warmup(iterations = 4, time = 300, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 7, time = 400, timeUnit = TimeUnit.MILLISECONDS)
@State(Scope.Thread)
-public class PrimaryExpressionBenchmark {
- @Param({"control", "typical", "specialForms", "postfixChain",
"wideProjection"})
+public class IdentifierPostProcessorBenchmark {
+ @Param({"control", "typical", "wide", "nonReserved", "quoted"})
public String workload;
private final DorisSqlParser facade = new DorisSqlParser();
- private final PostProcessor postProcessor = new PostProcessor();
private final ParseErrorListener errorListener = new ParseErrorListener();
private String sql;
@@ -64,33 +62,7 @@ public class PrimaryExpressionBenchmark {
@Setup(Level.Trial)
public void setUp() {
- switch (workload) {
- case "control":
- sql = "SELECT 1";
- break;
- case "typical":
- sql = "SELECT a + b * c FROM t "
- + "WHERE (d = 1 OR e[2].f > 3) AND g IS NOT NULL";
- break;
- case "specialForms":
- sql = "SELECT CASE a WHEN 1 THEN CONVERT(b USING utf8) "
- + "WHEN 2 THEN CAST(c AS BIGINT) ELSE d + 1 END FROM t
"
- + "WHERE CASE WHEN e > 0 THEN TRUE ELSE FALSE END";
- break;
- case "postfixChain":
- sql = "SELECT fn(a)[1:2][3].field[4].nested[5:6].leaf "
- + "COLLATE utf8_general_ci FROM t";
- break;
- case "wideProjection":
- sql = "SELECT " + IntStream.range(0, 64)
- .mapToObj(i -> "c" + i + " + " + i)
- .collect(Collectors.joining(", "))
- + " FROM t WHERE key_col[1].field > 0";
- break;
- default:
- throw new IllegalArgumentException("Unknown workload: " +
workload);
- }
-
+ sql = statement(workload);
CommonTokenStream stream = new CommonTokenStream(facade.newLexer(sql));
stream.fill();
tokens = List.copyOf(stream.getTokens());
@@ -105,10 +77,32 @@ public class PrimaryExpressionBenchmark {
public Object parsePreTokenized() {
CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(tokens));
DorisParser parser = new DorisParser(stream);
- parser.addParseListener(postProcessor);
parser.removeErrorListeners();
parser.addErrorListener(errorListener);
parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
return parser.singleStatement();
}
+
+ private static String statement(String workload) {
+ switch (workload) {
+ case "control":
+ return "SELECT 1";
+ case "typical":
+ return "SELECT customer_id, sum(amount) AS total FROM sales AS
s "
+ + "WHERE region = 'east' GROUP BY customer_id ORDER BY
total DESC LIMIT 10";
+ case "wide":
+ return "SELECT " + IntStream.range(0, 64)
+ .mapToObj(index -> "c" + index + " AS alias" + index)
+ .collect(Collectors.joining(", "))
+ + " FROM catalog.db.fact_table";
+ case "nonReserved":
+ return "SELECT action, branch, cache, catalog, connection,
engine, format, global, name "
+ + "FROM aggregate AS alias WHERE action = 1";
+ case "quoted":
+ return "SELECT `AD``D`, `Mixed Name`, `select` FROM
`db``name`.`table name` AS `t``1` "
+ + "WHERE `t``1`.`AD``D` > 0";
+ default:
+ throw new IllegalArgumentException("Unknown workload: " +
workload);
+ }
+ }
}
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
index fca29e73bc4..0938338b1c4 100644
---
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/PrimaryExpressionBenchmark.java
@@ -19,7 +19,6 @@ package org.apache.doris.sqlparser.benchmark;
import org.apache.doris.nereids.DorisParser;
import org.apache.doris.nereids.parser.ParseErrorListener;
-import org.apache.doris.nereids.parser.PostProcessor;
import org.apache.doris.sqlparser.DorisSqlParser;
import org.antlr.v4.runtime.CommonTokenStream;
@@ -56,7 +55,6 @@ public class PrimaryExpressionBenchmark {
public String workload;
private final DorisSqlParser facade = new DorisSqlParser();
- private final PostProcessor postProcessor = new PostProcessor();
private final ParseErrorListener errorListener = new ParseErrorListener();
private String sql;
@@ -105,7 +103,6 @@ public class PrimaryExpressionBenchmark {
public Object parsePreTokenized() {
CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(tokens));
DorisParser parser = new DorisParser(stream);
- parser.addParseListener(postProcessor);
parser.removeErrorListeners();
parser.addErrorListener(errorListener);
parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
diff --git
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
index b2700dce7f8..9efd78abbb9 100644
---
a/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
+++
b/fe/fe-sql-parser-benchmark/src/main/java/org/apache/doris/sqlparser/benchmark/QueryOrDmlCommonPrefixBenchmark.java
@@ -19,7 +19,6 @@ package org.apache.doris.sqlparser.benchmark;
import org.apache.doris.nereids.DorisParser;
import org.apache.doris.nereids.parser.ParseErrorListener;
-import org.apache.doris.nereids.parser.PostProcessor;
import org.apache.doris.sqlparser.DorisSqlParser;
import org.antlr.v4.runtime.CommonTokenStream;
@@ -56,7 +55,6 @@ public class QueryOrDmlCommonPrefixBenchmark {
public String workload;
private final DorisSqlParser facade = new DorisSqlParser();
- private final PostProcessor postProcessor = new PostProcessor();
private final ParseErrorListener errorListener = new ParseErrorListener();
private String sql;
@@ -99,7 +97,6 @@ public class QueryOrDmlCommonPrefixBenchmark {
public Object parsePreTokenized() {
CommonTokenStream stream = new CommonTokenStream(new
ListTokenSource(tokens));
DorisParser parser = new DorisParser(stream);
- parser.addParseListener(postProcessor);
parser.removeErrorListeners();
parser.addErrorListener(errorListener);
parser.getInterpreter().setPredictionMode(PredictionMode.SLL);
diff --git a/fe/fe-sql-parser/README.md b/fe/fe-sql-parser/README.md
index 73a5662d3b9..afc3dd8fa4e 100644
--- a/fe/fe-sql-parser/README.md
+++ b/fe/fe-sql-parser/README.md
@@ -35,7 +35,7 @@ fe-sql-parser/
├── org/apache/doris/nereids/
│ ├── parser/ # Parser support: CaseInsensitiveStream,
│ │ # Origin, OriginAware, ParserUtils,
- │ │ # ParseErrorListener, PostProcessor
+ │ │ # ParseErrorListener
│ ├── exceptions/ # ParseException, SyntaxParseException
│ └── errors/QueryParsingErrors.java
└── org/apache/doris/sqlparser/
@@ -423,7 +423,7 @@ public class AuditListener extends DorisParserBaseListener {
### Example 3: Live `ParseTreeListener` — fire during parsing
-Most cases are covered by Examples 1 and 2. If you need to intervene **while
the parser is building each node** (mutating tokens, injecting metadata,
streaming work), attach a listener with `parser.addParseListener(...)`. This is
exactly how `fe-sql-parser`'s internal `PostProcessor` rewrites identifier case
at parse time.
+Most cases are covered by Examples 1 and 2. If you need to intervene **while
the parser is building each node** (mutating tokens, injecting metadata,
streaming work), attach a listener with `parser.addParseListener(...)`.
`DorisSqlParser.parseStatement` does not expose the parser instance; use
`newLexer` + `newParser` to take ownership:
@@ -453,7 +453,7 @@ DorisParser.SingleStatementContext tree =
parser.singleStatement();
System.out.println(hintListener.hints);
```
-`newParser` already attaches `PostProcessor` and `ParseErrorListener`; your
listener is added on top.
+`newParser` already attaches `ParseErrorListener`; your listener is added on
top. Identifier normalization is handled locally by the grammar.
### Example 4: Wrap the facade — metrics, caching, rewriting
diff --git
a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index b03f6835717..3ec5d2b6dc0 100644
--- a/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -24,6 +24,31 @@ options { tokenVocab = DorisLexer; }
@members {
public boolean ansiSQLSyntax = false;
+ private void replaceTokenByIdentifier(ParserRuleContext ctx, int
stripMargins,
+ boolean unescapeBackticks) {
+ if (!getBuildParseTree()) {
+ return;
+ }
+ ParserRuleContext parent = ctx.getParent();
+ parent.removeLastChild();
+ Token token = (Token) ctx.getChild(0).getPayload();
+ CommonToken identifier = new CommonToken(
+ new Pair<>(token.getTokenSource(), token.getInputStream()),
+ IDENTIFIER,
+ token.getChannel(),
+ token.getStartIndex() + stripMargins,
+ token.getStopIndex() - stripMargins);
+ if (unescapeBackticks) {
+ identifier.setText(identifier.getText().replace("``", "`"));
+ }
+ parent.addChild(new TerminalNodeImpl(identifier));
+ }
+
+ private void reportUnquotedIdentifier(ErrorIdentContext ctx) {
+ throw
org.apache.doris.nereids.errors.QueryParsingErrors.unquotedIdentifierError(
+ ctx.getParent().getText(), ctx);
+ }
+
private boolean isTupleLambdaBody() {
if (_input.LA(1) != LEFT_PAREN) {
return false;
@@ -2171,6 +2196,11 @@ errorCapturingIdentifierExtra
: (SUBTRACT identifier)+ #errorIdent
| #realIdent
;
+finally {
+ if ($ctx instanceof ErrorIdentContext) {
+ reportUnquotedIdentifier((ErrorIdentContext) $ctx);
+ }
+}
identifier
: strictIdentifier
@@ -2183,6 +2213,7 @@ strictIdentifier
;
quotedIdentifier
+@after { replaceTokenByIdentifier($ctx, 1, true); }
: BACKQUOTED_IDENTIFIER
;
@@ -2201,6 +2232,7 @@ dollarQuotedString
// The non-reserved keywords are listed in `nonReserved`.
// TODO: need to stay consistent with the legacy
nonReserved
+@after { replaceTokenByIdentifier($ctx, 0, false); }
//--DEFAULT-NON-RESERVED-START
: ACTIONS
| AFTER
diff --git
a/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
deleted file mode 100644
index d94fb545081..00000000000
---
a/fe/fe-sql-parser/src/main/java/org/apache/doris/nereids/parser/PostProcessor.java
+++ /dev/null
@@ -1,71 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.parser;
-
-import org.apache.doris.nereids.DorisParser;
-import org.apache.doris.nereids.DorisParser.ErrorIdentContext;
-import org.apache.doris.nereids.DorisParser.NonReservedContext;
-import org.apache.doris.nereids.DorisParser.QuotedIdentifierContext;
-import org.apache.doris.nereids.DorisParserBaseListener;
-import org.apache.doris.nereids.errors.QueryParsingErrors;
-
-import org.antlr.v4.runtime.CommonToken;
-import org.antlr.v4.runtime.ParserRuleContext;
-import org.antlr.v4.runtime.Token;
-import org.antlr.v4.runtime.tree.TerminalNodeImpl;
-
-import java.util.function.Function;
-
-/**
- * Do some post processor after parse to facilitate subsequent analysis.
- */
-public class PostProcessor extends DorisParserBaseListener {
- @Override
- public void exitErrorIdent(ErrorIdentContext ctx) {
- String ident = ctx.getParent().getText();
- throw QueryParsingErrors.unquotedIdentifierError(ident, ctx);
- }
-
- @Override
- public void exitQuotedIdentifier(QuotedIdentifierContext ctx) {
- replaceTokenByIdentifier(ctx, 1, token -> {
- // Remove the double back ticks in the string.
- token.setText(token.getText().replace("``", "`"));
- return token;
- });
- }
-
- @Override
- public void exitNonReserved(NonReservedContext ctx) {
- replaceTokenByIdentifier(ctx, 0, i -> i);
- }
-
- private void replaceTokenByIdentifier(ParserRuleContext ctx, int
stripMargins,
- Function<CommonToken, CommonToken> f) {
- ParserRuleContext parent = ctx.getParent();
- parent.removeLastChild();
- Token token = (Token) (ctx.getChild(0).getPayload());
- CommonToken newToken = new CommonToken(
- new org.antlr.v4.runtime.misc.Pair<>(token.getTokenSource(),
token.getInputStream()),
- DorisParser.IDENTIFIER,
- token.getChannel(),
- token.getStartIndex() + stripMargins,
- token.getStopIndex() - stripMargins);
- parent.addChild(new TerminalNodeImpl(f.apply(newToken)));
- }
-}
diff --git
a/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
index e006228cf3f..6d62a273448 100644
---
a/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
+++
b/fe/fe-sql-parser/src/main/java/org/apache/doris/sqlparser/DorisSqlParser.java
@@ -25,7 +25,6 @@ import
org.apache.doris.nereids.DorisParser.MultiStatementsContext;
import org.apache.doris.nereids.DorisParser.SingleStatementContext;
import org.apache.doris.nereids.parser.CaseInsensitiveStream;
import org.apache.doris.nereids.parser.ParseErrorListener;
-import org.apache.doris.nereids.parser.PostProcessor;
import org.antlr.v4.runtime.CharStreams;
import org.antlr.v4.runtime.CommonTokenStream;
@@ -41,7 +40,6 @@ import java.util.function.Function;
*/
public final class DorisSqlParser {
private static final ParseErrorListener PARSE_ERROR_LISTENER = new
ParseErrorListener();
- private static final PostProcessor POST_PROCESSOR = new PostProcessor();
private final boolean noBackslashEscapes;
private final boolean ansiSqlSyntax;
@@ -116,7 +114,6 @@ public final class DorisSqlParser {
private DorisParser configure(DorisParser parser) {
parser.ansiSQLSyntax = ansiSqlSyntax;
- parser.addParseListener(POST_PROCESSOR);
parser.removeErrorListeners();
parser.addErrorListener(PARSE_ERROR_LISTENER);
return parser;
diff --git
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
index 928160e85c4..362c0d90404 100644
---
a/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
+++
b/fe/fe-sql-parser/src/test/java/org/apache/doris/sqlparser/DorisSqlParserTest.java
@@ -17,14 +17,21 @@
package org.apache.doris.sqlparser;
+import org.apache.doris.nereids.DorisParser;
import org.apache.doris.nereids.DorisParser.ExpressionContext;
import org.apache.doris.nereids.DorisParser.MultiStatementsContext;
import org.apache.doris.nereids.DorisParser.SingleStatementContext;
import org.apache.doris.nereids.exceptions.ParseException;
+import org.antlr.v4.runtime.Token;
+import org.antlr.v4.runtime.tree.ParseTree;
+import org.antlr.v4.runtime.tree.TerminalNode;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import java.util.ArrayList;
+import java.util.List;
+
class DorisSqlParserTest {
private final DorisSqlParser parser = new DorisSqlParser();
@@ -72,4 +79,84 @@ class DorisSqlParserTest {
void rejectsTrailingGarbageInExpression() {
Assertions.assertThrows(ParseException.class, () ->
parser.parseExpression("1 + 2 BAD GARBAGE"));
}
+
+ @Test
+ void normalizesIdentifiersInGrammarRules() {
+ String sql = "SELECT ordinary, actions, `AD``D` FROM
catalog.`db``name`.`table name`";
+ List<String> expected = List.of("ordinary", "actions", "AD`D",
"catalog", "db`name", "table name");
+
+ SingleStatementContext facadeTree = parser.parseStatement(sql);
+ Assertions.assertEquals(expected, identifierTexts(facadeTree));
+
+ DorisParser generatedParser = parser.newParser(parser.newLexer(sql));
+ Assertions.assertTrue(generatedParser.getParseListeners().isEmpty());
+ SingleStatementContext generatedTree =
generatedParser.singleStatement();
+ Assertions.assertEquals(expected, identifierTexts(generatedTree));
+
+ List<Token> identifiers = identifierTokens(facadeTree);
+ Token nonReserved = identifiers.get(1);
+ Assertions.assertEquals(DorisParser.IDENTIFIER, nonReserved.getType());
+ Assertions.assertEquals(sql.indexOf("actions"),
nonReserved.getStartIndex());
+ Assertions.assertEquals(sql.indexOf("actions") + "actions".length() -
1, nonReserved.getStopIndex());
+
+ Token quoted = identifiers.get(2);
+ Assertions.assertEquals(DorisParser.IDENTIFIER, quoted.getType());
+ Assertions.assertEquals(sql.indexOf("`AD``D`") + 1,
quoted.getStartIndex());
+ Assertions.assertEquals(sql.indexOf("`AD``D`") + "`AD``D`".length() -
2, quoted.getStopIndex());
+ }
+
+ @Test
+ void rejectsUnquotedIdentifiersInGrammarRule() {
+ ParseException reservedSuffixException = Assertions.assertThrows(
+ ParseException.class, () -> parser.parseStatement("SELECT *
FROM test-table"));
+ Assertions.assertTrue(reservedSuffixException.getMessage().contains(
+ "Possibly unquoted identifier test- detected"));
+
+ String sql = "SELECT * FROM test-tbl";
+ ParseException facadeException = Assertions.assertThrows(
+ ParseException.class, () -> parser.parseStatement(sql));
+
+ DorisParser generatedParser = parser.newParser(parser.newLexer(sql));
+ ParseException generatedException = Assertions.assertThrows(
+ ParseException.class, generatedParser::singleStatement);
+
+ Assertions.assertEquals(facadeException.getMessage(),
generatedException.getMessage());
+ Assertions.assertTrue(facadeException.getMessage().contains(
+ "Possibly unquoted identifier test-tbl detected"));
+ }
+
+ @Test
+ void parsesWithoutBuildingParseTree() {
+ DorisParser generatedParser = parser.newParser(
+ parser.newLexer("SELECT actions, `AD``D` FROM t"));
+ generatedParser.setBuildParseTree(false);
+ Assertions.assertDoesNotThrow(generatedParser::singleStatement);
+ }
+
+ private static List<String> identifierTexts(ParseTree tree) {
+ List<String> texts = new ArrayList<>();
+ for (Token token : identifierTokens(tree)) {
+ texts.add(token.getText());
+ }
+ return texts;
+ }
+
+ private static List<Token> identifierTokens(ParseTree tree) {
+ List<Token> tokens = new ArrayList<>();
+ collectIdentifierTokens(tree, tokens);
+ return tokens;
+ }
+
+ private static void collectIdentifierTokens(ParseTree tree, List<Token>
tokens) {
+ if (tree instanceof TerminalNode) {
+ Token token = ((TerminalNode) tree).getSymbol();
+ if (token.getType() == DorisParser.IDENTIFIER) {
+ tokens.add(token);
+ }
+ return;
+ }
+ for (int index = 0; index < tree.getChildCount(); index++) {
+ collectIdentifierTokens(tree.getChild(index), tokens);
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
