This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 00948a1ab5a [fix](nereids) Fix SQL masking and view rewrites after
supplementary characters (#70366)
00948a1ab5a is described below
commit 00948a1ab5a4a8e767c75c399ad0db0ec5b909ad
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Nov 16 11:25:03 2026 +0800
[fix](nereids) Fix SQL masking and view rewrites after supplementary
characters (#70366)
### Background
Several FE features rewrite the text of a statement by cutting the
original SQL string at token positions from the parser. Audit masking
replaces credential properties and passwords with `*XXX` before the
statement reaches `fe.audit.log`, `__internal_schema.audit_log` and the
FE log; CREATE / ALTER VIEW and CREATE MATERIALIZED VIEW expand `*` and
qualify names in the definition they store; the parser reads `/*+ ...
*/` hints from comment tokens. The lexer counts its input by Unicode
code point, while a Java `String` holds a supplementary character, such
as an emoji, as two `char`s.
### What problem does this PR solve?
Issue Number: None
Related PR: #47595 (introduced the bug in audit masking), #32743
(introduced the bug in view rewriting), #70338 (its review found the
masking leak)
PR Type: Bugfix
Problem Summary:
After each supplementary character in a statement, these token positions
point one `char` too early, so the text is cut at the wrong place. A
credential after or inside such characters stays partly or fully in
clear text in the audit log, CREATE VIEW fails with a syntax error,
CREATE MATERIALIZED VIEW stores a broken query, and a hint after such a
character is silently ignored. This PR converts the positions to
`String` offsets, or reads the text through the token, at the five
places that cut the original SQL. Tests are 89% of the diff.
For example, the eight emoji in this catalog's comment move both ends of
the span that masking replaces eight characters to the left, so the
audit log loses `ERTIES (` and keeps `"secret"` in clear text:
```sql
CREATE CATALOG hive COMMENT 'ππππππππ'
PROPERTIES ("type" = "hms", "hive.metastore.uris" = "thrift://hms:9083",
"s3.secret_key" = "secret");
```
| | Line 2 in the audit log (line 1 is unchanged) |
|---|---|
| Without the fix | `PROP"type" = "hms", "hive.metastore.uris" =
"thrift://hms:9083", "s3.secret_key" = "*XXX""secret")` |
| With the fix | `PROPERTIES ("type" = "hms", "hive.metastore.uris" =
"thrift://hms:9083", "s3.secret_key" = "*XXX")` |
### Details
- **Change map:** production code +33 -33 in 3 files; tests +240 in 5
files (FE unit tests and one regression suite); generated `.out` +16.
(1) `BaseViewInfo.rewriteSql`, shared by audit masking, views and MTMV,
converts its positions; the sync-MV builder drops its private copy of
it, and `rewriteProjectsToUserDefineAlias` replaces the select list
through it. (2) `NereidsParser.getHintMap` reads hints from the token
text. (3) `NereidsParser.tryParseExplainPlan` converts the start of the
EXPLAIN PLAN body.
- **Frequency:** Always, when a character above U+FFFF (an emoji, CJK
Extension B) comes before or inside the rewritten part, as in the
example above.
- **Root cause:** `CaseInsensitiveStream.fromString` keeps ANTLR's
code-point stream for any input with a surrogate (before #67452 every
input used `CharStreams.fromString`), so `Token.getStartIndex()` and
`getStopIndex()` count code points. The five places in the Change map
used them as `String` indexes.
- **Why existing tests missed it:** Unit tests: the masking
(`EncryptSQLTest`, `RepositoryAuditEncryptionTest`), view and MTMV
(`CreateViewTest`, `CreateMTMVCommandTest`) and hint tests
(`LeanTokenModeTest`) use only BMP text, where both kinds of position
coincide; `CaseInsensitiveStreamTest` uses an emoji but checks only the
stream's own positions. Regression:
`audit/test_external_catalog_credential_masking` and the
`ddl_p0/create_view_nereids` suites have no supplementary characters,
and the masking suite only checks that the whole credential is absent,
which a leaked tail passes.
- **Fix:** Convert where the text is cut. `rewriteSql` converts each
position with `String.offsetByCodePoints`, walking the sorted positions
once so the cost stays linear; `rewriteProjectsToUserDefineAlias` reads
expressions through the token's input stream; `getHintMap` reads the
hint from the token text, as `removeCommentAndTrimBlank` does. Changing
the stream to report `String` positions was rejected: #67452 made
code-point positions part of `CaseInsensitiveStream`'s contract, and
`CaseInsensitiveStreamTest` pins them.
- **Other paths and branches:** Fixed here: the 19 masking visitors of
`LogicalPlanBuilderForEncryption` with every caller of
`geneEncryptionSQL` (audit log, FE statement log, `_query_plan`) and the
streaming insert job; the view, MTMV and sync-MV rewrites. Not affected:
the other uses of token positions read text through the token's own
stream (such as `getOriginSql`) or only compare positions with each
other. branch-3.1, branch-4.0, branch-4.1 and branch-4.2 have the same
code. Introduced by #32743 (views), #47595 (masking), #33699 (sync MV)
and #33262 (EXPLAIN PLAN); hints since at least #39590.
### Testing
- New unit tests fail on master `f1bc8b3690c` and pass with the fix, run
locally: `EncryptSQLTest#testEncryptionAfterSupplementaryCharacters`,
through `MysqlConnectProcessor` to the audit event (emoji inside a
credential, in a catalog COMMENT, in the first of two property clauses,
in a user name before `PASSWORD(...)`; master left `secret`, `abc`,
`123` and `123456` in clear text);
`CreateViewTest#testCreateViewRewritesAfterSupplementaryCharacters`
(master: `mismatched input '`e`'`);
`CreateMTMVCommandTest#testCreateMTMVRewritesQuerySqlAfterSupplementaryCharacters`
(master kept ``SELECT 'ππ' A`e` e`internal`...``);
`SupplementaryCharacterOffsetTest` (master dropped the hint, cut the
EXPLAIN PLAN body to `AN SELECT 100`, and rewrote the sync MV to
`avg_stateg(k)))`).
- New regression suite
`ddl_p0/create_view_nereids/test_create_view_supplementary_chars`: fails
without the fix with the same syntax error and passes with it; the
`.out` is generated by the script. Run locally on one FE and one BE, an
older master FE build with the three changed classes swapped in.
- 27 existing FE UT classes around the change (467 tests) pass.
Reverting each part of the fix alone, moving the end of a replaced span
by one, or not advancing the position cursor each fails a new test; the
off-by-one also fails existing ASCII cases.
- By hand on that cluster, without the fix: the secret stayed in
`audit_log`, `/* π */ select /*+ SET_VAR(query_timeout=1) */ sleep(3)`
ran for 3 s, and an MTMV over `select 'ππ' as e, ...` was created but
failed to refresh (`Nereids parse failed`). With the fix:
`"s3.secret_key" = "*XXX")`, and the query times out after 1 s.
### Notes for Reviewers
- `getHintMap` loses its `sql` parameter, which nothing reads any more;
its three callers are in `NereidsParser`.
- `rewriteProjectsToUserDefineAlias` replaces the select list through
`rewriteSql` instead of `StringUtils.overlay`, so the conversion lives
in one method.
- Stored definitions are not repaired: an MTMV created by an affected
version from such a query keeps its broken definition and has to be
recreated.
---
.../parser/LogicalPlanBuilderForSyncMv.java | 17 +---
.../apache/doris/nereids/parser/NereidsParser.java | 19 +++--
.../trees/plans/commands/info/BaseViewInfo.java | 30 ++++---
.../org/apache/doris/catalog/CreateViewTest.java | 20 +++++
.../doris/nereids/parser/EncryptSQLTest.java | 92 ++++++++++++++++++++++
.../parser/SupplementaryCharacterOffsetTest.java | 64 +++++++++++++++
.../nereids/trees/plans/CreateMTMVCommandTest.java | 19 +++++
.../test_create_view_supplementary_chars.out | 16 ++++
.../test_create_view_supplementary_chars.groovy | 45 +++++++++++
9 files changed, 289 insertions(+), 33 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForSyncMv.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForSyncMv.java
index 1a3e19c8e0d..c53f1392d45 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForSyncMv.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilderForSyncMv.java
@@ -33,6 +33,7 @@ import
org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunctio
import
org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.commands.CreateMTMVCommand;
+import org.apache.doris.nereids.trees.plans.commands.info.BaseViewInfo;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.util.PlanUtils;
@@ -109,7 +110,7 @@ public class LogicalPlanBuilderForSyncMv extends
LogicalPlanBuilder {
for (Expression expr : outputs) {
aggStateFunctionFinder.find(expr, indexInSqlToString);
}
- querySql = Optional.of(rewriteSql(originSql, indexInSqlToString));
+ querySql = Optional.of(BaseViewInfo.rewriteSql(indexInSqlToString,
originSql));
} else {
querySql = Optional.of(originSql);
}
@@ -193,18 +194,4 @@ public class LogicalPlanBuilderForSyncMv extends
LogicalPlanBuilder {
return unboundFunction;
}
}
-
- private static String rewriteSql(String querySql,
- Map<Pair<Integer, Integer>, String> indexStringSqlMap) {
- StringBuilder builder = new StringBuilder();
- int beg = 0;
- for (Map.Entry<Pair<Integer, Integer>, String> entry :
indexStringSqlMap.entrySet()) {
- Pair<Integer, Integer> index = entry.getKey();
- builder.append(querySql, beg, index.first);
- builder.append(entry.getValue());
- beg = index.second + 1;
- }
- builder.append(querySql, beg, querySql.length());
- return builder.toString();
- }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
index 7a30b276da1..0484c0145dc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/NereidsParser.java
@@ -200,7 +200,10 @@ public class NereidsParser {
return Optional.empty();
}
ExplainOptions explainOptions = new ExplainOptions(explainLevel,
showPlanProcess);
- return Optional.of(Pair.of(explainOptions,
sql.substring(explainPlanBody.getStartIndex())));
+ // The parser counts a supplementary character such as an emoji as
one character, while sql
+ // counts it as two.
+ return Optional.of(Pair.of(explainOptions,
+ sql.substring(sql.offsetByCodePoints(0,
explainPlanBody.getStartIndex()))));
} catch (Throwable t) {
return Optional.empty();
}
@@ -349,7 +352,7 @@ public class NereidsParser {
CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream, parseFunction);
LogicalPlanBuilder realLogicalPlanBuilder = logicalPlanBuilder == null
- ? new LogicalPlanBuilder(getHintMap(sql, tokenStream,
DorisParser::selectHint))
+ ? new LogicalPlanBuilder(getHintMap(tokenStream,
DorisParser::selectHint))
: logicalPlanBuilder;
return (T) realLogicalPlanBuilder.visit(tree);
}
@@ -358,7 +361,7 @@ public class NereidsParser {
CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream,
DorisParser::singleStatement);
LogicalPlanBuilder realLogicalPlanBuilder = new
LogicalPlanBuilderForCreateView(
- getHintMap(sql, tokenStream, DorisParser::selectHint));
+ getHintMap(tokenStream, DorisParser::selectHint));
return (LogicalPlan) realLogicalPlanBuilder.visit(tree);
}
@@ -378,13 +381,13 @@ public class NereidsParser {
CommonTokenStream tokenStream = parseLeanTokens(sql);
ParserRuleContext tree = toAst(tokenStream,
DorisParser::singleStatement);
LogicalPlanBuilderForSyncMv logicalPlanBuilderForSyncMv = new
LogicalPlanBuilderForSyncMv(
- getHintMap(sql, tokenStream, DorisParser::selectHint));
+ getHintMap(tokenStream, DorisParser::selectHint));
logicalPlanBuilderForSyncMv.visit(tree);
return logicalPlanBuilderForSyncMv.getQuerySql();
}
/** get hint map */
- public static Map<Integer, ParserRuleContext> getHintMap(String sql,
CommonTokenStream hintTokenStream,
+ public static Map<Integer, ParserRuleContext> getHintMap(CommonTokenStream
hintTokenStream,
Function<DorisParser, ParserRuleContext> parseFunction) {
// parse hint first round
Map<Integer, ParserRuleContext> selectHintMap = Maps.newHashMap();
@@ -392,8 +395,10 @@ public class NereidsParser {
Iterator<Token> tokenIterator = hintTokenStream.getTokens().iterator();
Token hintToken = tokenIterator.hasNext() ? tokenIterator.next() :
null;
while (hintToken != null && hintToken.getType() != DorisLexer.EOF) {
- if (hintToken.getChannel() == 2 &&
sql.charAt(hintToken.getStartIndex() + 2) == '+') {
- String hintSql = sql.substring(hintToken.getStartIndex() + 3,
hintToken.getStopIndex() + 1);
+ // Token positions count a supplementary character such as an
emoji as one character, so read
+ // the hint from the token instead of cutting the SQL string.
+ if (hintToken.getChannel() == 2 &&
hintToken.getText().startsWith("/*+")) {
+ String hintSql = hintToken.getText().substring(3);
DorisLexer newHintLexer = new
DorisLexer(CaseInsensitiveStream.fromString(hintSql));
CommonTokenStream newHintTokenStream = new
CommonTokenStream(newHintLexer);
DorisParser hintParser = new DorisParser(newHintTokenStream);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
index 54b1969db56..892c67a4e48 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/BaseViewInfo.java
@@ -81,9 +81,9 @@ import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import org.antlr.v4.runtime.ParserRuleContext;
+import org.antlr.v4.runtime.misc.Interval;
import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.RuleNode;
-import org.apache.commons.lang3.StringUtils;
import java.util.List;
import java.util.Map;
@@ -128,18 +128,25 @@ public class BaseViewInfo {
* Add the full path to the field
*
* @param indexStringSqlMap key is the start and end position of the sql
substring that needs to be replaced,
- * and value is the new string used for replacement.
+ * and value is the new string used for replacement. The positions
are the token positions the
+ * parser reports for querySql.
* @param querySql origin query sql
* @return sql rewritten sql
*/
public static String rewriteSql(TreeMap<Pair<Integer, Integer>, String>
indexStringSqlMap, String querySql) {
StringBuilder builder = new StringBuilder();
+ // The parser counts a supplementary character such as an emoji as one
character, while querySql
+ // counts it as two. beg and begTokenPosition are the same place on
both scales; the keys ascend,
+ // so each position is converted by scanning forward from the previous
one.
int beg = 0;
+ int begTokenPosition = 0;
for (Map.Entry<Pair<Integer, Integer>, String> entry :
indexStringSqlMap.entrySet()) {
Pair<Integer, Integer> index = entry.getKey();
- builder.append(querySql, beg, index.first);
+ int start = querySql.offsetByCodePoints(beg, index.first -
begTokenPosition);
+ builder.append(querySql, beg, start);
builder.append(entry.getValue());
- beg = index.second + 1;
+ beg = querySql.offsetByCodePoints(start, index.second + 1 -
index.first);
+ begTokenPosition = index.second + 1;
}
builder.append(querySql, beg, querySql.length());
return builder.toString();
@@ -167,10 +174,11 @@ public class BaseViewInfo {
List<NamedExpressionContext> namedExpressionContexts =
finder.getNamedExpressionContexts();
StringBuilder replaceWithColsBuilder = new StringBuilder();
for (int i = 0; i < namedExpressionContexts.size(); ++i) {
- NamedExpressionContext namedExpressionContext =
namedExpressionContexts.get(i);
- int start =
namedExpressionContext.expression().start.getStartIndex();
- int stop = namedExpressionContext.expression().stop.getStopIndex();
- replaceWithColsBuilder.append(resSql, start, stop + 1);
+ ParserRuleContext expression =
namedExpressionContexts.get(i).expression();
+ // Token positions count a supplementary character such as an
emoji as one character, so read
+ // the expression from the parser's input instead of cutting
resSql.
+
replaceWithColsBuilder.append(expression.start.getInputStream().getText(
+ Interval.of(expression.start.getStartIndex(),
expression.stop.getStopIndex())));
replaceWithColsBuilder.append(" AS `");
String escapeBacktick = finalColNames.get(i).replace("`", "``");
replaceWithColsBuilder.append(escapeBacktick);
@@ -179,9 +187,9 @@ public class BaseViewInfo {
replaceWithColsBuilder.append(", ");
}
}
- String replaceWithCols = replaceWithColsBuilder.toString();
- return StringUtils.overlay(resSql, replaceWithCols,
finder.getIndex().first,
- finder.getIndex().second + 1);
+ TreeMap<Pair<Integer, Integer>, String> selectColumns = new
TreeMap<>(new Pair.PairComparator<>());
+ selectColumns.put(finder.getIndex(),
replaceWithColsBuilder.toString());
+ return rewriteSql(selectColumns, resSql);
}
protected void createFinalCols(List<Slot> outputs) throws
org.apache.doris.common.AnalysisException {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateViewTest.java
b/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateViewTest.java
index bec3e35a626..b3ea280fb17 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateViewTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/CreateViewTest.java
@@ -268,6 +268,26 @@ public class CreateViewTest extends TestWithFeService {
view.getInlineViewDef());
}
+ @Test
+ public void testCreateViewRewritesAfterSupplementaryCharacters() throws
Exception {
+ // The parser counts a supplementary character such as an emoji as one
character, while
+ // the Java string of the view definition counts it as two.
+ ExceptionChecker.expectThrowsNoException(() -> createView(
+ "create view test.supplementary_char_view as select 'ππ' as e,
k1 from test.tbl1;"));
+ ExceptionChecker.expectThrowsNoException(() -> createView(
+ "create view test.supplementary_char_alias_view(c1, c2) as
select 'ππ', k1 from test.tbl1;"));
+
+ Database db =
Env.getCurrentInternalCatalog().getDbOrDdlException("test");
+ View view = (View)
db.getTableOrDdlException("supplementary_char_view");
+ Assertions.assertEquals(
+ "select 'ππ' as `e`, `internal`.`test`.`tbl1`.`k1` from
`internal`.`test`.`tbl1`",
+ view.getInlineViewDef());
+ View aliasView = (View)
db.getTableOrDdlException("supplementary_char_alias_view");
+ Assertions.assertEquals(
+ "select 'ππ' AS `c1`, `internal`.`test`.`tbl1`.`k1` AS `c2`
from `internal`.`test`.`tbl1`",
+ aliasView.getInlineViewDef());
+ }
+
@Test
public void testViewPreservesVarbinary() throws Exception {
createView("create view test.vb_view as select X'AB' as vb;");
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/EncryptSQLTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/EncryptSQLTest.java
index 5ccb3ca9400..dab80292149 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/EncryptSQLTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/EncryptSQLTest.java
@@ -505,6 +505,98 @@ public class EncryptSQLTest extends ParserTestBase {
}
}
+ @Test
+ public void testEncryptionAfterSupplementaryCharacters() throws Exception {
+ ctx.setDatabase("test");
+ try (MockedConstruction<StmtExecutor> ignored =
Mockito.mockConstruction(StmtExecutor.class,
+
Mockito.withSettings().defaultAnswer(Mockito.CALLS_REAL_METHODS),
+ (mock, context) -> {
+ Mockito.doReturn(false).when(mock).isForwardToMaster();
+ Profile profile = new Profile(false, 0, 0);
+ Deencapsulation.setField(mock, "profile", profile);
+ Mockito.doReturn(profile).when(mock).getProfile();
+ Mockito.doReturn(ctx).when(mock).getContext();
+ Mockito.doNothing().when(mock).execute();
+ Deencapsulation.setField(mock, "context", ctx);
+ if (context.arguments().size() >= 2
+ && context.arguments().get(1) instanceof
StatementBase) {
+ Deencapsulation.setField(mock, "parsedStmt",
+ context.arguments().get(1));
+ }
+ if (ctx.getStatementContext() == null) {
+ ctx.setStatementContext(new StatementContext());
+ }
+ })) {
+ ctx.setEnv(env);
+ ctx.setCurrentUserIdentity(UserIdentity.ROOT);
+ // The parser counts a supplementary character such as an emoji as
one character, while
+ // the Java string of the SQL counts it as two.
+ String sql = "CREATE CATALOG ctl "
+ + "PROPERTIES("
+ + " \"type\" = \"iceberg\","
+ + " \"iceberg.catalog.type\" = \"hadoop\","
+ + " \"warehouse\" = \"s3://bucket/dir/key\","
+ + " \"s3.access_key\" = \"abc\","
+ + " \"s3.secret_key\" = \"πππππππsecret\""
+ + ")";
+ String res = "CREATE CATALOG ctl "
+ + "PROPERTIES("
+ + " \"type\" = \"iceberg\","
+ + " \"iceberg.catalog.type\" = \"hadoop\","
+ + " \"warehouse\" = \"s3://bucket/dir/key\","
+ + " \"s3.access_key\" = \"*XXX\","
+ + " \"s3.secret_key\" = \"*XXX\""
+ + ")";
+ parseAndCheck(sql, res);
+
+ sql = "CREATE CATALOG ctl COMMENT \"ππππππππ\" "
+ + "PROPERTIES("
+ + " \"type\" = \"iceberg\","
+ + " \"iceberg.catalog.type\" = \"hadoop\","
+ + " \"warehouse\" = \"s3://bucket/π/key\","
+ + " \"s3.access_key\" = \"abc\","
+ + " \"s3.secret_key\" = \"abc\""
+ + ")";
+ res = "CREATE CATALOG ctl COMMENT \"ππππππππ\" "
+ + "PROPERTIES("
+ + " \"type\" = \"iceberg\","
+ + " \"iceberg.catalog.type\" = \"hadoop\","
+ + " \"warehouse\" = \"s3://bucket/π/key\","
+ + " \"s3.access_key\" = \"*XXX\","
+ + " \"s3.secret_key\" = \"*XXX\""
+ + ")";
+ parseAndCheck(sql, res);
+
+ sql = "CREATE EXTERNAL TABLE broker_tbl("
+ + " k1 tinyint) "
+ + "ENGINE=broker "
+ + "PROPERTIES("
+ + " \"broker_name\" = \"hdfs\","
+ + " \"path\" = \"hdfs://abc/ππππππππ/a.txt\""
+ + ") "
+ + "BROKER PROPERTIES("
+ + " \"username\" = \"root\","
+ + " \"password\" = \"123\""
+ + ")";
+ res = "CREATE EXTERNAL TABLE broker_tbl("
+ + " k1 tinyint) "
+ + "ENGINE=broker "
+ + "PROPERTIES("
+ + " \"broker_name\" = \"hdfs\","
+ + " \"path\" = \"hdfs://abc/ππππππππ/a.txt\""
+ + ") "
+ + "BROKER PROPERTIES("
+ + " \"username\" = \"root\","
+ + " \"password\" = \"*XXX\""
+ + ")";
+ parseAndCheck(sql, res);
+
+ sql = "SET PASSWORD FOR 'πππππππππ' = PASSWORD('123456')";
+ res = "SET PASSWORD FOR 'πππππππππ' = PASSWORD('*XXX')";
+ parseAndCheck(sql, res);
+ }
+ }
+
private void parseAndCheck(String sql, String expected) throws Exception {
processor.executeQuery(sql);
AuditEvent event = auditEvents.get(auditEvents.size() - 1);
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/SupplementaryCharacterOffsetTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/SupplementaryCharacterOffsetTest.java
new file mode 100644
index 00000000000..b55a7e8a8cf
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/SupplementaryCharacterOffsetTest.java
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.nereids.properties.SelectHintOrdered;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSelectHint;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+
+/**
+ * The parser counts a supplementary character such as an emoji as one
character, while the Java
+ * string of the SQL counts it as two. These cases cut the SQL text after such
characters.
+ */
+class SupplementaryCharacterOffsetTest extends ParserTestBase {
+ private final NereidsParser parser = new NereidsParser();
+
+ @Test
+ void recognizesHintAfterSupplementaryCharacter() {
+ LogicalPlan plan = parser.parseSingle("/* π */ SELECT /*+ ORDERED */ a
FROM t");
+ List<LogicalSelectHint<Plan>> hintPlans =
plan.collectToList(LogicalSelectHint.class::isInstance);
+
+ Assertions.assertEquals(1, hintPlans.size());
+ Assertions.assertTrue(hintPlans.get(0).getHints().stream()
+ .anyMatch(SelectHintOrdered.class::isInstance));
+ }
+
+ @Test
+ void extractsExplainPlanBodyAfterSupplementaryCharacters() {
+ Assertions.assertEquals("SELECT 100",
+ NereidsParser.tryParseExplainPlan("EXPLAIN /* abc */ PLAN
SELECT 100").orElseThrow().second);
+ Assertions.assertEquals("SELECT 100",
+ NereidsParser.tryParseExplainPlan("EXPLAIN /* πππ */ PLAN
SELECT 100").orElseThrow().second);
+ }
+
+ @Test
+ void rewritesSyncMvAggregateAfterSupplementaryCharacter() {
+ Assertions.assertEquals("SELECT /* a */ k1, avg_union(avg_state(k2))
FROM t GROUP BY k1",
+ parser.parseForSyncMv("CREATE MATERIALIZED VIEW mv AS "
+ + "SELECT /* a */ k1, avg(k2) FROM t GROUP BY
k1").orElseThrow());
+ Assertions.assertEquals("SELECT /* π */ k1, avg_union(avg_state(k2))
FROM t GROUP BY k1",
+ parser.parseForSyncMv("CREATE MATERIALIZED VIEW mv AS "
+ + "SELECT /* π */ k1, avg(k2) FROM t GROUP BY
k1").orElseThrow());
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
index f0fe9fc3da2..df810ff920c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/CreateMTMVCommandTest.java
@@ -626,6 +626,25 @@ public class CreateMTMVCommandTest extends
TestWithFeService {
Assertions.assertTrue(createMTMVInfo.getQuerySql().contains("AS
`mv_score`"));
}
+ @Test
+ public void testCreateMTMVRewritesQuerySqlAfterSupplementaryCharacters()
throws Exception {
+ createTable("create table test.mtmv_supplementary_char_base (id int,
score int)\n"
+ + "duplicate key(id)\n"
+ + "distributed by hash(id) buckets 1\n"
+ + "properties('replication_num' = '1');");
+
+ // The parser counts a supplementary character such as an emoji as one
character, while
+ // the Java string of the query counts it as two.
+ CreateMTMVInfo createMTMVInfo = getPartitionTableInfo("CREATE
MATERIALIZED VIEW mtmv_supplementary_char\n"
+ + " BUILD DEFERRED REFRESH COMPLETE ON MANUAL\n"
+ + " PROPERTIES ('replication_num' = '1')\n"
+ + " AS\n"
+ + " SELECT 'ππ' AS e, id FROM mtmv_supplementary_char_base;");
+
+ Assertions.assertEquals("SELECT 'ππ' AS `e`,
`internal`.`test`.`mtmv_supplementary_char_base`.`id` "
+ + "FROM `internal`.`test`.`mtmv_supplementary_char_base`",
createMTMVInfo.getQuerySql());
+ }
+
@Test
public void
testCreateMTMVRewriteQuerySqlWithDefinedColumnsForProjectScanPlan() throws
Exception {
createTable("create table test.mtmv_project_scan_base (id int, score
int)\n"
diff --git
a/regression-test/data/ddl_p0/create_view_nereids/test_create_view_supplementary_chars.out
b/regression-test/data/ddl_p0/create_view_nereids/test_create_view_supplementary_chars.out
new file mode 100644
index 00000000000..05a50df32b2
--- /dev/null
+++
b/regression-test/data/ddl_p0/create_view_nereids/test_create_view_supplementary_chars.out
@@ -0,0 +1,16 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !show_view --
+test_create_view_supplementary_chars_view CREATE VIEW
`test_create_view_supplementary_chars_view`\n(`e`,`k1`,`v1`)\n AS select 'ππ'
as `e`,
`internal`.`regression_test_ddl_p0_create_view_nereids`.`test_create_view_supplementary_chars_base`.`k1`,
`internal`.`regression_test_ddl_p0_create_view_nereids`.`test_create_view_supplementary_chars_base`.`v1`
from
`internal`.`regression_test_ddl_p0_create_view_nereids`.`test_create_view_supplementary_chars_base`
where `internal`.`regression_test_dd [...]
+
+-- !select_view --
+ππ 1 aπ
+ππ 2 b
+
+-- !show_alias --
+test_create_view_supplementary_chars_alias CREATE VIEW
`test_create_view_supplementary_chars_alias`\n(`c1`,`c2`)\n AS select 'ππ' AS
`c1`,
`internal`.`regression_test_ddl_p0_create_view_nereids`.`test_create_view_supplementary_chars_base`.`k1`
AS `c2` from
`internal`.`regression_test_ddl_p0_create_view_nereids`.`test_create_view_supplementary_chars_base`;
utf8mb4 utf8mb4_0900_bin
+
+-- !select_alias --
+ππ 1
+ππ 2
+ππ 3
+
diff --git
a/regression-test/suites/ddl_p0/create_view_nereids/test_create_view_supplementary_chars.groovy
b/regression-test/suites/ddl_p0/create_view_nereids/test_create_view_supplementary_chars.groovy
new file mode 100644
index 00000000000..ea0e7636baa
--- /dev/null
+++
b/regression-test/suites/ddl_p0/create_view_nereids/test_create_view_supplementary_chars.groovy
@@ -0,0 +1,45 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// The parser counts a supplementary character such as an emoji as one
character, while the Java string
+// of the view query counts it as two; the view definition is rewritten on
that string.
+suite("test_create_view_supplementary_chars") {
+ sql "drop table if exists test_create_view_supplementary_chars_base"
+ sql """
+ create table test_create_view_supplementary_chars_base (k1 int, v1
varchar(32))
+ duplicate key(k1)
+ distributed by hash(k1) buckets 1
+ properties('replication_num' = '1')
+ """
+ sql "insert into test_create_view_supplementary_chars_base values (1,
'aπ'), (2, 'b'), (3, 'πππ')"
+
+ sql "drop view if exists test_create_view_supplementary_chars_view"
+ sql """
+ create view test_create_view_supplementary_chars_view as
+ select 'ππ' as e, k1, v1 from
test_create_view_supplementary_chars_base where v1 != 'πππ'
+ """
+ qt_show_view "show create view test_create_view_supplementary_chars_view"
+ order_qt_select_view "select * from
test_create_view_supplementary_chars_view"
+
+ sql "drop view if exists test_create_view_supplementary_chars_alias"
+ sql """
+ create view test_create_view_supplementary_chars_alias (c1, c2) as
+ select 'ππ', k1 from test_create_view_supplementary_chars_base
+ """
+ qt_show_alias "show create view test_create_view_supplementary_chars_alias"
+ order_qt_select_alias "select * from
test_create_view_supplementary_chars_alias"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]