github-actions[bot] commented on code in PR #68701:
URL: https://github.com/apache/doris/pull/68701#discussion_r4215623180
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java:
##########
@@ -99,6 +99,26 @@
public class NereidsParserTest extends ParserTestBase {
+ @Test
+ public void testParsePropertiesRejectsTrailingTokens() {
+ NereidsParser parser = new NereidsParser();
+ for (String suffix : new String[] {" trailing_tokens", " ;", " )", "
@", " \"extra\"=\"value\""}) {
+ Assertions.assertThrows(ParseException.class,
+ () -> parser.parseProperties("\"parser\" = \"unicode\"" +
suffix), suffix);
+ }
+ }
+
+ @Test
+ public void testParseProperties() {
+ NereidsParser parser = new NereidsParser();
+ Assertions.assertEquals(ImmutableMap.of("parser", "unicode"),
+ parser.parseProperties("\"parser\" = \"unicode\""));
+ Assertions.assertEquals(ImmutableMap.of("parser", "unicode",
"support_phrase", "true"),
+ parser.parseProperties("\"parser\" = \"unicode\",
\"support_phrase\" = \"true\""));
+ Assertions.assertEquals(ImmutableMap.of("parser", "unicode"),
+ parser.parseProperties(" \"parser\" = \"unicode\" /* comment
*/ \n -- comment\n"));
Review Comment:
[P2] Keep comment assignments out of BE property parsing. This new test
accepts trailing comments, but FE ignores them while BE regex-scans the
original `tokenize` string. With visible
`"parser"="unicode","char_filter_type"="char_replace","char_filter_pattern"=".","char_filter_replacement"="_"`
followed by `/* "char_filter_replacement"="xyz" */`, FE validates `_`, then BE
overwrites it with `xyz` and applies `x` as the replacement. A commented
property thus changes results and bypasses FE validation. Please make FE
validation and BE parsing agree on comments and cover an assignment inside a
comment in an execution test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
