LucaCappelletti94 commented on code in PR #2694:
URL:
https://github.com/apache/datafusion-sqlparser-rs/pull/2694#discussion_r4514618344
##########
src/dialect/clickhouse.rs:
##########
@@ -92,6 +97,29 @@ impl Dialect for ClickHouseDialect {
true
}
+ fn supports_numeric_prefix(&self) -> bool {
+ true
+ }
+
+ /// Undocumented; the lexer reads digits followed by word characters as
one bare word
+ /// unless the run is a number, see "Try to parse it to a identifier" in
`Lexer.cpp`.
+ fn is_digit_leading_identifier(
+ &self,
+ chars: &Peekable<Chars<'_>>,
+ prev_token: Option<&Token>,
+ ) -> bool {
+ let run: String = chars
Review Comment:
Would checking the run straight off `chars.clone()`, as the default
`is_digit_leading_identifier` does, be better than collecting it into a
`String`? Every ClickHouse numeric literal now costs one extra heap allocation.
Tokenizing `SELECT 0, 1, ..., 999` with `ClickHouseDialect` takes 2015
allocations on this branch and 1015 on `main`.
##########
src/tokenizer.rs:
##########
Review Comment:
You should leave the `L` suffix to dialects without
`supports_numeric_prefix`. `SELECT 1.5L` becomes a `long` number, and `SELECT
0.L1`, which parses on `main`, fails with `Expected: end of statement, found:
1`. MySQL 8.4 reads them as `1.5 AS L` and `0 AS L1`.
```suggestion
let long =
if chars.peek() == Some(&'L') &&
!self.dialect.supports_numeric_prefix() {
chars.next();
true
} else {
false
};
```
##########
tests/sqlparser_mysql.rs:
##########
@@ -5245,3 +5245,19 @@ fn parse_key_constraint_index_type_before_columns() {
);
}
}
+
+#[test]
+fn parse_digit_leading_identifiers() {
+ // Verified with MySQL 8.4: anything a number cannot absorb is an
identifier.
+ mysql().verified_stmt("SELECT 123abc, 1_a, 1e, 1ea, 0x1_a, 0xg, 0b01z,
100L FROM t");
+ mysql().verified_stmt("SELECT t.1, t.123, t.12e34, t.1col FROM t");
+ mysql().verified_stmt("SELECT 1 AS 1e, 2 AS 123abc");
+ // A float with an exponent ends the number, so what follows is an alias.
+ let canonical = if cfg!(feature = "bigdecimal") {
+ "SELECT 1000 AS a, 100000 AS x"
+ } else {
+ "SELECT 1e3 AS a, 1e5 AS x"
+ };
+ mysql().one_statement_parses_to("SELECT 1e3a, 1e5x", canonical);
+ mysql().one_statement_parses_to("SELECT 1.5abc", "SELECT 1.5 AS abc");
Review Comment:
Extended to be a red test
```suggestion
mysql().one_statement_parses_to("SELECT 1.5abc, 1.5L", "SELECT 1.5 AS
abc, 1.5 AS L");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
