tnakama created CALCITE-7921:
--------------------------------
Summary: BigQuery string literals should decode backslash escape
sequences
Key: CALCITE-7921
URL: https://issues.apache.org/jira/browse/CALCITE-7921
Project: Calcite
Issue Type: Bug
Components: core
Affects Versions: 1.42.0, 1.40.0
Reporter: tnakama
When SqlParser is configured with {{Lex.BIG_QUERY}}, backslash escapes other
than the enclosing quotation mark are not decoded in ordinary string literals.
This is reproducible with the released {{calcite-core:1.42.0}} artifact,
independently of any custom parser. It also occurs in 1.40.0. The current main
{{Parser.jj}} still calls {{SqlParserUtil.stripQuotes}} for
{{BIG_QUERY_QUOTED_STRING}}, {{BIG_QUERY_DOUBLE_QUOTED_STRING}}, and
{{SimpleStringLiteral}}.
h3. Minimal Java reproduction
{code:java}
import org.apache.calcite.config.Lex;
import org.apache.calcite.sql.SqlCharStringLiteral;
import org.apache.calcite.sql.SqlSelect;
import org.apache.calcite.sql.parser.SqlParser;
import org.apache.calcite.sql.validate.SqlConformanceEnum;
var config = SqlParser.config()
.withLex(Lex.BIG_QUERY)
.withConformance(SqlConformanceEnum.BIG_QUERY);
var select = (SqlSelect) SqlParser.create("SELECT 'a\\nb'",
config).parseQuery();
var literal = (SqlCharStringLiteral) select.getSelectList().get(0);
String value = literal.getValueAs(String.class);
System.out.println(value.length()); // actual: 4; expected: 3
value.codePoints().forEach(cp -> System.out.printf("U+%04X ", cp));
// actual: U+0061 U+005C U+006E U+0062
// expected: U+0061 U+000A U+0062
{code}
h3. SQL examples
The following examples show SQL source text, not Java string literals.
{noformat}
SELECT 'a\nb'
Expected: a + newline + b (3 characters)
Actual: a + backslash + n + b (4 characters)
SELECT 'a\\b'
Expected: a + one backslash + b (3 characters)
Actual: a + two backslashes + b (4 characters)
SELECT 'a\qb'
Expected under GoogleSQL rules: invalid escape sequence error
Actual: the backslash and q are preserved
{noformat}
The same issue affects double-quoted string literals. {{SimpleStringLiteral}}
should also apply the same decoding when used for typed literals such as DATE.
GoogleSQL treats backslash as an escape introducer in ordinary string literals,
including backslashes, newlines, tabs, and numerical/Unicode escapes. Quotation
removal alone does not produce the specified value.
{{SqlParserUtil.replaceEscapedChars}} can already decode several common
sequences, but follows PostgreSQL-style rules and differs from GoogleSQL for
some sequences and malformed input. Reusing it could address common cases;
exact GoogleSQL compatibility would require checking those differences. Raw
string support is a separate feature and is not required to reproduce this bug.
h3. Related issues
* CALCITE-4247 and CALCITE-4847 added BigQuery quotation handling, but do not
cover the remaining escape decoding.
* PR #5094 and CALCITE-7781 address SQL generation/unparsing, rather than
parsing input SQL.
h3. References
*
https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/lexical#escape_sequences_for_string_and_bytes_literals
*
https://github.com/apache/calcite/blob/main/core/src/main/codegen/templates/Parser.jj
* https://github.com/apache/calcite/pull/5094
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
