srielau opened a new pull request, #59868: URL: https://github.com/apache/spark/pull/59868
### What changes were proposed in this pull request? `parse_sql` reports positions in Unicode code points: - A statement's `start` (1-based, relative to the batch) and `length` are code points. Before, they were UTF-16 code units. - An error's `queryContext` `startIndex` and `stopIndex` are code points relative to the statement, and `fragment` is recomputed by code points. Parser origins are already ANTLR code-point offsets, but `SQLQueryContext.fragment` slices the text by UTF-16 units, so the fragment came out wrong after a supplementary character such as an emoji. - The fallback origin of a statement now ends at its last code point instead of its last UTF-16 unit. - Contexts of another object, such as a view body, are left unchanged. This is the first of a stack of PRs that split [#59844](https://github.com/apache/spark/pull/59844) (SPARK-60593, `parse_sql` ANALYZE mode). ### Why are the changes needed? Before this change, the statement span and the error indexes used different units. For a batch containing emoji, a consumer could not use `start`/`length` and `queryContext` together to locate text. Code points are also what SQL users and most editors count as characters. ### Does this PR introduce _any_ user-facing change? Yes, for the unreleased `parse_sql` function, which is behind `spark.sql.function.parseSql.enabled` (default `false`). For example, `get_json_object(parse_sql('SELECT ''<emoji>'';SELECT 2'), '$[1].start')` returns `12` instead of `13`. Text without supplementary characters is not affected. ### How was this patch tested? - Updated the code-point span assertions in `ParseSqlResultSuite`. - Added a test that the error `queryContext` indexes and `fragment` are code points. - Ran `ParseSqlResultSuite`, `ParseSqlSuite`, `ExpressionsSchemaSuite`, and `SQLQueryTestSuite -z parse-sql`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
