srielau opened a new pull request, #58530: URL: https://github.com/apache/spark/pull/58530
### What changes were proposed in this pull request? Extend the experimental `parse_sql` function so it can parse a batch of SQL statements (`'select 1; select 2'`) instead of a single statement. `parse_sql` now: - Splits the input with `SqlStatementSplitter` (the same splitter used by `SparkSqlParser.splitStatements`). - Parses each statement independently and returns a JSON **array** of statement objects. - Adds `start` (1-based offset in the original batch) and `length` (trimmed statement text, excluding surrounding whitespace and the terminating semicolon) on every statement object. - Continues after a parse failure so later statements are still described. Well-formed `BEGIN ... END` scripts remain a single array element. Nested error locations stay statement-relative; `start` is relative to the original batch. Empty or comment-only input returns `[]`. `NULL` still returns SQL `NULL`. The splitter now records source positions internally so spans are taken from token offsets rather than reconstructed with `indexOf` (which would mis-bind when a dropped comment repeats later statement text). ### Why are the changes needed? Users of the experimental `parse_sql` function asked to parse batches such as `'select 1; select 2'`. Source spans are needed so consumers can highlight each sub-statement in the original text. JIRA: https://issues.apache.org/jira/browse/SPARK-59255 ### Does this PR introduce _any_ user-facing change? Yes, behind `spark.sql.function.parseSql.enabled` (still off by default; the JSON contract is documented as evolving). Previously a successful parse returned one JSON object: ```json {"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]} ``` Now the same input is wrapped in an array and includes source spans: ```json [{"start":1,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]}] ``` A two-statement batch: ```sql SELECT parse_sql('select 1; select 2') ``` ```json [ {"start":1,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]}, {"start":11,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]} ] ``` JSON paths such as `$.statement_identifier` become `$[0].statement_identifier`. Empty SQL that previously produced a parse-failure object now returns `[]`. ### How was this patch tested? - `SqlStatementSplitterSuite` (including comment / empty-`;` span recovery) - `ParseSqlResultSuite` and `ParseSqlSuite` - `SQLQueryTestSuite -- -z parse-sql.sql` (goldens regenerated) - `ExpressionsSchemaSuite` and `ExpressionInfoSuite` example-output check - `catalyst/scalastyle` and `sql/scalastyle` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
