This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 62b7327c06b8 [SPARK-56147][SQL] Use parser-based SQL splitter in
`spark-sql` CLI
62b7327c06b8 is described below
commit 62b7327c06b83d75c1cdd05a66a2cdd738f555f7
Author: Cheng Pan <[email protected]>
AuthorDate: Fri Jul 10 10:29:04 2026 +0800
[SPARK-56147][SQL] Use parser-based SQL splitter in `spark-sql` CLI
### What changes were proposed in this pull request?
The `spark-sql` CLI replaces the regex/character-scanner `splitSemiColon`
(`StringUtils.splitSemiColon`) with a new parser-based splitter so it correctly
handles `;` inside quoted strings, single-line and bracketed comments, and SQL
scripting compound blocks (`BEGIN ... END`, `IF ... END IF`, `WHILE ... DO ...
END WHILE`, `CASE ... END CASE`).
Concretely:
- **New `SqlStatementSplitter` (in `sql/catalyst`)** — inspired by Trino's
`io.trino.cli.lexer.StatementSplitter`. The splitter tokenizes the input once
with the existing `SqlBaseLexer`, walks the token stream, and at each candidate
region asks the existing `SqlBaseParser`'s `compoundOrSingleStatement` rule
whether the prefix ending at the next `;` is a complete statement. This is the
same rule the normal parser uses, so a `BEGIN ... END` block whose body
contains semicolons is always [...]
- **`splitStatements` on `ParserInterface`** — added as an extension hook
(default implementation in `AbstractSqlParser` delegates to
`SqlStatementSplitter.split`). `SparkSqlParser` overrides it to thread its
variable-substitution preprocessor into the splitter's parser-validation step
(see below).
- **Placeholder-validation hybrid for `${...}` variable substitution** —
Spark SQL allows `SET key=value; SELECT '${key}'` so that `${key}` is
substituted using the value set by an earlier statement in the same batch.
Substituting `${...}` up-front (before splitting) breaks this ordering, and
also breaks block boundaries when the substituted value lands inside a `BEGIN
... END` whose variable was set by a `SET` outside the block. Instead, the
splitter accepts an optional `validationPr [...]
- **Interactive loop changes (`SparkSQLCLIDriver`)** — preserved line-level
"end with `;`" gating (matches the pre-PR behavior so multi-line bracketed
comments and other still-typing input keep buffering until the user signals
end-of-statement). When the user types a line ending with `;`, the splitter
runs over the accumulated buffer. Completed statements are sent to the backend;
any partial remainder (e.g. an open `BEGIN`) keeps buffering. An unclosed
bracketed comment whose buffer h [...]
- **Hive-compat `\;` line continuation** — restored at the CLI layer so the
legacy escape still defers execution to the next line. A new `CliSuite` test
(`SPARK-56147`) covers it.
Notable callouts in the diff for reviewers:
- `SqlStatementSplitter.tryParseRegion` extracts the candidate region from
the original source by char offsets (`Token.getStartIndex` / `getStopIndex`),
applies `validationPreprocess`, re-lexes, and parses with a fresh
`SqlBaseParser`. The splitter uses the same SLL → LL two-stage prediction
strategy as the normal parser for performance.
- `BailErrorStrategy` plus the parser's cursor position after the bail
distinguishes "still wanted more input" from "structural error", so the
splitter can keep buffering vs. fall back without inspecting exception text.
- The splitter does **not** install `PostProcessor` /
`UnclosedCommentProcessor`; the parse tree is discarded, and unclosed comments
are surfaced via `SqlStatementSplitResult.hasUnclosedComment` instead of being
thrown.
- Performance note in the splitter docstring: a single `BEGIN ... END`
block with k internal `;` triggers O(k) `tryParseRegion` calls on growing
prefixes (O(k²) worst case for an incomplete block being typed live). A non-EOF
terminated single-statement rule would drop this to O(n), but Spark's
`setResetStatement: SET .*?` / `RESET .*?` wildcards need an EOF anchor to
terminate deterministically, so the simple rule rewrite doesn't drop in
cleanly. Tracked as a follow-up.
### Why are the changes needed?
The `spark-sql` CLI uses semicolons to determine statement boundaries, both
for splitting multi-statement input (`splitSemiColon`) and for deciding when to
execute in interactive mode (line ends with `;`). SQL Scripting compound blocks
use semicolons as internal statement terminators (e.g. `BEGIN SELECT 1; SELECT
2; END;`), so the regex-based scanner incorrectly splits them. For example, in
interactive mode pre-PR:
```
spark-sql> BEGIN
> SELECT 1; <-- CLI fires here with incomplete "BEGIN\n
SELECT 1;"
```
The scanner also has correctness bugs that this PR fixes incidentally — see
the historical notes on the `CliSuite` tests:
- `SPARK-33100` case `/*/* multi-level bracketed*/ SELECT 'test';` was
passing only because the old scanner mis-handled `/*/*` (it counted the second
char `*` together with the third char `/` as a closing `*/`, prematurely
closing the outer comment). Under correct nesting rules this is an unclosed
comment. The test is updated to use truly-nested syntax that the new lexer
handles correctly.
- `SPARK-37471` was passing only because the old scanner treated `/*+` as a
nested comment opener. The new ANTLR lexer correctly treats `/*+` as a hint
marker (not a nested comment opener) — also documented by `SPARK-37555`'s test
docstring. The test is updated to use truly-nested syntax (`/* outer /* inner
*/ outer */`).
### Does this PR introduce _any_ user-facing change?
Yes. The `spark-sql` CLI now correctly accepts multi-line SQL Scripting
blocks in both interactive mode and file/`-e` mode without prematurely
splitting or executing them. Variable substitution with `${...}` now preserves
block boundaries even when the variable's value comes from a `SET` in the same
batch.
Example interactive session that previously fired prematurely:
```
$ build/sbt -Phive,hive-thriftserver clean package
$ SPARK_PREPEND_CLASSES=true bin/spark-sql
spark-sql (default)> BEGIN
> DECLARE counter INT DEFAULT 1;
> DECLARE total INT DEFAULT 0;
> WHILE counter <= 5 DO
> SET total = total + counter;
> SET counter = counter + 1;
> END WHILE;
> SELECT total AS sum_of_first_five;
> END;
15
Time taken: 0.363 seconds, Fetched 1 row(s)
```
The CLI also restores the Hive-compat `\;` line-continuation escape.
### How was this patch tested?
- New `SqlStatementSplitterSuite` in `sql/catalyst` (80 cases) covers
quoted strings, comments, nested bracketed comments, multi-statement input,
`BEGIN ... END` and other compound blocks, fall-back behavior on structural
errors, unclosed-comment detection, and the `validationPreprocess` hook.
- New `SparkSqlParserSuite` cases (6 added, 53 total) cover the
placeholder-validation hybrid: `${...}` preservation in emitted statements,
`SET`-then-`${var}` ordering across statements, `${var}` inside `BEGIN ... END`
with the `SET` outside the block (C3), and the disabled-substitution
passthrough.
- New `CliSuite` test `SPARK-56147: \; at end of line is a Hive-compat line
continuation marker`. Other `CliSuite` cases (`SPARK-33100`, `SPARK-37471`)
updated to reflect the new lexer's correct comment-nesting behavior (see `Why
are the changes needed?`).
- Full `CliSuite` passes locally (45 tests).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.7
Closes #54946 from pan3793/SPARK-48338.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 49c2637d4fa9cc49ee63903e94fd226db65c065a)
Signed-off-by: Cheng Pan <[email protected]>
---
.../sql/catalyst/parser/AbstractSqlParser.scala | 10 +
.../sql/catalyst/parser/ParserInterface.scala | 28 +
.../sql/catalyst/parser/SqlStatementSplitter.scala | 416 +++++++++
.../parser/SqlStatementSplitterSuite.scala | 994 +++++++++++++++++++++
.../SparkConnectWithSessionExtensionSuite.scala | 5 +-
.../spark/sql/execution/SparkSqlParser.scala | 57 ++
.../spark/sql/SparkSessionExtensionSuite.scala | 5 +-
.../spark/sql/execution/SparkSqlParserSuite.scala | 75 ++
.../sql/hive/thriftserver/SparkSQLCLIDriver.scala | 75 +-
.../spark/sql/hive/thriftserver/CliSuite.scala | 86 +-
10 files changed, 1714 insertions(+), 37 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala
index e6d53ecc1e25..378eab0c23fd 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala
@@ -121,6 +121,16 @@ abstract class AbstractSqlParser extends AbstractParser
with ParserInterface {
}
}
+ /**
+ * Default splitter implementation that defers directly to
[[SqlStatementSplitter]].
+ * Subclasses that perform additional input preprocessing (e.g. variable
+ * substitution) should override this to apply that preprocessing before
+ * splitting so the splitter sees the same tokens the parser would.
+ */
+ override def splitStatements(sqlText: String): SqlStatementSplitResult = {
+ SqlStatementSplitter.split(sqlText)
+ }
+
/**
* Parse the right-hand side of `SET PATH = ...` (a comma-separated list of
path elements).
* Used by [[org.apache.spark.sql.connector.catalog.CatalogManager]] to
honor the
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserInterface.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserInterface.scala
index 193d68e9a444..4456303ffef5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserInterface.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserInterface.scala
@@ -90,4 +90,32 @@ trait ParserInterface extends DataTypeParserInterface {
*/
@throws[ParseException]("Text cannot be parsed to routine parameters")
def parseRoutineParam(sqlText: String): StructType
+
+ /**
+ * Split a SQL string into individual statements at `;` boundaries.
+ *
+ * Designed for tooling such as the `spark-sql` CLI that needs to feed
multiple
+ * statements to the parser one at a time, while correctly handling quoted
+ * strings, single-line and bracketed comments, and SQL scripting compound
blocks
+ * (`BEGIN ... END`) so that semicolons inside them do not split the
surrounding
+ * statement.
+ *
+ * The method is fault-tolerant: it does not throw on incomplete or malformed
+ * input. Trailing text that does not yet form a complete statement is
returned
+ * in [[SqlStatementSplitResult.partialStatement]] so callers can buffer it
and
+ * read more input.
+ *
+ * Variable substitution: the emitted [[SqlStatement]] text is always the
+ * **original** input -- variable references such as `${var}` are NOT
+ * substituted at split time, because their values may come from an earlier
+ * `SET` in the same batch that has not executed yet when the splitter runs.
+ * Substitution is deferred to `parsePlan` at execution time, where each
+ * complete statement is substituted individually after all preceding
statements
+ * have run. Implementations whose grammar allows `${...}` in syntactically
+ * significant positions (e.g. block boundaries) may use a validation-only
+ * preprocessor to help the parser recognize complete statements -- see
+ * [[org.apache.spark.sql.catalyst.parser.SqlStatementSplitter.split]]'s
+ * `validationPreprocess` hook.
+ */
+ def splitStatements(sqlText: String): SqlStatementSplitResult
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala
new file mode 100644
index 000000000000..16112402350e
--- /dev/null
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala
@@ -0,0 +1,416 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.catalyst.parser
+
+import scala.collection.mutable
+
+import org.antlr.v4.runtime._
+import org.antlr.v4.runtime.atn.PredictionMode
+import org.antlr.v4.runtime.misc.ParseCancellationException
+
+import org.apache.spark.sql.internal.SqlApiConf
+
+/**
+ * Represents a single complete SQL statement together with the delimiter that
+ * terminated it (always `";"` for now).
+ *
+ * @param statement the SQL statement text, with surrounding whitespace
trimmed and
+ * without the terminator
+ * @param terminator the delimiter string that terminated the statement
+ */
+case class SqlStatement(statement: String, terminator: String) {
+ override def toString: String = statement + terminator
+}
+
+/**
+ * Result of splitting a SQL string into individual statements.
+ *
+ * @param completeStatements statements that are fully terminated by `;`
+ * @param partialStatement trailing text after the last `;` that has not yet
+ * formed a complete statement; an empty string
when the
+ * input ends with `;` or contains no significant
+ * trailing content
+ * @param hasUnclosedComment true when [[partialStatement]] contains an
unclosed
+ * bracketed comment (`/* ...` with no matching
`*/`).
+ * Interactive CLIs may want to flush the partial
to the
+ * backend in this case (so the user sees a parse
error)
+ * rather than keep buffering, since the input
cannot be
+ * completed simply by appending more SQL.
+ */
+case class SqlStatementSplitResult(
+ completeStatements: Seq[SqlStatement],
+ partialStatement: String,
+ hasUnclosedComment: Boolean = false) {
+ def isEmpty: Boolean = completeStatements.isEmpty && partialStatement.isEmpty
+}
+
+/**
+ * A parser-based SQL statement splitter, inspired by Trino's
+ * `io.trino.cli.lexer.StatementSplitter`.
+ *
+ * Each candidate statement is consumed and confirmed by the ANTLR-generated
+ * [[SqlBaseParser]] via the existing `compoundOrSingleStatement` rule (the
same
+ * rule that the normal Spark SQL parser uses). The splitter:
+ *
+ * 1. Tokenizes the input once.
+ * 2. Walks through the token stream. At each significant position, the
+ * splitter asks the parser whether the prefix ending at the next `;` is a
+ * complete statement; if not, it extends the prefix to the next `;` and
+ * re-tries. This is how SQL scripting `BEGIN ... END` blocks (whose body
+ * contains semicolons) end up emitted as a single statement: only the
+ * prefix that includes a matching `END` is accepted by the parser.
+ * 3. When the parser fails because it reached EOF mid-rule (e.g. an
+ * un-terminated `BEGIN ... END`, a SELECT with a missing operand), the
+ * remaining input is treated as a partial statement so an interactive
+ * caller can keep buffering.
+ * 4. When the parser fails on a non-EOF token (the input is structurally
+ * invalid), the splitter falls back to splitting at the next `;` so the
+ * surrounding delimiters still emit chunks and the backend can report
+ * the error per chunk.
+ *
+ * Quoted strings, single-line and bracketed (nested) comments are honored
+ * throughout. An unterminated bracketed comment is surfaced via
+ * [[SqlStatementSplitResult.hasUnclosedComment]].
+ *
+ * The optional `validationPreprocess` parameter lets the caller transform the
+ * candidate region before the parser sees it -- this exists for the
+ * placeholder-substitution hybrid in
[[org.apache.spark.sql.execution.SparkSqlParser]]
+ * where `${...}` references are replaced with a constant identifier so the
+ * parser can confirm block boundaries even when the real variable values are
+ * not available yet (they are produced by an earlier `SET` in the same batch).
+ * Emission always uses the original (un-preprocessed) token text, so the
+ * variable references survive into the emitted statement and are substituted
+ * for real at execution time. When `validationPreprocess` is `identity`
+ * (the default), the splitter behaves as a pure original-text splitter.
+ *
+ * Performance note: for a single `BEGIN ... END` block with k internal `;`,
+ * the splitter calls `tryParseRegion` O(k) times on growing prefixes -- an
+ * O(k^2) cost in the worst case (incomplete block on every keystroke in
+ * interactive mode). Ordinary non-scripting SQL is O(n). A non-EOF terminated
+ * single-statement rule (read `ctx.getStop` once per region) would make this
+ * O(n), but Spark's `setResetStatement` has `SET .*?` / `RESET .*?` wildcards
+ * that need an EOF anchor to terminate deterministically, so such a
+ * single-statement rule-rewrite does not drop in cleanly. Tracked as a
+ * follow-up.
+ */
+object SqlStatementSplitter {
+
+ /** Split the given SQL text into individual statements at `;` boundaries. */
+ def split(sqlText: String): SqlStatementSplitResult =
+ split(sqlText, identity)
+
+ /**
+ * Split the given SQL text, applying `validationPreprocess` to each
candidate
+ * region before parser validation. The emitted [[SqlStatement]] text is
+ * always the original (un-preprocessed) input; the preprocessor only affects
+ * whether the parser accepts a candidate region as a complete statement.
+ *
+ * Pass `identity` for a pure original-text splitter (the default).
+ */
+ def split(sqlText: String, validationPreprocess: String => String):
SqlStatementSplitResult = {
+ require(sqlText != null, "sqlText must not be null")
+ require(validationPreprocess != null, "validationPreprocess must not be
null")
+
+ val lexer = new SqlBaseLexer(new
UpperCaseCharStream(CharStreams.fromString(sqlText)))
+ lexer.removeErrorListeners()
+ val tokenStream = new CommonTokenStream(lexer)
+ tokenStream.fill()
+
+ val numTokens = tokenStream.size()
+ // Pre-compute the positions of `;` tokens (on the default channel).
+ val delimiterPositions: Array[Int] = {
+ val acc = mutable.ArrayBuffer.empty[Int]
+ var i = 0
+ while (i < numTokens) {
+ if (tokenStream.get(i).getType == SqlBaseLexer.SEMICOLON) acc += i
+ i += 1
+ }
+ acc.toArray
+ }
+
+ val completeStatements = mutable.ArrayBuffer.empty[SqlStatement]
+ val buffer = new StringBuilder()
+ // Whether `buffer` contains any non-hidden token (i.e. any actual SQL
content
+ // beyond whitespace and comments). Chunks that only contain
whitespace/comments
+ // are dropped, matching the spark-sql CLI's long-standing behavior.
+ var bufferHasContent = false
+ var index = 0
+ var stopOuter = false
+ // The first index in `delimiterPositions` that is still > our cursor.
+ var delimSearchStart = 0
+
+ // Snapshot the session config once -- the splitter is short-lived and the
+ // splitter's parser must agree with the session config on grammar
+ // interpretation (e.g. `double_quoted_identifiers`).
+ val conf = SqlApiConf.get
+
+ while (!stopOuter && index < numTokens) {
+ val startIdx = nextSignificantTokenIndex(tokenStream, index)
+ if (startIdx < 0) {
+ // Just hidden trailing tokens (whitespace / closed comments). Drain
+ // them into the buffer; if any are an unclosed comment, the lexer
+ // flag will surface it as a partial.
+ while (index < numTokens) {
+ val tok = tokenStream.get(index)
+ index += 1
+ if (tok.getType != Token.EOF) buffer.append(tok.getText)
+ }
+ stopOuter = true
+ } else if (tokenStream.get(startIdx).getType == SqlBaseLexer.SEMICOLON) {
+ // The next significant token is itself a `;`. This is an empty
+ // statement (e.g. `;;` or leading `;`); drop everything from the
+ // cursor through this delimiter and continue.
+ index = startIdx + 1
+ } else {
+ // Advance the search for delimiters past our current cursor.
+ while (delimSearchStart < delimiterPositions.length &&
+ delimiterPositions(delimSearchStart) <= startIdx) {
+ delimSearchStart += 1
+ }
+
+ // Try increasingly long prefixes (each ending at a `;`) until the
+ // parser accepts one as a complete statement, or we exhaust the
+ // delimiters / hit a structural error.
+ var parsedOk = false
+ var failedNonEof = false
+ var matchedDelimIdx = -1
+ var d = delimSearchStart
+ while (!parsedOk && !failedNonEof && d < delimiterPositions.length) {
+ val candidateEnd = delimiterPositions(d)
+ tryParseRegion(
+ sqlText, tokenStream, startIdx, candidateEnd,
validationPreprocess, conf) match {
+ case ParsedOk =>
+ parsedOk = true
+ matchedDelimIdx = candidateEnd
+ case FailedAtEof =>
+ // Statement body so far is valid but expected more; try a longer
+ // prefix that includes the next `;`.
+ d += 1
+ case FailedNonEof =>
+ // Structurally invalid; stop extending.
+ failedNonEof = true
+ }
+ }
+
+ if (parsedOk) {
+ // Emit `[startIdx .. matchedDelimIdx)` as one statement, with the
+ // delimiter's text as the terminator. Tokens from the cursor up to
+ // `startIdx` are leading whitespace/comments that we preserve in the
+ // buffer too (so the emitted statement text matches the original
+ // input shape, modulo trimming).
+ val terminator = tokenStream.get(matchedDelimIdx).getText
+ while (index < matchedDelimIdx) {
+ val tok = tokenStream.get(index)
+ if (tok.getChannel != Token.HIDDEN_CHANNEL) bufferHasContent = true
+ buffer.append(tok.getText)
+ index += 1
+ }
+ if (bufferHasContent) {
+ val stmt = buffer.toString.trim
+ if (stmt.nonEmpty) {
+ completeStatements += SqlStatement(stmt, terminator)
+ }
+ }
+ buffer.setLength(0)
+ bufferHasContent = false
+ index = matchedDelimIdx + 1
+ delimSearchStart = d + 1
+ } else if (failedNonEof) {
+ // Fall back to a single delimiter step so the broken statement is
+ // surfaced and the rest can still be split.
+ //
+ // This is also the path taken for SQL that uses grammar extensions
+ // (e.g. Iceberg's `ALTER TABLE ... ADD PARTITION FIELD
bucket(...)`):
+ // the vanilla parser rejects the prefix, we fall back, and we walk
+ // tokens looking for the next `SEMICOLON` token. Because the
+ // ANTLR lexer tokenizes `;` inside strings, comments, and
+ // back-ticked identifiers as part of the surrounding STRING_LITERAL
+ // / SIMPLE_COMMENT / BRACKETED_COMMENT / BACKQUOTED_IDENTIFIER
+ // token rather than as a separate SEMICOLON token, those embedded
+ // `;` are NOT split-points. A valid `BEGIN ... END`
+ // block is always confirmed by the parser (ParsedOk), so this
+ // fall-back never splits a well-formed compound block at an
+ // internal `;`.
+ var stopInner = false
+ while (!stopInner && index < numTokens) {
+ val token = tokenStream.get(index)
+ index += 1
+ if (token.getType == Token.EOF) {
+ stopInner = true
+ } else if (token.getType == SqlBaseLexer.SEMICOLON) {
+ if (bufferHasContent) {
+ val stmt = buffer.toString.trim
+ if (stmt.nonEmpty) {
+ completeStatements += SqlStatement(stmt, token.getText)
+ }
+ }
+ buffer.setLength(0)
+ bufferHasContent = false
+ stopInner = true
+ } else {
+ if (token.getChannel != Token.HIDDEN_CHANNEL) bufferHasContent =
true
+ buffer.append(token.getText)
+ }
+ }
+ } else {
+ // All extension attempts failed at EOF (or there were no more
+ // delimiters to try). The remainder is a partial statement that the
+ // caller may complete by appending more input.
+ while (index < numTokens) {
+ val tok = tokenStream.get(index)
+ index += 1
+ if (tok.getType != Token.EOF) {
+ if (tok.getChannel != Token.HIDDEN_CHANNEL) bufferHasContent =
true
+ buffer.append(tok.getText)
+ }
+ }
+ stopOuter = true
+ }
+ }
+ }
+
+ val unclosed = lexer.has_unclosed_bracketed_comment
+ val partial =
+ if (bufferHasContent || unclosed) buffer.toString.trim else ""
+ SqlStatementSplitResult(completeStatements.toSeq, partial, unclosed &&
partial.nonEmpty)
+ }
+
+ /** Outcome of attempting to parse one statement candidate. */
+ private sealed trait ParseOutcome
+ private case object ParsedOk extends ParseOutcome
+ private case object FailedAtEof extends ParseOutcome
+ private case object FailedNonEof extends ParseOutcome
+
+ /**
+ * Try to parse the region of `sqlText` corresponding to the original token
+ * slice `[startIdx, endIdx]` (inclusive on both ends -- `endIdx` is the
+ * position of the trailing `;` token whose char range belongs to the
+ * region) as a complete top-level Spark SQL statement.
+ *
+ * The region is extracted from the original source by char-offset
+ * (`Token.getStartIndex` / `getStopIndex`), `validationPreprocess` is
+ * applied to it, and the result is re-lexed and parsed with a fresh
+ * [[SqlBaseParser]]. This isolation means the splitter's parser sees a
+ * sub-stream whose EOF lands right after the trailing `;`, so the existing
+ * `compoundOrSingleStatement` rule (which requires `SEMICOLON* EOF`) acts
+ * as the per-statement validator without any custom grammar rule.
+ *
+ * Uses the same two-stage SLL -> LL prediction strategy as the main parser
+ * for performance (most statements parse cleanly with the faster SLL stage).
+ *
+ * On failure, we distinguish "the parser still wanted more input" from "the
+ * parser saw something it couldn't make sense of" by looking at the
+ * remaining input after the bail: if it has been consumed up to EOF, the
+ * statement is structurally valid but incomplete -- the caller should try a
+ * longer prefix or treat it as partial. Otherwise, the input is invalid and
+ * the caller should fall back to delimiter-based splitting.
+ *
+ * Deeply nested input that throws [[StackOverflowError]] during parsing is
+ * treated as [[FailedNonEof]] so the splitter still surfaces the broken
+ * input to the backend (which will report the error properly via
+ * `QueryParsingErrors.parserStackOverflow`).
+ */
+ private def tryParseRegion(
+ sqlText: String,
+ stream: CommonTokenStream,
+ startIdx: Int,
+ endIdx: Int,
+ validationPreprocess: String => String,
+ conf: SqlApiConf): ParseOutcome = {
+ val firstTok = stream.get(startIdx)
+ val lastTok = stream.get(endIdx)
+ val regionStart = firstTok.getStartIndex
+ // Token.getStopIndex is inclusive, substring's upper bound is exclusive.
+ val regionEnd = lastTok.getStopIndex + 1
+ val original = sqlText.substring(regionStart, regionEnd)
+ val preprocessed = validationPreprocess(original)
+
+ val regionLexer = new SqlBaseLexer(
+ new UpperCaseCharStream(CharStreams.fromString(preprocessed)))
+ regionLexer.removeErrorListeners()
+ val regionTokens = new CommonTokenStream(regionLexer)
+ regionTokens.fill()
+ val parser = new SqlBaseParser(regionTokens)
+ configureSplitterParser(parser, conf)
+
+ // Two-stage parse: SLL first (fast), then LL on failure. Matches the
+ // normal parser's `AbstractParser.executeWithTwoStageStrategy`.
+ try {
+ parser.getInterpreter.setPredictionMode(PredictionMode.SLL)
+ parser.compoundOrSingleStatement()
+ ParsedOk
+ } catch {
+ case _: ParseCancellationException =>
+ // SLL bailed (could be a false positive). Rewind and retry with LL.
+ regionTokens.seek(0)
+ parser.reset()
+ parser.getInterpreter.setPredictionMode(PredictionMode.LL)
+ try {
+ parser.compoundOrSingleStatement()
+ ParsedOk
+ } catch {
+ case _: ParseCancellationException =>
+ // Use the parser's cursor position (rather than the exception's
+ // offending token, which is unreliable under LL adaptive
+ // prediction) to tell apart the two failure modes.
+ if (regionTokens.LA(1) == Token.EOF) FailedAtEof else FailedNonEof
+ case _: StackOverflowError => FailedNonEof
+ }
+ case _: StackOverflowError => FailedNonEof
+ }
+ }
+
+ /**
+ * Configure a fresh [[SqlBaseParser]] for splitter use: install the managed
+ * caches so candidate parses share ANTLR DFA state across calls, apply the
+ * session's behavior flags so the splitter agrees with the session parser
+ * on grammar interpretation (e.g. `double_quoted_identifiers`), and install
+ * a bail error strategy so failures throw immediately.
+ *
+ * Notably, the splitter does NOT install [[PostProcessor]] or
+ * [[UnclosedCommentProcessor]] -- the former mutates the parse tree (which
+ * we throw away) and the latter would convert unclosed comments into thrown
+ * exceptions, but the splitter surfaces them via the
+ * [[SqlStatementSplitResult.hasUnclosedComment]] flag instead.
+ */
+ private def configureSplitterParser(parser: SqlBaseParser, conf:
SqlApiConf): Unit = {
+ if (conf.manageParserCaches) AbstractParser.installCaches(parser)
+
+ parser.legacy_setops_precedence_enabled = conf.setOpsPrecedenceEnforced
+ parser.legacy_exponent_literal_as_decimal_enabled =
conf.exponentLiteralAsDecimalEnabled
+ parser.SQL_standard_keyword_behavior = conf.enforceReservedKeywords
+ parser.double_quoted_identifiers = conf.doubleQuotedIdentifiers
+ parser.parameter_substitution_enabled =
!conf.legacyParameterSubstitutionConstantsOnly
+ parser.legacy_identifier_clause_only = conf.legacyIdentifierClauseOnly
+ parser.single_character_pipe_operator_enabled =
conf.singleCharacterPipeOperatorEnabled
+
+ parser.removeErrorListeners()
+ parser.setErrorHandler(new BailErrorStrategy)
+ }
+
+ /** Returns the index of the next non-hidden, non-EOF token at or after
`from`, or -1. */
+ private def nextSignificantTokenIndex(stream: CommonTokenStream, from: Int):
Int = {
+ var i = from
+ while (i < stream.size()) {
+ val tok = stream.get(i)
+ if (tok.getType == Token.EOF) return -1
+ if (tok.getChannel != Token.HIDDEN_CHANNEL) return i
+ i += 1
+ }
+ -1
+ }
+}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitterSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitterSuite.scala
new file mode 100644
index 000000000000..2e091aeff974
--- /dev/null
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitterSuite.scala
@@ -0,0 +1,994 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.parser
+
+import org.apache.spark.SparkFunSuite
+
+class SqlStatementSplitterSuite extends SparkFunSuite {
+
+ private def split(sql: String): SqlStatementSplitResult =
SqlStatementSplitter.split(sql)
+
+ /** Helper to build a [[SqlStatement]] with the default `;` terminator. */
+ private def statement(text: String): SqlStatement = SqlStatement(text, ";")
+
+ //
----------------------------------------------------------------------------------
+ // Basic splitting
+ //
----------------------------------------------------------------------------------
+
+ test("incomplete - trailing input without delimiter") {
+ val result = split(" select * FROM foo ")
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == "select * FROM foo")
+ }
+
+ test("empty input") {
+ val result = split("")
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement.isEmpty)
+ assert(result.isEmpty)
+ }
+
+ test("input with only semicolons") {
+ val result = split(";;;")
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("single complete statement") {
+ val result = split("select * from foo;")
+ assert(result.completeStatements == Seq(statement("select * from foo")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("multiple statements with trailing partial") {
+ val result = split(" select * from foo ; select * from t; select * from ")
+ assert(result.completeStatements == Seq(
+ statement("select * from foo"),
+ statement("select * from t")))
+ assert(result.partialStatement == "select * from")
+ }
+
+ test("multiple statements with empty separators") {
+ val result = split("; select * from foo ; select * from t;;;select * from
")
+ assert(result.completeStatements == Seq(
+ statement("select * from foo"),
+ statement("select * from t")))
+ assert(result.partialStatement == "select * from")
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Error tolerance (mirrors Trino behavior)
+ //
----------------------------------------------------------------------------------
+
+ test("unrecognized character before complete statement") {
+ val result = split(" select * from z# oops ; select ")
+ assert(result.completeStatements == Seq(statement("select * from z#
oops")))
+ assert(result.partialStatement == "select")
+ }
+
+ test("unrecognized character in partial after complete") {
+ val result = split("select * from foo; select z# oops ")
+ assert(result.completeStatements == Seq(statement("select * from foo")))
+ assert(result.partialStatement == "select z# oops")
+ }
+
+ test("invalid select with comma before delimiter") {
+ val result = split("select abc, ; select 456;")
+ assert(result.completeStatements == Seq(
+ statement("select abc,"),
+ statement("select 456")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Quoted strings
+ //
----------------------------------------------------------------------------------
+
+ test("quoted string with embedded semicolon") {
+ val sql = "select ';foo;bar;' x from dual"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == sql)
+ }
+
+ test("complete statement with quoted string containing embedded semicolon") {
+ val sql = "select 'foo;bar' x from dual;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(statement("select 'foo;bar' x from
dual")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("unterminated quoted string is partial") {
+ val sql = "select 'foo', 'bar"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == sql)
+ }
+
+ test("escaped single quote inside string literal") {
+ val sql = "select 'hello''world' from dual"
+ val result = split(sql + ";")
+ assert(result.completeStatements == Seq(statement(sql)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("quoted identifier with backticks containing semicolon") {
+ val sql = "select `f;o;o`, `b``ar` from t"
+ val result = split(sql + ";")
+ assert(result.completeStatements == Seq(statement(sql)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("multiple statements with quoted-string semicolons") {
+ val result = split("SELECT 'a;b'; SELECT \"c;d\";")
+ assert(result.completeStatements == Seq(
+ statement("SELECT 'a;b'"),
+ statement("SELECT \"c;d\"")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Comments
+ //
----------------------------------------------------------------------------------
+
+ test("single-line comments - comment-only chunks are dropped") {
+ // Unlike Trino, the Spark splitter drops comment-only chunks (consistent
with
+ // the long-standing spark-sql CLI behavior); only the second chunk that
has
+ // actual SQL content is emitted as a complete statement.
+ val result =
+ split("--empty\n;-- start\nselect * -- junk\n-- hi\nfrom foo; -- done")
+ assert(result.completeStatements == Seq(
+ statement("-- start\nselect * -- junk\n-- hi\nfrom foo")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("multi-line comments - comment-only chunks are dropped") {
+ val result =
+ split("/* empty */;/* start */ select * /* middle */ from foo; /* end
*/")
+ assert(result.completeStatements == Seq(
+ statement("/* start */ select * /* middle */ from foo")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("single-line comment as partial is dropped (no SQL content)") {
+ val sql = "-- start\nselect * -- junk\n-- hi\nfrom foo -- done"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ // The whole input has SQL content ("select * from foo"), so it is
preserved
+ // as the partial statement.
+ assert(result.partialStatement == sql)
+ }
+
+ test("multi-line comment as partial preserves SQL content") {
+ val sql = "/* start */ select * /* middle */ from foo /* end */"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == sql)
+ }
+
+ test("nested bracketed comments - comment-only chunk between delimiters
dropped") {
+ val result = split("SELECT 1; /* outer /* inner */ */;")
+ assert(result.completeStatements == Seq(statement("SELECT 1")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("statement ending with bracketed comment retained") {
+ val result = split("SELECT 1; SELECT 2 /* trailer */")
+ assert(result.completeStatements == Seq(statement("SELECT 1")))
+ assert(result.partialStatement == "SELECT 2 /* trailer */")
+ }
+
+ test("unterminated bracketed comment as partial") {
+ val result = split("SELECT 1; /* unterminated")
+ assert(result.completeStatements == Seq(statement("SELECT 1")))
+ assert(result.partialStatement == "/* unterminated")
+ }
+
+ //
----------------------------------------------------------------------------------
+ // BEGIN..END compound blocks (Spark SQL scripting)
+ //
----------------------------------------------------------------------------------
+
+ test("BEGIN..END block is a single statement") {
+ val block = "BEGIN SELECT 1; SELECT 2; END"
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block followed by another statement") {
+ val block = "BEGIN SELECT 1; END"
+ val result = split(block + "; SELECT 99;")
+ assert(result.completeStatements == Seq(
+ statement(block),
+ statement("SELECT 99")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block preceded by another statement") {
+ val block = "BEGIN SELECT 1; SELECT 2; END"
+ val result = split("SELECT 11; " + block + ";")
+ assert(result.completeStatements == Seq(
+ statement("SELECT 11"),
+ statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN NOT ATOMIC block") {
+ val block = "BEGIN NOT ATOMIC SELECT 1; SELECT 2; END"
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("nested BEGIN..END block") {
+ val inner = "BEGIN SELECT 11; SELECT 22; END"
+ val block = s"BEGIN $inner; SELECT 33; END"
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with control flow (IF / WHILE / DECLARE)") {
+ val block =
+ """BEGIN
+ | DECLARE c INT;
+ | SET c = 0;
+ | WHILE c < 5 DO
+ | SET c = c + 1;
+ | END WHILE;
+ | IF c = 5 THEN
+ | SELECT c;
+ | END IF;
+ |END""".stripMargin
+ val result = split(block + ";\nSELECT 100;")
+ assert(result.completeStatements == Seq(
+ statement(block),
+ statement("SELECT 100")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with EXIT HANDLER FOR SQLEXCEPTION") {
+ // The exception handler body is itself a `BEGIN ... END` containing a `;`.
+ // The outer `END` is the boundary of the whole compound block.
+ val block =
+ """BEGIN
+ | DECLARE EXIT HANDLER FOR SQLEXCEPTION
+ | BEGIN
+ | VALUES('Hello from the exception handler.');
+ | END;
+ |
+ | SELECT * FROM does_not_exist;
+ |END""".stripMargin
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with CONTINUE HANDLER FOR SQLSTATE") {
+ val block =
+ """BEGIN
+ | DECLARE CONTINUE HANDLER FOR SQLSTATE '22012'
+ | SELECT 'caught divide by zero';
+ | SELECT 1 / 0;
+ | SELECT 'after handler';
+ |END""".stripMargin
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with single-statement handler body (SET)") {
+ // The handler body can be a single statement instead of a `BEGIN ... END`
+ // per the grammar:
+ // DECLARE ... HANDLER FOR ...
+ // (beginEndCompoundBlock | statement | setStatementInsideSqlScript)
+ val block =
+ """BEGIN
+ | DECLARE flag INT DEFAULT 0;
+ | DECLARE EXIT HANDLER FOR SQLEXCEPTION SET flag = 1;
+ | SELECT * FROM does_not_exist;
+ |END""".stripMargin
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with handlers for multiple condition values") {
+ val block =
+ """BEGIN
+ | DECLARE EXIT HANDLER FOR SQLEXCEPTION, SQLSTATE '22012', NOT FOUND
+ | BEGIN
+ | SELECT 'caught';
+ | END;
+ | SELECT 1 / 0;
+ |END""".stripMargin
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with nested compound + handler followed by another
statement") {
+ val block =
+ """BEGIN
+ | DECLARE EXIT HANDLER FOR SQLEXCEPTION
+ | BEGIN
+ | VALUES('Hello from the exception handler.');
+ | END;
+ |
+ | SELECT * FROM does_not_exist;
+ |END""".stripMargin
+ val result = split(s"$block;\nSELECT 'after';")
+ assert(result.completeStatements == Seq(
+ statement(block),
+ statement("SELECT 'after'")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("incomplete BEGIN..END block with EXIT HANDLER - treated as partial") {
+ // Same body as the canonical handler example, but missing the outer `END`.
+ // The parser cannot confirm the block, and -- critically -- the embedded
+ // `END;` of the handler body does NOT cause us to emit a premature
+ // statement; the entire input is buffered as a partial.
+ val sql =
+ """BEGIN
+ | DECLARE EXIT HANDLER FOR SQLEXCEPTION
+ | BEGIN
+ | VALUES('Hello from the exception handler.');
+ | END;
+ |
+ | SELECT * FROM does_not_exist;""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == sql)
+ }
+
+ test("BEGIN..END block ending without trailing delimiter is partial") {
+ val block = "BEGIN SELECT 1; END"
+ val result = split(block)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == block)
+ }
+
+ test("incomplete BEGIN..END block (no END) - treated as partial") {
+ val sql = "BEGIN SELECT 1; SELECT 2;"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == "BEGIN SELECT 1; SELECT 2;")
+ }
+
+ test("incomplete BEGIN..END block across multiple inputs - absorbs
everything") {
+ // Simulates an interactive CLI accumulating input. As long as END is
missing
+ // the splitter holds everything as partial so the caller can keep
buffering.
+ val r1 = split("BEGIN\n")
+ assert(r1.completeStatements.isEmpty)
+ assert(r1.partialStatement == "BEGIN")
+ val r2 = split("BEGIN\nSELECT 1;\nSELECT 2;\n")
+ assert(r2.completeStatements.isEmpty)
+ assert(r2.partialStatement.startsWith("BEGIN"))
+ assert(r2.partialStatement.endsWith(";"))
+ val r3 = split("BEGIN\nSELECT 1;\nSELECT 2;\nEND;\n")
+ assert(r3.completeStatements ==
+ Seq(statement("BEGIN\nSELECT 1;\nSELECT 2;\nEND")))
+ assert(r3.partialStatement.isEmpty)
+ }
+
+ test("BEGIN..END block with embedded line comments and block comments") {
+ val block =
+ """BEGIN -- start
+ | /* declare */
+ | SELECT 1; -- pick a number
+ | SELECT 2; /* another */
+ |END /* tail */""".stripMargin
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN identifier (lower-cased) is matched") {
+ // BEGIN keyword is non-reserved; it can be written in any case.
+ val block = "begin select 1; select 2; end"
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN used as identifier in middle of statement does not trigger block
parse") {
+ // BEGIN is non-reserved, so it may appear as a column reference. As long
as
+ // it is NOT the first significant token in a statement, block detection
+ // should not engage.
+ val sql = "SELECT begin FROM t WHERE x = 1;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(statement("SELECT begin FROM t
WHERE x = 1")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN as identifier in first statement, followed by compound block") {
+ // The first statement uses `begin` as a column identifier (valid since
BEGIN
+ // is a non-reserved keyword). The second statement is a real compound
block.
+ // The splitter must correctly disambiguate: BEGIN at the *start* of a
+ // statement triggers compound-block detection, otherwise it's just a
token.
+ val block = "BEGIN SELECT 1; SELECT 2; END"
+ val sql = s"select begin from t; $block;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("select begin from t"),
+ statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("multiple BEGIN..END blocks separated by other statements using BEGIN
as identifier") {
+ val block1 = "BEGIN SELECT 1; END"
+ val block2 = "BEGIN SELECT 2; END"
+ val sql =
+ s"select begin from t1; $block1; select begin as begin from t2; $block2;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("select begin from t1"),
+ statement(block1),
+ statement("select begin as begin from t2"),
+ statement(block2)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("BEGIN-led invalid body cannot be confirmed by parser and becomes
partial") {
+ // With the parser-based splitter, a BEGIN-led input that the parser cannot
+ // confirm as a valid compound block (no matching END found) is buffered
+ // as a partial statement. Interactive callers either type END to complete
+ // it, or trigger an EOF flush which forwards the partial to the backend
+ // for a proper parse error.
+ val sql = "BEGIN FROM t; SELECT 1;"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == sql)
+ }
+
+ test("BEGIN immediately followed by ; - failedNonEof path falls through
safely") {
+ // `BEGIN;` is structurally invalid: the parser bails as soon as it sees
+ // `;` right after `BEGIN` (LA(1) after fail = `;`, i.e. FailedNonEof).
+ // The fallback delimiter walk emits `BEGIN` as one (broken) statement and
+ // continues with the next.
+ //
+ // This exercises the failedNonEof branch with a BEGIN-led input -- it is
+ // important that this case does NOT incorrectly split a *valid* compound
+ // block. Valid blocks always return ParsedOk, so the fall-through here
+ // can only happen for inputs that aren't valid compound blocks anyway.
+ val result = split("BEGIN; SELECT 1;")
+ assert(result.completeStatements == Seq(
+ statement("BEGIN"),
+ statement("SELECT 1")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Valid BEGIN..END block is never split at internal ;") {
+ // A valid `BEGIN ... END` block must be confirmed in full -- the splitter
+ // must never emit at an internal `;`, even if a shorter prefix happens to
+ // look like a valid statement to a less-careful splitter.
+ val block =
+ """BEGIN
+ | SELECT 1;
+ | SELECT 2;
+ | SELECT 3;
+ |END""".stripMargin
+ val result = split(block + ";")
+ assert(result.completeStatements == Seq(statement(block)))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Deeply nested expression does not propagate StackOverflowError") {
+ // Deeply nested parenthesized expressions can cause the ANTLR parser to
+ // throw [[StackOverflowError]] during recursive descent. The splitter
+ // must catch it (and not propagate it out of `split`) so the caller
+ // still gets a result rather than a crash.
+ val deepExpr = "(" * 5000 + "1" + ")" * 5000
+ val sql = s"SELECT $deepExpr; SELECT 2;"
+ // Even if the deeply-nested first statement fails to parse, the splitter
+ // must still return without throwing -- the backend can later report the
+ // actual error via `QueryParsingErrors.parserStackOverflow`.
+ val result = split(sql)
+ // We don't assert on the specific contents -- depending on parser limits
+ // the first chunk may be emitted as a broken statement or buffered as a
+ // partial. What we DO require is that the call returns normally.
+ assert(result != null)
+ }
+
+ test("empty compound block BEGIN END is a single statement") {
+ val result = split("BEGIN END; SELECT 1;")
+ assert(result.completeStatements == Seq(
+ statement("BEGIN END"),
+ statement("SELECT 1")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Multiple statements in mixed form
+ //
----------------------------------------------------------------------------------
+
+ test("multiple statements - mix of regular and BEGIN..END") {
+ val block = "BEGIN SELECT 1; SELECT 2; END"
+ val sql = s"SELECT 11; $block; SELECT 22; $block; SELECT 33;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("SELECT 11"),
+ statement(block),
+ statement("SELECT 22"),
+ statement(block),
+ statement("SELECT 33")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // CLI-flavored edge cases for spark-sql
+ //
----------------------------------------------------------------------------------
+
+ test("CLI: dollar-quoted string is preserved") {
+ val sql = "SELECT $tag$ a;b;c $tag$ FROM t;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(statement("SELECT $tag$ a;b;c
$tag$ FROM t")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("CLI: SET command with trailing semicolons") {
+ val result = split("SET spark.foo=1; SET spark.bar=2;")
+ assert(result.completeStatements == Seq(
+ statement("SET spark.foo=1"),
+ statement("SET spark.bar=2")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("CLI: SPARK-37906 - comment-only trailing input is dropped") {
+ // The CLI relies on these results not producing spurious empty statements.
+ assert(split("SELECT 1; --comment").completeStatements ==
+ Seq(statement("SELECT 1")))
+ assert(split("-- comment ").completeStatements.isEmpty)
+ assert(split("/* comment */ ").completeStatements.isEmpty)
+ assert(split("-- comment \nSELECT 1").partialStatement == "-- comment
\nSELECT 1")
+ }
+
+ test("CLI: SPARK-54876 - assorted comment / split edge cases") {
+ assert(split("SELECT 1; SELECT 2 /* comment */") ==
+ SqlStatementSplitResult(Seq(statement("SELECT 1")), "SELECT 2 /* comment
*/"))
+ assert(split("-- foo\n/* bar */") ==
+ SqlStatementSplitResult(Nil, ""))
+ assert(split("SELECT 1; -- foo\n /* bar */") ==
+ SqlStatementSplitResult(Seq(statement("SELECT 1")), ""))
+ assert(split("SELECT 1; /* outer /* inner */ */") ==
+ SqlStatementSplitResult(Seq(statement("SELECT 1")), ""))
+ assert(split("/* a */ -- foo\n/* b */") ==
+ SqlStatementSplitResult(Nil, ""))
+ // Unterminated comments set hasUnclosedComment = true.
+ assert(split("SELECT 1; /* unterminated") ==
+ SqlStatementSplitResult(
+ Seq(statement("SELECT 1")), "/* unterminated", hasUnclosedComment =
true))
+ assert(split("'unterminated") ==
+ SqlStatementSplitResult(Nil, "'unterminated"))
+ assert(split("SELECT 1; 'unterminated string") ==
+ SqlStatementSplitResult(Seq(statement("SELECT 1")), "'unterminated
string"))
+ assert(split("/* only a comment that never closes") ==
+ SqlStatementSplitResult(
+ Nil, "/* only a comment that never closes", hasUnclosedComment = true))
+ assert(split("SELECT * FROM `t;a`; SELECT 1") ==
+ SqlStatementSplitResult(Seq(statement("SELECT * FROM `t;a`")), "SELECT
1"))
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Validation
+ //
----------------------------------------------------------------------------------
+
+ test("rejects null SQL text") {
+ intercept[IllegalArgumentException](split(null))
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Integration with ParserInterface
+ //
----------------------------------------------------------------------------------
+
+ test("ParserInterface.splitStatements delegates to SqlStatementSplitter") {
+ val parser = new CatalystSqlParser
+ val result = parser.splitStatements("SELECT 1; SELECT 2;")
+ assert(result.completeStatements == Seq(statement("SELECT 1"),
statement("SELECT 2")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("SqlStatement.toString concatenates statement and terminator") {
+ assert(SqlStatement("SELECT 1", ";").toString == "SELECT 1;")
+ }
+
+ test("SqlStatementSplitResult.isEmpty") {
+ assert(SqlStatementSplitResult(Nil, "").isEmpty)
+ assert(!SqlStatementSplitResult(Seq(SqlStatement("x", ";")), "").isEmpty)
+ assert(!SqlStatementSplitResult(Nil, "x").isEmpty)
+ }
+
+ test("hasUnclosedComment flag - set only for unclosed bracketed comments in
partial") {
+ assert(split("/* unclosed").hasUnclosedComment)
+ assert(split("SELECT 1; /* unclosed").hasUnclosedComment)
+ assert(split("SELECT 1; /* unclosed comment SELECT 2;").hasUnclosedComment)
+ // Closed comments do not set the flag.
+ assert(!split("/* closed */").hasUnclosedComment)
+ assert(!split("SELECT 1; /* closed */").hasUnclosedComment)
+ assert(!split("/* nested /* still nested */ outer */").hasUnclosedComment)
+ // No comments at all.
+ assert(!split("SELECT 1").hasUnclosedComment)
+ assert(!split("SELECT 1;").hasUnclosedComment)
+ }
+
+ test("SPARK-31595 - unescaped quote mark in quoted string") {
+ // The `"` inside a single-quoted string is a literal `"`, not an escape.
+ val result = split("SELECT '\"legal string a';select 1 + 234;")
+ assert(result.completeStatements == Seq(
+ statement("SELECT '\"legal string a'"),
+ statement("select 1 + 234")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Extension grammars (e.g. Iceberg's `IcebergSparkSqlExtensionsParser`)
+ //
----------------------------------------------------------------------------------
+ //
+ // The splitter validates each candidate statement against Spark's *vanilla*
+ // SQL grammar, not against any session-injected parser extension. Inputs
that
+ // the vanilla parser rejects still need to be split correctly so that the
+ // CLI can hand each chunk to the extended parser for execution.
+ //
+ // The splitter achieves this via its `FailedNonEof` fall-back: when the
+ // vanilla parser cannot consume a candidate prefix, the splitter reverts to
+ // a token-level walk and emits at the next `;`. Quoted strings and
+ // comments are still honored (they are part of the lexer, not the parser),
+ // so semicolons inside them remain non-delimiters.
+
+ test("Iceberg-style ALTER TABLE single statement splits correctly") {
+ // `ADD PARTITION FIELD bucket(...)` is not in vanilla Spark SQL but is
+ // provided by Iceberg's grammar extension. The vanilla parser rejects it
+ // (the splitter's per-candidate `compoundOrSingleStatement` call fails),
+ // so the splitter falls back to delimiter splitting and the extended
+ // parser then handles the actual execution.
+ val sql = "ALTER TABLE prod.db.sample ADD PARTITION FIELD bucket(16, id);"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE prod.db.sample ADD PARTITION FIELD bucket(16,
id)")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Iceberg-style statement followed by vanilla statement") {
+ val sql =
+ "ALTER TABLE t ADD PARTITION FIELD bucket(16, id); SELECT 1;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t ADD PARTITION FIELD bucket(16, id)"),
+ statement("SELECT 1")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Iceberg-style WRITE ORDERED BY clause splits correctly") {
+ val sql = "ALTER TABLE t WRITE ORDERED BY id;"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(statement("ALTER TABLE t WRITE
ORDERED BY id")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Multiple Iceberg-style statements split at every ;") {
+ val sql =
+ """ALTER TABLE t1 ADD PARTITION FIELD bucket(16, id);
+ |ALTER TABLE t2 ADD PARTITION FIELD truncate(10, name);
+ |ALTER TABLE t3 WRITE ORDERED BY id;""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t1 ADD PARTITION FIELD bucket(16, id)"),
+ statement("ALTER TABLE t2 ADD PARTITION FIELD truncate(10, name)"),
+ statement("ALTER TABLE t3 WRITE ORDERED BY id")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension statement with ; inside a quoted string is not split") {
+ // The lexer correctly tokenizes the embedded `;` as part of the
+ // STRING_LITERAL, so even when the splitter falls back to delimiter
+ // scanning, semicolons inside quotes remain non-delimiters.
+ val sql = "ALTER TABLE t SET TBLPROPERTIES ('foo' = 'a; b; c');"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t SET TBLPROPERTIES ('foo' = 'a; b; c')")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension fall-back: ; inside a STRING_LITERAL is not a delimiter") {
+ // The vanilla parser fails on `ADD PARTITION FIELD names(...)` so the
+ // splitter takes the failedNonEof fall-back. The fall-back walks tokens
+ // and checks `token.getType == SEMICOLON`; the `;`s inside `'a;b;c'` are
+ // part of a single STRING_LITERAL token (not separate SEMICOLON tokens),
+ // so they are correctly preserved as part of the statement text.
+ val sql = "ALTER TABLE t ADD PARTITION FIELD names('a;b;c');"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t ADD PARTITION FIELD names('a;b;c')")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension fall-back: ; inside a `backtick-quoted identifier` is not a
delimiter") {
+ val sql = "ALTER TABLE `t;weird` ADD PARTITION FIELD bucket(16, id);"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE `t;weird` ADD PARTITION FIELD bucket(16, id)")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension fall-back: ; inside a single-line comment is not a
delimiter") {
+ val sql =
+ """ALTER TABLE t ADD PARTITION FIELD bucket(16, id); -- trailing ;
comment
+ |ALTER TABLE t WRITE ORDERED BY id;""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t ADD PARTITION FIELD bucket(16, id)"),
+ statement("-- trailing ; comment\nALTER TABLE t WRITE ORDERED BY id")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension fall-back: ; inside a bracketed comment is not a delimiter")
{
+ val sql = "ALTER TABLE t ADD PARTITION FIELD bucket(/* in ; comment */ 16,
id);"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t ADD PARTITION FIELD bucket(/* in ; comment */
16, id)")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension fall-back: ; inside a nested bracketed comment is not a
delimiter") {
+ val sql = "ALTER TABLE t ADD PARTITION FIELD bucket(/* outer /* inner ; */
; */ 16, id);"
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement(
+ "ALTER TABLE t ADD PARTITION FIELD bucket(/* outer /* inner ; */ ; */
16, id)")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Extension statement without trailing ; is partial") {
+ // No `;` means there are no delimiter candidates to try, so the splitter
+ // falls into the "no more delimiters" branch and surfaces the input as a
+ // partial. The CLI buffers and waits for more input.
+ val sql = "ALTER TABLE t ADD PARTITION FIELD bucket(16, id)"
+ val result = split(sql)
+ assert(result.completeStatements.isEmpty)
+ assert(result.partialStatement == sql)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // Mixed vanilla SQL, vanilla SQL scripting, and extension SQL
+ //
----------------------------------------------------------------------------------
+ //
+ // Inputs that mix all three flavors must split correctly:
+ // - Vanilla statements (SELECT, INSERT, ...) -> ParsedOk via vanilla
parser.
+ // - Vanilla compound `BEGIN ... END` blocks -> ParsedOk after extending
the
+ // prefix to include the closing `END;` (embedded `;` are not
split-points).
+ // - Extension statements (e.g. Iceberg) -> FailedNonEof at the first
+ // unexpected token, then the fall-back walk splits at the next `;`
+ // while honoring strings/comments.
+
+ test("Mix: vanilla + extension + vanilla") {
+ val sql =
+ """SELECT 1;
+ |ALTER TABLE prod.db.t ADD PARTITION FIELD bucket(16, id);
+ |SELECT 2;""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("SELECT 1"),
+ statement("ALTER TABLE prod.db.t ADD PARTITION FIELD bucket(16, id)"),
+ statement("SELECT 2")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Mix: vanilla BEGIN..END block + extension + vanilla statement") {
+ // The compound block must remain one statement (its internal `;` are
+ // *not* split-points) even when followed by an extension statement that
+ // forces the fall-back walk for the rest of the input.
+ val block = "BEGIN SELECT 1; SELECT 2; END"
+ val sql =
+ s"""$block;
+ |ALTER TABLE t ADD PARTITION FIELD bucket(16, id);
+ |SELECT 99;""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement(block),
+ statement("ALTER TABLE t ADD PARTITION FIELD bucket(16, id)"),
+ statement("SELECT 99")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Mix: extension + vanilla BEGIN..END + extension") {
+ // Order reversed: extension first, then a real compound block, then
+ // another extension. The compound block must still be confirmed as one
+ // statement by the parser, not split at its internal `;`.
+ val block = "BEGIN SELECT 1; SELECT 2; END"
+ val sql =
+ s"""ALTER TABLE t1 ADD PARTITION FIELD bucket(16, id);
+ |$block;
+ |ALTER TABLE t2 WRITE ORDERED BY id;""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement("ALTER TABLE t1 ADD PARTITION FIELD bucket(16, id)"),
+ statement(block),
+ statement("ALTER TABLE t2 WRITE ORDERED BY id")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Mix: vanilla SQL scripting compound block with EXIT HANDLER +
extension after") {
+ // SQL scripting block (vanilla) with an exception handler containing its
+ // own embedded `;`s, followed by an extension statement. The compound
+ // block is ParsedOk in full; the trailing extension statement falls back
+ // to the delimiter walk.
+ val block =
+ """BEGIN
+ | DECLARE EXIT HANDLER FOR SQLEXCEPTION
+ | BEGIN
+ | VALUES('Hello from the exception handler.');
+ | END;
+ |
+ | SELECT * FROM does_not_exist;
+ |END""".stripMargin
+ val sql =
+ s"""$block;
+ |ALTER TABLE t ADD PARTITION FIELD bucket(16, id);""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements == Seq(
+ statement(block),
+ statement("ALTER TABLE t ADD PARTITION FIELD bucket(16, id)")))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Mix: extensions, BEGIN..END, comments, and strings interleaved") {
+ // A torture-test mix: line comments, block comments, single- and
+ // double-quoted strings, back-ticked identifiers, vanilla DML, an
+ // extension DDL with embedded `;` inside a comment, and a vanilla
+ // BEGIN..END compound block, all with various trailing `;`s.
+ val block = "BEGIN SELECT 'inside ; block'; SELECT 2; END"
+ val sql =
+ s"""-- prelude
+ |/* multi
+ | line
+ | comment with ; in it */
+ |SELECT 'a;b;c', "x;y;z", `col;name` FROM `t;weird`;
+ |
+ |ALTER TABLE t ADD PARTITION FIELD bucket(/* ; in comment */ 16, id);
+ |
+ |$block;
+ |
+ |-- trailing comment with ; in it
+ |""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements.map(_.statement) === Seq(
+ // The leading line comment + block comment carry over into the first
+ // emitted statement (the splitter preserves leading hidden tokens when
+ // a `parsedOk` candidate is found).
+ """-- prelude
+ |/* multi
+ | line
+ | comment with ; in it */
+ |SELECT 'a;b;c', "x;y;z", `col;name` FROM `t;weird`""".stripMargin,
+ "ALTER TABLE t ADD PARTITION FIELD bucket(/* ; in comment */ 16, id)",
+ block))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Mix: multiple BEGIN..END blocks with extensions and vanilla in
between") {
+ val block1 = "BEGIN SELECT 1; SELECT 2; END"
+ val block2 = "BEGIN SELECT 3; SELECT 4; END"
+ val sql =
+ s"""SELECT 'first';
+ |$block1;
+ |ALTER TABLE t1 ADD PARTITION FIELD bucket(16, id);
+ |INSERT INTO t2 VALUES (1);
+ |$block2;
+ |ALTER TABLE t3 WRITE ORDERED BY id;
+ |SELECT 'last';""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements.map(_.statement) === Seq(
+ "SELECT 'first'",
+ block1,
+ "ALTER TABLE t1 ADD PARTITION FIELD bucket(16, id)",
+ "INSERT INTO t2 VALUES (1)",
+ block2,
+ "ALTER TABLE t3 WRITE ORDERED BY id",
+ "SELECT 'last'"))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("Mix: incomplete extension at end keeps the rest as partial-friendly
buffer") {
+ // The vanilla statements before the trailing extension are emitted; the
+ // un-terminated extension at the end becomes the partial. (No `;` after
+ // the last `bucket(...)` -> "no more delimiters" branch.)
+ val sql =
+ """SELECT 1;
+ |BEGIN SELECT 2; END;
+ |ALTER TABLE t ADD PARTITION FIELD bucket(16, id)""".stripMargin
+ val result = split(sql)
+ assert(result.completeStatements.map(_.statement) === Seq(
+ "SELECT 1",
+ "BEGIN SELECT 2; END"))
+ assert(result.partialStatement ==
+ "ALTER TABLE t ADD PARTITION FIELD bucket(16, id)")
+ }
+
+ test("Mix: extension inside a BEGIN..END body splits at internal ;
(documented limitation)") {
+ // This input is meant to be a valid SQL-scripting block whose body uses
+ // an extension statement. The splitter's vanilla parser cannot recognise
+ // the extension, so it cannot confirm the whole block as a single
+ // compound statement and falls back to per-`;` splitting. The result is
+ // multiple broken chunks rather than one block -- a known limitation:
+ // preserving compound-block boundaries requires a parser that recognises
+ // the extension grammar, which the vanilla splitter does not use.
+ val sql =
+ """BEGIN
+ | ALTER TABLE t ADD PARTITION FIELD bucket(16, id);
+ | SELECT 1;
+ |END;""".stripMargin
+ val result = split(sql)
+ // We don't assert the exact split here because the precise chunk
+ // boundaries depend on where the vanilla parser bails. What we DO
+ // require is that the call returns -- the splitter must remain
+ // fault-tolerant for invalid (or extension-laden) bodies.
+ assert(result != null)
+ // The input is at least split at SOME `;` boundary; we don't get one
+ // single statement back.
+ assert(result.completeStatements.size + (if
(result.partialStatement.isEmpty) 0 else 1) >= 2)
+ }
+
+ //
----------------------------------------------------------------------------------
+ // validationPreprocess hook
+ //
----------------------------------------------------------------------------------
+
+ test("validationPreprocess: identity preserves single-text-splitter
behavior") {
+ // Smoke test that `split(text, identity)` behaves identically to
+ // `split(text)` for a non-trivial mix of vanilla and compound input.
+ val sql = "SELECT 1; BEGIN SELECT 2; SELECT 3; END;"
+ val a = SqlStatementSplitter.split(sql)
+ val b = SqlStatementSplitter.split(sql, identity)
+ assert(a === b)
+ assert(b.completeStatements.map(_.statement) ===
+ Seq("SELECT 1", "BEGIN SELECT 2; SELECT 3; END"))
+ }
+
+ test("validationPreprocess: emitted text is from the *original* input, not
the preprocessed") {
+ // Replace every `XXX` with `1` for validation only. The boundary-finding
+ // still sees the original tokens, and emission must reproduce the
+ // original substring -- the preprocessed value never leaks out.
+ val sql = "SELECT XXX FROM t; SELECT XXX + 1;"
+ val preprocess: String => String = _.replace("XXX", "1")
+ val result = SqlStatementSplitter.split(sql, preprocess)
+ // Both candidates must validate (the preprocessed text is valid SQL).
+ assert(result.completeStatements.map(_.statement) ===
+ Seq("SELECT XXX FROM t", "SELECT XXX + 1"))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("validationPreprocess: lets the parser confirm a region the original
cannot parse") {
+ // The original region uses `@@@` (unrecognized by Spark SQL) where a
column
+ // expression would go. With `identity` the region fails validation and
+ // the splitter falls back to a delimiter walk; with a preprocessor that
+ // maps `@@@` -> `1`, the region validates as a single compound block.
+ val sql = "BEGIN SELECT @@@; SELECT 2; END;"
+ val identityResult = SqlStatementSplitter.split(sql, identity)
+ // Identity preprocess: the BEGIN..END can't be confirmed, so it falls
+ // through to per-`;` splitting (a known behavior under the fall-back).
+ assert(identityResult.completeStatements.size > 1)
+
+ val mapped: String => String = _.replace("@@@", "1")
+ val mappedResult = SqlStatementSplitter.split(sql, mapped)
+ // With the preprocessor, the whole block is one statement -- and the
+ // *emitted* text still contains the original `@@@`.
+ assert(mappedResult.completeStatements.map(_.statement) ===
+ Seq("BEGIN SELECT @@@; SELECT 2; END"))
+ assert(mappedResult.partialStatement.isEmpty)
+ }
+}
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectWithSessionExtensionSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectWithSessionExtensionSuite.scala
index 82c8192fe070..cb396e67143a 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectWithSessionExtensionSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectWithSessionExtensionSuite.scala
@@ -23,7 +23,7 @@ import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst._
import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.parser.{ParserInterface,
SqlStatementSplitResult}
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.classic
import org.apache.spark.sql.connect.SparkConnectTestUtils
@@ -58,6 +58,9 @@ class SparkConnectWithSessionExtensionSuite extends
SparkFunSuite {
override def parseRoutineParam(sqlText: String): StructType =
delegate.parseRoutineParam(sqlText)
+
+ override def splitStatements(sqlText: String): SqlStatementSplitResult =
+ delegate.splitStatements(sqlText)
}
test("Parse table name with test parser") {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
index 2eb2c7e91441..0f689c1705bf 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
@@ -103,6 +103,27 @@ class SparkSqlParser extends AbstractSqlParser {
parseInternal(command, None)(toResult)
}
+ /**
+ * Split a SQL string into individual statements at `;` boundaries.
+ *
+ * The emitted statements are pieces of the original input -- `${...}`
variable
+ * references are NOT expanded here, so they survive into each emitted
statement
+ * and are substituted for real per-statement at parse time (see
[[parseInternal]]).
+ * This is important for the `spark-sql` CLI's batch mode: a `${x}` whose
value
+ * comes from a `SET x=...` in the same batch must resolve to the post-`SET`
+ * value, which is only available *after* the earlier statement executes.
+ *
+ * To still recognize `BEGIN ... END` compound block boundaries when a body
+ * contains a `${...}` reference, the splitter is invoked with a validation
+ * preprocessor that replaces every `${...}` with a fixed valid identifier.
+ * The placeholder makes the candidate region parse-equivalent regardless of
+ * whether (or how) the variable is set, while the emitted statement keeps
+ * the original `${...}` so per-statement substitution at execution still
+ * runs against the real session config (with earlier `SET`s applied).
+ */
+ override def splitStatements(sqlText: String): SqlStatementSplitResult =
+ SqlStatementSplitter.split(sqlText,
SparkSqlParser.substituteVariablesForValidation)
+
/**
* Internal parse method that handles both parameter substitution and
regular parsing.
*
@@ -169,6 +190,42 @@ class SparkSqlParser extends AbstractSqlParser {
}
}
+object SparkSqlParser {
+
+ /**
+ * Identifier used to stand in for every `${...}` variable reference when
+ * the splitter validates a candidate compound block. It is intentionally
+ * shaped so that a real user identifier cannot collide with it; it is also
+ * a structurally well-formed Spark SQL identifier (letters + underscores
+ * only) so the parser accepts it in nearly every position a `${...}` might
+ * appear.
+ */
+ private[execution] val VariableRefPlaceholder = "__SPARK_VAR_PLACEHOLDER__"
+
+ /**
+ * Regex matching a `${...}` variable reference. Mirrors
+ * [[org.apache.spark.internal.config.ConfigReader.REF_RE]]: an optional
+ * `prefix:` followed by a non-whitespace name, enclosed in `${ }`.
+ */
+ private val variableRefRe = """\$\{(?:\w+?:)?\S+?\}""".r
+
+ /**
+ * Validation-time preprocessor for the splitter (see
+ * [[SparkSqlParser.splitStatements]]): replaces every `${...}` reference
+ * with [[VariableRefPlaceholder]] so candidate regions parse regardless of
+ * whether the referenced variable is set. The emitted statement keeps the
+ * original `${...}` and is substituted for real at execution time. If
+ * variable substitution is disabled in the session conf, the input is
+ * returned unchanged (matching the behavior of the real substituter).
+ */
+ private[execution] def substituteVariablesForValidation(input: String):
String = {
+ if (input == null || input.isEmpty || !input.contains("${")) return input
+ if (!SQLConf.get.variableSubstituteEnabled) return input
+ variableRefRe.replaceAllIn(
+ input, _ =>
scala.util.matching.Regex.quoteReplacement(VariableRefPlaceholder))
+ }
+}
+
/**
* Builder that converts an ANTLR ParseTree into a
LogicalPlan/Expression/TableIdentifier.
*/
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
index d9b29b757b76..05619efa285f 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
@@ -31,7 +31,7 @@ import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
import org.apache.spark.sql.catalyst.expressions._
import
org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression,
Final, Max, Partial}
-import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser,
ParserInterface}
+import org.apache.spark.sql.catalyst.parser.{CatalystSqlParser,
ParserInterface, SqlStatementSplitResult}
import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, AggregateHint,
ColumnStat, Limit, LocalRelation, LogicalPlan, Project, Range, Sort, SortHint,
Statistics, UnresolvedHint}
import org.apache.spark.sql.catalyst.plans.physical.{Partitioning,
SinglePartition}
@@ -720,6 +720,9 @@ case class MyParser(spark: SparkSession, delegate:
ParserInterface) extends Pars
override def parseRoutineParam(sqlText: String): StructType = {
delegate.parseRoutineParam(sqlText)
}
+
+ override def splitStatements(sqlText: String): SqlStatementSplitResult =
+ delegate.splitStatements(sqlText)
}
object MyExtensions {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala
index 3b2b08365c6e..9762feeecf54 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/SparkSqlParserSuite.scala
@@ -1207,4 +1207,79 @@ class SparkSqlParserSuite extends AnalysisTest with
SharedSparkSession {
parser.parsePlan(
"SELECT CAST(null AS STRUCT<>), CAST(null AS MAP<STRING, ARRAY<INT>>), 2
>> 1")
}
+
+ test("splitStatements preserves ${...} variable references in emitted
statements") {
+ // The splitter must NOT pre-expand variable references at split time --
+ // doing so in batch mode would resolve `${x}` against the pre-`SET` value
+ // when an earlier statement in the same batch is `SET x=...`.
Per-statement
+ // substitution at execution time (in `parseInternal`) handles that case
+ // correctly. The reference must therefore survive into the emitted
+ // statement text.
+ withSQLConf("spark.x.stmt" -> "abc") {
+ val result = parser.splitStatements("SELECT '${spark.x.stmt}';")
+ assert(result.completeStatements.map(_.statement) === Seq("SELECT
'${spark.x.stmt}'"))
+ assert(result.partialStatement.isEmpty)
+ }
+ }
+
+ test("splitStatements: BEGIN..END containing ${...} is confirmed as one
statement (C2)") {
+ // The block's body references a pre-set conf variable. The splitter must
+ // confirm the whole `BEGIN..END;` as a single statement -- it does so by
+ // replacing the `${...}` with a placeholder identifier purely for the
+ // parse-validation step, while emitting the original text with `${...}`
+ // intact.
+ withSQLConf("spark.x.t" -> "my_table") {
+ val sql = "BEGIN SELECT 1 FROM ${spark.x.t}; SELECT 2; END;"
+ val result = parser.splitStatements(sql)
+ assert(result.completeStatements.map(_.statement) ===
+ Seq("BEGIN SELECT 1 FROM ${spark.x.t}; SELECT 2; END"))
+ assert(result.partialStatement.isEmpty)
+ }
+ }
+
+ test("splitStatements: SET-then-${var} across statements (C4)") {
+ // The variable is `SET` by an earlier statement in the same batch. The
+ // splitter must not try to resolve `${x}` against its pre-`SET` value;
+ // it must emit the reference verbatim and let it resolve at execution
+ // time after the `SET` has run.
+ val result = parser.splitStatements("SET x=1; SELECT '${x}';")
+ assert(result.completeStatements.map(_.statement) ===
+ Seq("SET x=1", "SELECT '${x}'"))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("splitStatements: SET-in-batch-then-${var}-inside-BEGIN..END (C3)") {
+ // The block's body references `${x}`, whose value comes from the earlier
+ // `SET x=1` in the same batch. Substituting `${x}` up front (before the
+ // `SET` runs) would replace it with the pre-`SET` (default / empty) value
+ // and break the block's parse. The placeholder-validation hybrid leaves
+ // `${x}` in the emitted text but still recognizes the block boundary.
+ val sql = "SET x=1; BEGIN SELECT ${x}; SELECT 1; END;"
+ val result = parser.splitStatements(sql)
+ assert(result.completeStatements.map(_.statement) ===
+ Seq("SET x=1", "BEGIN SELECT ${x}; SELECT 1; END"))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("splitStatements: ${...} in a non-block statement preserved (no
premature substitute)") {
+ // Same shape as C4 but without the SET -- the variable need not exist.
+ // The reference must still survive into the emitted statement so the
+ // backend substituter can either resolve it from session config or
+ // leave it literal (per the substituter's contract).
+ val result = parser.splitStatements("SELECT '${unset.var}';")
+ assert(result.completeStatements.map(_.statement) === Seq("SELECT
'${unset.var}'"))
+ assert(result.partialStatement.isEmpty)
+ }
+
+ test("splitStatements: when variable substitution is disabled, ${...} still
passes through") {
+ // Disabling variable substitution should disable the placeholder pass
+ // too -- the parser then sees the literal `${...}` and (typically) bails
+ // non-EOF, so the splitter falls back to a plain delimiter walk. The
+ // emitted text still keeps the `${...}` intact.
+ withSQLConf(SQLConf.VARIABLE_SUBSTITUTE_ENABLED.key -> "false") {
+ val result = parser.splitStatements("SELECT '${spark.x}';")
+ assert(result.completeStatements.map(_.statement) === Seq("SELECT
'${spark.x}'"))
+ assert(result.partialStatement.isEmpty)
+ }
+ }
}
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
index f34ac25ef78f..e1553ce9b368 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
@@ -41,7 +41,7 @@ import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
-import org.apache.spark.sql.catalyst.util.{SQLKeywordUtils, StringUtils}
+import org.apache.spark.sql.catalyst.util.SQLKeywordUtils
import org.apache.spark.sql.hive.client.HiveClientImpl
import org.apache.spark.sql.hive.security.HiveDelegationTokenProvider
import
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.closeHiveSessionStateIfStarted
@@ -239,7 +239,15 @@ private[hive] object SparkSQLCLIDriver extends Logging {
}
var ret = 0
- var prefix = ""
+ // Accumulated input that has not yet formed a complete statement. The
+ // interactive loop is line-level: we buffer input until the user types a
+ // line ending with `;`, then ask the parser-based [[SqlStatementSplitter]]
+ // to split the buffered text into statements. Line-level buffering keeps
+ // multi-line input accumulating until the user signals end-of-statement
+ // with `;`, so constructs that span several lines -- an un-closed
+ // bracketed comment (SPARK-33100, SPARK-37471) or a `BEGIN ... END`
+ // scripting block -- are assembled in full before being split.
+ var buffer = ""
def currentDB = {
if (!SparkSQLEnv.sparkSession.sessionState.conf
@@ -255,6 +263,7 @@ private[hive] object SparkSQLCLIDriver extends Logging {
def continuedPromptWithDBSpaces: String = continuedPrompt +
ReflectionUtils.invokeStatic(
classOf[CliDriver], "spacesForString", classOf[String] -> currentDB)
+ val sqlParser = SparkSQLEnv.sparkSession.sessionState.sqlParser
var currentPrompt = promptWithCurrentDB
var line = reader.readLine(currentPrompt + "> ")
@@ -262,23 +271,49 @@ private[hive] object SparkSQLCLIDriver extends Logging {
// SPARK-55198: call line.trim to also skip comment line with leading
whitespaces,
// this keeps the behavior align with HIVE-8396
if (!line.trim.startsWith("--")) {
- if (prefix.nonEmpty) {
- prefix += '\n'
- }
-
- if (line.trim().endsWith(";") && !line.trim().endsWith("\\;")) {
- line = prefix + line
- ret = cli.processLine(line, true)
- prefix = ""
- currentPrompt = promptWithCurrentDB
+ val trimmed = line.trim
+ if (trimmed.endsWith(";") && !trimmed.endsWith("\\;")) {
+ // The line ends with `;` (and not the Hive-compat `\;` continuation
+ // escape) -- the user signaled end of statement. Ask the splitter
+ // to split the accumulated buffer into statements. The splitter
+ // correctly handles `;` inside quoted strings, comments, and SQL
+ // scripting compound blocks (`BEGIN ... END`).
+ val candidate = if (buffer.isEmpty) line else buffer + "\n" + line
+ val splitResult = sqlParser.splitStatements(candidate)
+ // An un-terminated bracketed comment cannot be completed by
+ // appending more SQL; flush it to the backend so the user sees a
+ // parse error rather than staying stuck on the continuation prompt
+ // forever (SPARK-37555).
+ val flushPartial =
+ splitResult.hasUnclosedComment &&
splitResult.partialStatement.nonEmpty
+ if (splitResult.completeStatements.nonEmpty || flushPartial) {
+ val parts =
+ splitResult.completeStatements.iterator.map(s => s.statement +
s.terminator).toBuffer
+ if (flushPartial) parts += splitResult.partialStatement
+ ret = cli.processLine(parts.mkString("\n"), true)
+ }
+ buffer = if (flushPartial) "" else splitResult.partialStatement
+ currentPrompt =
+ if (buffer.isEmpty) promptWithCurrentDB else
continuedPromptWithDBSpaces
} else {
- prefix = prefix + line
+ // The line does not signal end of statement (no trailing `;`, or
+ // the trailing `;` is escaped as `\;`). Accumulate and wait for
+ // more input. The `\;` escape survives into the buffer and is
+ // reattached at execution time by `processLine` (via its
+ // `oneCmd.endsWith("\\")` branch).
+ buffer = if (buffer.isEmpty) line else buffer + "\n" + line
currentPrompt = continuedPromptWithDBSpaces
}
}
line = reader.readLine(currentPrompt + "> ")
}
+ // Stdin closed with un-terminated trailing input. Pass it to the backend
so the
+ // user gets a parse error rather than silently dropping their input.
+ if (buffer.nonEmpty) {
+ ret = cli.processLine(buffer, true)
+ }
+
closeHiveSessionStateIfStarted(sessionState)
exit(ret)
@@ -583,7 +618,7 @@ private[hive] class SparkSQLCLIDriver extends CliDriver
with Logging {
var lastRet: Int = 0
// we can not use "split" function directly as ";" may be quoted
- val commands = splitSemiColon(line).asScala
+ val commands = splitStatements(line)
var command: String = ""
for (oneCmd <- commands) {
if (oneCmd.endsWith("\\")) {
@@ -613,9 +648,15 @@ private[hive] class SparkSQLCLIDriver extends CliDriver
with Logging {
}
}
- // Splits SQL into individual statements by top-level semicolons. See
- // [[StringUtils.splitSemiColon]] for the implementation.
+ // Splits SQL into individual statements via the parser-based
+ // [[org.apache.spark.sql.catalyst.parser.SqlStatementSplitter]]. The
returned
+ // list contains the text of every complete statement (without its
terminator)
+ // followed by the trailing partial statement, if any -- the shape that
+ // [[processLine]] consumes.
// Note: [SPARK-31595], [SPARK-33100], [SPARK-54876]
- private[hive] def splitSemiColon(line: String): JList[String] =
- StringUtils.splitSemiColon(line, enableSqlScripting = false).asJava
+ private[hive] def splitStatements(line: String): List[String] = {
+ val result =
SparkSQLEnv.sparkSession.sessionState.sqlParser.splitStatements(line)
+ val complete = result.completeStatements.iterator.map(_.statement).toList
+ if (result.partialStatement.isEmpty) complete else complete :+
result.partialStatement
+ }
}
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
index 5debfd168404..571060ab5653 100644
---
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
@@ -25,7 +25,6 @@ import java.util.concurrent.CountDownLatch
import scala.collection.mutable.ArrayBuffer
import scala.concurrent.Promise
import scala.concurrent.duration._
-import scala.jdk.CollectionConverters._
import org.apache.hadoop.hive.cli.CliSessionState
import org.apache.hadoop.hive.ql.session.SessionState
@@ -588,7 +587,11 @@ class CliSuite extends SparkFunSuite {
"SELECT 'test'; /* SELECT 'test';*/;" -> "test",
"/*$meta chars{^\\;}*/ SELECT 'test';" -> "test",
"/*\nmulti-line\n*/ SELECT 'test';" -> "test",
- "/*/* multi-level bracketed*/ SELECT 'test';" -> "test"
+ // Nested bracketed comments are honored: a `;` inside any nesting
+ // level is ignored as long as every level is closed, so the fully
+ // closed `/* outer; /* inner; */ outer; */` is skipped and only the
+ // trailing `SELECT 'test'` executes.
+ "/* outer; /* inner; */ outer; */ SELECT 'test';" -> "test"
)
}
@@ -619,24 +622,33 @@ class CliSuite extends SparkFunSuite {
}
test("SPARK-37471: spark-sql support nested bracketed comment ") {
+ // `/* outer /* inner */ outer */` is a truly-nested bracketed comment: the
+ // inner `/* ... */` opens and closes an extra nesting level, and the outer
+ // comment closes only at the final `*/`. The whole comment is skipped and
+ // the following `SELECT 'nested-comment'` executes. (Note `/*+` is a hint
+ // marker, not a nested comment opener, so nesting must use `/*`.)
runCliWithin(1.minute)(
"""
- |/* SELECT /*+ HINT() */ 4; */
- |SELECT 1;
- |""".stripMargin -> "SELECT 1"
+ |/* outer /* inner */ outer */
+ |SELECT 'nested-comment';
+ |""".stripMargin -> "nested-comment"
)
}
testRetry("SPARK-37555: spark-sql should pass last unclosed comment to
backend") {
runCliWithin(1.minute)(
- // Only unclosed comment.
- "/* SELECT /*+ HINT() 4; */;".stripMargin -> "Syntax error at or near
';'",
- // Unclosed nested bracketed comment.
+ // A fully closed bracketed comment. `/*+` inside it is a hint marker,
+ // not a nested comment opener, so the comment closes at the first `*/`;
+ // the comment is skipped and the trailing `SELECT 1` executes.
"/* SELECT /*+ HINT() 4; */ SELECT 1;".stripMargin -> "1",
- // Unclosed comment with query.
+ // Genuinely unclosed comment with a query inside it. The splitter
+ // detects the un-terminated bracketed comment and flushes the partial
+ // input to the backend so the parser reports the error rather than the
+ // CLI buffering forever.
"/* Here is a unclosed bracketed comment SELECT 1;"->
"Found an unclosed bracketed comment. Please, append */ at the end of
the comment.",
- // Whole comment.
+ // Whole comment with `/*+` hint marker inside, followed by trailing
+ // tokens that are not a valid statement -- backend reports the error.
"/* SELECT /*+ HINT() */ 4; */;".stripMargin -> ""
)
}
@@ -658,25 +670,27 @@ class CliSuite extends SparkFunSuite {
val sessionState = new CliSessionState(cliConf)
SessionState.setCurrentSessionState(sessionState)
val cli = new SparkSQLCLIDriver
+ // SPARK-56147: the parser-based splitter trims surrounding whitespace from
+ // each chunk and drops chunks that contain only comments / whitespace.
Seq("SELECT 1; --comment" -> Seq("SELECT 1"),
"SELECT 1; /* comment */" -> Seq("SELECT 1"),
- "SELECT 1; /* comment" -> Seq("SELECT 1", " /* comment"),
- "SELECT 1; /* comment select 1;" -> Seq("SELECT 1", " /* comment select
1;"),
+ "SELECT 1; /* comment" -> Seq("SELECT 1", "/* comment"),
+ "SELECT 1; /* comment select 1;" -> Seq("SELECT 1", "/* comment select
1;"),
"/* This is a comment without end symbol SELECT 1;" ->
Seq("/* This is a comment without end symbol SELECT 1;"),
"SELECT 1; --comment\n" -> Seq("SELECT 1"),
"SELECT 1; /* comment */\n" -> Seq("SELECT 1"),
- "SELECT 1; /* comment\n" -> Seq("SELECT 1", " /* comment\n"),
- "SELECT 1; /* comment select 1;\n" -> Seq("SELECT 1", " /* comment
select 1;\n"),
+ "SELECT 1; /* comment\n" -> Seq("SELECT 1", "/* comment"),
+ "SELECT 1; /* comment select 1;\n" -> Seq("SELECT 1", "/* comment select
1;"),
"/* This is a comment without end symbol SELECT 1;\n" ->
- Seq("/* This is a comment without end symbol SELECT 1;\n"),
+ Seq("/* This is a comment without end symbol SELECT 1;"),
"/* comment */ SELECT 1;" -> Seq("/* comment */ SELECT 1"),
"SELECT /* comment */ 1;" -> Seq("SELECT /* comment */ 1"),
"-- comment " -> Seq(),
"-- comment \nSELECT 1" -> Seq("-- comment \nSELECT 1"),
"/* comment */ " -> Seq(),
// SPARK-54876: statement after semicolon ending with block comment
should not be dropped
- "SELECT 1; SELECT 2 /* comment */" -> Seq("SELECT 1", " SELECT 2 /*
comment */"),
+ "SELECT 1; SELECT 2 /* comment */" -> Seq("SELECT 1", "SELECT 2 /*
comment */"),
// SPARK-54876: line comment followed by block comment should produce
empty result
"-- foo\n/* bar */" -> Seq(),
"SELECT 1; -- foo\n /* bar */" -> Seq("SELECT 1"),
@@ -685,9 +699,9 @@ class CliSuite extends SparkFunSuite {
// SPARK-54876: preceding closed block comment + line comment (no SQL
statement)
"/* a */ -- foo\n/* b */" -> Seq(),
// SPARK-54876: semicolons inside backtick-quoted identifiers are not
split points
- "SELECT * FROM `t;a`; SELECT 1" -> Seq("SELECT * FROM `t;a`", " SELECT
1")
+ "SELECT * FROM `t;a`; SELECT 1" -> Seq("SELECT * FROM `t;a`", "SELECT 1")
).foreach { case (query, ret) =>
- assert(cli.splitSemiColon(query).asScala === ret)
+ assert(cli.splitStatements(query) === ret)
}
sessionState.close()
SparkSQLEnv.stop()
@@ -898,4 +912,40 @@ class CliSuite extends SparkFunSuite {
runCliWithin(2.minutes, extraArgs = Seq("-f", sqlFilePath.toString))(""
-> "x\t1")
}
}
+
+ test("SPARK-56147: `\\;` at end of line is a Hive-compat line continuation
marker") {
+ // `\;` at the end of an interactive line is a Hive-compat continuation
+ // marker: it defers execution and shows the continuation prompt for the
+ // next line. The joined input is then sent to the backend; the `\;` chunk
+ // reattachment in `processLine` reinjects a literal `;` in the middle,
+ // so the backend reports a parse error. Asserting the parse error here
+ // also confirms the continuation prompt for the second line was reached
+ // (otherwise the queryEcho for `' second';` would carry the main prompt
+ // and the first expected echo would not match).
+ runCliWithin(2.minutes, errorResponses = Nil)(
+ """SELECT 'first' \;
+ |, 'second';""".stripMargin -> "PARSE_SYNTAX_ERROR"
+ )
+ }
+
+ test("SPARK-56147: interactive BEGIN ... END block buffers across lines " +
+ "and runs as a single statement") {
+ // The interactive loop is line-level. A multi-line `BEGIN ... END` SQL
+ // scripting block carries internal `;` delimiters (after `DECLARE`, `SET`,
+ // the `WHILE ... END WHILE`, etc.), but the parser-based splitter
recognises
+ // the whole block as one compound statement, so the CLI keeps buffering on
+ // the continuation prompt until the closing `END;` instead of firing early
+ // at an internal `;`. Feeding the block line-by-line and asserting the
+ // trailing `SELECT` produces the value computed by the loop confirms the
+ // block executed as a single statement.
+ runCliWithin(2.minutes)(
+ """BEGIN
+ | DECLARE c INT = 0;
+ | WHILE c < 3 DO
+ | SET c = c + 1;
+ | END WHILE;
+ | SELECT 'loop-ran-' || CAST(c AS STRING) AS r;
+ |END;""".stripMargin -> "loop-ran-3"
+ )
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
