This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 0f76201e6cbf [SPARK-60672][SQL] Treat SQL query-context indexes as
Unicode code points
0f76201e6cbf is described below
commit 0f76201e6cbf20530ba773775e8c0b67f99f6d4c
Author: Serge Rielau <[email protected]>
AuthorDate: Sat Oct 31 02:24:10 2026 +0000
[SPARK-60672][SQL] Treat SQL query-context indexes as Unicode code points
### What changes were proposed in this pull request?
`SQLQueryContext` treated parser `startIndex`/`stopIndex` as UTF-16
offsets. Parser origins are ANTLR Unicode code points, so any error after a
supplementary character reported the wrong fragment.
This PR makes every SQL error surface use code points:
- `SQLQueryContext.fragment`, `summary`, and `isValid` interpret indexes as
code points. Text with no supplementary characters keeps the previous path,
which is identical.
- Whole-text origins for parameterized queries, `EXECUTE IMMEDIATE`, and
SQL-scripting CASE conditions set `stopIndex` from `codePointCount`.
- `PositionMapper` measures substituted parameter values in code points, so
an error after a value that contains a supplementary character maps back to the
original SQL.
- Statement-text recovery (`SingleStatement.getText`, `EXECUTE IMMEDIATE`
original-text extraction, pipeline file offsets) slices by code points.
- `parse_sql` statement `start` and `length` change from UTF-16 code units
to Unicode code points (the SPARK-60620 work this PR also carries). For the
documented example, `$[1].start` changes from 13 to 12. Nested error objects
keep the shared renderer's fragment and only widen a context that falls outside
the statement.
The public `QueryContext.startIndex()` / `stopIndex()` docs now say these
values are 0-based Unicode code-point offsets, with an inclusive stop.
### Why are the changes needed?
For `SELECT '😀' AS a, DATE 'bad'`, Spark currently reports fragment ` DATE
'bad`. Query results are not affected. The PRETTY caret line for that
single-line context is unchanged (the UTF-16 shift and the extra emoji column
cancel); the caret changes only when the context is on a later line than a
supplementary character, or when start-truncation walks past one.
### Does this PR introduce _any_ user-facing change?
Yes. For SQL that contains supplementary characters, error fragments become
the code-point range of the context. PRETTY carets change for a context on a
later line than a supplementary character, or a start-truncated context with
one before it. Whole-text `stopIndex` values and mapped positions after a
supplementary parameter value also become code points. `parse_sql` statement
`start`/`length` are code points rather than UTF-16 units. SQL without
supplementary characters is unchanged.
### How was this patch tested?
- `SQLQueryContextSuite`: fragment, invalid overshoot `stopIndex`,
same-line and later-line carets, and `sqlTextSlice` after a supplementary
character.
- `QueryContextSuite`: regular SQL, ASCII no-change, named parameter value
containing an emoji, `EXECUTE IMMEDIATE` typed-literal and syntax errors.
- `ParameterSubstitutionSuite`: `PositionMapper` after a supplementary
replacement, and whole-text origin `stopIndex` mapping.
- `SqlScriptingParserSuite`: recovered statement text after an emoji.
- `SqlQueryOriginSuite`: pipeline SQL origin after a comment that contains
an emoji.
- `ParseSqlResultSuite`: parse_sql error-path spans and scripting-context
fragments after a supplementary character.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
Closes #59868 from srielau/SPARK-60620-parse-sql-unicode.
Authored-by: Serge Rielau <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit b474a93760154c092dec8e4885f4124af83bba4a)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../main/java/org/apache/spark/QueryContext.java | 10 +-
python/pyspark/errors/exceptions/base.py | 8 +-
.../spark/sql/catalyst/parser/PositionMapper.scala | 5 +-
.../spark/sql/catalyst/trees/QueryContexts.scala | 76 ++++++++++---
.../apache/spark/sql/catalyst/trees/origin.scala | 10 ++
.../plans/logical/SqlScriptingLogicalPlans.scala | 4 +-
.../parser/ParameterSubstitutionSuite.scala | 27 +++++
.../catalyst/parser/SqlScriptingParserSuite.scala | 8 ++
.../sql/catalyst/trees/SQLQueryContextSuite.scala | 65 ++++++++++++
.../analysis/ResolveExecuteImmediate.scala | 11 +-
.../spark/sql/catalyst/expressions/ParseSql.scala | 7 +-
.../spark/sql/catalyst/parser/ParseSqlResult.scala | 118 +++++++++++++++++----
.../spark/sql/execution/SparkSqlParser.scala | 2 +-
.../sql/scripting/SqlScriptingExecutionNode.scala | 6 +-
.../sql/catalyst/parser/ParseSqlResultSuite.scala | 57 +++++++++-
.../spark/sql/errors/QueryContextSuite.scala | 69 +++++++++++-
.../execution/ExecuteImmediateEndToEndSuite.scala | 14 +++
.../scripting/SqlScriptingInterpreterSuite.scala | 21 ++++
.../graph/SqlGraphRegistrationContext.scala | 24 +++--
.../sql/pipelines/graph/SqlQueryOriginSuite.scala | 23 ++++
20 files changed, 498 insertions(+), 67 deletions(-)
diff --git a/common/utils-java/src/main/java/org/apache/spark/QueryContext.java
b/common/utils-java/src/main/java/org/apache/spark/QueryContext.java
index 45e38c8cfe0f..f05f8cf14afd 100644
--- a/common/utils-java/src/main/java/org/apache/spark/QueryContext.java
+++ b/common/utils-java/src/main/java/org/apache/spark/QueryContext.java
@@ -40,10 +40,16 @@ public interface QueryContext {
// Otherwise, it should be the object name. For example, a view name "V1".
String objectName();
- // The starting index in the query text which throws the exception. The
index starts from 0.
+ /**
+ * The starting index in the query text which throws the exception.
+ * This is a 0-based Unicode code-point offset.
+ */
int startIndex();
- // The stopping index in the query which throws the exception. The index
starts from 0.
+ /**
+ * The stopping index in the query text which throws the exception.
+ * This is a 0-based Unicode code-point offset, inclusive.
+ */
int stopIndex();
// The corresponding fragment of the query which throws the exception.
diff --git a/python/pyspark/errors/exceptions/base.py
b/python/pyspark/errors/exceptions/base.py
index aeabb46c2579..206f1f2ddde7 100644
--- a/python/pyspark/errors/exceptions/base.py
+++ b/python/pyspark/errors/exceptions/base.py
@@ -434,14 +434,18 @@ class QueryContext(ABC):
@abstractmethod
def startIndex(self) -> int:
"""
- The starting index in the query text which throws the exception. The
index starts from 0.
+ The starting index in the query text which throws the exception.
+
+ The index is a 0-based Unicode code-point offset.
"""
...
@abstractmethod
def stopIndex(self) -> int:
"""
- The stopping index in the query which throws the exception. The index
starts from 0.
+ The stopping index in the query text which throws the exception.
+
+ The index is a 0-based Unicode code-point offset, inclusive.
"""
...
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala
index c8a67cb3203f..d88f0de8eb66 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/PositionMapper.scala
@@ -132,7 +132,10 @@ case class PositionMapper(
val newSubstitutedPos = substitutedPos + textBeforeLength
// Create range for the substitution
- val substitutedLength = substitution.replacement.length
+ // Parser and mapper indexes are Unicode code points. A supplementary
+ // character is one code point, not String.length's two UTF-16 units.
+ val replacement = substitution.replacement
+ val substitutedLength = replacement.codePointCount(0,
replacement.length)
val originalLength = substitution.end - substitution.start
val offsetDelta = originalLength - substitutedLength
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala
index d5be65a2f36c..a893de58f648 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala
@@ -19,6 +19,25 @@ package org.apache.spark.sql.catalyst.trees
import org.apache.spark.{QueryContext, QueryContextType,
SparkUnsupportedOperationException}
+object SQLQueryContext {
+
+ /**
+ * Inclusive Unicode code-point slice `[from, to]` of `text`. Indexes are
clamped to the text.
+ * Parser origins store ANTLR code-point offsets, so this is the unit every
SQL-text recovery
+ * path must use.
+ */
+ private[sql] def sliceByCodePoints(text: String, from: Int, to: Int): String
= {
+ val count = text.codePointCount(0, text.length)
+ val begin = math.max(0, math.min(from, count))
+ val end = math.max(begin, math.min(to + 1, count))
+ text.substring(text.offsetByCodePoints(0, begin),
text.offsetByCodePoints(0, end))
+ }
+
+ /** True when `text` contains a supplementary character (two UTF-16 units).
*/
+ private def hasSupplementary(text: String): Boolean =
+ text.length != text.codePointCount(0, text.length)
+}
+
/** The class represents error context of a SQL query. */
case class SQLQueryContext(
line: Option[Int],
@@ -62,8 +81,18 @@ case class SQLQueryContext(
builder ++= s"== SQL$objectContext$positionContext ==\n"
val text = sqlText.get
- val start = math.max(originStartIndex.get, 0)
- val stop = math.min(originStopIndex.getOrElse(text.length - 1),
text.length - 1)
+ // Parser origins are code-point indexes. Convert to UTF-16 for line
walking
+ // when the text has supplementary characters; otherwise the two units
match.
+ val startCp = math.max(originStartIndex.get, 0)
+ val stopCp = originStopIndex.get
+ val (start, stop) = if (SQLQueryContext.hasSupplementary(text)) {
+ val count = text.codePointCount(0, text.length)
+ val from = text.offsetByCodePoints(0, startCp)
+ val to = text.offsetByCodePoints(0, math.min(stopCp + 1, count)) - 1
+ (from, to)
+ } else {
+ (startCp, math.min(stopCp, text.length - 1))
+ }
// Ideally we should show all the lines which contains the SQL text
context of the current
// node:
// [additional text] [current tree node] [additional text]
@@ -99,16 +128,31 @@ case class SQLQueryContext(
text.substring(lineStartIndex + 1, lineStopIndex) +
(if (stopTruncated) truncatedText else "")
val lines = truncatedSubText.split("\n")
+ val oneCaretPerCodePoint = SQLQueryContext.hasSupplementary(text)
lines.foreach { lineText =>
builder ++= lineText + "\n"
currentIndex += 1
- (0 until lineText.length).foreach { _ =>
- if (currentIndex < start) {
- builder ++= " "
- } else if (currentIndex >= start && currentIndex <= stop) {
- builder ++= "^"
+ if (oneCaretPerCodePoint) {
+ var i = 0
+ while (i < lineText.length) {
+ val width = Character.charCount(lineText.codePointAt(i))
+ if (currentIndex < start) {
+ builder ++= " "
+ } else if (currentIndex <= stop) {
+ builder ++= "^"
+ }
+ currentIndex += width
+ i += width
+ }
+ } else {
+ (0 until lineText.length).foreach { _ =>
+ if (currentIndex < start) {
+ builder ++= " "
+ } else if (currentIndex >= start && currentIndex <= stop) {
+ builder ++= "^"
+ }
+ currentIndex += 1
}
- currentIndex += 1
}
builder ++= "\n"
}
@@ -120,15 +164,23 @@ case class SQLQueryContext(
lazy val fragment: String = {
if (!isValid) {
""
- } else {
+ } else if (!SQLQueryContext.hasSupplementary(sqlText.get)) {
sqlText.get.substring(originStartIndex.get, originStopIndex.get + 1)
+ } else {
+ SQLQueryContext.sliceByCodePoints(sqlText.get, originStartIndex.get,
originStopIndex.get)
}
}
def isValid: Boolean = {
- sqlText.isDefined && originStartIndex.isDefined &&
originStopIndex.isDefined &&
- originStartIndex.get >= 0 && originStopIndex.get < sqlText.get.length &&
- originStartIndex.get <= originStopIndex.get
+ sqlText.isDefined && originStartIndex.isDefined &&
originStopIndex.isDefined && {
+ val text = sqlText.get
+ val start = originStartIndex.get
+ val stop = originStopIndex.get
+ val limit =
+ if (SQLQueryContext.hasSupplementary(text)) text.codePointCount(0,
text.length)
+ else text.length
+ start >= 0 && stop < limit && start <= stop
+ }
}
override def callSite: String = throw SparkUnsupportedOperationException()
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala
index a1ed62f4359f..64c2a645169d 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/origin.scala
@@ -71,6 +71,16 @@ case class Origin(
case _ => true
}.toArray
}
+
+ /**
+ * The SQL text covered by this origin. Indexes are Unicode code points,
matching parser
+ * origins.
+ */
+ def sqlTextSlice: Option[String] = (sqlText, startIndex, stopIndex) match {
+ case (Some(text), Some(from), Some(to)) =>
+ Some(SQLQueryContext.sliceByCodePoints(text, from, to))
+ case _ => None
+ }
}
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala
index fdd5524b536e..f26bf0b35bf1 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala
@@ -48,8 +48,8 @@ case class SingleStatement(parsedPlan: LogicalPlan)
* SQL query text.
*/
def getText: String = {
- assert(origin.sqlText.isDefined && origin.startIndex.isDefined &&
origin.stopIndex.isDefined)
- origin.sqlText.get.substring(origin.startIndex.get, origin.stopIndex.get +
1)
+ assert(origin.sqlTextSlice.isDefined)
+ origin.sqlTextSlice.get
}
override def output: Seq[Attribute] = parsedPlan.output
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ParameterSubstitutionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ParameterSubstitutionSuite.scala
index 8baf3a921eaf..220e2c08b18c 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ParameterSubstitutionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ParameterSubstitutionSuite.scala
@@ -18,6 +18,7 @@ package org.apache.spark.sql.catalyst.parser
import org.apache.spark.SparkFunSuite
import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.trees.{Origin, SQLQueryContext}
/**
* Test suite for the new parameter substitution architecture.
@@ -158,6 +159,32 @@ class ParameterSubstitutionSuite extends SparkFunSuite {
assert(result === s"SELECT '${flexed}', '${tada}', '${flexed}${flexed}',
99")
}
+ test("whole-text origin stopIndex after substitution is a code-point
offset") {
+ val emoji = new String(Character.toChars(0x1F600))
+ val original = s"SELECT '$emoji' AS a, :p AS b"
+ val (substituted, mapper) = ParameterHandler.substituteParameters(
+ original, NamedParameterContext(Map("p" -> Literal(1))))
+ val origin = Origin(
+ sqlText = Some(substituted),
+ startIndex = Some(0),
+ stopIndex = Some(substituted.codePointCount(0, substituted.length) - 1),
+ positionMapper = Some(mapper))
+ val ctx = origin.context.asInstanceOf[SQLQueryContext]
+ assert(ctx.isValid)
+ assert(ctx.fragment === original)
+ assert(ctx.startIndex === 0)
+ assert(ctx.stopIndex === original.codePointCount(0, original.length) - 1)
+ }
+
+ test("PositionMapper maps past a supplementary replacement in code points") {
+ val emoji = new String(Character.toChars(0x1F600))
+ val original = "SELECT :p AS a, DATE 'bad'"
+ val (substituted, mapper) = ParameterHandler.substituteParameters(
+ original, NamedParameterContext(Map("p" -> Literal(emoji))))
+ val dateInSubstituted = substituted.codePointCount(0,
substituted.indexOf("DATE"))
+ assert(mapper.mapToOriginal(dateInSubstituted) ===
original.indexOf("DATE"))
+ }
+
test("Large parameter set") {
val largeParamMap = (1 to 100).map(i => s"param$i" -> Literal(i)).toMap
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala
index 8c4c9044248d..2eee0607f10e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/SqlScriptingParserSuite.scala
@@ -65,6 +65,14 @@ class SqlScriptingParserSuite extends SparkFunSuite with
SQLHelper {
assert(e.getMessage.contains("SELECT"))
}
+ test("statement text after a supplementary character") {
+ val emoji = new String(Character.toChars(0x1F600))
+ val sqlScriptText = s"BEGIN SELECT '$emoji'; SELECT 2; END"
+ val tree = parsePlan(sqlScriptText).asInstanceOf[CompoundBody]
+ val statements =
tree.collection.map(_.asInstanceOf[SingleStatement].getText)
+ assert(statements === Seq(s"SELECT '$emoji'", "SELECT 2"))
+ }
+
test("multi select") {
val sqlScriptText = "BEGIN SELECT 1;SELECT 2; END"
val tree = parsePlan(sqlScriptText).asInstanceOf[CompoundBody]
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/SQLQueryContextSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/SQLQueryContextSuite.scala
index 61fe90093d29..36bbfb09b960 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/SQLQueryContextSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/trees/SQLQueryContextSuite.scala
@@ -45,5 +45,70 @@ class SQLQueryContextSuite extends SparkFunSuite with
SQLHelper {
).foreach { case ((sqlText, start, stop), expectedFragment) =>
assert(getFragment(sqlText, start, stop) === expectedFragment)
}
+
+ val emoji = new String(Character.toChars(0x1F600))
+ val sql = s"SELECT '$emoji' AS a, DATE 'bad'"
+ val start = sql.codePointCount(0, sql.indexOf("DATE"))
+ val stop = sql.codePointCount(0, sql.length) - 1
+ assert(getFragment(sql, start, stop) === "DATE 'bad'")
+ // A UTF-16 slice of the same indexes would include a leading space and
drop the last quote.
+ assert(sql.substring(start, stop + 1) !== "DATE 'bad'")
+
+ val overshoot = SQLQueryContext(
+ line = None,
+ startPosition = None,
+ originStartIndex = Some(0),
+ originStopIndex = Some(sql.codePointCount(0, sql.length)),
+ sqlText = Some(sql),
+ originObjectType = None,
+ originObjectName = None)
+ assert(overshoot.fragment === "")
+ assert(overshoot.summary === "")
+ }
+
+ test("summary caret follows code-point indexes after a supplementary
character") {
+ def summaryLines(sql: String, line: Int, startPos: Int): Array[String] = {
+ val start = sql.codePointCount(0, sql.indexOf("DATE"))
+ val stop = sql.codePointCount(0, sql.length) - 1
+ SQLQueryContext(
+ line = Some(line),
+ startPosition = Some(startPos),
+ originStartIndex = Some(start),
+ originStopIndex = Some(stop),
+ sqlText = Some(sql),
+ originObjectType = None,
+ originObjectName = None).summary.split('\n')
+ }
+
+ val emoji = new String(Character.toChars(0x1F600))
+ val sameLine = s"SELECT '$emoji' AS a, DATE 'bad'"
+ val sameStart = sameLine.codePointCount(0, sameLine.indexOf("DATE"))
+ val sameLines = summaryLines(sameLine, 1, sameStart)
+ assert(sameLines(1) === sameLine)
+ assert(sameLines(2) === " " * sameStart + "^" * "DATE 'bad'".length)
+
+ // A later line is where UTF-16 caret placement no longer cancels with the
emoji width.
+ val nextLine = s"SELECT '$emoji' AS a,\nDATE 'bad'"
+ val nextLines = summaryLines(nextLine, 2, 0)
+ assert(nextLines(1) === "DATE 'bad'")
+ assert(nextLines(2) === "^" * "DATE 'bad'".length)
+
+ val indented = s"SELECT '$emoji' AS a,\n b, DATE 'bad'"
+ val indentedLines = summaryLines(indented, 2,
indented.split('\n')(1).indexOf("DATE"))
+ assert(indentedLines(1) === " b, DATE 'bad'")
+ val col = indentedLines(1).indexOf("DATE")
+ assert(indentedLines(2) === " " * col + "^" * "DATE 'bad'".length)
+ }
+
+ test("sqlTextSlice uses code-point indexes after a supplementary character")
{
+ val emoji = new String(Character.toChars(0x1F600))
+ val sql = s"BEGIN SELECT '$emoji'; SELECT 2; END"
+ val start = sql.codePointCount(0, sql.indexOf("SELECT 2"))
+ val stop = start + "SELECT 2".length - 1
+ val origin = Origin(
+ startIndex = Some(start),
+ stopIndex = Some(stop),
+ sqlText = Some(sql))
+ assert(origin.sqlTextSlice.contains("SELECT 2"))
}
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala
index e0314e837108..7d1a3e0365d9 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveExecuteImmediate.scala
@@ -139,7 +139,7 @@ object ResolveExecuteImmediate {
objectName = None, // No named object for EXECUTE IMMEDIATE, unlike views
sqlText = Some(sqlString),
startIndex = Some(0),
- stopIndex = Some(sqlString.length - 1)
+ stopIndex = Some(sqlString.codePointCount(0, sqlString.length) - 1)
)
// Parse and analyze the inner statement with local variables hidden and
EXECUTE IMMEDIATE
@@ -227,14 +227,7 @@ object ResolveExecuteImmediate {
private def extractOriginalText(queryExpr: Expression): String = {
val origin = queryExpr.origin
// Try to extract the original text from the origin information
- (origin.sqlText, origin.startIndex, origin.stopIndex) match {
- case (Some(sqlText), Some(startIndex), Some(stopIndex)) =>
- // Extract the substring from the original SQL text
- sqlText.substring(startIndex, stopIndex + 1)
- case _ =>
- // Fallback to the SQL representation if origin information is not
available
- queryExpr.sql
- }
+ origin.sqlTextSlice.getOrElse(queryExpr.sql)
}
/**
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala
index 9551b9e10acd..616a76e0f8b5 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala
@@ -41,7 +41,7 @@ import org.apache.spark.unsafe.types.UTF8String
@ExpressionDescription(
usage = """_FUNC_(sqlStmt) - Splits `sqlStmt` into SQL statements, parses
each with
the stock Spark SQL parser, and returns a JSON array describing them
(1-based
- UTF-16 code-unit `start` and UTF-16 code-unit `length`, parse success,
Table 39
+ Unicode code-point `start` and code-point `length`, parse success, Table 39
statement identifier/code, target and source table references for lineage,
select-list column names, and parameter markers).
Statement length excludes surrounding whitespace and the terminating
semicolon.
@@ -49,7 +49,8 @@ import org.apache.spark.unsafe.types.UTF8String
Requires spark.sql.function.parseSql.enabled=true. On syntax / parse error
returns JSON
for that statement with `parse_success` false, source location, and a
nested STANDARD
error object instead of throwing or stopping the remaining statements.
Nested error
- locations are statement-relative. An empty or closed-comment-only batch
returns `[]`.""",
+ locations are statement-relative Unicode code points. An empty or
closed-comment-only
+ batch returns `[]`.""",
arguments = """
Arguments:
* sqlStmt - A SQL batch string to split and parse.
@@ -60,7 +61,7 @@ import org.apache.spark.unsafe.types.UTF8String
> SELECT _FUNC_('SELECT 1;SELECT 2');
[{"start":1,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]},{"start":10,"length":8,"parse_success":true,"statement_identifier":"SELECT","statement_code":21,"select_list":[{"name":[]}]}]
> SELECT get_json_object(_FUNC_('SELECT ''😀'';SELECT 2'), '$[1].start');
- 13
+ 12
> SELECT get_json_object(_FUNC_('SELEC'), '$[0].error.errorClass');
PARSE_SYNTAX_ERROR
""",
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
index 3155fca0fd50..1989866b2d8f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
@@ -43,9 +43,9 @@ import
org.apache.spark.sql.execution.datasources.CreateTempViewUsing
* executors without a session, so only the stock parser is available under
* distributed eval.
*
- * Every statement object includes its 1-based UTF-16 code-unit `start` in the
- * original batch and its UTF-16 code-unit `length`, excluding surrounding
- * whitespace and the terminating semicolon. On success it also includes
+ * Every statement object includes its 1-based Unicode code-point `start` in
the
+ * original batch and its code-point `length`, excluding surrounding whitespace
+ * and the terminating semicolon. On success it also includes
* `parse_success`, the statement
* identifier/code (ISO/IEC 9075-2:2023 Table 39), and omits unused optional
* fields (`target_table_references`, `source_table_references`,
@@ -53,7 +53,8 @@ import
org.apache.spark.sql.execution.datasources.CreateTempViewUsing
* parse failure the statement object contains `parse_success: false` with
* source location and a nested STANDARD-format error object, and parsing
* continues with later statements. Nested error locations are relative to the
- * individual statement, while `start` is relative to the original batch. An
+ * individual statement, while `start` is relative to the original batch; error
+ * `queryContext` indexes are code points as well. An
* empty or closed-comment-only batch produces an empty array. Only
* [[ParseException]] / [[SqlScriptingException]] are converted to JSON;
* unexpected / internal failures propagate so the function fails.
@@ -67,38 +68,49 @@ object ParseSqlResult {
def fromSql(sql: String): String = {
val split = parser.get().splitStatementsWithPositions(sql)
val statements = split.completeStatements ++ split.partialStatement
- compact(render(JArray(statements.map(parseStatement).toList)))
+ val starts = charStarts(sql, statements)
+ compact(render(JArray(statements.zip(starts).map {
+ case (segment, start) => parseStatement(segment, start)
+ }.toList)))
}
- private def parseStatement(segment: PositionedSqlStatement): JObject = {
+ private def parseStatement(segment: PositionedSqlStatement, charStart: Int):
JObject = {
val sql = segment.statement
+ val length = sql.codePointCount(0, sql.length)
try {
// Do not inherit the outer query's origin from the parse_sql expression.
- // Errors and parsed nodes refer to the individual statement.
+ // Errors and parsed nodes refer to the individual statement. Parser
+ // origins are code-point offsets, so this one is too.
val origin = if (sql.nonEmpty) {
- Origin(startIndex = Some(0), stopIndex = Some(sql.length - 1), sqlText
= Some(sql))
+ Origin(
+ startIndex = Some(0),
+ stopIndex = Some(length - 1),
+ sqlText = Some(sql))
} else {
Origin(sqlText = Some(sql))
}
CurrentOrigin.withOrigin(origin) {
val plan = parser.get().parsePlan(sql)
- successJson(plan, segment)
+ successJson(plan, charStart, length)
}
} catch {
// User-facing parse / scripting failures become JSON; everything else
fails.
case e: ParseException =>
- errorJson(e, segment)
+ errorJson(e, segment, charStart, length)
case e: SqlScriptingException =>
- errorJson(e, segment)
+ errorJson(e, segment, charStart, length)
}
}
/** Build success JSON from an already-parsed unresolved plan. */
- private def successJson(plan: LogicalPlan, segment: PositionedSqlStatement):
JObject = {
+ private def successJson(
+ plan: LogicalPlan,
+ charStart: Int,
+ length: Int): JObject = {
val classification = SqlStatementCodes.classify(plan)
val fields = mutable.ListBuffer.empty[JField]
- fields += "start" -> JInt(segment.start + 1)
- fields += "length" -> JInt(segment.length)
+ fields += "start" -> JInt(charStart)
+ fields += "length" -> JInt(length)
fields += "parse_success" -> JBool(true)
fields += "statement_identifier" ->
JString(classification.statementIdentifier)
fields += "statement_code" -> JInt(classification.statementCode)
@@ -123,9 +135,76 @@ object ParseSqlResult {
JObject(fields.toList)
}
+ /**
+ * 1-based Unicode code-point starts for `statements` in `batch`. Statements
+ * are in ascending UTF-16 start order, so one left-to-right pass is enough:
+ * each count covers only the gap since the previous start.
+ */
+ private def charStarts(
+ batch: String,
+ statements: Seq[PositionedSqlStatement]): Seq[Int] = {
+ var utf16 = 0
+ var codePoints = 0
+ statements.map { segment =>
+ val startUtf16 = math.max(0, math.min(segment.start, batch.length))
+ if (startUtf16 >= utf16) {
+ codePoints += batch.codePointCount(utf16, startUtf16)
+ } else {
+ codePoints = batch.codePointCount(0, startUtf16)
+ }
+ utf16 = startUtf16
+ codePoints + 1
+ }
+ }
+
+ /**
+ * Keep statement-level `queryContext` indexes inside the statement.
+ * In-range contexts keep the shared renderer's fragment. A context that
+ * falls outside the statement is widened to cover all of it. Contexts of
+ * another object (a view body) are left as is.
+ */
+ private def rewriteErrorCharIndexes(
+ errorObj: JObject,
+ statement: String,
+ length: Int): JObject = {
+ JObject(errorObj.obj.map {
+ case ("queryContext", JArray(contexts)) =>
+ "queryContext" -> JArray(contexts.map {
+ case ctx: JObject => rewriteQueryContextIndexes(ctx, statement,
length)
+ case other => other
+ }.toList)
+ case other => other
+ })
+ }
+
+ private def rewriteQueryContextIndexes(
+ ctx: JObject,
+ statement: String,
+ length: Int): JObject = {
+ val objectType = ctx.obj.collectFirst { case ("objectType", JString(t)) =>
t }
+ val start = ctx.obj.collectFirst { case ("startIndex", JInt(n)) => n.toInt
}
+ val stop = ctx.obj.collectFirst { case ("stopIndex", JInt(n)) => n.toInt }
+ (objectType, start, stop) match {
+ case (Some(""), Some(from), Some(to)) =>
+ if (from >= 1 && to >= from && to <= length) {
+ ctx
+ } else {
+ JObject(ctx.obj.map {
+ case ("startIndex", _) => "startIndex" -> JInt(1)
+ case ("stopIndex", _) => "stopIndex" -> JInt(length)
+ case ("fragment", _) => "fragment" -> JString(statement)
+ case other => other
+ })
+ }
+ case _ => ctx
+ }
+ }
+
private def errorJson(
e: SparkThrowable with Throwable,
- segment: PositionedSqlStatement): JObject = {
+ segment: PositionedSqlStatement,
+ charStart: Int,
+ length: Int): JObject = {
val errorObj = parseJson(
SparkThrowableHelper.getMessage(e,
ErrorMessageFormat.STANDARD)).asInstanceOf[JObject]
val origin = e match {
@@ -140,10 +219,13 @@ object ParseSqlResult {
origin.toSeq.flatMap(queryContextField)
}
JObject(
- "start" -> JInt(segment.start + 1),
- "length" -> JInt(segment.length),
+ "start" -> JInt(charStart),
+ "length" -> JInt(length),
"parse_success" -> JBool(false),
- "error" -> JObject(errorObj.obj ++ contextFields ++ locationFields)
+ "error" -> rewriteErrorCharIndexes(
+ JObject(errorObj.obj ++ contextFields ++ locationFields),
+ segment.statement,
+ length)
)
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
index 54c953e9f8ce..0458540e962f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
@@ -180,7 +180,7 @@ class SparkSqlParser extends AbstractSqlParser {
currentOrigin.copy(
sqlText = Some(paramSubstituted),
startIndex = Some(0),
- stopIndex = Some(paramSubstituted.length - 1),
+ stopIndex = Some(paramSubstituted.codePointCount(0,
paramSubstituted.length) - 1),
positionMapper = Some(positionMapper)
)
} else {
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala
b/sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala
index 7a2bd26f18c9..be188b3c8697 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecutionNode.scala
@@ -185,8 +185,8 @@ class SingleStatementExec(
* SQL query text.
*/
def getText: String = {
- assert(origin.sqlText.isDefined && origin.startIndex.isDefined &&
origin.stopIndex.isDefined)
- origin.sqlText.get.substring(origin.startIndex.get, origin.stopIndex.get +
1)
+ assert(origin.sqlTextSlice.isDefined)
+ origin.sqlTextSlice.get
}
/**
@@ -826,7 +826,7 @@ class SimpleCaseStatementExec(
condition,
Origin(sqlText = Some(conditionText),
startIndex = Some(0),
- stopIndex = Some(conditionText.length - 1),
+ stopIndex = Some(conditionText.codePointCount(0,
conditionText.length) - 1),
line = caseVariableExec.origin.line),
Map.empty,
isInternal = true,
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResultSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResultSuite.scala
index a04c8b775385..ea5cee50f8ce 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResultSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResultSuite.scala
@@ -35,6 +35,13 @@ class ParseSqlResultSuite extends SparkFunSuite {
case other => fail(s"expected JSON array, got: $other")
}
+ /** Slice `s` using 1-based Unicode code-point start and length. */
+ private def sliceByCodePoints(s: String, start: Int, length: Int): String = {
+ val begin = s.offsetByCodePoints(0, start - 1)
+ val end = s.offsetByCodePoints(begin, length)
+ s.substring(begin, end)
+ }
+
private def obj(sql: String): JObject = objs(sql) match {
case value :: Nil => value
case other => fail(s"expected one statement, got ${other.size}: $other")
@@ -103,9 +110,9 @@ class ParseSqlResultSuite extends SparkFunSuite {
val JInt(length) = statement \ "length"
(start.toInt, length.toInt)
}
- assert(spans === Seq((1, 13), (16, 8)))
+ assert(spans === Seq((1, 11), (14, 8)))
assert(spans.map { case (start, length) =>
- unicodeSql.substring(start - 1, start - 1 + length)
+ sliceByCodePoints(unicodeSql, start, length)
} === Seq(s"SELECT '$emoji$emoji'", "SELECT 2"))
val nbsp = 0xA0.toChar
@@ -246,4 +253,50 @@ class ParseSqlResultSuite extends SparkFunSuite {
assert(j \ "parse_success" === JBool(false))
assert(j \ "error" \ "errorClass" === JString("PARSE_SYNTAX_ERROR"))
}
+
+ test("error queryContext indexes are Unicode code points") {
+ val emoji = new String(Character.toChars(0x1F600))
+ val sql = s"SELECT '$emoji' FRUM t"
+ val j = obj(sql)
+ assert(j \ "parse_success" === JBool(false))
+ val ctx = (j \ "error" \ "queryContext")(0)
+ val JString(fragment) = ctx \ "fragment"
+ val JInt(start) = ctx \ "startIndex"
+ val JInt(stop) = ctx \ "stopIndex"
+ val JInt(position) = j \ "error" \ "position"
+ assert(fragment === sql)
+ assert(start.toInt === 1)
+ assert(stop.toInt === sql.codePointCount(0, sql.length))
+ assert(sliceByCodePoints(sql, start.toInt, stop.toInt - start.toInt + 1)
=== fragment)
+ // ANTLR column is already a 0-based code-point offset on the line.
+ assert(position.toInt === sql.codePointCount(0, sql.lastIndexOf('t')))
+
+ val literalSql = s"SELECT '$emoji' AS e, DATE 'bad'"
+ val literal = obj(literalSql)
+ assert(literal \ "error" \ "errorClass" ===
JString("INVALID_TYPED_LITERAL"))
+ val literalCtx = (literal \ "error" \ "queryContext")(0)
+ val JInt(literalStart) = literalCtx \ "startIndex"
+ val JInt(literalStop) = literalCtx \ "stopIndex"
+ assert(literalCtx \ "fragment" === JString("DATE 'bad'"))
+ assert(literalStart.toInt === literalSql.codePointCount(0,
literalSql.indexOf("DATE")) + 1)
+ assert(literalStop.toInt === literalSql.codePointCount(0,
literalSql.length))
+
+ val failedAfterEmoji = s"SELECT '$emoji'; SELECT '$emoji' FRUM t"
+ val failed = objs(failedAfterEmoji)
+ assert(failed.map(_ \ "parse_success") === Seq(JBool(true), JBool(false)))
+ val JInt(failedStart) = failed(1) \ "start"
+ val JInt(failedLength) = failed(1) \ "length"
+ assert((failedStart.toInt, failedLength.toInt) === (13, 17))
+ assert(sliceByCodePoints(failedAfterEmoji, failedStart.toInt,
failedLength.toInt) ===
+ s"SELECT '$emoji' FRUM t")
+
+ val scriptSql = s"BEGIN SELECT '$emoji'; lbl_begin: BEGIN SELECT 1; END
lbl_end; END"
+ val script = obj(scriptSql)
+ assert(script \ "error" \ "errorClass" === JString("LABELS_MISMATCH"))
+ val scriptCtx = (script \ "error" \ "queryContext")(0)
+ val JInt(scriptStart) = scriptCtx \ "startIndex"
+ val JInt(scriptStop) = scriptCtx \ "stopIndex"
+ assert(scriptCtx \ "fragment" === JString("lbl_begin:"))
+ assert((scriptStart.toInt, scriptStop.toInt) === (19, 28))
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryContextSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryContextSuite.scala
index dea02f3aa93d..fc2ae3798d3c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryContextSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryContextSuite.scala
@@ -16,7 +16,9 @@
*/
package org.apache.spark.sql.errors
-import org.apache.spark.{SparkArithmeticException, SparkConf}
+import org.apache.spark.{QueryContext, SparkArithmeticException, SparkConf,
SparkThrowable}
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.parser.{NamedParameterContext,
ParseException}
import org.apache.spark.sql.functions.{col, lit}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
@@ -34,12 +36,75 @@ class QueryContextSuite extends SharedSparkSession {
assert(e.getQueryContext.head.summary() ==
"""== DataFrame ==
|"div" was called from
-
|org.apache.spark.sql.errors.QueryContextSuite.$anonfun$new$3(QueryContextSuite.scala:32)
+
|org.apache.spark.sql.errors.QueryContextSuite.$anonfun$new$3(QueryContextSuite.scala:34)
|org.scalatest.Assertions.intercept(Assertions.scala:749)
|""".stripMargin)
}
}
+ test("SQL query context indexes are Unicode code points") {
+ val emoji = new String(Character.toChars(0x1F600))
+
+ def contextOf(thunk: => Any): QueryContext = {
+ val e = intercept[SparkThrowable] { thunk }
+ e.getQueryContext.head
+ }
+
+ val literalSql = s"SELECT '$emoji' AS a, DATE 'bad'"
+ val literal = contextOf(spark.sql(literalSql).collect())
+ val literalStart = literalSql.codePointCount(0, literalSql.indexOf("DATE"))
+ assert(literal.fragment() === "DATE 'bad'")
+ assert(literal.startIndex() === literalStart)
+ assert(literal.stopIndex() === literalSql.codePointCount(0,
literalSql.length) - 1)
+ val literalCaret = literal.summary().split('\n')(2)
+ assert(literalCaret === " " * literalStart + "^" * "DATE 'bad'".length)
+
+ val asciiSql = "SELECT 'x' AS a, DATE 'bad'"
+ val ascii = contextOf(spark.sql(asciiSql).collect())
+ assert(ascii.fragment() === "DATE 'bad'")
+ assert(ascii.startIndex() === asciiSql.indexOf("DATE"))
+
+ val namedSql = "SELECT :p AS a, DATE 'bad'"
+ val named = contextOf(spark.sql(namedSql, Map("p" -> emoji)).collect())
+ assert(named.fragment() === "DATE 'bad'")
+ assert(named.startIndex() === namedSql.indexOf("DATE"))
+
+ val namedAfterSql = s"SELECT :p AS a, '$emoji' AS b, DATE 'bad'"
+ val namedAfter = contextOf(spark.sql(namedAfterSql, Map("p" ->
1)).collect())
+ assert(namedAfter.fragment() === "DATE 'bad'")
+ assert(namedAfter.startIndex() ===
+ namedAfterSql.codePointCount(0, namedAfterSql.indexOf("DATE")))
+
+ // A non-empty parameter context still installs the whole-text origin even
when
+ // the SQL has no markers, so a syntax error observes
SparkSqlParser.stopIndex.
+ val syntaxSql = s"SELECT '$emoji' AS a, '$emoji' AS b FRUM t"
+ val syntaxEx = intercept[ParseException] {
+ spark.sessionState.sqlParser.parsePlanWithParameters(
+ syntaxSql,
+ NamedParameterContext(Map("p" -> Literal(1))))
+ }
+ assert(syntaxEx.getQueryContext.nonEmpty)
+ val syntax = syntaxEx.getQueryContext.head
+ assert(syntax.fragment() === syntaxSql)
+ assert(syntax.startIndex() === 0)
+ assert(syntax.stopIndex() === syntaxSql.codePointCount(0,
syntaxSql.length) - 1)
+
+ val execSql = s"SELECT '$emoji' AS a, DATE 'bad'"
+ val exec = contextOf(
+ spark.sql(s"EXECUTE IMMEDIATE '${execSql.replace("'",
"''")}'").collect())
+ assert(exec.fragment() === "DATE 'bad'")
+ assert(exec.objectType() === "EXECUTE IMMEDIATE")
+ assert(exec.startIndex() === execSql.codePointCount(0,
execSql.indexOf("DATE")))
+
+ val execSyntaxSql = s"SELECT '$emoji' FRUM t"
+ val execSyntax = contextOf(
+ spark.sql(s"EXECUTE IMMEDIATE '${execSyntaxSql.replace("'",
"''")}'").collect())
+ assert(execSyntax.fragment() === execSyntaxSql)
+ assert(execSyntax.objectType() === "EXECUTE IMMEDIATE")
+ assert(execSyntax.stopIndex() ===
+ execSyntaxSql.codePointCount(0, execSyntaxSql.length) - 1)
+ }
+
test("SPARK-50290: Add a flag to disable DataFrame context") {
withSQLConf(SQLConf.DATA_FRAME_QUERY_CONTEXT_ENABLED.key -> "false") {
val df = spark.range(1).select(lit(1) / col("id"))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/ExecuteImmediateEndToEndSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/ExecuteImmediateEndToEndSuite.scala
index 6fd0f95abcd3..8cbbfef7e187 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/ExecuteImmediateEndToEndSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/ExecuteImmediateEndToEndSuite.scala
@@ -23,6 +23,20 @@ import org.apache.spark.sql.test.SharedSparkSession
class ExecuteImmediateEndToEndSuite extends SharedSparkSession {
+ test("NULL_QUERY_STRING_EXECUTE_IMMEDIATE names the variable after a
supplementary character") {
+ val emoji = new String(Character.toChars(0x1F600))
+ withSessionVariable("q") {
+ spark.sql("DECLARE VARIABLE q STRING")
+ checkError(
+ exception = intercept[AnalysisException] {
+ spark.sql(s"EXECUTE IMMEDIATE -- $emoji\nq")
+ },
+ condition = "NULL_QUERY_STRING_EXECUTE_IMMEDIATE",
+ parameters = Map("varName" -> "`q`")
+ )
+ }
+ }
+
test("SPARK-47033: EXECUTE IMMEDIATE USING does not recognize session
variable names") {
try {
spark.sql("DECLARE parm = 'Hello';")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala
index 1469564ffec1..babe32d5658c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreterSuite.scala
@@ -598,6 +598,27 @@ class SqlScriptingInterpreterSuite
verifySqlScriptResult(commands, expected)
}
+ test("IF non-boolean condition after a supplementary character") {
+ val emoji = new String(Character.toChars(0x1F600))
+ val commands =
+ s"""
+ |BEGIN
+ | SELECT '$emoji';
+ | IF 1 THEN
+ | SELECT 1;
+ | END IF;
+ |END
+ |""".stripMargin
+
+ checkError(
+ exception = intercept[SqlScriptingException] (
+ runSqlScript(commands)
+ ),
+ condition = "INVALID_BOOLEAN_STATEMENT",
+ parameters = Map("invalidStatement" -> "1")
+ )
+ }
+
test("searched case with non boolean condition - constant") {
val commands =
"""
diff --git
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
index 288cf9a0eb0e..c119df4a7a5c 100644
---
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
+++
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
@@ -910,17 +910,21 @@ object SqlGraphRegistrationContext {
// The actual start position of the SQL query in this sqlText string.
Within sqlText it's
// possible that whitespace or comments precede the start of the query,
and is accounted for
// in the parsed logical plan's origin.
- val sqlStatementStartIdxInString =
logicalPlanFromSqlQuery.origin.startIndex.getOrElse(
- throw new SparkRuntimeException(
- errorClass = "INTERNAL_ERROR",
- messageParameters = Map(
- "message" ->
- s"""Unable to retrieve start index of logical plan parsed by the
following
- |SQL text:
- |
- |$rawSqlStatementText""".stripMargin)
+ val sqlStatementStartIdxInString = {
+ val startCp = logicalPlanFromSqlQuery.origin.startIndex.getOrElse(
+ throw new SparkRuntimeException(
+ errorClass = "INTERNAL_ERROR",
+ messageParameters = Map(
+ "message" ->
+ s"""Unable to retrieve start index of logical plan parsed by
the following
+ |SQL text:
+ |
+ |$rawSqlStatementText""".stripMargin)
+ )
)
- )
+ rawSqlStatementText.offsetByCodePoints(
+ 0, math.min(startCp, rawSqlStatementText.codePointCount(0,
rawSqlStatementText.length)))
+ }
// The actual start position of the SQL query in the entire file.
val sqlStatementStartIndexInFile = currentCharIndexInFile +
sqlStatementStartIdxInString
diff --git
a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/SqlQueryOriginSuite.scala
b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/SqlQueryOriginSuite.scala
index 8b004bd9fe7c..02bfbb48f18e 100644
---
a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/SqlQueryOriginSuite.scala
+++
b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/SqlQueryOriginSuite.scala
@@ -129,6 +129,29 @@ class SqlQueryOriginSuite extends PipelineTest with
SharedSparkSession {
))
}
+ test("Preceding comment with a supplementary character is omitted from sql
text") {
+ val emoji = new String(Character.toChars(0x1F600))
+ val sqlQueryOrigins = SqlGraphRegistrationContext.splitSqlFileIntoQueries(
+ spark,
+ sqlFilePath = "file.sql",
+ sqlFileText =
+ s"""
+ |-- $emoji
+ |CREATE MATERIALIZED VIEW mv AS SELECT 1;
+ |""".stripMargin
+ ).map(_.queryOrigin)
+ assert(sqlQueryOrigins == Seq(
+ QueryOrigin(
+ sqlText = Option("CREATE MATERIALIZED VIEW mv AS SELECT 1"),
+ filePath = Option("file.sql"),
+ language = Option(Language.Sql()),
+ // 1 newline, "-- " (3), emoji (2 UTF-16 units), 1 newline
+ startPosition = Option(7),
+ line = Option(3)
+ )
+ ))
+ }
+
test("Preceeding comment is ommitted from sql text") {
val sqlQueryOrigins = SqlGraphRegistrationContext.splitSqlFileIntoQueries(
spark,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]