srielau commented on code in PR #58530:
URL: https://github.com/apache/spark/pull/58530#discussion_r3941807738
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala:
##########
@@ -27,34 +27,37 @@ import org.apache.spark.sql.types.{AbstractDataType,
DataType, StringType}
import org.apache.spark.unsafe.types.UTF8String
/**
- * Parses a SQL statement string and returns a compact JSON description of the
- * unresolved statement (identifier/code, lineage references, select-list
names,
- * parameters), or a STANDARD-format error object when the statement does not
- * parse.
+ * Parses a SQL batch string and returns a compact JSON array describing its
+ * unresolved statements (source position, identifier/code, lineage references,
+ * select-list names, parameters). A statement that does not parse is
represented
+ * by a STANDARD-format error object at its position in the array.
*
* Behind [[SQLConf.PARSE_SQL_ENABLED]] while the JSON contract is still
* evolving. Designed for batch evaluation over DataFrames of SQL text.
* User-facing parse errors become JSON; unexpected internal failures
propagate.
*/
// scalastyle:off line.size.limit
@ExpressionDescription(
- usage = """_FUNC_(sqlStmt) - Parses `sqlStmt` with the stock Spark SQL
parser and
- returns a JSON string describing the statement (parse success, Table 39
statement
- identifier/code, target and source table references for lineage,
select-list column
- names, and parameter markers). Session parser extensions are not applied.
+ usage = """_FUNC_(sqlStmt) - Splits `sqlStmt` into SQL statements, parses
each with
+ the stock Spark SQL parser, and returns a JSON array describing them
(1-based start
Review Comment:
Documented in c15a60657c5. Usage and class docs now say `start` is a 1-based
UTF-16 code-unit offset and `length` is a UTF-16 code-unit count. Added a
non-BMP example: `parse_sql('SELECT ''😀'';SELECT 2')` reports `$[1].start` as
13 (not a code-point 11).
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:
##########
@@ -158,6 +190,36 @@ object SqlStatementSplitter {
// interpretation (e.g. `double_quoted_identifiers`).
val conf = SqlApiConf.get
+ def appendToken(token: Token): Unit = {
+ if (buffer.isEmpty) {
+ // CodePointCharStream token offsets count Unicode code points, while
+ // String offsets and lengths count UTF-16 code units.
+ bufferStart = sqlText.offsetByCodePoints(0, token.getStartIndex)
Review Comment:
Fixed in c15a60657c5. `utf16Offsets` now builds one code-point-to-UTF-16 map
for the whole batch, so token start/stop lookup is O(1) and splitting stays
linear in the input length.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:
##########
@@ -158,6 +190,36 @@ object SqlStatementSplitter {
// interpretation (e.g. `double_quoted_identifiers`).
val conf = SqlApiConf.get
+ def appendToken(token: Token): Unit = {
+ if (buffer.isEmpty) {
+ // CodePointCharStream token offsets count Unicode code points, while
+ // String offsets and lengths count UTF-16 code units.
+ bufferStart = sqlText.offsetByCodePoints(0, token.getStartIndex)
+ }
+ buffer.append(token.getText)
+ }
+
+ def resetBuffer(): Unit = {
+ buffer.setLength(0)
+ bufferStart = -1
+ bufferHasContent = false
+ }
+
+ def positionedStatement(terminator: String):
Option[PositionedSqlStatement] = {
+ val raw = buffer.toString
+ val statement = raw.trim
Review Comment:
Fixed in c15a60657c5. Spans are trimmed with the Spark SQL lexer `WS` set
(including U+00A0 and the other Unicode spaces), not `String.trim`.
`parse_sql("\u00a0SELECT 1\u00a0;")` now reports start 2 / length 8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
