This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 271386574a42 [SPARK-59218][SQL] Support escape sequences in Variant
path parsing
271386574a42 is described below
commit 271386574a422140c55942a5a345ede858afb5cc
Author: bojana-db <[email protected]>
AuthorDate: Fri Oct 23 15:25:54 2026 +0000
[SPARK-59218][SQL] Support escape sequences in Variant path parsing
### What changes were proposed in this pull request?
`VariantPathParser` now decodes escape sequences in the single-quoted
`['...']` and double-quoted `["..."]` bracket form — \', \\, the short control
escapes `\b` `\t` `\n` `\f` `\r`, and `\uXXXX`, and escapes of the enclosing
quote (`\'` in `['...']`, `\"` in `["..."]`).
Notes:
- `\/` and valid Unicode surrogate pairs are also decoded. Unknown escapes
and unpaired surrogate escapes remain literal.
- Existing paths may match different keys. `$['a\']` and `$["a\"]` now
error because the backslash escapes the closing quote; double the backslash to
keep it literal.
- Set `spark.sql.legacy.variantPathBackslashAsLiteral=true` before creating
the query to restore the previous behavior.
- Analyzed plans and `EXPLAIN` output include an extra boolean for the
captured parser mode.
### Why are the changes needed?
There was no support before and generated recursive `variant_explode paths`
must round-trip through variant path consumers..
### Does this PR introduce _any_ user-facing change?
Yes. Inside both bracket forms of a variant path, a backslash now
introduces an escape. Examples:
- Previously the content was literal, e.g. `$['a\tb']` addressed the key
`a\tb` (backslash + t).
- Now `$['a\tb']` addresses a+TAB+b; `$['a\'b']` addresses a'b (previously
rejected); `$["a\"b"]` addresses `a"b`; `$['a\\b']` / `$["a\\b"]` address `a\b`.
### How was this patch tested?
Added unit tests.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #59097 from bojana-db/variant-path-parser-escapes.
Authored-by: bojana-db <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../sql/connector/expressions/VariantGet.java | 4 +-
.../sql/connector/read/VariantExtraction.java | 3 +
.../variant/VariantExpressionEvalUtils.scala | 75 +++++---
.../expressions/variant/variantExpressions.scala | 197 ++++++++++++++++-----
.../sql/catalyst/util/V2ExpressionBuilder.scala | 7 +-
.../org/apache/spark/sql/internal/SQLConf.scala | 11 ++
.../variant/VariantExpressionSuite.scala | 192 +++++++++++++++++++-
.../function_try_variant_array_append.explain | 2 +-
.../function_try_variant_get.explain | 2 +-
.../function_try_variant_insert.explain | 2 +-
.../function_try_variant_set.explain | 2 +-
..._try_variant_set_with_create_if_missing.explain | 2 +-
.../function_variant_array_append.explain | 2 +-
.../function_variant_delete.explain | 2 +-
.../explain-results/function_variant_get.explain | 2 +-
.../function_variant_insert.explain | 2 +-
.../explain-results/function_variant_pick.explain | 2 +-
.../explain-results/function_variant_set.explain | 2 +-
...tion_variant_set_with_create_if_missing.explain | 2 +-
.../datasources/PushVariantIntoScan.scala | 23 ++-
.../datasources/parquet/ParquetFilters.scala | 2 +-
.../sql-tests/analyzer-results/listagg.sql.out | 10 +-
.../variant-field-extractions.sql.out | 74 ++++----
.../scala/org/apache/spark/sql/VariantSuite.scala | 18 ++
.../datasources/PushVariantIntoScanSuite.scala | 16 +-
.../datasources/parquet/ParquetFilterSuite.scala | 18 ++
.../datasources/v2/DataSourceV2StrategySuite.scala | 26 +++
28 files changed, 568 insertions(+), 133 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 8642fed0d085..f385cfac3696 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -28,6 +28,7 @@ license: |
## Upgrading from Spark SQL 4.3 to 4.4
+- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
- Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before
Spark 4.2. With columnar input, they kept the columns that they pass through
until the results of their batch came back from Python, and failed or returned
wrong results when the operator reused or released the vectors of the batch in
the meantime. To enab [...]
- Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default
`false`) is a preview feature that may change in future releases. When true, it
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family,
explicit casts can return constrained types, least-common-type resolution for
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string
expressions that transform thei [...]
- Since Spark 4.4, when `spark.sql.charVarchar.standardSemantics.enabled` is
true, `from_json` length-checks JSON object names used as `MAP<CHAR(n), _>` or
`MAP<VARCHAR(n), _>` keys without padding or trimming them. A `CHAR(n)` key
must already be exactly `n` characters, and a `VARCHAR(n)` key must already be
at most `n` characters; duplicate names are kept, exactly as for STRING keys,
and `spark.sql.mapKeyDedupPolicy` is not applied. A key that fails the check
makes the enclosing map a [...]
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/VariantGet.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/VariantGet.java
index b2eb71c89f7b..256156d39c37 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/VariantGet.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/VariantGet.java
@@ -44,7 +44,9 @@ public class VariantGet extends ExpressionWithToString {
/**
* Creates VariantGet expression.
* @param child variant column reference
- * @param path JSON path string
+ * @param path extraction path using Spark's default Variant path grammar.
See
+ * {@link
org.apache.spark.sql.connector.read.VariantExtraction#metadata()} with
+ * {@code legacyPathBackslashAsLiteral=false} for bracket-key
escape rules.
* @param targetType expected result type
* @param failOnError whether to throw on cast failure ({@code variant_get})
or return null
* ({@code try_variant_get})
diff --git
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/VariantExtraction.java
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/VariantExtraction.java
index 64987299934a..d4d20120556c 100644
---
a/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/VariantExtraction.java
+++
b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/VariantExtraction.java
@@ -58,6 +58,9 @@ public interface VariantExtraction extends Serializable {
* - "failOnError": whether the extraction to expected data type should
throw an exception
* or return null if the cast fails.
* - "timeZoneId": a string identifier of a time zone. It is required by
timestamp-related casts.
+ * - "legacyPathBackslashAsLiteral": whether backslashes in bracket-quoted
keys are literal.
+ * If false, decodes \\, \/, \b, \f, \n, \r, \t, Unicode and
enclosing-quote escapes.
+ * Unknown escapes and unpaired surrogate escapes stay literal.
Dot-notation keys are unchanged.
*
*/
Metadata metadata();
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
index c0ce22f3b3a7..7f0f26278f5c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala
@@ -84,8 +84,9 @@ object VariantExpressionEvalUtils {
def parseVariantPath(
pathValue: String,
functionName: String,
+ legacyPathBackslashAsLiteral: Boolean,
allowRoot: Boolean = false): Array[VariantPathSegment] = {
- val parsed = VariantPathParser.parse(pathValue).getOrElse {
+ val parsed = VariantPathParser.parse(pathValue,
legacyPathBackslashAsLiteral).getOrElse {
throw QueryExecutionErrors.invalidVariantPath(pathValue, functionName)
}
if (!allowRoot && parsed.isEmpty) {
@@ -95,7 +96,9 @@ object VariantExpressionEvalUtils {
}
/** Render a parsed path prefix back to a JSONPath string for error
messages. */
- private def renderVariantPath(segments: Array[VariantBuilder.PathSegment]):
String = {
+ private def renderVariantPath(
+ segments: Array[VariantBuilder.PathSegment],
+ legacyPathBackslashAsLiteral: Boolean): String = {
val sb = new StringBuilder("$")
segments.foreach {
case o: VariantBuilder.ObjectKeySegment =>
@@ -104,10 +107,13 @@ object VariantExpressionEvalUtils {
// must use bracket notation so the rendered path round-trips to the
same segments.
if (key.nonEmpty && !key.contains('.') && !key.contains('[')) {
sb.append('.').append(key)
- } else if (!key.contains('\'')) {
- sb.append("['").append(key).append("']")
} else {
- sb.append("[\"").append(key).append("\"]")
+ val quote = if (key.contains('\'')) '"' else '\''
+ if (legacyPathBackslashAsLiteral) {
+ sb.append('[').append(quote).append(key).append(quote).append(']')
+ } else {
+ sb.append(VariantPathParser.quoteKey(key, quote))
+ }
}
case a: VariantBuilder.ArrayIndexSegment =>
sb.append('[').append(a.index).append(']')
}
@@ -130,8 +136,12 @@ object VariantExpressionEvalUtils {
new VariantVal(out.getValue, out.getMetadata)
}
- def deleteAtPath(input: VariantVal, path: UTF8String): VariantVal =
- deleteAtPath(input, toJavaSegments(parseVariantPath(path.toString,
"variant_delete")))
+ def deleteAtPath(
+ input: VariantVal,
+ path: UTF8String,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal =
+ deleteAtPath(input, toJavaSegments(
+ parseVariantPath(path.toString, "variant_delete",
legacyPathBackslashAsLiteral)))
/**
* Insert `value` into `input` at `javaSegments`. `path` is the source
string used in error
@@ -148,7 +158,8 @@ object VariantExpressionEvalUtils {
value: Any,
valueDataType: DataType,
functionName: String,
- failOnError: Boolean): VariantVal = {
+ failOnError: Boolean,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal = {
val v = new Variant(input.getValue, input.getMetadata)
try {
val valVal = castToVariant(value, valueDataType)
@@ -161,7 +172,8 @@ object VariantExpressionEvalUtils {
null
case e: VariantPathTypeMismatchException =>
throw QueryExecutionErrors.variantPathTypeMismatch(
- path, renderVariantPath(javaSegments.take(e.depth)), functionName)
+ path, renderVariantPath(javaSegments.take(e.depth),
legacyPathBackslashAsLiteral),
+ functionName)
case _: VariantSizeLimitException =>
throw
QueryExecutionErrors.variantSizeLimitError(VariantUtil.SIZE_LIMIT, functionName)
}
@@ -173,10 +185,13 @@ object VariantExpressionEvalUtils {
value: Any,
valueDataType: DataType,
functionName: String,
- failOnError: Boolean): VariantVal = {
+ failOnError: Boolean,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal = {
val pathStr = path.toString
- val javaSegments = toJavaSegments(parseVariantPath(pathStr, functionName))
- insertAtPath(input, javaSegments, pathStr, value, valueDataType,
functionName, failOnError)
+ val javaSegments = toJavaSegments(
+ parseVariantPath(pathStr, functionName, legacyPathBackslashAsLiteral))
+ insertAtPath(input, javaSegments, pathStr, value, valueDataType,
functionName, failOnError,
+ legacyPathBackslashAsLiteral)
}
/**
@@ -195,7 +210,8 @@ object VariantExpressionEvalUtils {
valueDataType: DataType,
createIfMissing: Boolean,
functionName: String,
- failOnError: Boolean): VariantVal = {
+ failOnError: Boolean,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal = {
val v = new Variant(input.getValue, input.getMetadata)
try {
val valVal = castToVariant(value, valueDataType)
@@ -206,7 +222,8 @@ object VariantExpressionEvalUtils {
case _: VariantPathTypeMismatchException if !failOnError => null
case e: VariantPathTypeMismatchException =>
throw QueryExecutionErrors.variantPathTypeMismatch(
- path, renderVariantPath(javaSegments.take(e.depth)), functionName)
+ path, renderVariantPath(javaSegments.take(e.depth),
legacyPathBackslashAsLiteral),
+ functionName)
case _: VariantSizeLimitException =>
throw
QueryExecutionErrors.variantSizeLimitError(VariantUtil.SIZE_LIMIT, functionName)
}
@@ -219,11 +236,13 @@ object VariantExpressionEvalUtils {
valueDataType: DataType,
createIfMissing: Boolean,
functionName: String,
- failOnError: Boolean): VariantVal = {
+ failOnError: Boolean,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal = {
val pathStr = path.toString
- val javaSegments = toJavaSegments(parseVariantPath(pathStr, functionName))
+ val javaSegments = toJavaSegments(
+ parseVariantPath(pathStr, functionName, legacyPathBackslashAsLiteral))
setAtPath(input, javaSegments, pathStr, value, valueDataType,
createIfMissing, functionName,
- failOnError)
+ failOnError, legacyPathBackslashAsLiteral)
}
/**
@@ -241,7 +260,8 @@ object VariantExpressionEvalUtils {
value: Any,
valueDataType: DataType,
functionName: String,
- failOnError: Boolean): VariantVal = {
+ failOnError: Boolean,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal = {
val v = new Variant(input.getValue, input.getMetadata)
try {
val valVal = castToVariant(value, valueDataType)
@@ -252,7 +272,8 @@ object VariantExpressionEvalUtils {
case _: VariantPathTypeMismatchException if !failOnError => null
case e: VariantPathTypeMismatchException =>
throw QueryExecutionErrors.variantPathTypeMismatch(
- path, renderVariantPath(javaSegments.take(e.depth)), functionName)
+ path, renderVariantPath(javaSegments.take(e.depth),
legacyPathBackslashAsLiteral),
+ functionName)
case _: VariantSizeLimitException =>
throw
QueryExecutionErrors.variantSizeLimitError(VariantUtil.SIZE_LIMIT, functionName)
}
@@ -264,10 +285,13 @@ object VariantExpressionEvalUtils {
value: Any,
valueDataType: DataType,
functionName: String,
- failOnError: Boolean): VariantVal = {
+ failOnError: Boolean,
+ legacyPathBackslashAsLiteral: Boolean): VariantVal = {
val pathStr = path.toString
- val javaSegments = toJavaSegments(parseVariantPath(pathStr, functionName,
allowRoot = true))
- arrayAppendAtPath(input, javaSegments, pathStr, value, valueDataType,
functionName, failOnError)
+ val javaSegments = toJavaSegments(
+ parseVariantPath(pathStr, functionName, legacyPathBackslashAsLiteral,
allowRoot = true))
+ arrayAppendAtPath(input, javaSegments, pathStr, value, valueDataType,
functionName, failOnError,
+ legacyPathBackslashAsLiteral)
}
def stripNulls(input: VariantVal, includeArrays: Boolean): VariantVal = {
@@ -276,8 +300,11 @@ object VariantExpressionEvalUtils {
new VariantVal(out.getValue, out.getMetadata)
}
- def parsePickPath(path: UTF8String): Array[VariantBuilder.PathSegment] =
- toJavaSegments(parseVariantPath(path.toString, "variant_pick", allowRoot =
true))
+ def parsePickPath(
+ path: UTF8String,
+ legacyPathBackslashAsLiteral: Boolean):
Array[VariantBuilder.PathSegment] =
+ toJavaSegments(parseVariantPath(
+ path.toString, "variant_pick", legacyPathBackslashAsLiteral, allowRoot =
true))
def pickAtPaths(
input: VariantVal,
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
index da0c45a981c8..f4aeacc4ddb7 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.catalyst.expressions.variant
import java.time.ZoneId
+import scala.util.matching.Regex
import scala.util.parsing.combinator.RegexParsers
import org.apache.spark.SparkRuntimeException
@@ -369,7 +370,7 @@ object VariantPathParser extends RegexParsers {
override def skipWhitespace: Boolean = false
// Parse key segment like `.name`, `['name']`, or `["name"]`.
- private def key: Parser[VariantPathSegment] =
+ private def literalBackslashKey: Parser[VariantPathSegment] =
for {
key <- '.' ~> "[^\\.\\[]+".r | "['" ~> "[^']*".r <~ "']" |
"[\"" ~> """[^"]*""".r <~ "\"]"
@@ -377,9 +378,79 @@ object VariantPathParser extends RegexParsers {
ObjectExtraction(key)
}
- private val parser: Parser[List[VariantPathSegment]] = phrase(root ~>
rep(key | index))
+ // Only bracket keys decode escapes.
+ private val escapedKey: Parser[VariantPathSegment] =
+ '.' ~> "[^\\.\\[]+".r ^^ ObjectExtraction |
+ "['" ~> """[^'\\]*+(?:\\[\s\S][^'\\]*+)*+""".r <~ "']" ^^ { k =>
+ ObjectExtraction(unescapeQuotedKey(k, '\''))
+ } |
+ "[\"" ~> """[^"\\]*+(?:\\[\s\S][^"\\]*+)*+""".r <~ "\"]" ^^ { k =>
+ ObjectExtraction(unescapeQuotedKey(k, '"'))
+ }
+
+ // Match a surrogate pair, `\uXXXX`, or a backslash followed by one
character.
+ private val escapeSeq =
+ ("""\\(u[dD][89abAB][0-9a-fA-F]{2}\\u[dD][c-fC-F][0-9a-fA-F]{2}|""" +
+ """u[0-9a-fA-F]{4}|[\s\S])""").r
+
+ // Unrecognized escapes remain literal.
+ private def unescapeQuotedKey(raw: String, quote: Char): String =
+ if (raw.indexOf('\\') < 0) raw
+ else escapeSeq.replaceAllIn(raw, m =>
+ Regex.quoteReplacement(decodeEscape(m.group(1), quote)))
+
+ private def decodeEscape(esc: String, quote: Char): String = esc match {
+ case "b" => "\b"
+ case "f" => "\f"
+ case "n" => "\n"
+ case "r" => "\r"
+ case "t" => "\t"
+ case "/" => "/"
+ case "\\" => "\\"
+ case "'" if quote == '\'' => "'"
+ case "\"" if quote == '"' => "\""
+ case escape if escape.length == 11 =>
+ val high = Integer.parseInt(escape.substring(1, 5), 16).toChar
+ val low = Integer.parseInt(escape.substring(7), 16).toChar
+ s"$high$low"
+ case escape if escape.length == 5 =>
+ val decoded = Integer.parseInt(escape.substring(1), 16).toChar
+ if (Character.isSurrogate(decoded)) "\\" + escape else decoded.toString
+ case other => "\\" + other
+ }
+
+ // Escape and quote a key using bracket notation.
+ private[variant] def quoteKey(key: String, quote: Char): String = {
+ val result = new java.lang.StringBuilder(key.length +
4).append('[').append(quote)
+ var i = 0
+ while (i < key.length) {
+ key.charAt(i) match {
+ case '\\' => result.append("\\\\")
+ case c if c == quote => result.append('\\').append(c)
+ case '\b' => result.append("\\b")
+ case '\f' => result.append("\\f")
+ case '\n' => result.append("\\n")
+ case '\r' => result.append("\\r")
+ case '\t' => result.append("\\t")
+ case c if c < 0x20 => result.append("\\u%04x".format(c.toInt))
+ case c => result.append(c)
+ }
+ i += 1
+ }
+ result.append(quote).append(']').toString
+ }
- def parse(str: String): Option[Array[VariantPathSegment]] = {
+ private val escapedParser = phrase(root ~> rep(escapedKey | index))
+ private val literalBackslashParser = phrase(root ~> rep(literalBackslashKey
| index))
+
+ def parse(
+ str: String,
+ legacyPathBackslashAsLiteral: Boolean):
Option[Array[VariantPathSegment]] = {
+ val parser = if (legacyPathBackslashAsLiteral) {
+ literalBackslashParser
+ } else {
+ escapedParser
+ }
try {
this.parseAll(parser, str) match {
case Success(result, _) => Some(result.toArray)
@@ -408,6 +479,7 @@ object VariantPathArg {
private def parseFoldableExpression(
child: Expression,
functionName: String,
+ legacyPathBackslashAsLiteral: Boolean,
allowRoot: Boolean): Option[FoldableVariantPath] = {
val value = child.eval()
if (value == null) {
@@ -415,16 +487,18 @@ object VariantPathArg {
} else {
val pathStr = value.asInstanceOf[UTF8String].toString
Some(FoldableVariantPath(
- VariantExpressionEvalUtils.parseVariantPath(pathStr, functionName,
allowRoot), pathStr))
+ VariantExpressionEvalUtils.parseVariantPath(
+ pathStr, functionName, legacyPathBackslashAsLiteral, allowRoot),
pathStr))
}
}
def fromExpression(
child: Expression,
functionName: String,
+ legacyPathBackslashAsLiteral: Boolean,
allowRoot: Boolean = false): Option[VariantPathArg] = {
if (child.foldable) {
- parseFoldableExpression(child, functionName, allowRoot)
+ parseFoldableExpression(child, functionName,
legacyPathBackslashAsLiteral, allowRoot)
} else {
Some(DynamicVariantPath(child))
}
@@ -433,9 +507,10 @@ object VariantPathArg {
def foldableFromExpression(
child: Expression,
functionName: String,
+ legacyPathBackslashAsLiteral: Boolean,
allowRoot: Boolean = false): Option[FoldableVariantPath] = {
if (child.foldable) {
- parseFoldableExpression(child, functionName, allowRoot)
+ parseFoldableExpression(child, functionName,
legacyPathBackslashAsLiteral, allowRoot)
} else {
None
}
@@ -448,8 +523,10 @@ object VariantManipulationExpressionUtils extends
QueryErrorsBase {
def foldablePath(
path: Expression,
functionName: String,
+ legacyPathBackslashAsLiteral: Boolean,
allowRoot: Boolean = false): Option[FoldableVariantPath] =
- VariantPathArg.foldableFromExpression(path, functionName, allowRoot)
+ VariantPathArg.foldableFromExpression(
+ path, functionName, legacyPathBackslashAsLiteral, allowRoot)
def checkValueType(dataType: DataType): TypeCheckResult = {
if (dataType == NullType || VariantGet.checkDataType(dataType,
allowStructsAndMaps = false)) {
@@ -489,7 +566,9 @@ case class VariantGet(
path: Expression,
targetType: DataType,
failOnError: Boolean,
- timeZoneId: Option[String] = None)
+ timeZoneId: Option[String] = None,
+ legacyPathBackslashAsLiteral: Boolean =
+ SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
extends BinaryExpression
with TimeZoneAwareExpression
with ExpectsInputTypes
@@ -513,7 +592,7 @@ case class VariantGet(
@transient private lazy val parsedPath: Option[Array[VariantPathSegment]] = {
if (path.foldable) {
val pathValue = path.eval().toString
- Some(VariantGet.getParsedPath(pathValue, prettyName))
+ Some(VariantGet.getParsedPath(pathValue, prettyName,
legacyPathBackslashAsLiteral))
} else {
None
}
@@ -544,7 +623,7 @@ case class VariantGet(
VariantGet.variantGet(input.asInstanceOf[VariantVal], pp, dataType,
castArgs)
case _ =>
VariantGet.variantGet(input.asInstanceOf[VariantVal],
path.asInstanceOf[UTF8String], dataType,
- castArgs, prettyName)
+ castArgs, prettyName, legacyPathBackslashAsLiteral)
}
protected override def doGenCode(ctx: CodegenContext, ev: ExprCode):
ExprCode = {
@@ -569,7 +648,9 @@ case class VariantGet(
if (!${ev.isNull}) {
Object $tmp =
org.apache.spark.sql.catalyst.expressions.variant.VariantGet.variantGet(
${childCode.value}, $parsedPathArg, $dataTypeArg,
- $castArgsArg${if (parsedPath.isEmpty) s""", "$prettyName"""" else
""});
+ $castArgsArg${if (parsedPath.isEmpty) {
+ s""", "$prettyName", $legacyPathBackslashAsLiteral"""
+ } else ""});
if ($tmp == null) {
${ev.isNull} = true;
} else {
@@ -620,8 +701,11 @@ case object VariantGet {
/**
* Get parsed Array[VariantPathSegment] from string representing path
*/
- def getParsedPath(pathValue: String, prettyName: String):
Array[VariantPathSegment] = {
- VariantPathParser.parse(pathValue).getOrElse {
+ def getParsedPath(
+ pathValue: String,
+ prettyName: String,
+ legacyPathBackslashAsLiteral: Boolean): Array[VariantPathSegment] = {
+ VariantPathParser.parse(pathValue, legacyPathBackslashAsLiteral).getOrElse
{
throw QueryExecutionErrors.invalidVariantGetPath(pathValue, prettyName)
}
}
@@ -652,9 +736,10 @@ case object VariantGet {
path: UTF8String,
dataType: DataType,
castArgs: VariantCastArgs,
- prettyName: String): Any = {
+ prettyName: String,
+ legacyPathBackslashAsLiteral: Boolean): Any = {
val pathValue = path.toString
- val parsedPath = VariantGet.getParsedPath(pathValue, prettyName)
+ val parsedPath = VariantGet.getParsedPath(pathValue, prettyName,
legacyPathBackslashAsLiteral)
variantGet(input, parsedPath, dataType, castArgs)
}
@@ -969,10 +1054,16 @@ object TryVariantGetExpressionBuilder extends
VariantGetExpressionBuilderBase(fa
group = "variant_funcs"
)
// scalastyle:on line.size.limit
-case class VariantDelete(children: Seq[Expression])
+case class VariantDelete(
+ children: Seq[Expression],
+ legacyPathBackslashAsLiteral: Boolean =
+ SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
extends Expression
with ExpectsInputTypes {
+ def this(children: Seq[Expression]) =
+ this(children,
SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
+
override def dataType: DataType = VariantType
override def nullable: Boolean = children.headOption.forall(_.nullable)
@@ -996,7 +1087,8 @@ case class VariantDelete(children: Seq[Expression])
private def pathChildren: Seq[Expression] = children.tail
@transient private lazy val pathArgs: Seq[VariantPathArg] =
- pathChildren.flatMap(VariantPathArg.fromExpression(_, prettyName))
+ pathChildren.flatMap(
+ VariantPathArg.fromExpression(_, prettyName,
legacyPathBackslashAsLiteral))
override def eval(input: InternalRow): Any = {
val inputVariant = variantChild.eval(input).asInstanceOf[VariantVal]
@@ -1011,7 +1103,8 @@ case class VariantDelete(children: Seq[Expression])
case DynamicVariantPath(expr) =>
val pathVal = expr.eval(input).asInstanceOf[UTF8String]
if (pathVal != null) {
- current = VariantExpressionEvalUtils.deleteAtPath(current, pathVal)
+ current = VariantExpressionEvalUtils.deleteAtPath(
+ current, pathVal, legacyPathBackslashAsLiteral)
}
}
i += 1
@@ -1034,7 +1127,8 @@ case class VariantDelete(children: Seq[Expression])
s"""
|${pCode.code}
|if (!${pCode.isNull}) {
- | $current = $cls.deleteAtPath($current, ${pCode.value});
+ | $current = $cls.deleteAtPath(
+ | $current, ${pCode.value}, $legacyPathBackslashAsLiteral);
|}
""".stripMargin
}.mkString("\n")
@@ -1093,10 +1187,16 @@ case class VariantDelete(children: Seq[Expression])
group = "variant_funcs"
)
// scalastyle:on line.size.limit
-case class VariantPick(children: Seq[Expression])
+case class VariantPick(
+ children: Seq[Expression],
+ legacyPathBackslashAsLiteral: Boolean =
+ SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
extends Expression
with ExpectsInputTypes {
+ def this(children: Seq[Expression]) =
+ this(children,
SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
+
override def dataType: DataType = VariantType
override def nullable: Boolean = children.headOption.forall(_.nullable)
@@ -1119,7 +1219,8 @@ case class VariantPick(children: Seq[Expression])
private def pathChildren: Seq[Expression] = children.tail
@transient private lazy val pathArgs: Seq[VariantPathArg] =
- pathChildren.flatMap(VariantPathArg.fromExpression(_, prettyName,
allowRoot = true))
+ pathChildren.flatMap(VariantPathArg.fromExpression(
+ _, prettyName, legacyPathBackslashAsLiteral, allowRoot = true))
// When every path is constant, the keep-tree is the same for all rows, so
build it once here and
// reuse it rather than rebuilding it per row. `None` means at least one
path is dynamic.
@@ -1148,7 +1249,8 @@ case class VariantPick(children: Seq[Expression])
case DynamicVariantPath(expr) =>
val pathVal = expr.eval(input).asInstanceOf[UTF8String]
if (pathVal != null) {
- paths.add(VariantExpressionEvalUtils.parsePickPath(pathVal))
+ paths.add(VariantExpressionEvalUtils.parsePickPath(
+ pathVal, legacyPathBackslashAsLiteral))
}
}
i += 1
@@ -1191,7 +1293,7 @@ case class VariantPick(children: Seq[Expression])
s"""
|${pCode.code}
|if (!${pCode.isNull}) {
- | $paths.add($cls.parsePickPath(${pCode.value}));
+ | $paths.add($cls.parsePickPath(${pCode.value},
$legacyPathBackslashAsLiteral));
|}
""".stripMargin
}.mkString("\n")
@@ -1223,7 +1325,9 @@ case class VariantInsert(
input: Expression,
path: Expression,
value: Expression,
- failOnError: Boolean = true)
+ failOnError: Boolean = true,
+ legacyPathBackslashAsLiteral: Boolean =
+ SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
extends TernaryExpression
with ExpectsInputTypes {
@@ -1248,7 +1352,7 @@ case class VariantInsert(
}
@transient private lazy val foldablePath: Option[FoldableVariantPath] =
- VariantManipulationExpressionUtils.foldablePath(path, prettyName)
+ VariantManipulationExpressionUtils.foldablePath(path, prettyName,
legacyPathBackslashAsLiteral)
override def eval(input: InternalRow): Any = {
val _ = foldablePath
@@ -1261,11 +1365,11 @@ case class VariantInsert(
case Some(parsed) =>
VariantExpressionEvalUtils.insertAtPath(
inputVariant, parsed.javaSegments, parsed.pathStr, valValue,
value.dataType, prettyName,
- failOnError)
+ failOnError, legacyPathBackslashAsLiteral)
case None =>
VariantExpressionEvalUtils.insertAtPath(
inputVariant, p.asInstanceOf[UTF8String], valValue, value.dataType,
prettyName,
- failOnError)
+ failOnError, legacyPathBackslashAsLiteral)
}
}
@@ -1277,9 +1381,11 @@ case class VariantInsert(
val parsedArg = ctx.addReferenceObj("insertPath", parsed)
s"""$cls.insertAtPath(
| $vVal, $parsedArg.javaSegments(), $parsedArg.pathStr(), $valVal,
$fromArg,
- | "$prettyName", $failOnError)""".stripMargin
+ | "$prettyName", $failOnError,
$legacyPathBackslashAsLiteral)""".stripMargin
case None =>
- s"""$cls.insertAtPath($vVal, $pVal, $valVal, $fromArg, "$prettyName",
$failOnError)"""
+ s"""$cls.insertAtPath(
+ | $vVal, $pVal, $valVal, $fromArg, "$prettyName", $failOnError,
+ | $legacyPathBackslashAsLiteral)""".stripMargin
}
nullSafeCodeGen(ctx, ev, (vVal, pVal, valVal) =>
VariantManipulationExpressionUtils.assignVariantResult(
@@ -1385,7 +1491,9 @@ case class VariantSet(
path: Expression,
value: Expression,
createIfMissing: Expression,
- failOnError: Boolean = true)
+ failOnError: Boolean = true,
+ legacyPathBackslashAsLiteral: Boolean =
+ SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
extends QuaternaryExpression
with ExpectsInputTypes
with QueryErrorsBase {
@@ -1423,7 +1531,7 @@ case class VariantSet(
}
@transient private lazy val foldablePath: Option[FoldableVariantPath] =
- VariantManipulationExpressionUtils.foldablePath(path, prettyName)
+ VariantManipulationExpressionUtils.foldablePath(path, prettyName,
legacyPathBackslashAsLiteral)
override def eval(input: InternalRow): Any = {
val _ = foldablePath
@@ -1437,11 +1545,11 @@ case class VariantSet(
case Some(parsed) =>
VariantExpressionEvalUtils.setAtPath(
inputVariant, parsed.javaSegments, parsed.pathStr, valValue,
value.dataType,
- createIfMissingValue, prettyName, failOnError)
+ createIfMissingValue, prettyName, failOnError,
legacyPathBackslashAsLiteral)
case None =>
VariantExpressionEvalUtils.setAtPath(
inputVariant, p.asInstanceOf[UTF8String], valValue, value.dataType,
- createIfMissingValue, prettyName, failOnError)
+ createIfMissingValue, prettyName, failOnError,
legacyPathBackslashAsLiteral)
}
}
@@ -1454,10 +1562,12 @@ case class VariantSet(
val parsedArg = ctx.addReferenceObj("setPath", parsed)
s"""$cls.setAtPath(
| $vVal, $parsedArg.javaSegments(), $parsedArg.pathStr(),
$valVal, $fromArg,
- | $createVal, "$prettyName", $failOnError)""".stripMargin
+ | $createVal, "$prettyName", $failOnError,
+ | $legacyPathBackslashAsLiteral)""".stripMargin
case None =>
s"""$cls.setAtPath(
- | $vVal, $pVal, $valVal, $fromArg, $createVal, "$prettyName",
$failOnError)"""
+ | $vVal, $pVal, $valVal, $fromArg, $createVal, "$prettyName",
$failOnError,
+ | $legacyPathBackslashAsLiteral)"""
.stripMargin
}
nullSafeCodeGen(ctx, ev, (vVal, pVal, valVal, createVal) =>
@@ -1578,7 +1688,9 @@ case class VariantArrayAppend(
input: Expression,
path: Expression,
value: Expression,
- failOnError: Boolean = true)
+ failOnError: Boolean = true,
+ legacyPathBackslashAsLiteral: Boolean =
+ SQLConf.get.getConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL))
extends TernaryExpression
with ExpectsInputTypes {
@@ -1603,7 +1715,8 @@ case class VariantArrayAppend(
}
@transient private lazy val foldablePath: Option[FoldableVariantPath] =
- VariantManipulationExpressionUtils.foldablePath(path, prettyName,
allowRoot = true)
+ VariantManipulationExpressionUtils.foldablePath(
+ path, prettyName, legacyPathBackslashAsLiteral, allowRoot = true)
override def eval(input: InternalRow): Any = {
val _ = foldablePath
@@ -1616,11 +1729,11 @@ case class VariantArrayAppend(
case Some(parsed) =>
VariantExpressionEvalUtils.arrayAppendAtPath(
inputVariant, parsed.javaSegments, parsed.pathStr, valValue,
value.dataType, prettyName,
- failOnError)
+ failOnError, legacyPathBackslashAsLiteral)
case None =>
VariantExpressionEvalUtils.arrayAppendAtPath(
inputVariant, p.asInstanceOf[UTF8String], valValue, value.dataType,
prettyName,
- failOnError)
+ failOnError, legacyPathBackslashAsLiteral)
}
}
@@ -1632,9 +1745,11 @@ case class VariantArrayAppend(
val parsedArg = ctx.addReferenceObj("appendPath", parsed)
s"""$cls.arrayAppendAtPath(
| $vVal, $parsedArg.javaSegments(), $parsedArg.pathStr(), $valVal,
$fromArg,
- | "$prettyName", $failOnError)""".stripMargin
+ | "$prettyName", $failOnError,
$legacyPathBackslashAsLiteral)""".stripMargin
case None =>
- s"""$cls.arrayAppendAtPath($vVal, $pVal, $valVal, $fromArg,
"$prettyName", $failOnError)"""
+ s"""$cls.arrayAppendAtPath(
+ | $vVal, $pVal, $valVal, $fromArg, "$prettyName", $failOnError,
+ | $legacyPathBackslashAsLiteral)""".stripMargin
}
nullSafeCodeGen(ctx, ev, (vVal, pVal, valVal) =>
VariantManipulationExpressionUtils.assignVariantResult(
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
index 1d61cabd0afb..4fa4a327c792 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
@@ -22,7 +22,7 @@ import org.apache.spark.internal.LogKeys.EXPR
import org.apache.spark.sql.catalyst.expressions._
import
org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression,
AggregateFunction, Complete}
import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
-import org.apache.spark.sql.catalyst.expressions.variant.VariantGet
+import org.apache.spark.sql.catalyst.expressions.variant.{VariantGet,
VariantPathParser}
import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
import org.apache.spark.sql.connector.expressions.{Cast => V2Cast, Expression
=> V2Expression, Extract => V2Extract, FieldReference, GeneralScalarExpression,
GetArrayItem => V2GetArrayItem, LiteralValue, NullOrdering, SortDirection,
SortValue, UserDefinedScalarFunc, VariantGet => V2VariantGet}
@@ -345,7 +345,10 @@ class V2ExpressionBuilder(e: Expression, isPredicate:
Boolean = false) extends L
}
case v: VariantGet if v.path.foldable =>
(Option(v.path.eval()).map(_.toString), generateExpression(v.child))
match {
- case (Some(path), Some(colRef: FieldReference)) =>
+ case (Some(path), Some(colRef: FieldReference))
+ if !v.legacyPathBackslashAsLiteral ||
+ VariantPathParser.parse(path, legacyPathBackslashAsLiteral =
true).map(_.toSeq) ==
+ VariantPathParser.parse(path, legacyPathBackslashAsLiteral =
false).map(_.toSeq) =>
val vg = new V2VariantGet(colRef, path, v.targetType, v.failOnError,
v.timeZoneId.orNull)
if (isPredicate && v.dataType.isInstanceOf[BooleanType]) {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 570657bec19e..417ee14831e3 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -7492,6 +7492,17 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL =
+ buildConf("spark.sql.legacy.variantPathBackslashAsLiteral")
+ .internal()
+ .doc("When true, backslashes in quoted Variant path keys are treated as
literal " +
+ "characters, restoring the behavior before Spark 4.4. When false,
quoted keys decode " +
+ "recognized JSONPath escape sequences.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(false)
+
val VARIANT_ALLOW_READING_SHREDDED =
buildConf("spark.sql.variant.allowReadingShredded")
.internal()
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
index 634100197987..c298278106d8 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionSuite.scala
@@ -737,6 +737,51 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
// scalastyle:on nonascii
testVariantGet("[1, 2, 3]", "$[2147483647]", IntegerType, null)
+ testVariantGet("""{"a'b": 1}""", """$['a\'b']""", IntegerType, 1)
+ testVariantGet("""{"a']b": 1}""", """$['a\']b']""", IntegerType, 1)
+ testVariantGet("""{"a\nb":1,"a\\nb":2}""", """$.a\nb""", IntegerType, 2)
+ testVariantGet("""{"a\"b": 2}""", """$['a"b']""", IntegerType, 2)
+ testVariantGet("""{"a\\b": 4}""", """$['a\\b']""", IntegerType, 4)
+ testVariantGet("""{"a\tb": 5}""", """$['a\tb']""", IntegerType, 5)
+ testVariantGet("""{"a\nb": 6}""", """$['a\nb']""", IntegerType, 6)
+ testVariantGet("""{"a\bb": 11}""", """$['a\bb']""", IntegerType, 11)
+ testVariantGet("""{"a\fb": 12}""", """$['a\fb']""", IntegerType, 12)
+ testVariantGet("""{"a\rb": 13}""", """$['a\rb']""", IntegerType, 13)
+ testVariantGet("""{"a/b": 14}""", """$['a\/b']""", IntegerType, 14)
+ // Split the Unicode escape so Scala does not decode it.
+ testVariantGet("{\"a\\" + "u000bb\": 7}", "$['a\\" + "u000bb']",
IntegerType, 7)
+ testVariantGet("""{"a\"b": 8}""", """$["a\"b"]""", IntegerType, 8)
+ testVariantGet("""{"a'b": 9}""", """$["a'b"]""", IntegerType, 9)
+ testVariantGet("""{"a\\xb": 15}""", """$['a\xb']""", IntegerType, 15)
+ testVariantGet("""{"a\\u12G4b": 17}""", "$['a\\" + "u12G4b']",
IntegerType, 17)
+ testVariantGet("""{"a\\\"b": 18}""", """$['a\"b']""", IntegerType, 18)
+ testVariantGet("""{"a\\'b": 19}""", """$["a\'b"]""", IntegerType, 19)
+ testVariantGet(
+ "{\"a\\" + "uD83D\\" + "uDE00b\": 20}",
+ "$['a\\" + "uD83D\\" + "uDE00b']",
+ IntegerType,
+ 20)
+ testVariantGet(
+ "{\"\\\\" + "uD800\\" + "uD83D\\" + "uDE00\":21}",
+ "$['\\" + "uD800\\" + "uD83D\\" + "uDE00']",
+ IntegerType,
+ 21)
+ testVariantGet(
+ "{\"a\\" + "uD83D\\" + "uDE00b\": 22}",
+ "$['a\\" + "ud83d\\" + "ude00b']",
+ IntegerType,
+ 22)
+ testVariantGet(
+ "{\"a\\" + "uDBFF\\" + "uDFFFb\": 23}",
+ "$['a\\" + "uDBFF\\" + "uDFFFb']",
+ IntegerType,
+ 23)
+ Seq("uD800", "uDC00").foreach { escape =>
+ val path = "$['\\" + escape + "']"
+ testVariantGet("""{"?":1}""", path, IntegerType, null)
+ testVariantGet("{\"?\":1,\"\\\\" + escape + "\":2}", path, IntegerType,
2)
+ }
+
Seq("variant_get" -> true, "try_variant_get" -> false).foreach {
case (name, failOnError) =>
checkErrorInExpression[SparkRuntimeException](
@@ -756,7 +801,78 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkInvalidPath("""$['"]""")
checkInvalidPath("$[\"\"\"]")
- checkInvalidPath("$[\"\\\"\"]")
+ checkInvalidPath("""$['a\']""")
+ checkInvalidPath("""$["a\"]""")
+ }
+
+ test("legacy Variant paths treat backslashes literally") {
+ val input = """{"a\nb":1,"a\\nb":2}"""
+ val path = "$['a\\nb']"
+
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
"false") {
+ testVariantGet(input, path, IntegerType, 1)
+ }
+
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
"true") {
+ testVariantGet(input, path, IntegerType, 2)
+
+ val escapedQuote = """$['a\'b']"""
+ checkErrorInExpression[SparkRuntimeException](
+ variantGet("""{"a'b":1}""", escapedQuote, IntegerType),
+ "INVALID_VARIANT_GET_PATH",
+ Map("path" -> escapedQuote, "functionName" -> "`variant_get`"))
+ }
+ }
+
+ gridTest("Variant path functions use the captured parser mode")(Seq(false,
true)) { legacy =>
+ val conf = SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key
+ val input = Literal(parseJson("""{"a\nb":[1],"a\\nb":[2]}"""))
+ val path = "$['a\\nb']"
+ withSQLConf(conf -> legacy.toString) {
+ Seq(Literal(path), BoundReference(0, StringType, nullable =
false)).foreach { pathExpr =>
+ val get = VariantGet(input, pathExpr, ArrayType(IntegerType),
failOnError = true)
+ val cases = Seq(
+ VariantDelete(Seq(input, pathExpr)) ->
+ (if (legacy) """{"a\nb":[1]}""" else """{"a\\nb":[2]}"""),
+ VariantPick(Seq(input, pathExpr)) ->
+ (if (legacy) """{"a\\nb":[2]}""" else """{"a\nb":[1]}"""),
+ VariantInsert(input, Concat(Seq(pathExpr, Literal("[0]"))),
Literal(3)) ->
+ (if (legacy) """{"a\nb":[1],"a\\nb":[3,2]}""" else
"""{"a\nb":[3,1],"a\\nb":[2]}"""),
+ VariantSet(input, pathExpr, Literal(3), Literal(true)) ->
+ (if (legacy) """{"a\nb":[1],"a\\nb":3}""" else
"""{"a\nb":3,"a\\nb":[2]}"""),
+ VariantArrayAppend(input, pathExpr, Literal(3)) ->
+ (if (legacy) """{"a\nb":[1],"a\\nb":[2,3]}""" else
"""{"a\nb":[1,3],"a\\nb":[2]}"""))
+ withSQLConf(conf -> (!legacy).toString) {
+ val row = InternalRow(UTF8String.fromString(path))
+ checkEvaluation(get, Array(if (legacy) 2 else 1), row)
+ cases.foreach { case (expr, expected) =>
+ checkEvaluation(
+ ResolveTimeZone.resolveTimeZones(Cast(expr, StringType)),
expected, row)
+ }
+ }
+ }
+ }
+ }
+
+ gridTest("Variant diagnostics use the captured parser mode")(Seq(false,
true)) { legacy =>
+ val conf = SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key
+ val path = """$['a.\nb'].c"""
+ val input = Literal(parseJson("""{"a.\nb":5,"a.\\nb":6}"""))
+ val expressions = withSQLConf(conf -> legacy.toString) {
+ Seq(
+ VariantInsert(input, Literal(path), Literal(2)),
+ VariantSet(input, Literal(path), Literal(2), Literal(true)),
+ VariantArrayAppend(input, Literal(path), Literal(2)))
+ }
+ withSQLConf(conf -> (!legacy).toString) {
+ expressions.foreach { expression =>
+ checkErrorInExpression[SparkRuntimeException](
+ expression,
+ "VARIANT_PATH_TYPE_MISMATCH",
+ Map("path" -> path, "failedAt" -> """$['a.\nb']""",
+ "functionName" -> s"`${expression.prettyName}`"))
+ }
+ }
}
test("SPARK-58672: validate char/varchar target types in variant_get") {
@@ -1487,6 +1603,18 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
"INVALID_VARIANT_PATH",
Map("path" -> ".a", "functionName" -> "`variant_delete`"))
+ checkErrorInExpression[SparkRuntimeException](
+ ResolveTimeZone.resolveTimeZones(
+ VariantDelete(Seq(Literal(parseJson("""{"a": 1}""")),
Literal("""$['a\']""")))),
+ "INVALID_VARIANT_PATH",
+ Map("path" -> """$['a\']""", "functionName" -> "`variant_delete`"))
+
+ checkErrorInExpression[SparkRuntimeException](
+ ResolveTimeZone.resolveTimeZones(
+ VariantDelete(Seq(Literal(parseJson("""{"a": 1}""")),
Literal("""$["a\"]""")))),
+ "INVALID_VARIANT_PATH",
+ Map("path" -> """$["a\"]""", "functionName" -> "`variant_delete`"))
+
for (path <- Seq("$[2147483648]", "$[4294967296]")) {
checkErrorInExpression[SparkRuntimeException](
ResolveTimeZone.resolveTimeZones(
@@ -1580,6 +1708,11 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
// Special / edge-case keys.
checkInsert("""{"a": 1}""", "$['']", Literal(2), """{"":2,"a":1}""")
checkInsert("""{"a": 1}""", "$['?']", Literal(2), """{"?":2,"a":1}""")
+ Seq("uD800", "uDC00").foreach { escape =>
+ checkInsert(
+ """{"?":1}""", "$['\\" + escape + "']", Literal(2),
+ "{\"?\":1,\"\\\\" + escape + "\":2}")
+ }
checkInsert(
"""{"a": 1}""", "$['key with spaces']", Literal(2), """{"a":1,"key with
spaces":2}""")
checkInsert("""{"a": 1}""", "$.fb:testid", Literal(2),
"""{"a":1,"fb:testid":2}""")
@@ -1640,6 +1773,32 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkInsertRecoverableError("""{"a.b": 5}""", """$['a.b'].c""", Literal(2),
"VARIANT_PATH_TYPE_MISMATCH",
Map("path" -> "$['a.b'].c", "failedAt" -> "$['a.b']", "functionName" ->
"`variant_insert`"))
+ checkInsertRecoverableError("""{"a.'b": 5}""", """$["a.'b"].c""",
Literal(2),
+ "VARIANT_PATH_TYPE_MISMATCH",
+ Map("path" -> "$[\"a.'b\"].c", "failedAt" -> "$[\"a.'b\"]",
+ "functionName" -> "`variant_insert`"))
+ checkInsertRecoverableError("""{"a.'\"b": 5}""", """$['a.\'"b'].c""",
Literal(2),
+ "VARIANT_PATH_TYPE_MISMATCH",
+ Map(
+ "path" -> """$['a.\'"b'].c""",
+ "failedAt" -> """$["a.'\"b"]""",
+ "functionName" -> "`variant_insert`"))
+ checkInsertRecoverableError("""{"a.\\nb": 5}""", """$['a.\\nb'].c""",
Literal(2),
+ "VARIANT_PATH_TYPE_MISMATCH",
+ Map(
+ "path" -> """$['a.\\nb'].c""",
+ "failedAt" -> """$['a.\\nb']""",
+ "functionName" -> "`variant_insert`"))
+ Seq('\b' -> "\\b", '\f' -> "\\f", '\n' -> "\\n", '\r' -> "\\r", '\t' ->
"\\t",
+ 1.toChar -> ("\\" + "u0001")).foreach { case (control, escape) =>
+ val prefix = s"$$['a.${escape}b']"
+ checkInsertRecoverableError(s"""{"a.${escape}b":5}""", prefix + ".c",
Literal(2),
+ "VARIANT_PATH_TYPE_MISMATCH",
+ Map("path" -> (prefix + ".c"), "failedAt" -> prefix,
+ "functionName" -> "`variant_insert`"))
+ assert(VariantPathParser.parse(prefix, legacyPathBackslashAsLiteral =
false).get
+ .sameElements(Array(ObjectExtraction(s"a.${control}b"))))
+ }
// Structs and maps are rejected at analysis.
Seq(
@@ -1660,6 +1819,12 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkInsertUnrecoverableError("{}", "abc", Literal(1),
"INVALID_VARIANT_PATH",
name => Map("path" -> "abc", "functionName" -> s"`$name`"))
+ checkInsertUnrecoverableError("{}", """$['a\']""", Literal(1),
+ "INVALID_VARIANT_PATH",
+ name => Map("path" -> """$['a\']""", "functionName" -> s"`$name`"))
+ checkInsertUnrecoverableError("{}", """$["a\"]""", Literal(1),
+ "INVALID_VARIANT_PATH",
+ name => Map("path" -> """$["a\"]""", "functionName" -> s"`$name`"))
Seq("variant_insert" -> true, "try_variant_insert" -> false).foreach {
case (name, failOnError) =>
@@ -1742,6 +1907,11 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkSet("""{"a": 1}""", "$.a", Literal(2), """{"a":2}""")
checkSet("""{"a": 1}""", "$.b", Literal(3), """{"a":1,"b":3}""")
checkSet("""{"a": {"b": 1}}""", "$.a.b", Literal(9), """{"a":{"b":9}}""")
+ Seq("uD800", "uDC00").foreach { escape =>
+ checkSet(
+ """{"?":1}""", "$['\\" + escape + "']", Literal(2),
+ "{\"?\":1,\"\\\\" + escape + "\":2}")
+ }
// Missing intermediate keys are created when create_if_missing is true.
checkSet("{}", "$.a.b", Literal(1), """{"a":{"b":1}}""")
@@ -1882,6 +2052,12 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkSetUnrecoverableError("{}", "$", Literal(1),
"INVALID_VARIANT_PATH",
name => Map("path" -> "$", "functionName" -> s"`$name`"))
+ checkSetUnrecoverableError("{}", """$['a\']""", Literal(1),
+ "INVALID_VARIANT_PATH",
+ name => Map("path" -> """$['a\']""", "functionName" -> s"`$name`"))
+ checkSetUnrecoverableError("{}", """$["a\"]""", Literal(1),
+ "INVALID_VARIANT_PATH",
+ name => Map("path" -> """$["a\"]""", "functionName" -> s"`$name`"))
val tooBig = "x".repeat(16 * 1024 * 1024)
checkSetUnrecoverableError("{}", "$.a[2000000000]", Literal(1),
"VARIANT_SIZE_LIMIT",
@@ -2038,6 +2214,12 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
checkAppendUnrecoverableError("[]", "abc", Literal(1),
"INVALID_VARIANT_PATH",
name => Map("path" -> "abc", "functionName" -> s"`$name`"))
+ checkAppendUnrecoverableError("[]", """$['a\']""", Literal(1),
+ "INVALID_VARIANT_PATH",
+ name => Map("path" -> """$['a\']""", "functionName" -> s"`$name`"))
+ checkAppendUnrecoverableError("[]", """$["a\"]""", Literal(1),
+ "INVALID_VARIANT_PATH",
+ name => Map("path" -> """$["a\"]""", "functionName" -> s"`$name`"))
Seq("variant_array_append" -> true, "try_variant_array_append" ->
false).foreach {
case (name, failOnError) =>
@@ -2297,6 +2479,14 @@ class VariantExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper {
VariantPick(Seq(Literal(parseJson("""{"a": 1}""")), Literal("garbage"))),
"INVALID_VARIANT_PATH",
Map("path" -> "garbage", "functionName" -> "`variant_pick`"))
+ checkErrorInExpression[SparkRuntimeException](
+ VariantPick(Seq(Literal(parseJson("""{"a": 1}""")),
Literal("""$['a\']"""))),
+ "INVALID_VARIANT_PATH",
+ Map("path" -> """$['a\']""", "functionName" -> "`variant_pick`"))
+ checkErrorInExpression[SparkRuntimeException](
+ VariantPick(Seq(Literal(parseJson("""{"a": 1}""")),
Literal("""$["a\"]"""))),
+ "INVALID_VARIANT_PATH",
+ Map("path" -> """$["a\"]""", "functionName" -> "`variant_pick`"))
// A malformed constant path is rejected in both interpreted and codegen
modes even when the
// input variant is NULL: the path is parsed before the NULL short-circuit.
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_array_append.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_array_append.explain
index 0905c744ce77..61c6480f0ec6 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_array_append.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_array_append.explain
@@ -1,2 +1,2 @@
-Project
[try_variant_array_append(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, false) AS try_variant_array_append(parse_json(g),
$.a, 1)#0]
+Project
[try_variant_array_append(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, false, false) AS
try_variant_array_append(parse_json(g), $.a, 1)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_get.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_get.explain
index 68eace9d8322..4b259835f240 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_get.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_get.explain
@@ -1,2 +1,2 @@
-Project
[try_variant_get(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $, IntegerType, false, Some(America/Los_Angeles)) AS
try_variant_get(parse_json(g), $)#0]
+Project
[try_variant_get(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $, IntegerType, false, Some(America/Los_Angeles), false) AS
try_variant_get(parse_json(g), $)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_insert.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_insert.explain
index cec39e83751e..bf08f54122b4 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_insert.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_insert.explain
@@ -1,2 +1,2 @@
-Project
[try_variant_insert(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, false) AS try_variant_insert(parse_json(g), $.a,
1)#0]
+Project
[try_variant_insert(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, false, false) AS try_variant_insert(parse_json(g),
$.a, 1)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set.explain
index 30503b185e1e..7fb373161134 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set.explain
@@ -1,2 +1,2 @@
-Project
[try_variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, 1, true, false) AS try_variant_set(parse_json(g), $.a, 1,
true)#0]
+Project
[try_variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, 1, true, false, false) AS try_variant_set(parse_json(g),
$.a, 1, true)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set_with_create_if_missing.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set_with_create_if_missing.explain
index 831e27641964..c34ba4ad2e28 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set_with_create_if_missing.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_try_variant_set_with_create_if_missing.explain
@@ -1,2 +1,2 @@
-Project
[try_variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, 1, false, false) AS try_variant_set(parse_json(g), $.a, 1,
false)#0]
+Project
[try_variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, 1, false, false, false) AS try_variant_set(parse_json(g),
$.a, 1, false)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_array_append.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_array_append.explain
index 4fcec34c6c2d..ec7a5531cc49 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_array_append.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_array_append.explain
@@ -1,2 +1,2 @@
-Project
[variant_array_append(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, true) AS variant_array_append(parse_json(g), $.a,
1)#0]
+Project
[variant_array_append(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, true, false) AS
variant_array_append(parse_json(g), $.a, 1)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_delete.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_delete.explain
index 6b54167b66d6..43ead714ca74 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_delete.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_delete.explain
@@ -1,2 +1,2 @@
-Project
[variant_delete(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, $.b) AS variant_delete(parse_json(g), $.a, $.b)#0]
+Project
[variant_delete(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, $.b, false) AS variant_delete(parse_json(g), $.a, $.b)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_get.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_get.explain
index c13d45774492..0f1bb836b9d0 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_get.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_get.explain
@@ -1,2 +1,2 @@
-Project [variant_get(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $, IntegerType, true, Some(America/Los_Angeles)) AS
variant_get(parse_json(g), $)#0]
+Project [variant_get(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $, IntegerType, true, Some(America/Los_Angeles), false) AS
variant_get(parse_json(g), $)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_insert.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_insert.explain
index 583443ce13fc..5e034aaf896b 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_insert.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_insert.explain
@@ -1,2 +1,2 @@
-Project
[variant_insert(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, 1, true) AS variant_insert(parse_json(g), $.a, 1)#0]
+Project
[variant_insert(static_invoke(VariantExpressionEvalUtils.parseJson(g#0, false,
true, true)), $.a, 1, true, false) AS variant_insert(parse_json(g), $.a, 1)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_pick.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_pick.explain
index 6193b2fb67d2..9d46b4087ab6 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_pick.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_pick.explain
@@ -1,2 +1,2 @@
-Project [variant_pick(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, $.b) AS variant_pick(parse_json(g), $.a, $.b)#0]
+Project [variant_pick(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, $.b, false) AS variant_pick(parse_json(g), $.a,
$.b)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set.explain
index 37dbd79d5071..b01ebec53516 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set.explain
@@ -1,2 +1,2 @@
-Project [variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, true, true) AS variant_set(parse_json(g), $.a, 1,
true)#0]
+Project [variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, true, true, false) AS variant_set(parse_json(g),
$.a, 1, true)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set_with_create_if_missing.explain
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set_with_create_if_missing.explain
index c0a38e55e436..345b1d9857e7 100644
---
a/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set_with_create_if_missing.explain
+++
b/sql/connect/common/src/test/resources/query-tests/explain-results/function_variant_set_with_create_if_missing.explain
@@ -1,2 +1,2 @@
-Project [variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, false, true) AS variant_set(parse_json(g), $.a, 1,
false)#0]
+Project [variant_set(static_invoke(VariantExpressionEvalUtils.parseJson(g#0,
false, true, true)), $.a, 1, false, true, false) AS variant_set(parse_json(g),
$.a, 1, false)#0]
+- LocalRelation <empty>, [id#0L, a#0, b#0, d#0, e#0, f#0, g#0]
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala
index 69aa576f0c94..35c6a68e61af 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala
@@ -47,15 +47,19 @@ case class VariantMetadata(
// with the offending value when the paired data field's strict cast raises
// INVALID_VARIANT_CAST. We pair by NAME (not struct ordinal) because
later pruning or
// reordering of struct fields preserves names but may shift positions.
- castErrorFor: Option[String] = None) {
+ castErrorFor: Option[String] = None,
+ // Path parser mode captured from the originating VariantGet.
+ legacyPathBackslashAsLiteral: Boolean = false) {
// Produce a metadata contain one key-value pair. The key is the special
`METADATA_KEY`.
- // The value contains key-value pairs for `path`, `failOnError`,
`timeZoneId`, and -- for
- // companion fields only -- `castErrorFor`.
+ // The value contains key-value pairs for `path`, `failOnError`,
`timeZoneId`,
+ // `legacyPathBackslashAsLiteral`, and -- for companion fields only --
`castErrorFor`.
def toMetadata: Metadata = {
val inner = new MetadataBuilder()
.putString(VariantMetadata.PATH_KEY, path)
.putBoolean(VariantMetadata.FAIL_ON_ERROR_KEY, failOnError)
.putString(VariantMetadata.TIME_ZONE_ID_KEY, timeZoneId)
+ .putBoolean(
+ VariantMetadata.LEGACY_PATH_BACKSLASH_AS_LITERAL_KEY,
legacyPathBackslashAsLiteral)
castErrorFor.foreach { name =>
inner.putString(VariantMetadata.CAST_ERROR_FOR_KEY, name)
}
@@ -63,7 +67,7 @@ case class VariantMetadata(
}
def parsedPath(): Array[VariantPathSegment] = {
- VariantPathParser.parse(path).getOrElse {
+ VariantPathParser.parse(path, legacyPathBackslashAsLiteral).getOrElse {
val name = if (failOnError) "variant_get" else "try_variant_get"
throw QueryExecutionErrors.invalidVariantGetPath(path, name)
}
@@ -75,6 +79,7 @@ object VariantMetadata {
val PATH_KEY = "path"
val FAIL_ON_ERROR_KEY = "failOnError"
val TIME_ZONE_ID_KEY = "timeZoneId"
+ val LEGACY_PATH_BACKSLASH_AS_LITERAL_KEY = "legacyPathBackslashAsLiteral"
// Optional metadata key marking a struct field as a synthetic cast-error
companion. When
// present, the value is the NAME of the paired data field in the same
variant struct. We tag
// in metadata (rather than by a field-name convention or sentinel path) so
the marker can't
@@ -122,7 +127,8 @@ object VariantMetadata {
value.getString(PATH_KEY),
value.getBoolean(FAIL_ON_ERROR_KEY),
value.getString(TIME_ZONE_ID_KEY),
- castErrorFor
+ castErrorFor,
+ value.getBoolean(LEGACY_PATH_BACKSLASH_AS_LITERAL_KEY)
)
}
}
@@ -138,7 +144,8 @@ object RequestedVariantField {
def apply(v: VariantGet): RequestedVariantField = {
assert(v.path.foldable)
RequestedVariantField(
- VariantMetadata(v.path.eval().toString, v.failOnError,
v.timeZoneId.get), v.dataType)
+ VariantMetadata(v.path.eval().toString, v.failOnError, v.timeZoneId.get,
+ legacyPathBackslashAsLiteral = v.legacyPathBackslashAsLiteral),
v.dataType)
}
def apply(c: Cast): RequestedVariantField =
@@ -281,7 +288,7 @@ class VariantInRelation {
// fields, which also changes the struct type containing it, and it is
difficult to reconstruct
// the original struct value. This is not a big loss, because we need the
full variant anyway.
def collectRequestedFields(expr: Expression): Unit = expr match {
- case v@VariantGet(StructPathToVariant(fields), path, _, _, _) if
path.foldable =>
+ case v@VariantGet(StructPathToVariant(fields), path, _, _, _, _) if
path.foldable =>
addField(fields, RequestedVariantField(v))
case c@Cast(StructPathToVariant(fields), _, _, _) => addField(fields,
RequestedVariantField(c))
case IsNotNull(StructPath(_, _)) | IsNull(StructPath(_, _)) =>
@@ -336,7 +343,7 @@ class VariantInRelation {
// Rewrite patterns should be consistent with visit patterns in
`collectRequestedFields`.
expr.transformDown {
- case g@VariantGet(v@StructPathToVariant(fields), path, _, _, _) if
path.foldable =>
+ case g@VariantGet(v@StructPathToVariant(fields), path, _, _, _, _) if
path.foldable =>
// Rewrite the attribute in advance, rather than depending on the last
branch to rewrite it.
// Ww need to avoid the `v@StructPathToVariant(fields)` branch to
rewrite the child again.
accessRequestedField(fields, RequestedVariantField(g),
rewriteAttribute(v))
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
index bf7c67a03300..655f2447f273 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala
@@ -394,7 +394,7 @@ class ParquetFilters(
// `VariantPathParser.parse` returns None for an unparseable
path; use it directly
// rather than `parsedPath()` (which throws) so a bad path
is a clean no-push and
// we don't swallow unrelated exceptions.
- VariantPathParser.parse(meta.path) match {
+ VariantPathParser.parse(meta.path,
meta.legacyPathBackslashAsLiteral) match {
case None => Nil
case Some(segments) =>
// Only scalar object-extraction paths are eligible.
Reject array-index paths
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/listagg.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/listagg.sql.out
index f2e52a1be723..ab01b1a6e91f 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/listagg.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/listagg.sql.out
@@ -707,7 +707,7 @@ org.apache.spark.sql.catalyst.ExtendedAnalysisException
-- !query
SELECT listagg(DISTINCT v:a::string, ',') WITHIN GROUP (ORDER BY v:a::string)
FROM (SELECT parse_json('{"a": "x"}') v UNION ALL SELECT parse_json('{"a":
"y"}') UNION ALL SELECT parse_json('{"a": "x"}'))
-- !query analysis
-Aggregate [listagg(distinct cast(variant_get(v#x, $.a, VariantType, true,
Some(America/Los_Angeles)) as string), ,, cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles)) as string) ASC NULLS FIRST, 0, 0)
AS listagg(DISTINCT CAST(variant_get(v, $.a) AS a AS STRING), ,) WITHIN GROUP
(ORDER BY CAST(variant_get(v, $.a) AS a AS STRING) ASC NULLS FIRST)#x]
+Aggregate [listagg(distinct cast(variant_get(v#x, $.a, VariantType, true,
Some(America/Los_Angeles), false) as string), ,, cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles), false) as string) ASC NULLS
FIRST, 0, 0) AS listagg(DISTINCT CAST(variant_get(v, $.a) AS a AS STRING), ,)
WITHIN GROUP (ORDER BY CAST(variant_get(v, $.a) AS a AS STRING) ASC NULLS
FIRST)#x]
+- SubqueryAlias __auto_generated_subquery_name
+- Union false, false
:- Union false, false
@@ -722,7 +722,7 @@ Aggregate [listagg(distinct cast(variant_get(v#x, $.a,
VariantType, true, Some(A
-- !query
SELECT listagg(v:a::string, ',') WITHIN GROUP (ORDER BY v:a::string) FROM
(SELECT parse_json('{"a": "x"}') v UNION ALL SELECT parse_json('{"a": "y"}')
UNION ALL SELECT parse_json('{"a": "x"}'))
-- !query analysis
-Aggregate [listagg(cast(variant_get(v#x, $.a, VariantType, true,
Some(America/Los_Angeles)) as string), ,, cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles)) as string) ASC NULLS FIRST, 0, 0)
AS listagg(CAST(variant_get(v, $.a) AS a AS STRING), ,) WITHIN GROUP (ORDER BY
CAST(variant_get(v, $.a) AS a AS STRING) ASC NULLS FIRST)#x]
+Aggregate [listagg(cast(variant_get(v#x, $.a, VariantType, true,
Some(America/Los_Angeles), false) as string), ,, cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles), false) as string) ASC NULLS
FIRST, 0, 0) AS listagg(CAST(variant_get(v, $.a) AS a AS STRING), ,) WITHIN
GROUP (ORDER BY CAST(variant_get(v, $.a) AS a AS STRING) ASC NULLS FIRST)#x]
+- SubqueryAlias __auto_generated_subquery_name
+- Union false, false
:- Union false, false
@@ -737,7 +737,7 @@ Aggregate [listagg(cast(variant_get(v#x, $.a, VariantType,
true, Some(America/Lo
-- !query
SELECT listagg(DISTINCT v:a::string, ',') WITHIN GROUP (ORDER BY v:a::string
DESC) FROM (SELECT parse_json('{"a": "x"}') v UNION ALL SELECT
parse_json('{"a": "y"}') UNION ALL SELECT parse_json('{"a": "x"}'))
-- !query analysis
-Aggregate [listagg(distinct cast(variant_get(v#x, $.a, VariantType, true,
Some(America/Los_Angeles)) as string), ,, cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles)) as string) DESC NULLS LAST, 0, 0)
AS listagg(DISTINCT CAST(variant_get(v, $.a) AS a AS STRING), ,) WITHIN GROUP
(ORDER BY CAST(variant_get(v, $.a) AS a AS STRING) DESC NULLS LAST)#x]
+Aggregate [listagg(distinct cast(variant_get(v#x, $.a, VariantType, true,
Some(America/Los_Angeles), false) as string), ,, cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles), false) as string) DESC NULLS
LAST, 0, 0) AS listagg(DISTINCT CAST(variant_get(v, $.a) AS a AS STRING), ,)
WITHIN GROUP (ORDER BY CAST(variant_get(v, $.a) AS a AS STRING) DESC NULLS
LAST)#x]
+- SubqueryAlias __auto_generated_subquery_name
+- Union false, false
:- Union false, false
@@ -752,7 +752,7 @@ Aggregate [listagg(distinct cast(variant_get(v#x, $.a,
VariantType, true, Some(A
-- !query
SELECT listagg(DISTINCT v:a.b::string, ',') WITHIN GROUP (ORDER BY
v:a.b::string) FROM (SELECT parse_json('{"a": {"b": "x"}}') v UNION ALL SELECT
parse_json('{"a": {"b": "y"}}') UNION ALL SELECT parse_json('{"a": {"b":
"x"}}'))
-- !query analysis
-Aggregate [listagg(distinct cast(variant_get(v#x, $.a.b, VariantType, true,
Some(America/Los_Angeles)) as string), ,, cast(variant_get(v#x, $.a.b,
VariantType, true, Some(America/Los_Angeles)) as string) ASC NULLS FIRST, 0, 0)
AS listagg(DISTINCT CAST(variant_get(v, $.a.b) AS b AS STRING), ,) WITHIN GROUP
(ORDER BY CAST(variant_get(v, $.a.b) AS b AS STRING) ASC NULLS FIRST)#x]
+Aggregate [listagg(distinct cast(variant_get(v#x, $.a.b, VariantType, true,
Some(America/Los_Angeles), false) as string), ,, cast(variant_get(v#x, $.a.b,
VariantType, true, Some(America/Los_Angeles), false) as string) ASC NULLS
FIRST, 0, 0) AS listagg(DISTINCT CAST(variant_get(v, $.a.b) AS b AS STRING), ,)
WITHIN GROUP (ORDER BY CAST(variant_get(v, $.a.b) AS b AS STRING) ASC NULLS
FIRST)#x]
+- SubqueryAlias __auto_generated_subquery_name
+- Union false, false
:- Union false, false
@@ -767,7 +767,7 @@ Aggregate [listagg(distinct cast(variant_get(v#x, $.a.b,
VariantType, true, Some
-- !query
SELECT grp, listagg(DISTINCT v:a::string, ',') WITHIN GROUP (ORDER BY
v:a::string) FROM (SELECT 1 grp, parse_json('{"a": "x"}') v UNION ALL SELECT 1,
parse_json('{"a": "y"}') UNION ALL SELECT 2, parse_json('{"a": "x"}') UNION ALL
SELECT 2, parse_json('{"a": "x"}') UNION ALL SELECT 1, parse_json('{"a":
"x"}')) GROUP BY grp
-- !query analysis
-Aggregate [grp#x], [grp#x, listagg(distinct cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles)) as string), ,,
cast(variant_get(v#x, $.a, VariantType, true, Some(America/Los_Angeles)) as
string) ASC NULLS FIRST, 0, 0) AS listagg(DISTINCT CAST(variant_get(v, $.a) AS
a AS STRING), ,) WITHIN GROUP (ORDER BY CAST(variant_get(v, $.a) AS a AS
STRING) ASC NULLS FIRST)#x]
+Aggregate [grp#x], [grp#x, listagg(distinct cast(variant_get(v#x, $.a,
VariantType, true, Some(America/Los_Angeles), false) as string), ,,
cast(variant_get(v#x, $.a, VariantType, true, Some(America/Los_Angeles), false)
as string) ASC NULLS FIRST, 0, 0) AS listagg(DISTINCT CAST(variant_get(v, $.a)
AS a AS STRING), ,) WITHIN GROUP (ORDER BY CAST(variant_get(v, $.a) AS a AS
STRING) ASC NULLS FIRST)#x]
+- SubqueryAlias __auto_generated_subquery_name
+- Union false, false
:- Union false, false
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/variant-field-extractions.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/variant-field-extractions.sql.out
index 6137de3f665a..01bb9d20c04a 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/variant-field-extractions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/variant-field-extractions.sql.out
@@ -25,7 +25,7 @@ CreateViewCommand `variant_test_data`, SELECT
-- !query
select int_price_variant:price from variant_test_data
-- !query analysis
-Project [variant_get(int_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles)) AS price#x]
+Project [variant_get(int_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles), false) AS price#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -36,7 +36,7 @@ Project [variant_get(int_price_variant#x, $.price,
VariantType, true, Some(Ameri
-- !query
select int_price_variant:price::decimal(5, 2) from variant_test_data
-- !query analysis
-Project [cast(variant_get(int_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles)) as decimal(5,2)) AS price#x]
+Project [cast(variant_get(int_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles), false) as decimal(5,2)) AS price#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -47,7 +47,7 @@ Project [cast(variant_get(int_price_variant#x, $.price,
VariantType, true, Some(
-- !query
select int_price_variant:price::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(int_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles)) as string) AS price#x]
+Project [cast(variant_get(int_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles), false) as string) AS price#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -58,7 +58,7 @@ Project [cast(variant_get(int_price_variant#x, $.price,
VariantType, true, Some(
-- !query
select double_price_variant:price::decimal(3, 2) from variant_test_data
-- !query analysis
-Project [cast(variant_get(double_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles)) as decimal(3,2)) AS price#x]
+Project [cast(variant_get(double_price_variant#x, $.price, VariantType, true,
Some(America/Los_Angeles), false) as decimal(3,2)) AS price#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -69,7 +69,7 @@ Project [cast(variant_get(double_price_variant#x, $.price,
VariantType, true, So
-- !query
select multi_field_variant:name, multi_field_variant:age,
multi_field_variant:city from variant_test_data
-- !query analysis
-Project [variant_get(multi_field_variant#x, $.name, VariantType, true,
Some(America/Los_Angeles)) AS name#x, variant_get(multi_field_variant#x, $.age,
VariantType, true, Some(America/Los_Angeles)) AS age#x,
variant_get(multi_field_variant#x, $.city, VariantType, true,
Some(America/Los_Angeles)) AS city#x]
+Project [variant_get(multi_field_variant#x, $.name, VariantType, true,
Some(America/Los_Angeles), false) AS name#x, variant_get(multi_field_variant#x,
$.age, VariantType, true, Some(America/Los_Angeles), false) AS age#x,
variant_get(multi_field_variant#x, $.city, VariantType, true,
Some(America/Los_Angeles), false) AS city#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -80,7 +80,7 @@ Project [variant_get(multi_field_variant#x, $.name,
VariantType, true, Some(Amer
-- !query
select multi_field_variant:name::string, multi_field_variant:age::int,
multi_field_variant:active::boolean from variant_test_data
-- !query analysis
-Project [cast(variant_get(multi_field_variant#x, $.name, VariantType, true,
Some(America/Los_Angeles)) as string) AS name#x,
cast(variant_get(multi_field_variant#x, $.age, VariantType, true,
Some(America/Los_Angeles)) as int) AS age#x,
cast(variant_get(multi_field_variant#x, $.active, VariantType, true,
Some(America/Los_Angeles)) as boolean) AS active#x]
+Project [cast(variant_get(multi_field_variant#x, $.name, VariantType, true,
Some(America/Los_Angeles), false) as string) AS name#x,
cast(variant_get(multi_field_variant#x, $.age, VariantType, true,
Some(America/Los_Angeles), false) as int) AS age#x,
cast(variant_get(multi_field_variant#x, $.active, VariantType, true,
Some(America/Los_Angeles), false) as boolean) AS active#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -91,7 +91,7 @@ Project [cast(variant_get(multi_field_variant#x, $.name,
VariantType, true, Some
-- !query
select multi_field_variant:['name'] from variant_test_data
-- !query analysis
-Project [variant_get(multi_field_variant#x, $['name'], VariantType, true,
Some(America/Los_Angeles)) AS name#x]
+Project [variant_get(multi_field_variant#x, $['name'], VariantType, true,
Some(America/Los_Angeles), false) AS name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -102,7 +102,7 @@ Project [variant_get(multi_field_variant#x, $['name'],
VariantType, true, Some(A
-- !query
select multi_field_variant:['age'] from variant_test_data
-- !query analysis
-Project [variant_get(multi_field_variant#x, $['age'], VariantType, true,
Some(America/Los_Angeles)) AS age#x]
+Project [variant_get(multi_field_variant#x, $['age'], VariantType, true,
Some(America/Los_Angeles), false) AS age#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -113,7 +113,7 @@ Project [variant_get(multi_field_variant#x, $['age'],
VariantType, true, Some(Am
-- !query
select array_value_variant:item[0].model from variant_test_data
-- !query analysis
-Project [variant_get(array_value_variant#x, $.item[0].model, VariantType,
true, Some(America/Los_Angeles)) AS model#x]
+Project [variant_get(array_value_variant#x, $.item[0].model, VariantType,
true, Some(America/Los_Angeles), false) AS model#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -124,7 +124,7 @@ Project [variant_get(array_value_variant#x,
$.item[0].model, VariantType, true,
-- !query
select array_value_variant:item[0].price from variant_test_data
-- !query analysis
-Project [variant_get(array_value_variant#x, $.item[0].price, VariantType,
true, Some(America/Los_Angeles)) AS price#x]
+Project [variant_get(array_value_variant#x, $.item[0].price, VariantType,
true, Some(America/Los_Angeles), false) AS price#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -135,7 +135,7 @@ Project [variant_get(array_value_variant#x,
$.item[0].price, VariantType, true,
-- !query
select array_value_variant:item[1].model from variant_test_data
-- !query analysis
-Project [variant_get(array_value_variant#x, $.item[1].model, VariantType,
true, Some(America/Los_Angeles)) AS model#x]
+Project [variant_get(array_value_variant#x, $.item[1].model, VariantType,
true, Some(America/Los_Angeles), false) AS model#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -146,7 +146,7 @@ Project [variant_get(array_value_variant#x,
$.item[1].model, VariantType, true,
-- !query
select array_value_variant:item[1].price from variant_test_data
-- !query analysis
-Project [variant_get(array_value_variant#x, $.item[1].price, VariantType,
true, Some(America/Los_Angeles)) AS price#x]
+Project [variant_get(array_value_variant#x, $.item[1].price, VariantType,
true, Some(America/Los_Angeles), false) AS price#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -157,7 +157,7 @@ Project [variant_get(array_value_variant#x,
$.item[1].price, VariantType, true,
-- !query
select array_variant:[0].id from variant_test_data
-- !query analysis
-Project [variant_get(array_variant#x, $[0].id, VariantType, true,
Some(America/Los_Angeles)) AS id#x]
+Project [variant_get(array_variant#x, $[0].id, VariantType, true,
Some(America/Los_Angeles), false) AS id#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -168,7 +168,7 @@ Project [variant_get(array_variant#x, $[0].id, VariantType,
true, Some(America/L
-- !query
select array_variant:[0].name from variant_test_data
-- !query analysis
-Project [variant_get(array_variant#x, $[0].name, VariantType, true,
Some(America/Los_Angeles)) AS name#x]
+Project [variant_get(array_variant#x, $[0].name, VariantType, true,
Some(America/Los_Angeles), false) AS name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -179,7 +179,7 @@ Project [variant_get(array_variant#x, $[0].name,
VariantType, true, Some(America
-- !query
select array_variant:[1].id from variant_test_data
-- !query analysis
-Project [variant_get(array_variant#x, $[1].id, VariantType, true,
Some(America/Los_Angeles)) AS id#x]
+Project [variant_get(array_variant#x, $[1].id, VariantType, true,
Some(America/Los_Angeles), false) AS id#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -190,7 +190,7 @@ Project [variant_get(array_variant#x, $[1].id, VariantType,
true, Some(America/L
-- !query
select array_variant:[1].name from variant_test_data
-- !query analysis
-Project [variant_get(array_variant#x, $[1].name, VariantType, true,
Some(America/Los_Angeles)) AS name#x]
+Project [variant_get(array_variant#x, $[1].name, VariantType, true,
Some(America/Los_Angeles), false) AS name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -201,7 +201,7 @@ Project [variant_get(array_variant#x, $[1].name,
VariantType, true, Some(America
-- !query
select nested_variant:metadata.version from variant_test_data
-- !query analysis
-Project [variant_get(nested_variant#x, $.metadata.version, VariantType, true,
Some(America/Los_Angeles)) AS version#x]
+Project [variant_get(nested_variant#x, $.metadata.version, VariantType, true,
Some(America/Los_Angeles), false) AS version#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -212,7 +212,7 @@ Project [variant_get(nested_variant#x, $.metadata.version,
VariantType, true, So
-- !query
select nested_variant:metadata.tags[0] from variant_test_data
-- !query analysis
-Project [variant_get(nested_variant#x, $.metadata.tags[0], VariantType, true,
Some(America/Los_Angeles)) AS tags#x]
+Project [variant_get(nested_variant#x, $.metadata.tags[0], VariantType, true,
Some(America/Los_Angeles), false) AS tags#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -223,7 +223,7 @@ Project [variant_get(nested_variant#x, $.metadata.tags[0],
VariantType, true, So
-- !query
select nested_variant:metadata.nested.level from variant_test_data
-- !query analysis
-Project [variant_get(nested_variant#x, $.metadata.nested.level, VariantType,
true, Some(America/Los_Angeles)) AS level#x]
+Project [variant_get(nested_variant#x, $.metadata.nested.level, VariantType,
true, Some(America/Los_Angeles), false) AS level#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -234,7 +234,7 @@ Project [variant_get(nested_variant#x,
$.metadata.nested.level, VariantType, tru
-- !query
select nested_variant:metadata.nested.value from variant_test_data
-- !query analysis
-Project [variant_get(nested_variant#x, $.metadata.nested.value, VariantType,
true, Some(America/Los_Angeles)) AS value#x]
+Project [variant_get(nested_variant#x, $.metadata.nested.value, VariantType,
true, Some(America/Los_Angeles), false) AS value#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -245,7 +245,7 @@ Project [variant_get(nested_variant#x,
$.metadata.nested.value, VariantType, tru
-- !query
select nested_variant:metadata['version'] from variant_test_data
-- !query analysis
-Project [variant_get(nested_variant#x, $.metadata['version'], VariantType,
true, Some(America/Los_Angeles)) AS version#x]
+Project [variant_get(nested_variant#x, $.metadata['version'], VariantType,
true, Some(America/Los_Angeles), false) AS version#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -256,7 +256,7 @@ Project [variant_get(nested_variant#x,
$.metadata['version'], VariantType, true,
-- !query
select nested_variant:metadata['tags'] from variant_test_data
-- !query analysis
-Project [variant_get(nested_variant#x, $.metadata['tags'], VariantType, true,
Some(America/Los_Angeles)) AS tags#x]
+Project [variant_get(nested_variant#x, $.metadata['tags'], VariantType, true,
Some(America/Los_Angeles), false) AS tags#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -267,7 +267,7 @@ Project [variant_get(nested_variant#x, $.metadata['tags'],
VariantType, true, So
-- !query
select special_chars_variant:`field-name`::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(special_chars_variant#x, $.field-name, VariantType,
true, Some(America/Los_Angeles)) as string) AS field-name#x]
+Project [cast(variant_get(special_chars_variant#x, $.field-name, VariantType,
true, Some(America/Los_Angeles), false) as string) AS field-name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -278,7 +278,7 @@ Project [cast(variant_get(special_chars_variant#x,
$.field-name, VariantType, tr
-- !query
select special_chars_variant:['field-name']::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(special_chars_variant#x, $['field-name'],
VariantType, true, Some(America/Los_Angeles)) as string) AS field-name#x]
+Project [cast(variant_get(special_chars_variant#x, $['field-name'],
VariantType, true, Some(America/Los_Angeles), false) as string) AS field-name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -289,7 +289,7 @@ Project [cast(variant_get(special_chars_variant#x,
$['field-name'], VariantType,
-- !query
select special_chars_variant:field_name::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(special_chars_variant#x, $.field_name, VariantType,
true, Some(America/Los_Angeles)) as string) AS field_name#x]
+Project [cast(variant_get(special_chars_variant#x, $.field_name, VariantType,
true, Some(America/Los_Angeles), false) as string) AS field_name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -300,7 +300,7 @@ Project [cast(variant_get(special_chars_variant#x,
$.field_name, VariantType, tr
-- !query
select special_chars_variant:['field_name']::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(special_chars_variant#x, $['field_name'],
VariantType, true, Some(America/Los_Angeles)) as string) AS field_name#x]
+Project [cast(variant_get(special_chars_variant#x, $['field_name'],
VariantType, true, Some(America/Los_Angeles), false) as string) AS field_name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -311,7 +311,7 @@ Project [cast(variant_get(special_chars_variant#x,
$['field_name'], VariantType,
-- !query
select special_chars_variant:`field.name`::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(special_chars_variant#x, $.field.name, VariantType,
true, Some(America/Los_Angeles)) as string) AS name#x]
+Project [cast(variant_get(special_chars_variant#x, $.field.name, VariantType,
true, Some(America/Los_Angeles), false) as string) AS name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -322,7 +322,7 @@ Project [cast(variant_get(special_chars_variant#x,
$.field.name, VariantType, tr
-- !query
select special_chars_variant:['field.name']::string from variant_test_data
-- !query analysis
-Project [cast(variant_get(special_chars_variant#x, $['field.name'],
VariantType, true, Some(America/Los_Angeles)) as string) AS field.name#x]
+Project [cast(variant_get(special_chars_variant#x, $['field.name'],
VariantType, true, Some(America/Los_Angeles), false) as string) AS field.name#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -333,7 +333,7 @@ Project [cast(variant_get(special_chars_variant#x,
$['field.name'], VariantType,
-- !query
select multi_field_variant:scores[0]::int + multi_field_variant:scores[1]::int
from variant_test_data
-- !query analysis
-Project [(cast(variant_get(multi_field_variant#x, $.scores[0], VariantType,
true, Some(America/Los_Angeles)) as int) +
cast(variant_get(multi_field_variant#x, $.scores[1], VariantType, true,
Some(America/Los_Angeles)) as int)) AS (CAST(variant_get(multi_field_variant,
$.scores[0]) AS scores AS INT) + CAST(variant_get(multi_field_variant,
$.scores[1]) AS scores AS INT))#x]
+Project [(cast(variant_get(multi_field_variant#x, $.scores[0], VariantType,
true, Some(America/Los_Angeles), false) as int) +
cast(variant_get(multi_field_variant#x, $.scores[1], VariantType, true,
Some(America/Los_Angeles), false) as int)) AS
(CAST(variant_get(multi_field_variant, $.scores[0]) AS scores AS INT) +
CAST(variant_get(multi_field_variant, $.scores[1]) AS scores AS INT))#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -347,7 +347,7 @@ select count(*) from (select
explode(cast(multi_field_variant:scores as array<in
Aggregate [count(1) AS count(1)#xL]
+- SubqueryAlias __auto_generated_subquery_name
+- Project [score#x]
- +- Generate explode(cast(variant_get(multi_field_variant#x, $.scores,
VariantType, true, Some(America/Los_Angeles)) as array<int>)), false, [score#x]
+ +- Generate explode(cast(variant_get(multi_field_variant#x, $.scores,
VariantType, true, Some(America/Los_Angeles), false) as array<int>)), false,
[score#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x,
double_price_variant#x, multi_field_variant#x, array_value_variant#x,
array_variant#x, nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS
int_price_variant#x, cast(double_price_variant#x as variant) AS
double_price_variant#x, cast(multi_field_variant#x as variant) AS
multi_field_variant#x, cast(array_value_variant#x as variant) AS
array_value_variant#x, cast(array_variant#x as variant) AS array_variant#x,
cast(nested_variant#x as variant) AS nested_variant#x,
cast(special_chars_variant#x as variant) AS special_chars_variant#x]
@@ -383,7 +383,7 @@ org.apache.spark.sql.catalyst.parser.ParseException
-- !query
select typeof(multi_field_variant:name) from variant_test_data
-- !query analysis
-Project [typeof(variant_get(multi_field_variant#x, $.name, VariantType, true,
Some(America/Los_Angeles))) AS typeof(variant_get(multi_field_variant, $.name)
AS name)#x]
+Project [typeof(variant_get(multi_field_variant#x, $.name, VariantType, true,
Some(America/Los_Angeles), false)) AS typeof(variant_get(multi_field_variant,
$.name) AS name)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -394,7 +394,7 @@ Project [typeof(variant_get(multi_field_variant#x, $.name,
VariantType, true, So
-- !query
select typeof(multi_field_variant:age) from variant_test_data
-- !query analysis
-Project [typeof(variant_get(multi_field_variant#x, $.age, VariantType, true,
Some(America/Los_Angeles))) AS typeof(variant_get(multi_field_variant, $.age)
AS age)#x]
+Project [typeof(variant_get(multi_field_variant#x, $.age, VariantType, true,
Some(America/Los_Angeles), false)) AS typeof(variant_get(multi_field_variant,
$.age) AS age)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -405,7 +405,7 @@ Project [typeof(variant_get(multi_field_variant#x, $.age,
VariantType, true, Som
-- !query
select typeof(multi_field_variant:active) from variant_test_data
-- !query analysis
-Project [typeof(variant_get(multi_field_variant#x, $.active, VariantType,
true, Some(America/Los_Angeles))) AS typeof(variant_get(multi_field_variant,
$.active) AS active)#x]
+Project [typeof(variant_get(multi_field_variant#x, $.active, VariantType,
true, Some(America/Los_Angeles), false)) AS
typeof(variant_get(multi_field_variant, $.active) AS active)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -416,7 +416,7 @@ Project [typeof(variant_get(multi_field_variant#x,
$.active, VariantType, true,
-- !query
select typeof(multi_field_variant:scores) from variant_test_data
-- !query analysis
-Project [typeof(variant_get(multi_field_variant#x, $.scores, VariantType,
true, Some(America/Los_Angeles))) AS typeof(variant_get(multi_field_variant,
$.scores) AS scores)#x]
+Project [typeof(variant_get(multi_field_variant#x, $.scores, VariantType,
true, Some(America/Los_Angeles), false)) AS
typeof(variant_get(multi_field_variant, $.scores) AS scores)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -427,7 +427,7 @@ Project [typeof(variant_get(multi_field_variant#x,
$.scores, VariantType, true,
-- !query
select isnull(multi_field_variant:missing_field) from variant_test_data
-- !query analysis
-Project [isnull(variant_get(multi_field_variant#x, $.missing_field,
VariantType, true, Some(America/Los_Angeles))) AS
(variant_get(multi_field_variant, $.missing_field) AS missing_field IS NULL)#x]
+Project [isnull(variant_get(multi_field_variant#x, $.missing_field,
VariantType, true, Some(America/Los_Angeles), false)) AS
(variant_get(multi_field_variant, $.missing_field) AS missing_field IS NULL)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -438,7 +438,7 @@ Project [isnull(variant_get(multi_field_variant#x,
$.missing_field, VariantType,
-- !query
select isnotnull(multi_field_variant:name) from variant_test_data
-- !query analysis
-Project [isnotnull(variant_get(multi_field_variant#x, $.name, VariantType,
true, Some(America/Los_Angeles))) AS (variant_get(multi_field_variant, $.name)
AS name IS NOT NULL)#x]
+Project [isnotnull(variant_get(multi_field_variant#x, $.name, VariantType,
true, Some(America/Los_Angeles), false)) AS (variant_get(multi_field_variant,
$.name) AS name IS NOT NULL)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
@@ -449,7 +449,7 @@ Project [isnotnull(variant_get(multi_field_variant#x,
$.name, VariantType, true,
-- !query
select coalesce(multi_field_variant:missing_field, 'default_value') from
variant_test_data
-- !query analysis
-Project [coalesce(variant_get(multi_field_variant#x, $.missing_field,
VariantType, true, Some(America/Los_Angeles)), cast(default_value as variant))
AS coalesce(variant_get(multi_field_variant, $.missing_field) AS missing_field,
default_value)#x]
+Project [coalesce(variant_get(multi_field_variant#x, $.missing_field,
VariantType, true, Some(America/Los_Angeles), false), cast(default_value as
variant)) AS coalesce(variant_get(multi_field_variant, $.missing_field) AS
missing_field, default_value)#x]
+- SubqueryAlias variant_test_data
+- View (`variant_test_data`, [int_price_variant#x, double_price_variant#x,
multi_field_variant#x, array_value_variant#x, array_variant#x,
nested_variant#x, special_chars_variant#x])
+- Project [cast(int_price_variant#x as variant) AS int_price_variant#x,
cast(double_price_variant#x as variant) AS double_price_variant#x,
cast(multi_field_variant#x as variant) AS multi_field_variant#x,
cast(array_value_variant#x as variant) AS array_value_variant#x,
cast(array_variant#x as variant) AS array_variant#x, cast(nested_variant#x as
variant) AS nested_variant#x, cast(special_chars_variant#x as variant) AS
special_chars_variant#x]
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala
index 765edf1de809..e11d396542ad 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/VariantSuite.scala
@@ -212,6 +212,15 @@ class VariantSuite extends SharedSparkSession with
ExpressionEvalHelper {
sql("SELECT to_json(variant_delete(parse_json('[1, 2, 3]'), '$[1]'))"),
rows("[1,3]"))
+ Seq(false, true).foreach { legacy =>
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
legacy.toString) {
+ val input = Seq("""{"a\nb":1,"a\\nb":2}""").toDF("json")
+ checkAnswer(
+ input.selectExpr("""to_json(variant_delete(parse_json(json),
r"$['a\nb']"))"""),
+ rows(if (legacy) """{"a\nb":1}""" else """{"a\\nb":2}"""))
+ }
+ }
+
checkAnswer(
sql("SELECT to_json(variant_delete(CAST(NULL AS VARIANT), '$.a'))"),
rows(null))
@@ -776,6 +785,15 @@ class VariantSuite extends SharedSparkSession with
ExpressionEvalHelper {
sql("SELECT to_json(variant_pick(parse_json('{\"a\": 1}'), '$'))"),
rows("""{"a":1}"""))
+ Seq(false, true).foreach { legacy =>
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
legacy.toString) {
+ val input = Seq("""{"a\nb":1,"a\\nb":2}""").toDF("json")
+ checkAnswer(
+ input.selectExpr("""to_json(variant_pick(parse_json(json),
r"$['a\nb']"))"""),
+ rows(if (legacy) """{"a\\nb":2}""" else """{"a\nb":1}"""))
+ }
+ }
+
// The String* DataFrame-API overload compiles each path to a literal.
val df = Seq("""{"a": 1, "b": 2, "c": 3}""").toDF("json")
checkAnswer(
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScanSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScanSuite.scala
index 73590b039a01..58dd7aa1de41 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScanSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScanSuite.scala
@@ -18,7 +18,7 @@
package org.apache.spark.sql.execution.datasources
import org.apache.spark.SparkConf
-import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.{QueryTest, Row}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.variant._
import org.apache.spark.sql.catalyst.plans.logical._
@@ -156,6 +156,20 @@ trait PushVariantIntoScanSuiteBase extends
SharedSparkSession {
outputs.head
}
+ gridTest("Variant path escapes in pushed scans")(Seq(false, true)) { legacy
=>
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
legacy.toString) {
+ withVariantParquetData(
+ "v variant",
+ """(parse_json(r'{"a\nb":1,"a\\nb":2}'))""") {
+ val pushed = sql("""select variant_get(v, r'$["a\nb"]', 'int') from
T""")
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
(!legacy).toString) {
+
assertShreddedStruct(scanColumnType(pushed.queryExecution.optimizedPlan, "v"))
+ checkAnswer(pushed, Row(if (legacy) 2 else 1))
+ }
+ }
+ }
+ }
+
test("aggregate function argument variant_get is hoisted below the aggregate
and shredded") {
// `max(variant_get(v, '$.price'))` lives inside the aggregate function,
above the aggregate
// barrier that PhysicalOperation-based pushdown cannot see through.
PullOutVariantExtractions
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
index 342fc230ad71..e076e83dd9e6 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
@@ -2689,6 +2689,24 @@ abstract class ParquetFilterSuite extends ParquetTest
with SharedSparkSession {
s"Expected an isNull(leaf) guard arm in $s")
}
+ gridTest("shredded variant filter: quoted paths use the captured parser
mode")(
+ Seq(false, true)) { legacy =>
+ val leafType = new StructType().add("value",
BinaryType).add("typed_value", LongType)
+ val keys = new StructType().add("a\nb", leafType).add("a\\nb", leafType)
+ val variant = new StructType().add("value", BinaryType).add("typed_value",
keys)
+ val parquetSchema = new SparkToParquetSchemaConverter(conf)
+ .convert(new StructType().add("v", variant))
+ val metadata = VariantMetadata(
+ "$['a\\nb']", failOnError = true, "UTC", legacyPathBackslashAsLiteral =
legacy).toMetadata
+ val extraction = variantExtractionSchema("v", StructField("0", LongType,
metadata = metadata))
+ withSQLConf(SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key ->
(!legacy).toString) {
+ val pf = createParquetFilters(parquetSchema, variantExtractionSchema =
Some(extraction))
+ val filter = pf.createFilter(sources.GreaterThan("v.`0`", 1L))
+ val key = if (legacy) "a\\nb" else "a\nb"
+
assert(filter.exists(_.toString.contains(s"gt(v.typed_value.$key.typed_value,
1)")))
+ }
+ }
+
test("shredded variant filter: without variantExtractionSchema the logical
path is unknown") {
val parquetSchema =
"""message spark_schema {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
index fabebe4f4f69..5e62431b404d 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
@@ -968,6 +968,32 @@ class DataSourceV2StrategySuite extends SharedSparkSession
{
}
}
+ gridTest("VariantGet translation uses the captured path parser mode")(
+ Seq(false, true)) { legacy =>
+ val ref = AttributeReference("v", VariantType)()
+ val conf = SQLConf.LEGACY_VARIANT_PATH_BACKSLASH_AS_LITERAL.key
+ for {
+ (path, modeSensitive) <- Seq(
+ "$['city']" -> false,
+ """$.a\nb""" -> false,
+ """$['a\qb']""" -> false,
+ """$['a\nb']""" -> true,
+ """$["a\nb"]""" -> true,
+ """$['a\']""" -> true)
+ failOnError <- Seq(false, true)
+ } {
+ val expression = VariantGet(
+ ref, Literal(path), BooleanType, failOnError,
legacyPathBackslashAsLiteral = legacy)
+ val expected = Option.when(!legacy || !modeSensitive)(
+ new V2VariantGet(FieldReference("v"), path, BooleanType, failOnError,
null))
+ withSQLConf(conf -> (!legacy).toString) {
+ assert(new V2ExpressionBuilder(expression).build() == expected)
+ testTranslateFilter(expression,
+ expected.map(extraction => new Predicate("BOOLEAN_EXPRESSION",
Array(extraction))))
+ }
+ }
+ }
+
test("VariantGet with non-foldable path returns None") {
val ref = AttributeReference("v", VariantType)()
val s = AttributeReference("s", StringType)()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]