This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7ce5ae79327a [SPARK-57517][SQL] Fix schema_of_json to return proper
error on non-string literal input
7ce5ae79327a is described below
commit 7ce5ae79327a24135006b2663a96725e6cbdb8ae
Author: Jubin Soni <[email protected]>
AuthorDate: Fri Jun 19 07:57:01 2026 +0200
[SPARK-57517][SQL] Fix schema_of_json to return proper error on non-string
literal input
## What is the purpose of the change
Fixes [SPARK-57517](https://issues.apache.org/jira/browse/SPARK-57517) —
`schema_of_json` throws a `ClassCastException` during analysis when called with
a non-string literal (e.g., `SELECT schema_of_json(42)`), instead of surfacing
a clean `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` error.
The root cause is in `SchemaOfJson.checkInputDataTypes()`: it references a
`lazy val json = child.eval().asInstanceOf[UTF8String]` before verifying that
the child's type is `StringType`. For an integer literal, the
`asInstanceOf[UTF8String]` cast throws `ClassCastException` at analysis time
rather than producing a user-facing error.
The companion functions `schema_of_csv` and `schema_of_xml` were fixed for
the same issue in SPARK-52234, but `schema_of_json` was missed. This PR applies
the same fix: restructuring `checkInputDataTypes` to check `!foldable` →
`eval() == null` → `dataType != StringType` in safe order, and removing the
unsafe lazy val entirely.
## Brief change log
- `SchemaOfJson.checkInputDataTypes()`: removed the `lazy val json` that
performed an unsafe `asInstanceOf[UTF8String]` cast; restructured the condition
chain to check for non-foldable input, null input, and wrong type (adding a new
`UNEXPECTED_INPUT_TYPE` branch) before delegating to
`super.checkInputDataTypes()`
- Added `select schema_of_json(42)` to `json-functions.sql` input
- Added corresponding `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` expected
entries to `analyzer-results/json-functions.sql.out` and
`results/json-functions.sql.out`
## Verifying this change
This change is covered by golden file SQL query tests in
`SQLQueryTestSuite`:
- `select schema_of_json(42)` — verifies that a non-string integer literal
produces `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` at analysis time (previously
threw `ClassCastException`)
- Existing tests for `schema_of_json(null)` and
`schema_of_json(nonFoldableColumn)` continue to pass, confirming the null and
non-foldable branches are unaffected
## Does this pull request potentially affect one of the following parts
- **Dependencies** (does it add or upgrade a dependency): no
- **The public API, i.e., is any changed class annotated with
`Public(Evolving)`**: no — `SchemaOfJson` is an internal catalyst expression
- **The serializers**: no
- **The runtime per-record code paths (performance sensitive)**: no — only
affects the analysis-time type check path
- **Anything that affects deployment or recovery**: no
- **The S3 file system connector**: no
## Documentation
Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable
## Was generative AI tooling used to co-author this PR?
- [x] Yes — Claude Code was used as a pair-programming assistant. All code
was written, understood, and verified by the author.
Generated-by: Claude Opus 4.8
Closes #56582 from jubins/j-SPARK-57517-fix-class-cast-exception.
Authored-by: Jubin Soni <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
.../sql/catalyst/expressions/jsonExpressions.scala | 19 ++++++++++------
.../analyzer-results/json-functions.sql.out | 24 ++++++++++++++++++++
.../resources/sql-tests/inputs/json-functions.sql | 1 +
.../sql-tests/results/json-functions.sql.out | 26 ++++++++++++++++++++++
4 files changed, 63 insertions(+), 7 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
index 3ec17c6ca584..ed1b7191963c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
@@ -454,23 +454,28 @@ case class SchemaOfJson(
override def nullable: Boolean = false
- @transient
- private lazy val json = child.eval().asInstanceOf[UTF8String]
-
override def checkInputDataTypes(): TypeCheckResult = {
- if (child.foldable && json != null) {
- super.checkInputDataTypes()
- } else if (!child.foldable) {
+ if (!child.foldable) {
DataTypeMismatch(
errorSubClass = "NON_FOLDABLE_INPUT",
messageParameters = Map(
"inputName" -> toSQLId("json"),
"inputType" -> toSQLType(child.dataType),
"inputExpr" -> toSQLExpr(child)))
- } else {
+ } else if (child.eval() == null) {
DataTypeMismatch(
errorSubClass = "UNEXPECTED_NULL",
messageParameters = Map("exprName" -> "json"))
+ } else if (child.dataType != StringType) {
+ DataTypeMismatch(
+ errorSubClass = "UNEXPECTED_INPUT_TYPE",
+ messageParameters = Map(
+ "paramIndex" -> ordinalNumber(0),
+ "inputSql" -> toSQLExpr(child),
+ "inputType" -> toSQLType(child.dataType),
+ "requiredType" -> toSQLType(StringType)))
+ } else {
+ super.checkInputDataTypes()
}
}
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/json-functions.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/json-functions.sql.out
index 9afb3e1749ed..4fb1f0f04231 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/json-functions.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/json-functions.sql.out
@@ -446,6 +446,30 @@ org.apache.spark.sql.catalyst.ExtendedAnalysisException
}
+-- !query
+select schema_of_json(42)
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"42\"",
+ "inputType" : "\"INT\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"STRING\"",
+ "sqlExpr" : "\"schema_of_json(42)\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 25,
+ "fragment" : "schema_of_json(42)"
+ } ]
+}
+
+
-- !query
CREATE TEMPORARY VIEW jsonTable(jsonField, a) AS SELECT * FROM VALUES ('{"a":
1, "b": 2}', 'a')
-- !query analysis
diff --git a/sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
b/sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
index 71706d3c2e23..66134545107a 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/json-functions.sql
@@ -75,6 +75,7 @@ select to_json(array(array(1, 2, 3), array(4)));
select schema_of_json('{"c1":1}', map('primitivesAsString', 'true'));
select schema_of_json('{"c1":01, "c2":0.1}', map('allowNumericLeadingZeros',
'true', 'prefersDecimal', 'true'));
select schema_of_json(null);
+select schema_of_json(42);
CREATE TEMPORARY VIEW jsonTable(jsonField, a) AS SELECT * FROM VALUES ('{"a":
1, "b": 2}', 'a');
SELECT schema_of_json(jsonField) FROM jsonTable;
diff --git
a/sql/core/src/test/resources/sql-tests/results/json-functions.sql.out
b/sql/core/src/test/resources/sql-tests/results/json-functions.sql.out
index 53ade56684ad..96c6dab19dc7 100644
--- a/sql/core/src/test/resources/sql-tests/results/json-functions.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/json-functions.sql.out
@@ -496,6 +496,32 @@ org.apache.spark.sql.catalyst.ExtendedAnalysisException
}
+-- !query
+select schema_of_json(42)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE",
+ "sqlState" : "42K09",
+ "messageParameters" : {
+ "inputSql" : "\"42\"",
+ "inputType" : "\"INT\"",
+ "paramIndex" : "first",
+ "requiredType" : "\"STRING\"",
+ "sqlExpr" : "\"schema_of_json(42)\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 8,
+ "stopIndex" : 25,
+ "fragment" : "schema_of_json(42)"
+ } ]
+}
+
+
-- !query
CREATE TEMPORARY VIEW jsonTable(jsonField, a) AS SELECT * FROM VALUES ('{"a":
1, "b": 2}', 'a')
-- !query schema
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
