marcuslin123 opened a new pull request, #57211:
URL: https://github.com/apache/spark/pull/57211
### What changes were proposed in this pull request?
Add a new built-in SQL function `json_valid(jsonString) -> boolean` that
returns `true` if the input is a valid JSON string, `false` otherwise, and
`null` for null input.
```sql
SELECT json_valid('{"a": 1}'); -- true
SELECT json_valid('[1, 2, 3]'); -- true
SELECT json_valid('invalid'); -- false
SELECT json_valid('{"a":1} garbage'); -- false (trailing content)
SELECT json_valid(''); -- false
SELECT json_valid(null); -- null
```
The function is exposed across all surfaces: SQL, the Scala DataFrame API
(`json_valid(col)`), and PySpark (`F.json_valid(col)`), including Spark Connect.
**Implementation:** Uses a streaming parser (`JsonParser` + `skipChildren`)
via the existing shared `JsonExpressionUtils` pattern (same as
`json_array_length` / `json_object_keys`), rather than materializing a parse
tree. Validation confirms the input is exactly one complete JSON value with no
trailing content; empty/whitespace-only input returns `false`.
**On strict vs. lenient semantics:** This implementation reuses Spark's
existing `SharedFactory.jsonFactory()` (which enables `ALLOW_SINGLE_QUOTES` and
`ALLOW_UNESCAPED_CONTROL_CHARS` for Hive compatibility), keeping `json_valid`
consistent with `from_json` and `get_json_object`. This diverges from MySQL's
strict `JSON_VALID`. Happy to switch to strict RFC 8259 semantics if reviewers
prefer — flagging this as an open design decision per the JIRA.
### Why are the changes needed?
Spark SQL has no built-in JSON syntax validation. Users currently resort to
`get_json_object(col, '$') IS NOT NULL`, which is semantically different — it
returns null for some malformed JSON without proper validation. Other databases
provide this (MySQL 8.0+ `JSON_VALID()`, PostgreSQL). It is a common
data-quality check in ETL pipelines and performance-critical for large-scale
ingestion, which is why a built-in beats a UDF.
### Does this PR introduce _any_ user-facing change?
Yes. A new `json_valid` function is available in SQL, the Scala DataFrame
API, and PySpark (classic and Connect).
### How was this patch tested?
- Catalyst unit test in `JsonExpressionsSuite` covering null, valid values
(objects, arrays, scalars), and invalid inputs (empty, whitespace, malformed,
trailing content).
- End-to-end test in `JsonFunctionsSuite` exercising both the SQL string
form and the Scala function.
- SQL golden-file cases in `json-functions.sql` (regenerated `.sql.out`).
- PySpark doctest.
- Spark Connect plan golden files regenerated.
- Expression schema golden file regenerated.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for
implementation guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
