sankalpsthakur opened a new pull request, #59264:
URL: https://github.com/apache/spark/pull/59264
### What changes were proposed in this pull request?
`AstBuilder.visitRowFormatDelimited` maps every `ROW FORMAT DELIMITED`
clause to a Hive SerDe property except `NULL DEFINED AS`. The grammar captures
`nullDefinedAs`, but the visitor never reads it, and the method carries `//
TODO we need proper support for the NULL format.`
This PR stores the value under `serialization.null.format`, the key Hive's
`CreateTableDesc` writes and `LazySimpleSerDe` reads, and removes the TODO. It
applies to `CREATE`/`REPLACE TABLE [AS SELECT]`, `CREATE TABLE LIKE ... ROW
FORMAT` and `INSERT OVERWRITE [LOCAL] DIRECTORY ... ROW FORMAT DELIMITED`.
`SELECT TRANSFORM` already honours the clause through its own path
(SPARK-36156) and is unchanged.
### Why are the changes needed?
Spark accepts `NULL DEFINED AS` and silently drops it, so the same DDL over
the same files reads differently in Spark and Hive. With the repro from the
JIRA (`FIELDS TERMINATED BY '|' NULL DEFINED AS 'NA'`, data `2|NA`), the Hive
SerDe table's storage properties lack the null format and the row reads `'NA'`
instead of `NULL`.
This is an omission. SPARK-5824 (1.3) stored the null format for CTAS.
SPARK-6908 (1.4) dropped that path, and the TODO has been carried through every
refactor since (SPARK-7411, SPARK-14213).
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix.
- Hive-format tables and directories created with `NULL DEFINED AS 'x'` now
get `serialization.null.format=x`, so reads treat `x` as NULL and writes emit
`x`, as in Hive.
- Statements without the clause are unchanged.
- Existing tables' metadata is not modified (`ALTER TABLE ... SET
SERDEPROPERTIES` applies it).
- The value is unescaped like the other delimiter literals: `'\N'` is stored
as `N`, and `'\\N'` gives a literal `\N`.
### How was this patch tested?
- `PlanResolutionSuite` "create table(hive) - row format" previously
asserted a property map without the null-format key. It now expects
`serialization.null.format -> c`.
- New `DDLParserSuite` test covering `CREATE` and `REPLACE` with five clause
shapes, including `''`, `'\N'` and `'\\N'`.
- Both tests fail with the `AstBuilder` change reverted.
Run locally with JDK 17:
```
build/sbt catalyst/scalastyle catalyst/Test/scalastyle sql/scalastyle
sql/Test/scalastyle -> 0 errors
build/sbt "catalyst/testOnly *DDLParserSuite *PlanParserSuite"
-> 283 passed
build/sbt "sql/testOnly *PlanResolutionSuite *SparkSqlParserSuite
*SparkScriptTransformationSuite" -> 167 passed
build/sbt "sql/testOnly *SQLQueryTestSuite -- -z transform.sql"
-> 2 passed
```
The Hive module suites were not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
