Linhong Liu created SPARK-61555:
-----------------------------------
Summary: Single-column CSV write drops rows whose only value is
null; to_csv of a one-field null struct fails
Key: SPARK-61555
URL: https://issues.apache.org/jira/browse/SPARK-61555
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Linhong Liu
A CSV record whose only value is a null written as an empty token becomes an
empty line, which the univocity writer drops (CSVOptions.asWriterSettings sets
setSkipEmptyLines(true)). A single-column CSV write therefore silently loses
its null rows, and to_csv of a one-field struct holding null returns null /
throws NullPointerException.
Fix: write such a record as a quoted empty string ("") so it reads back as
null, and make to_csv return "" for any record the writer would drop as an
empty line. A new legacy conf
spark.sql.legacy.csv.skipSingleNullValueRecordOnWrite (default false) restores
the previous behavior.
Affects both Spark and Photon execution; found via correctness analysis.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]