Linhong Liu created SPARK-61555:
-----------------------------------

             Summary: Single-column CSV write drops rows whose only value is 
null; to_csv of a one-field null struct fails
                 Key: SPARK-61555
                 URL: https://issues.apache.org/jira/browse/SPARK-61555
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Linhong Liu


A CSV record whose only value is a null written as an empty token becomes an 
empty line, which the univocity writer drops (CSVOptions.asWriterSettings sets 
setSkipEmptyLines(true)). A single-column CSV write therefore silently loses 
its null rows, and to_csv of a one-field struct holding null returns null / 
throws NullPointerException.

Fix: write such a record as a quoted empty string ("") so it reads back as 
null, and make to_csv return "" for any record the writer would drop as an 
empty line. A new legacy conf 
spark.sql.legacy.csv.skipSingleNullValueRecordOnWrite (default false) restores 
the previous behavior.

Affects both Spark and Photon execution; found via correctness analysis.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to