Hossein Falaki commented on SPARK-17878:

I think moving it to another ticket is a good idea. One concern is that the API 
would be different between Scala and DDL.

> Support for multiple null values when reading CSV data
> ------------------------------------------------------
>                 Key: SPARK-17878
>                 URL: https://issues.apache.org/jira/browse/SPARK-17878
>             Project: Spark
>          Issue Type: Story
>          Components: SQL
>    Affects Versions: 2.0.1
>            Reporter: Hossein Falaki
> There are CSV files out there with multiple values that are supposed to be 
> interpreted as null. As a result, multiple spark users have asked for this 
> feature built into the CSV data source. It can be easily implemented in a 
> backwards compatible way:
> - Currently CSV data source supports an option named {{nullValue}}.
> - We can add logic in {{CSVOptions}} to understands option names that match 
> {{nullValue[\d]}}. This way user can specify a query with multiple or one 
> null value.
> {code}
> val df = spark.read.format("CSV").option("nullValue1", 
> "-").option("nullValue2", "*")....
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to