Hyukjin Kwon commented on SPARK-17878:

Yes, this might not be a reason to block this feature.
It seems not easy and needs to change a lot. Maybe, how about, for example, 
adding {{option(value: List[String])}} and then converting it into JSON array?
Actually, I can move this one into another JIRA as it might not be the reason 
to block this JIRA but just want to hear your thought before I will try.

> Support for multiple null values when reading CSV data
> ------------------------------------------------------
>                 Key: SPARK-17878
>                 URL: https://issues.apache.org/jira/browse/SPARK-17878
>             Project: Spark
>          Issue Type: Story
>          Components: SQL
>    Affects Versions: 2.0.1
>            Reporter: Hossein Falaki
> There are CSV files out there with multiple values that are supposed to be 
> interpreted as null. As a result, multiple spark users have asked for this 
> feature built into the CSV data source. It can be easily implemented in a 
> backwards compatible way:
> - Currently CSV data source supports an option named {{nullValue}}.
> - We can add logic in {{CSVOptions}} to understands option names that match 
> {{nullValue[\d]}}. This way user can specify a query with multiple or one 
> null value.
> {code}
> val df = spark.read.format("CSV").option("nullValue1", 
> "-").option("nullValue2", "*")....
> {code}

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to