[
https://issues.apache.org/jira/browse/SPARK-31747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
K Jaya Shankar Reddy updated SPARK-31747:
-----------------------------------------
Description:
There are CSV files out there with multiple values that are supposed to be
interpreted as null. Currently the `nullValue` option of spark.read is
accepting only one value.
Can we come with something like below:
{code:java}
spark.read.option("nullValue1", "abc").option("nullValue2", "def").csv("1.csv")
{code}
Or
{code:java}
spark.read.option("nullValue", Seq("abc","def")).csv("1.csv") {code}
Where the values "abc" and "def" are to be interpreted as NULLs.
was:
There are CSV files out there with multiple values that are supposed to be
interpreted as null. Currently the `nullValue` option of spark.read is
accepting only one value.
Can we come with something like below:
{code:java}
spark.read.option("nullValue1", "abc").option("nullValue2", "def").csv("1.csv")
{code}
Or
spark.read.option("nullValue", Seq("abc","def")).csv("1.csv")
Where the values "abc" and "def" are to be interpreted as NULLs.
> Support for multiple null values when reading CSV data
> ------------------------------------------------------
>
> Key: SPARK-31747
> URL: https://issues.apache.org/jira/browse/SPARK-31747
> Project: Spark
> Issue Type: Story
> Components: SQL
> Affects Versions: 2.3.2
> Reporter: K Jaya Shankar Reddy
> Priority: Major
>
> There are CSV files out there with multiple values that are supposed to be
> interpreted as null. Currently the `nullValue` option of spark.read is
> accepting only one value.
> Can we come with something like below:
> {code:java}
> spark.read.option("nullValue1", "abc").option("nullValue2",
> "def").csv("1.csv")
> {code}
> Or
> {code:java}
> spark.read.option("nullValue", Seq("abc","def")).csv("1.csv") {code}
> Where the values "abc" and "def" are to be interpreted as NULLs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
