Hyukjin Kwon created SPARK-14800:
------------------------------------
Summary: Dealing with null as a value in options for each internal
data source
Key: SPARK-14800
URL: https://issues.apache.org/jira/browse/SPARK-14800
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 2.0.0
Reporter: Hyukjin Kwon
Priority: Minor
This was found in https://github.com/apache/spark/pull/12494.
Currently, most of options in data sources throws {{NullPointerException}} when
given value is {{null}}.
For example, the codes below:
{code}
sqlContext.read
.format("csv")
.option("compression", null)
{code}
throws an exception below:
{code}
java.lang.NullPointerException was thrown.
java.lang.NullPointerException
at
org.apache.spark.sql.execution.datasources.csv.CSVOptions.getChar(CSVOptions.scala:32)
at
org.apache.spark.sql.execution.datasources.csv.CSVOptions.<init>(CSVOptions.scala:68)
at
org.apache.spark.sql.execution.datasources.csv.DefaultSource.prepareWrite(DefaultSource.scala:86)
...
{code}
while some of options such as {{nullValue}} in CSV data source accepts {{null}}
.
So, the options throwing {{NullPointerException}} when it sets to {{null}}
might have to be handled separately for each data source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
