[
https://issues.apache.org/jira/browse/SPARK-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Reynold Xin updated SPARK-15585:
--------------------------------
Description:
See email:
http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
We'd need to change DataFrameReader/DataFrameWriter in Python's
csv/json/parquet/... functions to put the actual default option values as
function parameters, rather than setting them to None. We can then in
CSVOptions.getChar (and JSONOptions, etc) to actually return null if the value
is null, rather than setting it to default value.
was:
See email:
http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
We'd need to change DataFrameReader/DataFrameWriter in Python's
csv/json/parquet/... functions to put the actual default option values as
function parameters, rather than setting them to None.
> Don't use null in data source options to indicate default value
> ---------------------------------------------------------------
>
> Key: SPARK-15585
> URL: https://issues.apache.org/jira/browse/SPARK-15585
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Reynold Xin
> Priority: Critical
>
> See email:
> http://apache-spark-developers-list.1001551.n3.nabble.com/changed-behavior-for-csv-datasource-and-quoting-in-spark-2-0-0-SNAPSHOT-td17704.html
> We'd need to change DataFrameReader/DataFrameWriter in Python's
> csv/json/parquet/... functions to put the actual default option values as
> function parameters, rather than setting them to None. We can then in
> CSVOptions.getChar (and JSONOptions, etc) to actually return null if the
> value is null, rather than setting it to default value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
