[
https://issues.apache.org/jira/browse/SPARK-15982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shixiong Zhu resolved SPARK-15982.
----------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
> Harmonize the behavior of DataFrameReader.text/csv/json/parquet/orc
> -------------------------------------------------------------------
>
> Key: SPARK-15982
> URL: https://issues.apache.org/jira/browse/SPARK-15982
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Tathagata Das
> Assignee: Tathagata Das
> Fix For: 2.0.0
>
>
> Issues with current reader behavior.
> - `text()` without args returns an empty DF with no columns -> inconsistent,
> its expected that text will always return a DF with `value` string field,
> - `textFile()` without args fails with exception because of the above reason,
> it expected the DF returned by `text()` to have a `value` field.
> - `orc()` does not have var args, inconsistent with others
> - `json(single-arg)` was removed, but that caused source compatibility issues
> - SPARK-16009
> - user specified schema was not respected when text/csv/... were used with no
> args - SPARK-16007
> The solution I am implementing is to do the following.
> 1. For each format, there will be a single argument method, and a vararg
> method. For json, parquet, csv, text, this means adding json(string), etc..
> For orc, this means adding orc(varargs).
> 2. Remove the special handling of text(), csv(), etc. that returns empty
> dataframe with no fields. Rather pass on the empty sequence of paths to the
> datasource, and let each datasource handle it right. For e.g, text data
> source, should return empty DF with schema (value: string)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
