GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/13604
[SPARK-15856][SQL] Revert API breaking changes made in DataFrameReader.text
and SQLContext.range
## What changes were proposed in this pull request?
In Spark 2.0, after unifying Datasets and DataFrames, we made two API
breaking changes:
1. DataFrameReader.text() now returns Dataset[String] instead of DataFrame
2. SQLContext.range() now returns Dataset[java.lang.Long] instead of
DataFrame
However, these two changes introduced several inconsistencies and problems:
1. spark.read.text() silently discards partitioned columns when reading a
partitioned table in text format since Dataset[String] only contains a single
field. Users have to use spark.read.format("text").load() to workaround this,
which is pretty confusing and error-prone.
2. All data source shortcut methods in `DataFrameReader` return DataFrame
(aka Dataset[Row]) except for DataFrameReader.text().
3. When applying typed operations over Datasets returned by spark.range(),
weird schema changes may happen. Please refer to SPARK-15632 for more details.
Due to these reasons, we decided to revert these two changes.
affected PRs:
https://github.com/apache/spark/pull/11730
https://github.com/apache/spark/pull/11731
https://github.com/apache/spark/pull/13104
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark revert
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13604.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13604
----
commit 788dd679fbf4cedfa72b84c56a0f005c908fa38b
Author: Wenchen Fan <[email protected]>
Date: 2016-06-10T17:38:31Z
revert API breaking changes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
