GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/11388
[SPARK-13457][SQL] Removes DataFrame RDD operations
## What changes were proposed in this pull request?
This is another try of PR #11323.
This PR removes DataFrame RDD operations except for `foreach` and
`foreachPartitions` (they are actions rather than transformations). Original
calls are now replaced by calls to methods of `DataFrame.rdd`.
PR #11323 was reverted because it introduced a regression: both
`DataFrame.foreach` and `DataFrame.foreachPartitions` wrap underlying RDD
operations with `withNewExecutionId` to track Spark jobs. But they are removed
in #11323.
## How was the this patch tested?
No extra tests are added. Existing tests should do the work.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark remove-df-rdd-ops
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11388.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11388
----
commit 7738d532f89302f1c3fa69537638c3677fa3f74e
Author: Cheng Lian <[email protected]>
Date: 2016-02-23T16:03:10Z
Removes DataFrame RDD operations
commit 406427ef48724f6ab2f6e1ecdf7511f06a075a0f
Author: Cheng Lian <[email protected]>
Date: 2016-02-24T01:15:56Z
Fixes styling issues
commit d530c758912241aebb979724476164df12418a7e
Author: Cheng Lian <[email protected]>
Date: 2016-02-25T08:36:52Z
Fixes compilation error introduced after rebasing
commit 2b4a95b54b985ee09997be9e047fe35213a7f481
Author: Cheng Lian <[email protected]>
Date: 2016-02-26T08:02:59Z
Don't remove foreach and foreachPartitions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
