GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/13890
[SPARK-16189][SQL] Add ExistingRDD logical plan for input with RDD to have
a chance to eliminate serialize/deserialize.
## What changes were proposed in this pull request?
Currently the input `RDD` of `Dataset` is always serialized to
`RDD[InternalRow]` prior to being as `Dataset`, but there is a case that we use
`map` or `mapPartitions` just after converted to `Dataset`.
In this case, serialize and then deserialize happens but it would not be
needed.
This pr adds `ExistingRDD` logical plan for input with `RDD` to have a
chance to eliminate serialize/deserialize.
## How was this patch tested?
Existing tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ueshin/apache-spark issues/SPARK-16189
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13890.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13890
----
commit 036d5fd944a2d91c17ab324ff45ceb50c8301eb2
Author: Takuya UESHIN <[email protected]>
Date: 2016-06-14T06:25:35Z
Add ExistingRDD logical plan for input with RDD to have a chance to
eliminate serialize/deserialize.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
