GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/21319
[SPARK-24267][SQL] explicitly keep DataSourceReader in DataSourceV2Relation
## What changes were proposed in this pull request?
To keep `DataSourceV2Relation` immutable, we don't put the
`DataSourceReader` in the constructor, but make it a `lazy val` instead. If we
think about how `lazy val` is implemented in Scala, we actually keep a
`DataSourceReader` instance in `DataSourceV2Relation`, and exclude it when
defining equality of `DataSourceV2Relation`.
This works, but have 2 problems:
1. after the pushdown rule, if `DataSourceV2Relation` get transformed and
return a new copy, we will re-do the pushdown and re-create the
`DataSourceReader`.
2. the pushdown logic is defined in `DataSourceV2Relation` instead of the
pushdown rule, which is a little counter-intuitive.
This PR proposes to implement the `lazy val` by ourselves: keep
`DataSourceReader` as an optional parameter in the constructor of
`DataSourceV2Relation`, exclude it in the equality definition but include it
when copying.
## How was this patch tested?
existing tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark refactor
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21319.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21319
----
commit ca6ccb24e8f2910a3ffc07a790f2ba7f57e79056
Author: Wenchen Fan <wenchen@...>
Date: 2018-04-24T17:26:19Z
do not create DataSourceReader many times
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
