GitHub user ron8hu opened a pull request:
https://github.com/apache/spark/pull/16334
estimate filter cardinality
## What changes were proposed in this pull request?
This is a WIP PR. In this version, we set up the framework to traverse
predicate and evaluate the equality (=) expression.
## How was this patch tested?
We just have a simple test case for now. More tests need to be added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ron8hu/spark filterEstimate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16334.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16334
----
commit f0bbb43f7b27c59a3181ce428ed2bb0a7c1fc89d
Author: wangzhenhua <[email protected]>
Date: 2016-12-09T06:34:41Z
estimation for inner join
commit 64603b589bd1b971d2370d848f1dd19f11b52928
Author: wangzhenhua <[email protected]>
Date: 2016-12-09T07:30:33Z
wrong position of comments
commit 02bdf875babac7676a732a01fa41c8b8416a58d3
Author: Ron Hu <[email protected]>
Date: 2016-12-18T02:10:28Z
implemented first version of filter cardinality estimation
commit 6c9d06085f30dd19eb8b0a06f62742bda6ace736
Author: Ron Hu <[email protected]>
Date: 2016-12-18T02:15:57Z
maintained a mutable colStats in Filter class
commit 9e92b4b4c5033441f63e0e76544a1d609ac2a39e
Author: Ron Hu <[email protected]>
Date: 2016-12-19T00:57:00Z
Used mutableColStats to reflect changes for column stats
commit 8e77acc8630df3696dbb12453ee3f469c03b1193
Author: Ron Hu <[email protected]>
Date: 2016-12-19T02:15:41Z
use getOrElse to return filter stats
commit 62315985cc6ec20fb26c8ed109b33160e9f3f80b
Author: Ron Hu <[email protected]>
Date: 2016-12-19T03:27:40Z
added the test case
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
