GitHub user saucam opened a pull request:
https://github.com/apache/spark/pull/2561
SPARK-3711: Optimize where in clause filter queries
The In case class is replaced by a InSet class in case all the filters are
literals, which uses a hashset instead of Sequence, thereby giving significant
performance improvement (earlier the seq was using a worst case linear match
(exists method) since expressions were assumed in the filter list) . Maximum
improvement should be visible in case small percentage of large data matches
the filter list.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/saucam/spark branch-1.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2561.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2561
----
commit bee98aadcea7cb8fa6402d72af45aef2a4de8c19
Author: Yash Datta <[email protected]>
Date: 2014-09-28T05:54:49Z
SPARK-3711: Optimize where in clause filter queries
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
