GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/2823
[CARBONDATA-3015] Support Lazy load in carbon vector
This PR depends on PR https://github.com/apache/carbondata/pull/2822
Even though we prune the pages as per min/max there is a high chance of
false positives in case of filters on high cardinality columns. So to avoid
that we can use the lazy loading design. It does not read/decompresses data and
fill the vector immediately when the call comes for data filling from
spark/presto.
First only reads the required filter columns give back to execution engine,
execution engine starts filtering on the filtered column vector and if it finds
some data need to be read from projection columns then only it starts reads the
projection columns and fills the vector on demand. It is the concept of presto
and same is integrated with spark 2.3. Older versions of spark cannot use this
advantage as ColumnVector interfaces are non-extendable.
For the above purpose added new classes 'LazyBlockletLoad' and
'LazyPageLoad' and changed the carbon vector interfaces.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance
test report.
- Any additional information to help reviewers in testing this
change.
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata perf-lazy-load
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2823.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2823
----
commit 5443ce85a91c4ce28454d73699d62ef47255a845
Author: ravipesala <ravi.pesala@...>
Date: 2018-10-16T05:02:18Z
Add carbon property to configure vector based row pruning push down
commit d024ccac82979a908a643b7198ba1f80d9c08e91
Author: ravipesala <ravi.pesala@...>
Date: 2018-10-16T06:00:43Z
Added support for full scan queries for vector direct fill.
commit 77203f344dbe9dca8ce2697a0dbc364bb8a01cdc
Author: ravipesala <ravi.pesala@...>
Date: 2018-10-16T09:23:14Z
Added support for pruning pages for vector direct fill.
commit 46578850c5c02c4927da32efe7a1df27cfe2ebd1
Author: ravipesala <ravi.pesala@...>
Date: 2018-10-16T11:07:18Z
Added support for inverted index and delete delta for direct scan queries
commit b08c254280198fbbcb41361c7b32a9eb51cf8473
Author: ravipesala <ravi.pesala@...>
Date: 2018-10-16T13:09:16Z
Support Lazy load in carbon vector
----
---
