GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1789
[WIP] Fix avoid reading of all block information in driver for old stores.
Problem:
For old stores prior to 1.2 version there is no blocklet information stored
in carbonindex file. So the new code needs to read all carbondata files footers
inside the driver to get the blocklet information. That makes the first time
queries become slower. As observed count(*) query was taking 2 swconds on old
version and after upgrade it takes very long time.
Solution:
If there is no information blocklet available in carbonindex file then
don't read carbondata files footer in driver side. Instead read carbondata
files in executor to get the blocklet information.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [X] Any interfaces changed?
- [X] Any backward compatibility impacted?
- [X] Document update required?
- [X] Testing done
- [X] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata
datamap-pld-store
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1789.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1789
----
commit f0aaf4d8d0e4761227bc1adff29d33798c88bd12
Author: ravipesala <ravi.pesala@...>
Date: 2018-01-10T15:35:48Z
Fix avoid reading of all block information in driver for old stores.
----
---
