GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1089
[CARBONDATA-1224] Added page level reader instead of reading whole blocklet
in V3.
Problem
In V3 format we read the whole blocklet at once to memory in order save IO
time. But it turns out to be costlier in case of parallel reading of more
carbondata files.
For example if we need to compact 50 segments then compactor need to open
the readers on all the 50 segments to do merge sort. But the memory consumption
is too high if each reader reads whole blocklet to the memory and there is high
chances of going out of memory.
Solution:
In this type of scenarios we can introduce new readers for V3 to read the
data page by page instead of reading whole blocklet at once to reduce the
memory footprint.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata
pagelevel-reader
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1089.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1089
----
commit 82e40556cbc9ff7fa872cfc0f39ba6998965acc2
Author: ravipesala <[email protected]>
Date: 2017-06-24T02:46:07Z
Added page by page reader instead of reading whole blocklet.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
