GitHub user ravipesala opened a pull request:
https://github.com/apache/carbondata/pull/1471
[WIP] Datamap FineGrain implementation
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
Implemented interfaces for FG datamap and integrated to filterscanner to
use the pruned bitset from FG datamap.
FG Query flow as follows.
1.The user can add FG datamap to any table and implement there interfaces.
2. Any filter query which hits the table with datamap will call prune
method of FGdatamap.
3. The prune method of FGDatamap return list FineGrainBlocklet , these
blocklets contain the information of block, blocklet, page and rowids
information as well.
4. The pruned blocklets are internally wriitten to file and returns only
the block , blocklet and filepath information as part of Splits.
5. Based on the splits scanrdd schedule the tasks.
6. In filterscanner we check the datamapwriterpath from split and reNoteads
the bitset if exists. And pass this bitset as input to it.
NOTE : This PR depends on https://github.com/apache/carbondata/pull/1410
- [X] Any interfaces changed?
Few changes to datamap interfaces only.
- [X] Any backward compatibility impacted? NO
- [X] Document update required?
Yes, Required to add to dev guide about interfaces
- [X] Testing done
Tests are added for both CG and FG
- [X] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ravipesala/incubator-carbondata
datamap-fg-impl
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1471.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1471
----
commit 8d727ffbc465191040274318a55dd0a69063bdfd
Author: ravipesala <[email protected]>
Date: 2017-10-11T13:37:22Z
Added FG interfaces
commit 0cc36e4b28d2747728013116cdacecb80beb0239
Author: ravipesala <[email protected]>
Date: 2017-10-17T12:23:09Z
Refactored code to remove path from blocklet
commit 3c204c2d2e69512f21f630a091df756255ccd6e8
Author: ravipesala <[email protected]>
Date: 2017-10-31T09:55:57Z
fgdatamap implement
commit 717ba3e41a6f3d1936ff14857a9b992cff13c8a6
Author: ravipesala <[email protected]>
Date: 2017-11-03T16:52:33Z
Added test for FGDatamap
commit e6093540c800c2e86c472787603c884a72e1d129
Author: ravipesala <[email protected]>
Date: 2017-11-04T06:21:21Z
Added testcase
----
---
