[
https://issues.apache.org/jira/browse/CARBONDATA-4162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal kumar ojha updated CARBONDATA-4162:
-----------------------------------------
Summary: Leverage Secondary Index till segment level with SI as datamap and
SI with plan rewrite (was: Leverage Secondary Index till segment level with
Spark plan rewrite)
> Leverage Secondary Index till segment level with SI as datamap and SI with
> plan rewrite
> ---------------------------------------------------------------------------------------
>
> Key: CARBONDATA-4162
> URL: https://issues.apache.org/jira/browse/CARBONDATA-4162
> Project: CarbonData
> Issue Type: New Feature
> Reporter: Nihal kumar ojha
> Priority: Major
> Attachments: Support SI at segment level.pdf
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> *Background:*
> Secondary index tables are created as indexes and managed as child tables
> internally by Carbondata. In the existing architecture, if the parent(main)
> table and SI table don’t
> have the same valid segments then we disable the SI table. And then from the
> next query onwards, we scan and prune only the parent table until we trigger
> the next load or REINDEX command (as these commands will make the
> parent and SI table segments in sync). Because of this, queries take more
> time to give the result when SI is disabled.
> *Proposed Solution:*
> We are planning to leverage SI till the segment level. It means at place
> of disabling the SI table(when parent and child table segments are not in
> sync)
> we will do pruning on SI tables for all the valid segments(segments with
> status
> success, marked for update and load partial success) and the rest of the
> segments will be pruned by the parent table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
