[
https://issues.apache.org/jira/browse/KYLIN-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041340#comment-15041340
]
hongbin ma commented on KYLIN-242:
----------------------------------
The feature is accomplished by a new design of aggregation groups. In the new
design, cubes designers are allowed to specify one or more Aggregation
Groups(briefed as AGG), each of which is essentially a selective partial cube.
An AGG can contain as few as 1 single cuboid, making it possible to specify a
white list by specifying multiple AGGs.
An AGG firstly scopes its dimensions, For example, suppose a cube has 10
dimensions, an AGG may select only 6 dimensions from it, all the cuboids
spawned by this AGG will NOT contain dimensions other than these 6 dimensions.
By amortizing dimensions to different AGGs we're effectively relieving the
problem of "curse of dimensions"
Within an AGG's included dimensions, there're three kinds of Select Rules to
further prune the spanning tree:
1. Mandatory, cube designers can set any number this AGG's included dimensions
as "Mandatory", this will make this AGG's spawning cuboids always contain the
mandatory columns. A valid Mandatory rule could be: [M,N]
2. Hierarchy, cube designers can set any sets of hierarchy dimensions. If
dimension [A,B,C] is a set of hierarchy dimensions, then only cuboid with (A)
or (A,B) or (A,B,C) or () will be spawned. Notice there can be more than one
sets of hierarchy dimensions in an AGG, however any two sets should not
interleave. So a valid hierarchy rule could be: [[A,B,C],[D,E,F]]
3. Joint, this is a new introduced feature, if (G,H) are specified as joint,
the in any cuboids spawned by the AGG, dimension G will always co-exist with
dimension H, in other words, only cuboid with (G,H) or () will be spawned.
Notice there can be more than one sets of joint dimensions in an AGG, however
any two sets should not interleave. So a valid hierarchy rule could be:
[[G,H],[I,J,K]]
To make things simple, within a AGG, dimensions belonging to different Select
Rules should NOT interleave.
> Kylin Cuboid Whitelist
> ----------------------
>
> Key: KYLIN-242
> URL: https://issues.apache.org/jira/browse/KYLIN-242
> Project: Kylin
> Issue Type: Improvement
> Components: General
> Reporter: Luke Han
> Assignee: hongbin ma
> Labels: github-import
> Fix For: v2.1
>
>
> Proposal from Hongbin about Cuboid White List:
> * More detail, please refer to Kylin Dev mailing list:
> [email protected]
> Logically, a cube contains cuboids representing all combinations of
> dimensions. Apparently, a naive cube building strategy that materializes
> all cuboids will easily meet curse-of-dimension problems. Currently Kylin
> leverages a strategy called "aggregation groups" to reduce the number of
> cuboids need being materialized.
> However, if the query pattern is simple and fixed, the "aggregation group"
> strategy is still not efficient enough. For example, suppose there're five
> dimensions, namely A,B,C,D and E. The data modeler is sure that only
> combinations (A,B,C), (D,E), (A,E) will be queried, so he’ll use the
> aggregation group tool to optimize his cube definition. However, whatever
> aggregation group he chooses, lots of useless combinations would be
> materialized.
> With a new strategy called "cuboid whitelist", data modelers can guide
> Kylin to only materialize the cuboids he's interested in. Depending on the
> whitelist, Kylin will materialize the minimal set of cuboids to cover each
> cuboid in the whitelist. To support this, the following functionalities
> should be added:
> 1. Front-end/UI for specifying whitelist members, and persistent them to
> cube description.
> 2. Enhanced job engine scheduler that will calculate a minimal spanning
> build tree based on the whitelist.
> 3. (OPTIONAL) Enhanced job engine to support dynamic whitelist, trigger new
> builds for lately added whitelist members.
> Hongbin Ma
> ---------------- Imported from GitHub ----------------
> Url: https://github.com/KylinOLAP/Kylin/issues/263
> Created by: [lukehan|https://github.com/lukehan]
> Labels: enhancement,
> Milestone: Backlog
> Assignee: [binmahone|https://github.com/binmahone]
> Created at: Thu Dec 25 13:17:11 CST 2014
> State: open
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)