Hi, Yuzhang:
I found out the reason why if includes = {D1, … , D10}, and mandatory = {D1, …,
D9}, then we only get one cuboid as {D1, … D10}, and {D1, …, D9} is NOT
generated.
It is caused by this code in
“core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java”:
public boolean getCubeAggrGroupIsMandatoryOnlyValid() {
return
Boolean.parseBoolean(getOptional("kylin.cube.aggrgroup.is-mandatory-only-valid",
"false"));
}
In kylin.properties, we did not config this parameter
“kylin.cube.aggrgroup.is-mandatory-only-valid" as “true”, and by default, it is
set to “false”. So {D1, …, D9} is so-called “mandatory-only”, and treated as
not valid.
Kang-sen
From: Lu, Kang-Sen <[email protected]>
Sent: Wednesday, March 20, 2019 8:32 AM
To: [email protected]; [email protected]
Subject: RE: question related to the aggregation groups configuration
________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________
Hi, Yuzhang:
Thank you for taking the time to respond. I did read this requirement for
“mandatory dimension”: ("if a dimension is specified as “mandatory”, then all
of the combinations without such dimension can be pruned"). That is the key
point information.
BTW: I am curious if there is an easy way to find out how many cuboids are
generated by kylin and every cuboid’s dimension set from kylin’s metadata. Your
finding is what I suspected. But I am not able to verify it as you did.
We can live with this fact as is, if it is documented. But it would be better
to fix the bug and allow the original description of mandatory stand as correct.
About Q2, I read the link you mentioned, it seems if hierarchy and joint are
both specified, then the joint is being treated as tag-alone restriction, say
D2 is in hierarchy and became “mandatory” in cuboids, if joint says D2 and D3
must be together, then D2 will pull D3 into the “mandatory” list. That is
elegant.
I am wondering why these three selection-rules can NOT be applied
independently. If we have D1-to-D10 in the includes set. Then the number of all
combination of D1-to-D10 is factorial(10). Now, we can apply “mandatory” to
“prune” some of the combination out. After that, we may further prune by
applying the hierarchy and joint rules. Isn’t it possible?
Thanks.
Kang-sen
From: yuzhang <[email protected]<mailto:[email protected]>>
Sent: Wednesday, March 20, 2019 1:00 AM
To: [email protected]<mailto:[email protected]>;
[email protected]<mailto:[email protected]>
Subject: Re: question related to the aggregation groups configuration
________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________
Hi kang-sen:
I do some test about Q1, {D1 to D10} have been included in an aggregation
group and {D1 to D9} have been added into mandatory dimension. Then kylin only
generates Cuboid {D1 to D10}(base Cuboid) which I expect {D1 to D10} and {D1 to
D9}. When I add {D1 to D8} in to mandatory dimension, kylin generates Cuboid
{D1 to D10}, {D1 to D8, D9} and {D1 to D8, D10} which I expect {D1 to D10}, {D1
to D8, D9}, {D1 to D8, D10} and {D1 to D8}. About your Q1, I think the answer
is ONLY ONE cuboid {D1 to D10} has been generated. But according the blog ("if
a dimension is specified as “mandatory”, then all of the combinations without
such dimension can be pruned"), the Cuboid {D1 to D9} should't been pruned.
Maybe someone else can give more detail.
Q2 is similar with this email
https://lists.apache.org/thread.html/3ccc8d7f98748d7c590c01c7da6ce666a16c4fe2b34be070940cae8f@%3Cuser.kylin.apache.org%3E<https://lists.apache.org/thread.html/3ccc8d7f98748d7c590c01c7da6ce666a16c4fe2b34be070940cae8f@%3Cuser.kylin.apache.org%3E>
and jira
https://issues.apache.org/jira/browse/KYLIN-2149<https://issues.apache.org/jira/browse/KYLIN-2149>
. Now kylin will prevent config overlapping hierachy, mandatory and joint.
Although the minds of three aggregation rule are different and even
contradictory, auto merging those rules into Cuboids is feasible. For now, the
restriction of aggregation group can't realize your requirement which I think
is common. May be the jira KYLIN-2149 can be resolved in the future.
Best regards
yuzhang
[Image removed by sender.]
yuzhang
[email protected]<mailto:[email protected]>
签名由 网易邮箱大师<https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
On 3/19/2019 23:09，Lu, Kang-Sen<[email protected]><mailto:[email protected]> wrote：
Hi, Yuzhang:
I would appreciate if you can provide answer to my 2 questions.
Thanks.
Kang-sen
From: Lu, Kang-Sen <[email protected]<mailto:[email protected]>>
Sent: Friday, March 15, 2019 8:15 AM
To: [email protected]<mailto:[email protected]>
Subject: RE: question related to the aggregation groups configuration
________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________
Hi, Yuzhang:
Thanks for taking time to reply.
I actually have read that article several times earlier before.
However, may be I missed some details or what, I am not clear about how those
rules actual work and how they interfere with each other.
From the article you pointed out, the hierarchy rule does have an example, so
it is less likely to be confused.
I did not find any discussion about the “mandatory rule”. It is supposed to be
very simple, but I am stuck by the details. Let’s say, “includes” is a set of
dim: { d1, d2, … d10}, and the “mandatory” is a set of dim: {D1, …, D9}.
So it is obvious that each cuboid generated from this agg group should all
include set of dim {D1, …, D9}.
Now, D10 could be either selected or not. So the natural guess is that this agg
group will generate two cuboids, i.e {D1,…,D9} and {D1,… D10}. Is this what
kylin will do?
Another detail I am not clear is the interaction of “joint rule” and the
“mandatory rule”. It seems that there is an interaction between these two
rules. I am not clear why, and it is not discussed in the article you mentioned.
That was my two original questions.
Thanks again.
Kang-sen
From: yuzhang <[email protected]<mailto:[email protected]>>
Sent: Friday, March 15, 2019 7:46 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: question related to the aggregation groups configuration
________________________________
NOTICE: This email was received from an EXTERNAL sender
________________________________
Hi kang-sen:
Here is a blog about the mind of aggregation group. I hope it will help you.
https://kylin.apache.org/blog/2016/02/18/new-aggregation-group/<https://kylin.apache.org/blog/2016/02/18/new-aggregation-group/>
Best regards
yuzhang
[Image removed by sender.]
yuzhang
[email protected]<mailto:[email protected]>
签名由 网易邮箱大师<https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
On 3/14/2019 21:21，Lu, Kang-Sen<[email protected]><mailto:[email protected]> wrote：
I am running kylin 2.5.1
I have two questions related to the aggregation group configuration. In the
kylin GUI, select “Model”, then try to edit a cube design, under “Grid”, select
“Advanced Setting”, we can enter multiple “Aggregation Groups”. Each
“Aggregation Group” can specify zero, one, or many cuboids, with the
combination of dimensions.
Q1: If I want one and only one cuboid to be created with dimensions set = {D1,
D2, … , D10}, then is it correct to enter D1-to-D10 in the “includes” list, and
“D1-to-D9 in the “Mandatory Dimensions” list? The key question is “will kylin
generate two cuboids, i.e. {D1, …, D9} and {D1, … , D10} or just one cuboid”?
Q2: If I entered D1-to-D10 into the “includes” list, and entered {D1, D2} in
the “Joint Dimensions” list, then I can’t enter either D1 or D2 into the
“Mandatory Dimensions” list? I was thinking if I entered {D1, D3, … , D9} in
the “Mandatory Dimensions”, and with {D1, D2} in the “Joint Dimensions”, then
there should only one cuboid generated for {D1, D2, …, D10}. Why is it not
allowed?
Maybe the doc have this information described. But it is not clear to me
exactly how does kylin process the info entered in the “includes”, “Mandatory
Dimensions”, and “Joint Dimensions”. Can someone either point me to some
document or answer the questions I mentioned above.
Thanks.
Kang-sen
________________________________
Notice: This e-mail together with any attachments may contain information of
Ribbon Communications Inc. that is confidential and/or proprietary for the sole
use of the intended recipient. Any review, disclosure, reliance or distribution
by others or forwarding without express permission is strictly prohibited. If
you are not the intended recipient, please notify the sender immediately and
then delete all copies, including any attachments.
________________________________
