[jira] [Commented] (KYLIN-2773) Should not push down join condition related columns are compatible while not consistent
[
https://issues.apache.org/jira/browse/KYLIN-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226179#comment-16226179
]
Shaofeng SHI commented on KYLIN-2773:
-
This patch isn't in master branch, so it won't be included in 2.2 release.
[~rogershi] Please review this change, and update status in JIRA. Thanks.
> Should not push down join condition related columns are compatible while not
> consistent
> ---
>
> Key: KYLIN-2773
> URL: https://issues.apache.org/jira/browse/KYLIN-2773
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2773.patch
>
>
> For sql,
> {code}
> select PART_DT, META_CATEG_NAME, sum(price)
> from KYLIN_SALES
> INNER JOIN KYLIN_CATEGORY_GROUPINGS ON KYLIN_SALES.LEAF_CATEG_ID =
> KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID
> AND KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
> INNER JOIN KYLIN_CAL_DT ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
> group by PART_DT, META_CATEG_NAME
> order by PART_DT, META_CATEG_NAME
> {code}
> the datatype of KYLIN_SALES.LEAF_CATEG_ID is bigint, while the one of
> KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID is integer.
> Then the plan transformed by kylin is as follows:
> {code}
> OLAPToEnumerableConverter
> OLAPLimitRel(fetch=[5])
> OLAPSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> OLAPAggregateRel(group=[{0, 1}], EXPR$2=[SUM($2)])
> OLAPProjectRel(PART_DT=[$0], META_CATEG_NAME=[$16], PRICE=[$5])
> OLAPJoinRel(condition=[=($0, $19)], joinType=[inner])
> {code}
> {color:#f79232}
> {code}
> OLAPProjectRel(expr#0..19=[{inputs}], PART_DT=[$t0],
> LSTG_FORMAT_NAME=[$t1], SLR_SEGMENT_CD=[$t2], LEAF_CATEG_ID=[$t3],
> LSTG_SITE_ID=[$t4], PRICE=[$t5], SELLER_ID=[$t6], COUNT__=[$t7],
> MIN_PRICE_=[$t8], COUNT_DISTINCT_SELLER_ID_=[$t9],
> USER_DEFINED_FIELD1=[$t10], USER_DEFINED_FIELD3=[$t11], UPD_DATE=[$t12],
> UPD_USER=[$t13], LEAF_CATEG_ID0=[$t14], SITE_ID=[$t15],
> META_CATEG_NAME=[$t16], CATEG_LVL2_NAME=[$t17], CATEG_LVL3_NAME=[$t18])
> OLAPJoinRel(condition=[AND(=($3, $19), =($4, $15))],
> joinType=[inner])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], fields=[[0, 1,
> 2, 3, 4, 5, 6, 7, 8, 9]])
> OLAPProjectRel(expr#0..8=[{inputs}],
> expr#9=[CAST($t4):BIGINT], USER_DEFINED_FIELD1=[$t0],
> USER_DEFINED_FIELD3=[$t1], UPD_DATE=[$t2], UPD_USER=[$t3],
> LEAF_CATEG_ID=[$t4], SITE_ID=[$t5], META_CATEG_NAME=[$t6],
> CATEG_LVL2_NAME=[$t7], CATEG_LVL3_NAME=[$t8], LEAF_CATEG_ID9=[$t9])
> {code}
> {color}
> {code}
> OLAPTableScan(table=[[DEFAULT, KYLIN_CATEGORY_GROUPINGS]],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
> OLAPTableScan(table=[[DEFAULT, KYLIN_CAL_DT]], fields=[[0, 1, 2,
> 3]])
> {code}
> However, what we expect is as follows:
> {code}
> OLAPToEnumerableConverter
> OLAPLimitRel(fetch=[5])
> OLAPSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> OLAPAggregateRel(group=[{0, 1}], EXPR$2=[SUM($2)])
> OLAPProjectRel(expr#0..21=[{inputs}], PART_DT=[$t0],
> META_CATEG_NAME=[$t17], PRICE=[$t4])
> OLAPJoinRel(condition=[=($0, $19)], joinType=[inner])
> {code}
> {color:#f79232}
> {code}
> OLAPJoinRel(condition=[AND(=($2, $10), =($3, $11))],
> joinType=[inner])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], fields=[[0, 1, 2,
> 3, 4, 5, 6, 7, 8, 9]])
> {code}
> {color}
> {code}
> OLAPTableScan(table=[[DEFAULT, KYLIN_CATEGORY_GROUPINGS]],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
> OLAPTableScan(table=[[DEFAULT, KYLIN_CAL_DT]], fields=[[0, 1, 2,
> 3, 4]])
> {code}
> The reason for this difference is as follows:
> * Although we remove the {{JoinPushExpressionsRule}} in {{OLAPTableScan}},
> the method {{RelOptUtil.pushDownJoinConditions()}} is still invoked when
> creating a join in {{SqlToRelConverter}}.
> * In the method of {{RelOptUtil.pushDownJoinConditions()}}, since the
> datatypes of the join related columns are not the same, *cast* function will
> be automatically assigned to KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID. Then a
> {{OLAPProjectRel}} will be introduced.
> In kylin, we don't need {{RelOptUtil.pushDownJoinConditions()}}. Therefore,
> the solution for this is just remove that logic.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2773) Should not push down join condition related columns are compatible while not consistent
[
https://issues.apache.org/jira/browse/KYLIN-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2773:
Fix Version/s: (was: v2.2.0)
v2.3.0
> Should not push down join condition related columns are compatible while not
> consistent
> ---
>
> Key: KYLIN-2773
> URL: https://issues.apache.org/jira/browse/KYLIN-2773
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2773.patch
>
>
> For sql,
> {code}
> select PART_DT, META_CATEG_NAME, sum(price)
> from KYLIN_SALES
> INNER JOIN KYLIN_CATEGORY_GROUPINGS ON KYLIN_SALES.LEAF_CATEG_ID =
> KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID
> AND KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
> INNER JOIN KYLIN_CAL_DT ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
> group by PART_DT, META_CATEG_NAME
> order by PART_DT, META_CATEG_NAME
> {code}
> the datatype of KYLIN_SALES.LEAF_CATEG_ID is bigint, while the one of
> KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID is integer.
> Then the plan transformed by kylin is as follows:
> {code}
> OLAPToEnumerableConverter
> OLAPLimitRel(fetch=[5])
> OLAPSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> OLAPAggregateRel(group=[{0, 1}], EXPR$2=[SUM($2)])
> OLAPProjectRel(PART_DT=[$0], META_CATEG_NAME=[$16], PRICE=[$5])
> OLAPJoinRel(condition=[=($0, $19)], joinType=[inner])
> {code}
> {color:#f79232}
> {code}
> OLAPProjectRel(expr#0..19=[{inputs}], PART_DT=[$t0],
> LSTG_FORMAT_NAME=[$t1], SLR_SEGMENT_CD=[$t2], LEAF_CATEG_ID=[$t3],
> LSTG_SITE_ID=[$t4], PRICE=[$t5], SELLER_ID=[$t6], COUNT__=[$t7],
> MIN_PRICE_=[$t8], COUNT_DISTINCT_SELLER_ID_=[$t9],
> USER_DEFINED_FIELD1=[$t10], USER_DEFINED_FIELD3=[$t11], UPD_DATE=[$t12],
> UPD_USER=[$t13], LEAF_CATEG_ID0=[$t14], SITE_ID=[$t15],
> META_CATEG_NAME=[$t16], CATEG_LVL2_NAME=[$t17], CATEG_LVL3_NAME=[$t18])
> OLAPJoinRel(condition=[AND(=($3, $19), =($4, $15))],
> joinType=[inner])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], fields=[[0, 1,
> 2, 3, 4, 5, 6, 7, 8, 9]])
> OLAPProjectRel(expr#0..8=[{inputs}],
> expr#9=[CAST($t4):BIGINT], USER_DEFINED_FIELD1=[$t0],
> USER_DEFINED_FIELD3=[$t1], UPD_DATE=[$t2], UPD_USER=[$t3],
> LEAF_CATEG_ID=[$t4], SITE_ID=[$t5], META_CATEG_NAME=[$t6],
> CATEG_LVL2_NAME=[$t7], CATEG_LVL3_NAME=[$t8], LEAF_CATEG_ID9=[$t9])
> {code}
> {color}
> {code}
> OLAPTableScan(table=[[DEFAULT, KYLIN_CATEGORY_GROUPINGS]],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
> OLAPTableScan(table=[[DEFAULT, KYLIN_CAL_DT]], fields=[[0, 1, 2,
> 3]])
> {code}
> However, what we expect is as follows:
> {code}
> OLAPToEnumerableConverter
> OLAPLimitRel(fetch=[5])
> OLAPSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> OLAPAggregateRel(group=[{0, 1}], EXPR$2=[SUM($2)])
> OLAPProjectRel(expr#0..21=[{inputs}], PART_DT=[$t0],
> META_CATEG_NAME=[$t17], PRICE=[$t4])
> OLAPJoinRel(condition=[=($0, $19)], joinType=[inner])
> {code}
> {color:#f79232}
> {code}
> OLAPJoinRel(condition=[AND(=($2, $10), =($3, $11))],
> joinType=[inner])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], fields=[[0, 1, 2,
> 3, 4, 5, 6, 7, 8, 9]])
> {code}
> {color}
> {code}
> OLAPTableScan(table=[[DEFAULT, KYLIN_CATEGORY_GROUPINGS]],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
> OLAPTableScan(table=[[DEFAULT, KYLIN_CAL_DT]], fields=[[0, 1, 2,
> 3, 4]])
> {code}
> The reason for this difference is as follows:
> * Although we remove the {{JoinPushExpressionsRule}} in {{OLAPTableScan}},
> the method {{RelOptUtil.pushDownJoinConditions()}} is still invoked when
> creating a join in {{SqlToRelConverter}}.
> * In the method of {{RelOptUtil.pushDownJoinConditions()}}, since the
> datatypes of the join related columns are not the same, *cast* function will
> be automatically assigned to KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID. Then a
> {{OLAPProjectRel}} will be introduced.
> In kylin, we don't need {{RelOptUtil.pushDownJoinConditions()}}. Therefore,
> the solution for this is just remove that logic.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Reopened] (KYLIN-2773) Should not push down join condition related columns are compatible while not consistent
[
https://issues.apache.org/jira/browse/KYLIN-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reopened KYLIN-2773:
-
> Should not push down join condition related columns are compatible while not
> consistent
> ---
>
> Key: KYLIN-2773
> URL: https://issues.apache.org/jira/browse/KYLIN-2773
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
> Attachments: APACHE-KYLIN-2773.patch
>
>
> For sql,
> {code}
> select PART_DT, META_CATEG_NAME, sum(price)
> from KYLIN_SALES
> INNER JOIN KYLIN_CATEGORY_GROUPINGS ON KYLIN_SALES.LEAF_CATEG_ID =
> KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID
> AND KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
> INNER JOIN KYLIN_CAL_DT ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
> group by PART_DT, META_CATEG_NAME
> order by PART_DT, META_CATEG_NAME
> {code}
> the datatype of KYLIN_SALES.LEAF_CATEG_ID is bigint, while the one of
> KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID is integer.
> Then the plan transformed by kylin is as follows:
> {code}
> OLAPToEnumerableConverter
> OLAPLimitRel(fetch=[5])
> OLAPSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> OLAPAggregateRel(group=[{0, 1}], EXPR$2=[SUM($2)])
> OLAPProjectRel(PART_DT=[$0], META_CATEG_NAME=[$16], PRICE=[$5])
> OLAPJoinRel(condition=[=($0, $19)], joinType=[inner])
> {code}
> {color:#f79232}
> {code}
> OLAPProjectRel(expr#0..19=[{inputs}], PART_DT=[$t0],
> LSTG_FORMAT_NAME=[$t1], SLR_SEGMENT_CD=[$t2], LEAF_CATEG_ID=[$t3],
> LSTG_SITE_ID=[$t4], PRICE=[$t5], SELLER_ID=[$t6], COUNT__=[$t7],
> MIN_PRICE_=[$t8], COUNT_DISTINCT_SELLER_ID_=[$t9],
> USER_DEFINED_FIELD1=[$t10], USER_DEFINED_FIELD3=[$t11], UPD_DATE=[$t12],
> UPD_USER=[$t13], LEAF_CATEG_ID0=[$t14], SITE_ID=[$t15],
> META_CATEG_NAME=[$t16], CATEG_LVL2_NAME=[$t17], CATEG_LVL3_NAME=[$t18])
> OLAPJoinRel(condition=[AND(=($3, $19), =($4, $15))],
> joinType=[inner])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], fields=[[0, 1,
> 2, 3, 4, 5, 6, 7, 8, 9]])
> OLAPProjectRel(expr#0..8=[{inputs}],
> expr#9=[CAST($t4):BIGINT], USER_DEFINED_FIELD1=[$t0],
> USER_DEFINED_FIELD3=[$t1], UPD_DATE=[$t2], UPD_USER=[$t3],
> LEAF_CATEG_ID=[$t4], SITE_ID=[$t5], META_CATEG_NAME=[$t6],
> CATEG_LVL2_NAME=[$t7], CATEG_LVL3_NAME=[$t8], LEAF_CATEG_ID9=[$t9])
> {code}
> {color}
> {code}
> OLAPTableScan(table=[[DEFAULT, KYLIN_CATEGORY_GROUPINGS]],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
> OLAPTableScan(table=[[DEFAULT, KYLIN_CAL_DT]], fields=[[0, 1, 2,
> 3]])
> {code}
> However, what we expect is as follows:
> {code}
> OLAPToEnumerableConverter
> OLAPLimitRel(fetch=[5])
> OLAPSortRel(sort0=[$0], sort1=[$1], dir0=[ASC], dir1=[ASC])
> OLAPAggregateRel(group=[{0, 1}], EXPR$2=[SUM($2)])
> OLAPProjectRel(expr#0..21=[{inputs}], PART_DT=[$t0],
> META_CATEG_NAME=[$t17], PRICE=[$t4])
> OLAPJoinRel(condition=[=($0, $19)], joinType=[inner])
> {code}
> {color:#f79232}
> {code}
> OLAPJoinRel(condition=[AND(=($2, $10), =($3, $11))],
> joinType=[inner])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], fields=[[0, 1, 2,
> 3, 4, 5, 6, 7, 8, 9]])
> {code}
> {color}
> {code}
> OLAPTableScan(table=[[DEFAULT, KYLIN_CATEGORY_GROUPINGS]],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
> OLAPTableScan(table=[[DEFAULT, KYLIN_CAL_DT]], fields=[[0, 1, 2,
> 3, 4]])
> {code}
> The reason for this difference is as follows:
> * Although we remove the {{JoinPushExpressionsRule}} in {{OLAPTableScan}},
> the method {{RelOptUtil.pushDownJoinConditions()}} is still invoked when
> creating a join in {{SqlToRelConverter}}.
> * In the method of {{RelOptUtil.pushDownJoinConditions()}}, since the
> datatypes of the join related columns are not the same, *cast* function will
> be automatically assigned to KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID. Then a
> {{OLAPProjectRel}} will be introduced.
> In kylin, we don't need {{RelOptUtil.pushDownJoinConditions()}}. Therefore,
> the solution for this is just remove that logic.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2753) Job duration may become negative
[ https://issues.apache.org/jira/browse/KYLIN-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2753: Summary: Job duration may become negative (was: Job duration may become negative after the job resumed from error or paused status) > Job duration may become negative > > > Key: KYLIN-2753 > URL: https://issues.apache.org/jira/browse/KYLIN-2753 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Fix For: v2.2.0 > > Attachments: > 0001-KYLIN-2753-Fix-for-the-special-case-that-jobs-not-re.patch, > 0001-KYLIN-2753-Job-duration-may-become-negative-after-th.patch, > KYLIN-2753-Job-duration-may-become-negative-after-th.patch, Screen Shot > 2017-08-02 at 13.30.51.png, Screen Shot 2017-08-02 at 14.01.13.png > > > After KYLIN-2655, we calculate the 'Interrupt time' when a job resumed from > error/stopped status to give more accurate job duration time. However, when > calculate the real duration time = 'Total time' - 'Interrupt time', we could > get a negative value. > The root cause is 'Interrupt time' calculated in > DefaultChainedExecutable.onExecuteStart method. The 'Interrupt time' value is > not only added with the true 'Interrupt time' when the job scheduled after > resumed, but also increased when subsequent sub tasks executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2718) Overflow when calculating combination amount based on static rules
[ https://issues.apache.org/jira/browse/KYLIN-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2718: Summary: Overflow when calculating combination amount based on static rules (was: overflow when calculating combination amount based on static rules) > Overflow when calculating combination amount based on static rules > -- > > Key: KYLIN-2718 > URL: https://issues.apache.org/jira/browse/KYLIN-2718 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: all >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.2.0 > > Attachments: APACHE-KYLIN-2718-Guava.patch > > > In extreme case, value of *combination* will exceed Long.MAX_VALUE leading to > the validation noneffective. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2716) Non-thread-safe WeakHashMap leading to high CPU
[ https://issues.apache.org/jira/browse/KYLIN-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2716: Summary: Non-thread-safe WeakHashMap leading to high CPU (was: Using non-thread-safe WeakHashMap leading to server high cpu) > Non-thread-safe WeakHashMap leading to high CPU > --- > > Key: KYLIN-2716 > URL: https://issues.apache.org/jira/browse/KYLIN-2716 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: all >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.2.0 > > Attachments: APACHE-KYLIN-2716.patch > > > Multiple threads invoke WeakHashMap.get() simultaneously may leading to a > dead loop in *WeakHashMap.get() -> getTable() -> expungeStaleEntries()*, > which finally resulting in server high cpu. There're two places using > WeakHashMap. > > One is used in the method *ClassUtil.forName()*. > We made an inner test by invoking the method *ClassUtil.forName()* 1M times, > the result is as follows: > * With cache: 20ms; > * Without cache: less than 2s. > By invoking the method *ClassUtil.forName()* with *newInstance()* 1M times, > the result is as follows: > * With cache: around 2s > * Without cache: around 3s. > Considering *ClassUtil.forName()* is always invoked with *newInstance()*, > there's no much downgrade without cache. Thus the fix is just to remove the > cache. > > Another is used in the method *CubeService.getHTableInfo()*. We changed the > WeakHashMap to Guava Cache with introducing size & time limitation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2705) Allow removing model's "partition_date_column" on web
[ https://issues.apache.org/jira/browse/KYLIN-2705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2705: Summary: Allow removing model's "partition_date_column" on web (was: Should allow user to remove partition_date_column for model in web) > Allow removing model's "partition_date_column" on web > - > > Key: KYLIN-2705 > URL: https://issues.apache.org/jira/browse/KYLIN-2705 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Minor > Fix For: v2.2.0 > > Attachments: KYLIN-2705.patch > > > Currently, User couldn't remove partition_date_column for model in web. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2703) Manage ACL through Apache Ranger
[ https://issues.apache.org/jira/browse/KYLIN-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2703: Summary: Manage ACL through Apache Ranger (was: kylin supports managing ACL through apache ranger.) > Manage ACL through Apache Ranger > > > Key: KYLIN-2703 > URL: https://issues.apache.org/jira/browse/KYLIN-2703 > Project: Kylin > Issue Type: New Feature > Components: General >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: newbie, patch, scope > Fix For: v2.2.0 > > Attachments: > 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch, > KylinAuditLog.jpg, KylinPlugins.jpg, KylinPolicies.jpg, > KylinServiceEntry.jpg, NewKylinPolicy.jpg, NewKylinService.jpg, > Ranger-PMS-hope.png > > > Ranger is a framework to enable, monitor and manage comprehensive data > security across the Hadoop platform. Apache Ranger has the following goals: > 1. Centralized security administration to manage all security related tasks > in a central UI or using REST APIs. > 2. Fine grained authorization to do a specific action and/or operation with > Hadoop component/tool and managed through a central administration tool > 3. Standardize authorization method across all Hadoop components. > 4. Enhanced support for different authorization methods - Role based access > control, attribute based access control etc. > 5. Centralize auditing of user access and administrative actions (security > related) within all the components of Hadoop. > Ranger has supported enable, monitor and manage following components: > 1. HDFS > 2. HIVE > 3. HBASE > 4. KNOX > 5. YARN > 6. STORM > 7. SOLR > 8. KAFKA > 9. ATLAS > In order to improve the flexibility of kylin privilege control and enhance > value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, > Kylin should also support that using Ranger to control access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/RANGER-1672 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2653) Spark Cubing read metadata from HDFS
[ https://issues.apache.org/jira/browse/KYLIN-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2653: Summary: Spark Cubing read metadata from HDFS (was: Spark cubing support HBase cluster with kerberos) > Spark Cubing read metadata from HDFS > > > Key: KYLIN-2653 > URL: https://issues.apache.org/jira/browse/KYLIN-2653 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.2.0 > > > Currently, Spark cubing doesn't support HBase cluster with kerberos. > Temporarily，we could support HBase cluster with kerberos on Yarn client mode, > because which is easy. > In the long term，we should avoid access HBase in Spark cubing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2844) Override "max-visit-scanrange" and "max-fuzzykey-scan" at cube level
[ https://issues.apache.org/jira/browse/KYLIN-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2844: Summary: Override "max-visit-scanrange" and "max-fuzzykey-scan" at cube level (was: override max-visit-scanrange and max-fuzzykey-scan at cube level) > Override "max-visit-scanrange" and "max-fuzzykey-scan" at cube level > > > Key: KYLIN-2844 > URL: https://issues.apache.org/jira/browse/KYLIN-2844 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Dayue Gao >Assignee: Dayue Gao > Fix For: v2.2.0 > > Attachments: KYLIN-2844.patch > > > User should be able to override "kylin.storage.hbase.max-fuzzykey-scan" and > "kylin.storage.hbase.max-visit-scanrange" at cube/project level. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2863) Double caret bug in sample.sh for old version bash
[
https://issues.apache.org/jira/browse/KYLIN-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2863:
Summary: Double caret bug in sample.sh for old version bash (was: fix the
double caret bug in sample.sh when executed in old bash version of some
enterprise OS)
> Double caret bug in sample.sh for old version bash
> --
>
> Key: KYLIN-2863
> URL: https://issues.apache.org/jira/browse/KYLIN-2863
> Project: Kylin
> Issue Type: Bug
> Components: Tools, Build and Test
>Affects Versions: v2.0.0, v2.1.0
> Environment: OS: SuSE 11 sp4
> Hortonworks hdp 2.6.1.0-129
> hadoop 2.7.3
> hbase 1.1.2
> hive 1.2.1
> jdk 1.8.0
>Reporter: TING WANG
>Assignee: hongbin ma
> Labels: patch, starter
> Fix For: v2.2.0
>
> Attachments: fixbug-tina-double-caret.patch
>
>
> When an old version bash is used, the following error will occur:
> bq. /apache-kylin-2.0.0-bin/bin # ./sample.sh
> bq. Retrieving hadoop conf dir...
> bq. Retrieving hadoop conf dir...
> bq. KYLIN_HOME is set to /usr/share/kylin/apache-kylin-2.0.0-bin
> bq. Loading sample data into HDFS tmp path: /tmp/kylin/sample_cube/data
> bq. ./sample.sh: line 40: ${sample_database^^}: bad substitution
> bq. Going to create sample tables in hive to database default by cli
> bq. ..
> If you ignore the error above, another error will occur when building cube:
> bq. java.lang.IllegalStateException: No resource found at --
> /table/DEFAULT.KYLIN_SALES.json
> bq. at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.dumpResources(AbstractHadoopJob.java:550)
> bq. at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.dumpKylinPropsAndMetadata(AbstractHadoopJob.java:505)
> bq. at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.attachCubeMetadata(AbstractHadoopJob.java:458)
> bq. at
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:122)
> bq. at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:102)
> bq. at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:123)
> bq. at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> bq. at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> bq. at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> bq. at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
> bq. at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> bq. at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> bq. at java.lang.Thread.run(Thread.java:745)
> bq. result code:2
> The reason is that the Bash version in SuSE 11 sp4 enterprise is quite old:
> bq. #: bash -version
> bq. bash -version
> bq. GNU bash, version 3.2.57(1)-release (x86_64-suse-linux-gnu)
> bq. Copyright (C) 2007 Free Software Foundation, Inc.
> Instead of using double carets to convert the string to uppercase, we can use
> the tr command to do the same job as shown in this pr which has higher
> compatibility for various kinds of OS.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2949) Couldn't get authorities with LDAP in RedHat Linux
[ https://issues.apache.org/jira/browse/KYLIN-2949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2949: Summary: Couldn't get authorities with LDAP in RedHat Linux (was: The kylin does not get right authorities when the LDAP authentication service was used in RedHat OS.) > Couldn't get authorities with LDAP in RedHat Linux > -- > > Key: KYLIN-2949 > URL: https://issues.apache.org/jira/browse/KYLIN-2949 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v2.1.0 >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: v2.2.0 > > Attachments: > 0001-KYLIN-2949-The-kylin-does-not-get-right-authorities-.patch > > > Scene: > 1. LDAP server(s) deployed in RedHat OS; > 2. Enable LDAP authentication in kylin; > 3. Add kylin_admin user and admin group in kylin. The kylin_admin user > belongs to the admin group. And admin group is the "admin" role. > 3. Login kylin using kylin_admin user. > The result is that kylin_admin user lost the authority of "ROLE_ADMIN", with > only default authorities returned. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2781) Make 'find-hadoop-conf-dir.sh' executable
[ https://issues.apache.org/jira/browse/KYLIN-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2781: Summary: Make 'find-hadoop-conf-dir.sh' executable (was: make find-hadoop-conf-dir.sh executable) > Make 'find-hadoop-conf-dir.sh' executable > - > > Key: KYLIN-2781 > URL: https://issues.apache.org/jira/browse/KYLIN-2781 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: Yang Hao >Assignee: Yang Hao > Fix For: v2.2.0 > > Attachments: KYLIN-2781.patch > > > Users may want to execute the shell script (find-hadoop-conf-dir.sh) directly > to get the hadoop-conf-dir, but now, it only support source it -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2819) Add "kylin.env.zookeeper-base-path" for zk path
[ https://issues.apache.org/jira/browse/KYLIN-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2819. - Resolution: Fixed > Add "kylin.env.zookeeper-base-path" for zk path > --- > > Key: KYLIN-2819 > URL: https://issues.apache.org/jira/browse/KYLIN-2819 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.3 >Reporter: Shen Yinjie >Assignee: Shen Yinjie > Fix For: v2.2.0 > > Attachments: KYLIN-2819_2.patch, kylin-2819_1.patch > > > currently kylin zkbasepath is hard-coded as "/kylin"，we met a scenario to > launch more kylin services on YARN, and should make their zkbasepaths > flexible and separable . > Add an attribute "kylin.zk.base.path" in kylin.properties and default value > is "/kylin". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2819) Add "kylin.env.zookeeper-base-path" for zk path
[ https://issues.apache.org/jira/browse/KYLIN-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2819: Summary: Add "kylin.env.zookeeper-base-path" for zk path (was: add configurable zk.base.path for kylin) > Add "kylin.env.zookeeper-base-path" for zk path > --- > > Key: KYLIN-2819 > URL: https://issues.apache.org/jira/browse/KYLIN-2819 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.3 >Reporter: Shen Yinjie >Assignee: Shen Yinjie > Fix For: v2.2.0 > > Attachments: KYLIN-2819_2.patch, kylin-2819_1.patch > > > currently kylin zkbasepath is hard-coded as "/kylin"，we met a scenario to > launch more kylin services on YARN, and should make their zkbasepaths > flexible and separable . > Add an attribute "kylin.zk.base.path" in kylin.properties and default value > is "/kylin". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Reopened] (KYLIN-2819) add configurable zk.base.path for kylin
[ https://issues.apache.org/jira/browse/KYLIN-2819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reopened KYLIN-2819: - > add configurable zk.base.path for kylin > --- > > Key: KYLIN-2819 > URL: https://issues.apache.org/jira/browse/KYLIN-2819 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.3 >Reporter: Shen Yinjie >Assignee: Shen Yinjie > Fix For: v2.2.0 > > Attachments: KYLIN-2819_2.patch, kylin-2819_1.patch > > > currently kylin zkbasepath is hard-coded as "/kylin"，we met a scenario to > launch more kylin services on YARN, and should make their zkbasepaths > flexible and separable . > Add an attribute "kylin.zk.base.path" in kylin.properties and default value > is "/kylin". -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2703) kylin supports managing ACL through apache ranger.
[ https://issues.apache.org/jira/browse/KYLIN-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2703: --- Summary: kylin supports managing ACL through apache ranger. (was: kylin supports managing access rights for project and cube through apache ranger.) > kylin supports managing ACL through apache ranger. > -- > > Key: KYLIN-2703 > URL: https://issues.apache.org/jira/browse/KYLIN-2703 > Project: Kylin > Issue Type: New Feature > Components: General >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: newbie, patch, scope > Fix For: v2.2.0 > > Attachments: > 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch, > KylinAuditLog.jpg, KylinPlugins.jpg, KylinPolicies.jpg, > KylinServiceEntry.jpg, NewKylinPolicy.jpg, NewKylinService.jpg, > Ranger-PMS-hope.png > > > Ranger is a framework to enable, monitor and manage comprehensive data > security across the Hadoop platform. Apache Ranger has the following goals: > 1. Centralized security administration to manage all security related tasks > in a central UI or using REST APIs. > 2. Fine grained authorization to do a specific action and/or operation with > Hadoop component/tool and managed through a central administration tool > 3. Standardize authorization method across all Hadoop components. > 4. Enhanced support for different authorization methods - Role based access > control, attribute based access control etc. > 5. Centralize auditing of user access and administrative actions (security > related) within all the components of Hadoop. > Ranger has supported enable, monitor and manage following components: > 1. HDFS > 2. HIVE > 3. HBASE > 4. KNOX > 5. YARN > 6. STORM > 7. SOLR > 8. KAFKA > 9. ATLAS > In order to improve the flexibility of kylin privilege control and enhance > value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, > Kylin should also support that using Ranger to control access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/RANGER-1672 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2979) Can kylin query the columns which were defined in model but not built in cube.
[ https://issues.apache.org/jira/browse/KYLIN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-2979. -- Resolution: Invalid > Can kylin query the columns which were defined in model but not built in cube. > -- > > Key: KYLIN-2979 > URL: https://issues.apache.org/jira/browse/KYLIN-2979 > Project: Kylin > Issue Type: Wish >Reporter: wuyingjun >Assignee: wuyingjun > > kylin(version 2.1.0) deployed with default property can only query the > columns built in cube, > while the rest columns in the model are not available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2960) We should submit a new feature that it support the authentication for user and role and the authentication for user and group when the LDAP authentication was enab
[
https://issues.apache.org/jira/browse/KYLIN-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224810#comment-16224810
]
peng.jianhua edited comment on KYLIN-2960 at 10/30/17 12:22 PM:
hi [~Aron.tao], thank you for your response, I have gotten your idea, and I
also agree very much, but I also have some questions.
1. Do you mean to use LDAPAuthoritiesPopulator to replace AuthoritiesPopulator?
I think AuthoritiesPopulator#getGroupMembershipRoles will return the roles of
user, and LDAPAuthoritiesPopulator #getGroupMembershipRoles will return the
groups of user, but you give me an example which will return not only groups
but also ROLE_ADMIN, so do you mean using LDAPAuthoritiesPopulator
#getGroupMembershipRoles to return all of groups and roles of user?
2. If AuthoritiesPopulator and LDAPAuthoritiesPopulator both exist, how I let
the Authentication#getAuthorities() to return result about AuthoritiesPopulator
or LDAPAuthoritiesPopulator?
So I am puzzled. In my comprehension, I think the LDAPAuthoritiesPopulator is
like this. then I can only get the groups of user by
LDAPAuthoritiesPopulator#getGroupMembershipRoles, which maybe meet my needs.
Thank you for your help, looking forward your response.
{code:java}
package org.apache.kylin.rest.security;
import java.util.Set;
import org.springframework.ldap.core.ContextSource;
import org.springframework.security.core.GrantedAuthority;
import
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator;
public class LDAPAuthoritiesPopulator extends DefaultLdapAuthoritiesPopulator {
//defaultRole may be helpful, it needs discuss
public LDAPAuthoritiesPopulator(ContextSource contextSource, String
groupSearchBase, String adminRole,
String defaultRole) {
super(contextSource, groupSearchBase);
setGroupSearchFilter("(|(member={0})(memberUid={1}))");
setConvertToUpperCase(false);
setRolePrefix("");
}
@Override
public Set getGroupMembershipRoles(String userDn, String
username) {
return super.getGroupMembershipRoles(userDn, username);
}
}
{code}
was (Author: peng.jianhua):
hi [~Aron.tao], thank you for your response, I have gotten your idea, and I
also agree very much, but I also have one question.
1. Do you mean to use LDAPAuthoritiesPopulator to replace AuthoritiesPopulator?
I think AuthoritiesPopulator#getGroupMembershipRoles will return the roles of
user, and LDAPAuthoritiesPopulator #getGroupMembershipRoles will return the
groups of user, but you give me an example which will return not only groups
but also ROLE_ADMIN, so do you mean using LDAPAuthoritiesPopulator
#getGroupMembershipRoles to return all of groups and roles of user?
2. If AuthoritiesPopulator and LDAPAuthoritiesPopulator both exist, how I let
the Authentication#getAuthorities() to return result about AuthoritiesPopulator
or LDAPAuthoritiesPopulator?
So I am puzzled. In my comprehension, I think the LDAPAuthoritiesPopulator is
like this. then I can only get the groups of user by
LDAPAuthoritiesPopulator#getGroupMembershipRoles, which maybe meet my needs.
Thank you for your help, looking forward your response.
{code:java}
package org.apache.kylin.rest.security;
import java.util.Set;
import org.springframework.ldap.core.ContextSource;
import org.springframework.security.core.GrantedAuthority;
import
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator;
public class LDAPAuthoritiesPopulator extends DefaultLdapAuthoritiesPopulator {
//defaultRole may be helpful, it needs discuss
public LDAPAuthoritiesPopulator(ContextSource contextSource, String
groupSearchBase, String adminRole,
String defaultRole) {
super(contextSource, groupSearchBase);
setGroupSearchFilter("(|(member={0})(memberUid={1}))");
setConvertToUpperCase(false);
setRolePrefix("");
}
@Override
public Set getGroupMembershipRoles(String userDn, String
username) {
return super.getGroupMembershipRoles(userDn, username);
}
}
{code}
> We should submit a new feature that it support the authentication for user
> and role and the authentication for user and group when the LDAP
> authentication was enabled.
> ---
>
> Key: KYLIN-2960
> URL: https://issues.apache.org/jira/browse/KYLIN-2960
> Project: Kylin
> Issue Type: New Feature
> Components: General
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Attachments:
> 0001-KYLIN-2960-We-should-submit-a-new-feature-that-it-su.patch
>
>
> Currently, the user authentication interface that was
[jira] [Commented] (KYLIN-2960) We should submit a new feature that it support the authentication for user and role and the authentication for user and group when the LDAP authentication was enabled.
[
https://issues.apache.org/jira/browse/KYLIN-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224810#comment-16224810
]
peng.jianhua commented on KYLIN-2960:
-
hi [~Aron.tao], thank you for your response, I have gotten your idea, and I
also agree very much, but I also have one question.
1. Do you mean to use LDAPAuthoritiesPopulator to replace AuthoritiesPopulator?
I think AuthoritiesPopulator#getGroupMembershipRoles will return the roles of
user, and LDAPAuthoritiesPopulator #getGroupMembershipRoles will return the
groups of user, but you give me an example which will return not only groups
but also ROLE_ADMIN, so do you mean using LDAPAuthoritiesPopulator
#getGroupMembershipRoles to return all of groups and roles of user?
2. If AuthoritiesPopulator and LDAPAuthoritiesPopulator both exist, how I let
the Authentication#getAuthorities() to return result about AuthoritiesPopulator
or LDAPAuthoritiesPopulator?
So I am puzzled. In my comprehension, I think the LDAPAuthoritiesPopulator is
like this. then I can only get the groups of user by
LDAPAuthoritiesPopulator#getGroupMembershipRoles, which maybe meet my needs.
Thank you for your help, looking forward your response.
{code:java}
package org.apache.kylin.rest.security;
import java.util.Set;
import org.springframework.ldap.core.ContextSource;
import org.springframework.security.core.GrantedAuthority;
import
org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator;
public class LDAPAuthoritiesPopulator extends DefaultLdapAuthoritiesPopulator {
//defaultRole may be helpful, it needs discuss
public LDAPAuthoritiesPopulator(ContextSource contextSource, String
groupSearchBase, String adminRole,
String defaultRole) {
super(contextSource, groupSearchBase);
setGroupSearchFilter("(|(member={0})(memberUid={1}))");
setConvertToUpperCase(false);
setRolePrefix("");
}
@Override
public Set getGroupMembershipRoles(String userDn, String
username) {
return super.getGroupMembershipRoles(userDn, username);
}
}
{code}
> We should submit a new feature that it support the authentication for user
> and role and the authentication for user and group when the LDAP
> authentication was enabled.
> ---
>
> Key: KYLIN-2960
> URL: https://issues.apache.org/jira/browse/KYLIN-2960
> Project: Kylin
> Issue Type: New Feature
> Components: General
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Attachments:
> 0001-KYLIN-2960-We-should-submit-a-new-feature-that-it-su.patch
>
>
> Currently, the user authentication interface that was provided by kylin to
> the third party only supports user and role authentication. However only user
> and group have authentication function when we use the LDAP authentication.
> In fact the authentication for user and role and the authentication for user
> and group have the same functional characteristics between different
> appplication system. So we should submit a new feature that it support the
> authentication for user and role and the authentication for user and group
> when the LDAP authentication was enabled.
> We supplied the checkPermission interface to implement the new feature. In
> the interface we set user groups information to the userRoles parameter when
> the LDAP was enabled, on the contrary we set user roles information to the
> userRoles parameter. The interface is as following:
> /**
> * Checks if a user has permission on an entity.
> *
> * @param user
> * @param userRoles
> * @param entityType String constants defined in AclEntityType
> * @param entityUuid
> * @param permission
> *
> * @return true if has permission
> */
> abstract public boolean checkPermission(String user, List userRoles,
> //
> String entityType, String entityUuid, Permission permission);
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (KYLIN-2974) joint group is Null lead to query error
[
https://issues.apache.org/jira/browse/KYLIN-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated KYLIN-2974:
Comment: was deleted
(was: The method isOnTree may satisfy it
{code:java}
if(!isOnTree(cuboidID)) {
return null;
}
public boolean isOnTree(long cuboidID) {
if (cuboidID <= 0) {
return false; //cuboid must be greater than 0
}
if ((cuboidID & ~partialCubeFullMask) != 0) {
return false; //a cuboid's parent within agg is at most
partialCubeFullMask
}
return checkMandatoryColumns(cuboidID) && checkHierarchy(cuboidID) &&
checkJoint(cuboidID);
}
{code}
)
> joint group is Null lead to query error
> ---
>
> Key: KYLIN-2974
> URL: https://issues.apache.org/jira/browse/KYLIN-2974
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Yang Hao
>Assignee: Yang Hao
> Attachments: KYLIN-2974.patch
>
>
> When query a sql, it turns to find all possible aggregation group. But it may
> cause error when the joint group is empty. Relevant code is
> {code:java}
> cuboidID = cuboidID | Collections.min(agg.getJoints(),
> cuboidSelectComparator);
> {code}
> In our env, when we have multiple aggregation groups, it may trigger the error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2974) joint group is Null lead to query error
[
https://issues.apache.org/jira/browse/KYLIN-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224764#comment-16224764
]
Yang Hao edited comment on KYLIN-2974 at 10/30/17 11:44 AM:
The method isOnTree may satisfy it
{code:java}
if(!isOnTree(cuboidID)) {
return null;
}
public boolean isOnTree(long cuboidID) {
if (cuboidID <= 0) {
return false; //cuboid must be greater than 0
}
if ((cuboidID & ~partialCubeFullMask) != 0) {
return false; //a cuboid's parent within agg is at most
partialCubeFullMask
}
return checkMandatoryColumns(cuboidID) && checkHierarchy(cuboidID) &&
checkJoint(cuboidID);
}
{code}
was (Author: yanghaogn):
The method isOnTree may satisfy it
{code:java}
public boolean isOnTree(long cuboidID) {
if (cuboidID <= 0) {
return false; //cuboid must be greater than 0
}
if ((cuboidID & ~partialCubeFullMask) != 0) {
return false; //a cuboid's parent within agg is at most
partialCubeFullMask
}
return checkMandatoryColumns(cuboidID) && checkHierarchy(cuboidID) &&
checkJoint(cuboidID);
}
{code}
> joint group is Null lead to query error
> ---
>
> Key: KYLIN-2974
> URL: https://issues.apache.org/jira/browse/KYLIN-2974
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Yang Hao
>Assignee: Yang Hao
> Attachments: KYLIN-2974.patch
>
>
> When query a sql, it turns to find all possible aggregation group. But it may
> cause error when the joint group is empty. Relevant code is
> {code:java}
> cuboidID = cuboidID | Collections.min(agg.getJoints(),
> cuboidSelectComparator);
> {code}
> In our env, when we have multiple aggregation groups, it may trigger the error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2974) joint group is Null lead to query error
[
https://issues.apache.org/jira/browse/KYLIN-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224764#comment-16224764
]
Yang Hao commented on KYLIN-2974:
-
The method isOnTree may satisfy it
{code:java}
public boolean isOnTree(long cuboidID) {
if (cuboidID <= 0) {
return false; //cuboid must be greater than 0
}
if ((cuboidID & ~partialCubeFullMask) != 0) {
return false; //a cuboid's parent within agg is at most
partialCubeFullMask
}
return checkMandatoryColumns(cuboidID) && checkHierarchy(cuboidID) &&
checkJoint(cuboidID);
}
{code}
> joint group is Null lead to query error
> ---
>
> Key: KYLIN-2974
> URL: https://issues.apache.org/jira/browse/KYLIN-2974
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Yang Hao
>Assignee: Yang Hao
> Attachments: KYLIN-2974.patch
>
>
> When query a sql, it turns to find all possible aggregation group. But it may
> cause error when the joint group is empty. Relevant code is
> {code:java}
> cuboidID = cuboidID | Collections.min(agg.getJoints(),
> cuboidSelectComparator);
> {code}
> In our env, when we have multiple aggregation groups, it may trigger the error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (KYLIN-2974) joint group is Null lead to query error
[
https://issues.apache.org/jira/browse/KYLIN-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Hao updated KYLIN-2974:
Comment: was deleted
(was: [~liyang.gmt8%40gmail.com] I think the method isOnTree may satisfy it
{code:java}
public boolean isOnTree(long cuboidID) {
if (cuboidID <= 0) {
return false; //cuboid must be greater than 0
}
if ((cuboidID & ~partialCubeFullMask) != 0) {
return false; //a cuboid's parent within agg is at most
partialCubeFullMask
}
return checkMandatoryColumns(cuboidID) && checkHierarchy(cuboidID) &&
checkJoint(cuboidID);
}
{code}
)
> joint group is Null lead to query error
> ---
>
> Key: KYLIN-2974
> URL: https://issues.apache.org/jira/browse/KYLIN-2974
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Yang Hao
>Assignee: Yang Hao
> Attachments: KYLIN-2974.patch
>
>
> When query a sql, it turns to find all possible aggregation group. But it may
> cause error when the joint group is empty. Relevant code is
> {code:java}
> cuboidID = cuboidID | Collections.min(agg.getJoints(),
> cuboidSelectComparator);
> {code}
> In our env, when we have multiple aggregation groups, it may trigger the error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2974) joint group is Null lead to query error
[
https://issues.apache.org/jira/browse/KYLIN-2974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224758#comment-16224758
]
Yang Hao commented on KYLIN-2974:
-
[~liyang.gmt8%40gmail.com] I think the method isOnTree may satisfy it
{code:java}
public boolean isOnTree(long cuboidID) {
if (cuboidID <= 0) {
return false; //cuboid must be greater than 0
}
if ((cuboidID & ~partialCubeFullMask) != 0) {
return false; //a cuboid's parent within agg is at most
partialCubeFullMask
}
return checkMandatoryColumns(cuboidID) && checkHierarchy(cuboidID) &&
checkJoint(cuboidID);
}
{code}
> joint group is Null lead to query error
> ---
>
> Key: KYLIN-2974
> URL: https://issues.apache.org/jira/browse/KYLIN-2974
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Yang Hao
>Assignee: Yang Hao
> Attachments: KYLIN-2974.patch
>
>
> When query a sql, it turns to find all possible aggregation group. But it may
> cause error when the joint group is empty. Relevant code is
> {code:java}
> cuboidID = cuboidID | Collections.min(agg.getJoints(),
> cuboidSelectComparator);
> {code}
> In our env, when we have multiple aggregation groups, it may trigger the error
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2979) Can kylin query the columns which were defined in model but not built in cube.
[ https://issues.apache.org/jira/browse/KYLIN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224638#comment-16224638 ] wuyingjun commented on KYLIN-2979: -- ok ，thank you. please close the issue > Can kylin query the columns which were defined in model but not built in cube. > -- > > Key: KYLIN-2979 > URL: https://issues.apache.org/jira/browse/KYLIN-2979 > Project: Kylin > Issue Type: Wish >Reporter: wuyingjun >Assignee: wuyingjun > > kylin(version 2.1.0) deployed with default property can only query the > columns built in cube, > while the rest columns in the model are not available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2979) Can kylin query the columns which were defined in model but not built in cube.
[ https://issues.apache.org/jira/browse/KYLIN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224620#comment-16224620 ] Shaofeng SHI commented on KYLIN-2979: - hi yingjun, Apache JIRA system is for bug and task tracking; For QA, we suggest send email to apache mailing list; See https://kylin.apache.org/community/ > Can kylin query the columns which were defined in model but not built in cube. > -- > > Key: KYLIN-2979 > URL: https://issues.apache.org/jira/browse/KYLIN-2979 > Project: Kylin > Issue Type: Wish >Reporter: wuyingjun >Assignee: wuyingjun > > kylin(version 2.1.0) deployed with default property can only query the > columns built in cube, > while the rest columns in the model are not available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2794) MultipleDictionaryValueEnumerator should output values in sorted order
[
https://issues.apache.org/jira/browse/KYLIN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224568#comment-16224568
]
Dong Li commented on KYLIN-2794:
Thanks [~leontong]. This patch made lots of modification on Kylin's low-level
code. And We need more validation to ensure no regression. So set fix version
to 'v2.3.0' to include this fix in next release.
> MultipleDictionaryValueEnumerator should output values in sorted order
> --
>
> Key: KYLIN-2794
> URL: https://issues.apache.org/jira/browse/KYLIN-2794
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
> Environment: hadoop hadoop-2.6.0-cdh5.8.2 hive 2.1 hbase 0.98
>Reporter: 翟玉勇
>Assignee: Yifei Wu
>Priority: Critical
> Labels: scope
> Fix For: v2.3.0
>
>
> Dictionary exception during merge of segments.
> {code}
> 2017-08-18 14:17:48,828 ERROR [pool-11-thread-1]
> threadpool.DistributedScheduler:188 : ExecuteException
> job:8d031b5f-2d3f-445f-a62b-7bc560d919ea in server: **
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 4 more
> Caused by: java.lang.IllegalStateException: Invalid input data. Unordered
> data cannot be split into multi trees
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78)
> at
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.addValue(DictionaryGenerator.java:212)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64)
> at
> org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104)
> at
> org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:267)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2794) MultipleDictionaryValueEnumerator should output values in sorted order
[
https://issues.apache.org/jira/browse/KYLIN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li updated KYLIN-2794:
---
Fix Version/s: (was: v2.2.0)
v2.3.0
> MultipleDictionaryValueEnumerator should output values in sorted order
> --
>
> Key: KYLIN-2794
> URL: https://issues.apache.org/jira/browse/KYLIN-2794
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
> Environment: hadoop hadoop-2.6.0-cdh5.8.2 hive 2.1 hbase 0.98
>Reporter: 翟玉勇
>Assignee: Yifei Wu
>Priority: Critical
> Labels: scope
> Fix For: v2.3.0
>
>
> Dictionary exception during merge of segments.
> {code}
> 2017-08-18 14:17:48,828 ERROR [pool-11-thread-1]
> threadpool.DistributedScheduler:188 : ExecuteException
> job:8d031b5f-2d3f-445f-a62b-7bc560d919ea in server: **
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 4 more
> Caused by: java.lang.IllegalStateException: Invalid input data. Unordered
> data cannot be split into multi trees
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78)
> at
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.addValue(DictionaryGenerator.java:212)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64)
> at
> org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104)
> at
> org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:267)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2979) Can kylin query the columns which were defined in model but not built in cube.
[ https://issues.apache.org/jira/browse/KYLIN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2979: - Description: kylin(version 2.1.0) deployed with default property can only query the columns built in cube, while the rest columns in the model are not available. was: kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 已经安装了kylin2.1.0版本，但是只能查询已经在cube中构建的字段，而模型中的其他字段并不能查询，配置使用的是默认配置，请问是否支持这种特性。 > Can kylin query the columns which were defined in model but not built in cube. > -- > > Key: KYLIN-2979 > URL: https://issues.apache.org/jira/browse/KYLIN-2979 > Project: Kylin > Issue Type: Wish >Reporter: wuyingjun >Assignee: wuyingjun > > kylin(version 2.1.0) deployed with default property can only query the > columns built in cube, > while the rest columns in the model are not available. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2794) MultipleDictionaryValueEnumerator should output values in sorted order
[
https://issues.apache.org/jira/browse/KYLIN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224475#comment-16224475
]
leontong edited comment on KYLIN-2794 at 10/30/17 7:55 AM:
---
thanks [~liyang.g...@gmail.com] for your advice. here is the PR link:
https://github.com/apache/kylin/pull/84. [~liyang.g...@gmail.com] and
[~lidong_sjtu], could you help review? thanks!
was (Author: leontong):
thanks [~liyang.g...@gmail.com] for your advice. here is the PR link:
https://github.com/apache/kylin/pull/84. [~liyang.g...@gmail.com] and
[~lidong_sjtu], could you help review? . thanks!
> MultipleDictionaryValueEnumerator should output values in sorted order
> --
>
> Key: KYLIN-2794
> URL: https://issues.apache.org/jira/browse/KYLIN-2794
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
> Environment: hadoop hadoop-2.6.0-cdh5.8.2 hive 2.1 hbase 0.98
>Reporter: 翟玉勇
>Assignee: Yifei Wu
>Priority: Critical
> Labels: scope
> Fix For: v2.2.0
>
>
> Dictionary exception during merge of segments.
> {code}
> 2017-08-18 14:17:48,828 ERROR [pool-11-thread-1]
> threadpool.DistributedScheduler:188 : ExecuteException
> job:8d031b5f-2d3f-445f-a62b-7bc560d919ea in server: **
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 4 more
> Caused by: java.lang.IllegalStateException: Invalid input data. Unordered
> data cannot be split into multi trees
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78)
> at
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.addValue(DictionaryGenerator.java:212)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64)
> at
> org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104)
> at
> org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:267)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2794) MultipleDictionaryValueEnumerator should output values in sorted order
[
https://issues.apache.org/jira/browse/KYLIN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224475#comment-16224475
]
leontong commented on KYLIN-2794:
-
thanks [~liyang.g...@gmail.com] for your advice. here is the PR link:
https://github.com/apache/kylin/pull/84. please help review and merge
[~liyang.g...@gmail.com] [~lidong_sjtu]. thanks!
> MultipleDictionaryValueEnumerator should output values in sorted order
> --
>
> Key: KYLIN-2794
> URL: https://issues.apache.org/jira/browse/KYLIN-2794
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
> Environment: hadoop hadoop-2.6.0-cdh5.8.2 hive 2.1 hbase 0.98
>Reporter: 翟玉勇
>Assignee: Yifei Wu
>Priority: Critical
> Labels: scope
> Fix For: v2.2.0
>
>
> Dictionary exception during merge of segments.
> {code}
> 2017-08-18 14:17:48,828 ERROR [pool-11-thread-1]
> threadpool.DistributedScheduler:188 : ExecuteException
> job:8d031b5f-2d3f-445f-a62b-7bc560d919ea in server: **
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 4 more
> Caused by: java.lang.IllegalStateException: Invalid input data. Unordered
> data cannot be split into multi trees
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78)
> at
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.addValue(DictionaryGenerator.java:212)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64)
> at
> org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104)
> at
> org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:267)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2979) kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建
[ https://issues.apache.org/jira/browse/KYLIN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16224438#comment-16224438 ] wuyingjun commented on KYLIN-2979: -- [~yimingliu] > kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 > --- > > Key: KYLIN-2979 > URL: https://issues.apache.org/jira/browse/KYLIN-2979 > Project: Kylin > Issue Type: Wish >Reporter: wuyingjun >Assignee: wuyingjun > > kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 > 已经安装了kylin2.1.0版本，但是只能查询已经在cube中构建的字段，而模型中的其他字段并不能查询，配置使用的是默认配置，请问是否支持这种特性。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2979) kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建
[ https://issues.apache.org/jira/browse/KYLIN-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun reassigned KYLIN-2979: Assignee: wuyingjun > kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 > --- > > Key: KYLIN-2979 > URL: https://issues.apache.org/jira/browse/KYLIN-2979 > Project: Kylin > Issue Type: Wish >Reporter: wuyingjun >Assignee: wuyingjun > > kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 > 已经安装了kylin2.1.0版本，但是只能查询已经在cube中构建的字段，而模型中的其他字段并不能查询，配置使用的是默认配置，请问是否支持这种特性。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2979) kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建
wuyingjun created KYLIN-2979: Summary: kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 Key: KYLIN-2979 URL: https://issues.apache.org/jira/browse/KYLIN-2979 Project: Kylin Issue Type: Wish Reporter: wuyingjun kylin是否可以查询模型的字段，这些字段并没有在该模型下的cube中进行构建 已经安装了kylin2.1.0版本，但是只能查询已经在cube中构建的字段，而模型中的其他字段并不能查询，配置使用的是默认配置，请问是否支持这种特性。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
