[jira] [Updated] (KYLIN-2703) kylin supports managing access rights for project and cube through apache ranger.
[ https://issues.apache.org/jira/browse/KYLIN-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2703: Attachment: NewKylinService.jpg NewKylinPolicy.jpg KylinServiceEntry.jpg KylinPolicies.jpg KylinPlugins.jpg KylinAuditLog.jpg > kylin supports managing access rights for project and cube through apache > ranger. > - > > Key: KYLIN-2703 > URL: https://issues.apache.org/jira/browse/KYLIN-2703 > Project: Kylin > Issue Type: New Feature > Components: General >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: newbie, patch > Attachments: > 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch, > KylinAuditLog.jpg, KylinPlugins.jpg, KylinPolicies.jpg, > KylinServiceEntry.jpg, NewKylinPolicy.jpg, NewKylinService.jpg > > > Ranger is a framework to enable, monitor and manage comprehensive data > security across the Hadoop platform. Apache Ranger has the following goals: > 1. Centralized security administration to manage all security related tasks > in a central UI or using REST APIs. > 2. Fine grained authorization to do a specific action and/or operation with > Hadoop component/tool and managed through a central administration tool > 3. Standardize authorization method across all Hadoop components. > 4. Enhanced support for different authorization methods - Role based access > control, attribute based access control etc. > 5. Centralize auditing of user access and administrative actions (security > related) within all the components of Hadoop. > Ranger has supported enable, monitor and manage following components: > 1. HDFS > 2. HIVE > 3. HBASE > 4. KNOX > 5. YARN > 6. STORM > 7. SOLR > 8. KAFKA > 9. ATLAS > In order to improve the flexibility of kylin privilege control and enhance > value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, > Kylin should also support that using Ranger to control access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/RANGER-1672 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099541#comment-16099541 ] Billy Liu commented on KYLIN-2755: -- You are welcomed to submit the patch. To learn more, http://kylin.apache.org/development/howto_contribute.html > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > Attachments: code modify.png > > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment a a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Attachment: (was: 飞信截图20170725134120.png) > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > Attachments: code modify.png > > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment a a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Attachment: code modify.png > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > Attachments: code modify.png > > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment a a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Attachment: 飞信截图20170725134120.png > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > Attachments: 飞信截图20170725134120.png > > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment a a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099490#comment-16099490 ] Billy Liu commented on KYLIN-2755: -- What do you proposal for "a better way"? > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment a a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-1936) Improve enable limit logic (exactAggregation is too strict)
[
https://issues.apache.org/jira/browse/KYLIN-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099487#comment-16099487
]
Zhong Yanghong commented on KYLIN-1936:
---
Hi [~mahongbin], why we need this precondition for limit push down, could you
give me an example? Thanks very much.
{code}
//if groupsD is clustered at "head" of the rowkey, then limit push down
is possible
int size = groupsD.size();
if (!groupsD.containsAll(cuboid.getColumns().subList(0, size))) {
possible = false;
logger.info("Storage limit push down is impossible because groupD
is not clustered at head, groupsD: " + groupsD //
+ " with cuboid columns: " + cuboid.getColumns());
}
{code}
> Improve enable limit logic (exactAggregation is too strict)
> ---
>
> Key: KYLIN-1936
> URL: https://issues.apache.org/jira/browse/KYLIN-1936
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v1.5.3
>Reporter: hongbin ma
>Assignee: hongbin ma
> Fix For: v1.5.4
>
>
> from zhaotians...@meizu.com:
> recently I got the following error while execute query on a cube which is not
> that big( about 400mb, 20milion record)
> ==
> Error while executing SQL "select FCRASHTIME,count(1) from
> UXIP.EDL_FDT_OUC_UPLOAD_FILES group by FCRASH_ANALYSIS_ID,FCRASHTIME limit
> 1": Scan row count exceeded threshold: 1000, please add filter condition
> to narrow down backend scan range, like where clause.
> I guess what it scan were the intermediate result, but It doesn't any order
> by,also the result count is limit to just 1.so it could scan to find any
> record with those two dimension and wala.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Description: I want to know how to integrate the kylin into hive datasource and hbase storage with kerberos. I have used hive beeline and modifid the hbase configuration initialization in the source code. Can the current kylin version support kerberos environment a a better way in mapreduce cubing? was: I want to know how to integrate the kylin into hive datasource and hbase storage with kerberos. I have used hive beeline and modifid the hbase configuration initialization in the source code. Can the current kylin version support kerberos environment by a better way in mapreduce cubing? > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment a a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authenticated with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Summary: Kylin support hive and hbase authenticated with Kerberos (was: Kylin support hive and hbase authentication with Kerberos) > Kylin support hive and hbase authenticated with Kerberos > > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment by a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authentication with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Description: I want to know how to integrate the kylin into hive datasource and hbase storage with kerberos. I have used hive beeline and modifid the hbase configuration initialization in the source code. Can the current kylin version support kerberos environment by a better way in mapreduce cubing? > Kylin support hive and hbase authentication with Kerberos > - > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > > I want to know how to integrate the kylin into hive datasource and hbase > storage with kerberos. > I have used hive beeline and modifid the hbase configuration initialization > in the source code. > Can the current kylin version support kerberos environment by a better way in > mapreduce cubing? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2755) Kylin support hive and hbase authentication with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-2755: - Request participants: Shaofeng SHI (was: ) > Kylin support hive and hbase authentication with Kerberos > - > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2755) Kylin support hive and hbase authentication with Kerberos
wuyingjun created KYLIN-2755: Summary: Kylin support hive and hbase authentication with Kerberos Key: KYLIN-2755 URL: https://issues.apache.org/jira/browse/KYLIN-2755 Project: Kylin Issue Type: New Feature Affects Versions: v2.0.0 Reporter: wuyingjun -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2755) Kylin support hive and hbase authentication with Kerberos
[ https://issues.apache.org/jira/browse/KYLIN-2755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun reassigned KYLIN-2755: Assignee: wuyingjun > Kylin support hive and hbase authentication with Kerberos > - > > Key: KYLIN-2755 > URL: https://issues.apache.org/jira/browse/KYLIN-2755 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: wuyingjun >Assignee: wuyingjun > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2753) Job duration may become negative after the job resumed from error or paused status
[ https://issues.apache.org/jira/browse/KYLIN-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-2753. -- Resolution: Fixed Fix Version/s: v2.1.0 LGTM. Merged. https://github.com/apache/kylin/commit/9706e30551ed47165ade083e5820d49e6cfa35be > Job duration may become negative after the job resumed from error or paused > status > -- > > Key: KYLIN-2753 > URL: https://issues.apache.org/jira/browse/KYLIN-2753 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Fix For: v2.1.0 > > Attachments: > 0001-KYLIN-2753-Job-duration-may-become-negative-after-th.patch > > > After KYLIN-2655, we calculate the 'Interrupt time' when a job resumed from > error/stopped status to give more accurate job duration time. However, when > calculate the real duration time = 'Total time' - 'Interrupt time', we could > get a negative value. > The root cause is 'Interrupt time' calculated in > DefaultChainedExecutable.onExecuteStart method. The 'Interrupt time' value is > not only added with the true 'Interrupt time' when the job scheduled after > resumed, but also increased when subsequent sub tasks executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2753) Job duration may become negative after the job resumed from error or paused status
[ https://issues.apache.org/jira/browse/KYLIN-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098412#comment-16098412 ] Billy Liu commented on KYLIN-2753: -- Thanks [~zhengd], I hit this bug today also. > Job duration may become negative after the job resumed from error or paused > status > -- > > Key: KYLIN-2753 > URL: https://issues.apache.org/jira/browse/KYLIN-2753 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Attachments: > 0001-KYLIN-2753-Job-duration-may-become-negative-after-th.patch > > > After KYLIN-2655, we calculate the 'Interrupt time' when a job resumed from > error/stopped status to give more accurate job duration time. However, when > calculate the real duration time = 'Total time' - 'Interrupt time', we could > get a negative value. > The root cause is 'Interrupt time' calculated in > DefaultChainedExecutable.onExecuteStart method. The 'Interrupt time' value is > not only added with the true 'Interrupt time' when the job scheduled after > resumed, but also increased when subsequent sub tasks executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2753) Job duration may become negative after the job resumed from error or paused status
[ https://issues.apache.org/jira/browse/KYLIN-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu reassigned KYLIN-2753: Assignee: zhengdong > Job duration may become negative after the job resumed from error or paused > status > -- > > Key: KYLIN-2753 > URL: https://issues.apache.org/jira/browse/KYLIN-2753 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Attachments: > 0001-KYLIN-2753-Job-duration-may-become-negative-after-th.patch > > > After KYLIN-2655, we calculate the 'Interrupt time' when a job resumed from > error/stopped status to give more accurate job duration time. However, when > calculate the real duration time = 'Total time' - 'Interrupt time', we could > get a negative value. > The root cause is 'Interrupt time' calculated in > DefaultChainedExecutable.onExecuteStart method. The 'Interrupt time' value is > not only added with the true 'Interrupt time' when the job scheduled after > resumed, but also increased when subsequent sub tasks executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2754) Sync issue when reload existing hive table
[ https://issues.apache.org/jira/browse/KYLIN-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-2754. -- Resolution: Fixed Fix Version/s: v2.1.0 > Sync issue when reload existing hive table > -- > > Key: KYLIN-2754 > URL: https://issues.apache.org/jira/browse/KYLIN-2754 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Fix For: v2.1.0 > > Attachments: > 0001-KYLIN-2754-Sync-issue-when-reload-existing-hive-tabl.patch > > > When loading hive table, after putting the TableExtDesc to resource store, we > should also update local and remote TableExtDesc cache to avoid overwriting > conflict exception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2754) Sync issue when reload existing hive table
[ https://issues.apache.org/jira/browse/KYLIN-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098404#comment-16098404 ] Billy Liu commented on KYLIN-2754: -- Thanks [~zhengd], LGTM. Patch merged. https://github.com/apache/kylin/commit/6519b18faa388ed017567cfb33aa094fcee70e8b > Sync issue when reload existing hive table > -- > > Key: KYLIN-2754 > URL: https://issues.apache.org/jira/browse/KYLIN-2754 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Attachments: > 0001-KYLIN-2754-Sync-issue-when-reload-existing-hive-tabl.patch > > > When loading hive table, after putting the TableExtDesc to resource store, we > should also update local and remote TableExtDesc cache to avoid overwriting > conflict exception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2754) Sync issue when reload existing hive table
[ https://issues.apache.org/jira/browse/KYLIN-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu reassigned KYLIN-2754: Assignee: zhengdong > Sync issue when reload existing hive table > -- > > Key: KYLIN-2754 > URL: https://issues.apache.org/jira/browse/KYLIN-2754 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong >Assignee: zhengdong > Attachments: > 0001-KYLIN-2754-Sync-issue-when-reload-existing-hive-tabl.patch > > > When loading hive table, after putting the TableExtDesc to resource store, we > should also update local and remote TableExtDesc cache to avoid overwriting > conflict exception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2754) Sync issue when reload existing hive table
[ https://issues.apache.org/jira/browse/KYLIN-2754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengdong updated KYLIN-2754: - Attachment: 0001-KYLIN-2754-Sync-issue-when-reload-existing-hive-tabl.patch > Sync issue when reload existing hive table > -- > > Key: KYLIN-2754 > URL: https://issues.apache.org/jira/browse/KYLIN-2754 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong > Attachments: > 0001-KYLIN-2754-Sync-issue-when-reload-existing-hive-tabl.patch > > > When loading hive table, after putting the TableExtDesc to resource store, we > should also update local and remote TableExtDesc cache to avoid overwriting > conflict exception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2754) Sync issue when reload existing hive table
zhengdong created KYLIN-2754: Summary: Sync issue when reload existing hive table Key: KYLIN-2754 URL: https://issues.apache.org/jira/browse/KYLIN-2754 Project: Kylin Issue Type: Bug Affects Versions: v2.1.0 Reporter: zhengdong When loading hive table, after putting the TableExtDesc to resource store, we should also update local and remote TableExtDesc cache to avoid overwriting conflict exception. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2753) Job duration may become negative after the job resumed from error or paused status
[ https://issues.apache.org/jira/browse/KYLIN-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengdong updated KYLIN-2753: - Attachment: 0001-KYLIN-2753-Job-duration-may-become-negative-after-th.patch > Job duration may become negative after the job resumed from error or paused > status > -- > > Key: KYLIN-2753 > URL: https://issues.apache.org/jira/browse/KYLIN-2753 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.1.0 >Reporter: zhengdong > Attachments: > 0001-KYLIN-2753-Job-duration-may-become-negative-after-th.patch > > > After KYLIN-2655, we calculate the 'Interrupt time' when a job resumed from > error/stopped status to give more accurate job duration time. However, when > calculate the real duration time = 'Total time' - 'Interrupt time', we could > get a negative value. > The root cause is 'Interrupt time' calculated in > DefaultChainedExecutable.onExecuteStart method. The 'Interrupt time' value is > not only added with the true 'Interrupt time' when the job scheduled after > resumed, but also increased when subsequent sub tasks executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2753) Job duration may become negative after the job resumed from error or paused status
zhengdong created KYLIN-2753: Summary: Job duration may become negative after the job resumed from error or paused status Key: KYLIN-2753 URL: https://issues.apache.org/jira/browse/KYLIN-2753 Project: Kylin Issue Type: Bug Affects Versions: v2.1.0 Reporter: zhengdong After KYLIN-2655, we calculate the 'Interrupt time' when a job resumed from error/stopped status to give more accurate job duration time. However, when calculate the real duration time = 'Total time' - 'Interrupt time', we could get a negative value. The root cause is 'Interrupt time' calculated in DefaultChainedExecutable.onExecuteStart method. The 'Interrupt time' value is not only added with the true 'Interrupt time' when the job scheduled after resumed, but also increased when subsequent sub tasks executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2749) Merge may remove old segments without saving merged segment
[
https://issues.apache.org/jira/browse/KYLIN-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098088#comment-16098088
]
Alexander Sterligov commented on KYLIN-2749:
It is reproduced again. It is caused by kylin OOM (system, not OutOfMemoryError)
Merge process is in progress, old segment tables are already removed, but cube
json contains references for them.
Merged segment is in NEW state and "to hfile" job is running.
It looks like it started job, which successfully worked almost up to the end
and then started a new one after OOM.
Does kylin run its jobs in separate process?
> Merge may remove old segments without saving merged segment
> ---
>
> Key: KYLIN-2749
> URL: https://issues.apache.org/jira/browse/KYLIN-2749
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
>
> Merge started to work on last 7 segments.
> During the process hbase had a failure because of spot-instances shutdown in
> Amazon. Data was not lost, because it is at S3.
> I stopped kylin and did hbase hbck --repair. During the report of repair I
> didn't see any information about lost data, just redistribution of regions.
> Then after kylin was started I cannot query data from the last 7 segments:
> {quote}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hbase.TableNotFoundException: Table 'KYLIN_7MMHCHKVVB' was
> not found, got: KYLIN_7H3WSPX1UJ.
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:67)
> at
> org.apache.kylin.storage.hbase.cube.v2.ExpectedSizeIterator.next(ExpectedSizeIterator.java:31)
> at
> com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:583)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator$2.hasNext(SegmentCubeTupleIterator.java:116)
> at
> org.apache.kylin.storage.gtrecord.SegmentCubeTupleIterator.hasNext(SegmentCubeTupleIterator.java:149)
> at com.google.common.collect.Iterators$6.hasNext(Iterators.java:582)
> at
> org.apache.kylin.storage.gtrecord.SequentialCubeTupleIterator.hasNext(SequentialCubeTupleIterator.java:129)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:67)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:826)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:331)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:294)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:553)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:544)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:193)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:607)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:600)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:615)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:148)
> ... 77 more
> Caused by: org.apache.hadoop.hbase.TableNotFoundException: Table
> 'KYLIN_7MMHCHKVVB' was not found, got: KYLIN_7H3WSPX1UJ.
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1310)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1189)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1173)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1130)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:965)
> at
>
[jira] [Updated] (KYLIN-2703) kylin supports managing access rights for project and cube through apache ranger.
[ https://issues.apache.org/jira/browse/KYLIN-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2703: Attachment: (was: 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch) > kylin supports managing access rights for project and cube through apache > ranger. > - > > Key: KYLIN-2703 > URL: https://issues.apache.org/jira/browse/KYLIN-2703 > Project: Kylin > Issue Type: New Feature > Components: General >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: newbie, patch > Attachments: > 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch > > > Ranger is a framework to enable, monitor and manage comprehensive data > security across the Hadoop platform. Apache Ranger has the following goals: > 1. Centralized security administration to manage all security related tasks > in a central UI or using REST APIs. > 2. Fine grained authorization to do a specific action and/or operation with > Hadoop component/tool and managed through a central administration tool > 3. Standardize authorization method across all Hadoop components. > 4. Enhanced support for different authorization methods - Role based access > control, attribute based access control etc. > 5. Centralize auditing of user access and administrative actions (security > related) within all the components of Hadoop. > Ranger has supported enable, monitor and manage following components: > 1. HDFS > 2. HIVE > 3. HBASE > 4. KNOX > 5. YARN > 6. STORM > 7. SOLR > 8. KAFKA > 9. ATLAS > In order to improve the flexibility of kylin privilege control and enhance > value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, > Kylin should also support that using Ranger to control access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/RANGER-1672 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2703) kylin supports managing access rights for project and cube through apache ranger.
[ https://issues.apache.org/jira/browse/KYLIN-2703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2703: Attachment: 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch > kylin supports managing access rights for project and cube through apache > ranger. > - > > Key: KYLIN-2703 > URL: https://issues.apache.org/jira/browse/KYLIN-2703 > Project: Kylin > Issue Type: New Feature > Components: General >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: newbie, patch > Attachments: > 0001-KYLIN-2703-kylin-supports-managing-access-rights-for.patch > > > Ranger is a framework to enable, monitor and manage comprehensive data > security across the Hadoop platform. Apache Ranger has the following goals: > 1. Centralized security administration to manage all security related tasks > in a central UI or using REST APIs. > 2. Fine grained authorization to do a specific action and/or operation with > Hadoop component/tool and managed through a central administration tool > 3. Standardize authorization method across all Hadoop components. > 4. Enhanced support for different authorization methods - Role based access > control, attribute based access control etc. > 5. Centralize auditing of user access and administrative actions (security > related) within all the components of Hadoop. > Ranger has supported enable, monitor and manage following components: > 1. HDFS > 2. HIVE > 3. HBASE > 4. KNOX > 5. YARN > 6. STORM > 7. SOLR > 8. KAFKA > 9. ATLAS > In order to improve the flexibility of kylin privilege control and enhance > value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, > Kylin should also support that using Ranger to control access rights for > project and cube. > Specific implementation plan is as following: > On the ranger website, administrators can configure policies to control user > access to projects and cube permissions. > Kylin provides an abstract class and authorization interfaces for use by the > ranger plugin. kylin instantiates ranger plugin’s implementation class when > starting(this class extends the abstract class provided by kylin). > Ranger plugin periodically polls ranger admin, updates the policy to the > local, and updates project and cube access rights based on policy information. > In the Kylin side： > 1. Kylin provides an abstract class that enables the ranger plugin's > implementation class to extend. > 2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin > implementation class's name. > 3. Instantiate the ranger plugin implementation class when starting kylin. > 4. kylin provides authorization interfaces for ranger plugin calls. > 5. According to the ranger authorization configuration item, hide kylin's > authorization management page. > 6. Using ranger manager access rights of the kylin does not affect kylin's > existing permissions functions and logic. > In the Ranger side： > 1. Ranger plugin will periodically polls ranger admin, updates the policy to > the local. > 2. The ranger plugin invoking the authorization interfaces provided by kylin > to updates the project and cube access rights based on the policy information. > reference link:https://issues.apache.org/jira/browse/RANGER-1672 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
