[jira] [Updated] (KYLIN-3388) Data may become not correct if mappers fail during the redistribute step, "distribute by rand()"
[ https://issues.apache.org/jira/browse/KYLIN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3388: Fix Version/s: v2.3.2 > Data may become not correct if mappers fail during the redistribute step, > "distribute by rand()" > > > Key: KYLIN-3388 > URL: https://issues.apache.org/jira/browse/KYLIN-3388 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Zhong Yanghong >Priority: Critical > Fix For: v2.4.0, v2.3.2 > > Attachments: Hive Issue - distribute by rand().png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1977) Update front end dependency (bower)
[ https://issues.apache.org/jira/browse/KYLIN-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pan, Julian closed KYLIN-1977. -- Resolution: Invalid > Update front end dependency (bower) > --- > > Key: KYLIN-1977 > URL: https://issues.apache.org/jira/browse/KYLIN-1977 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Zhong,Jason >Priority: Major > > 1. There is warning "Code incompatibilities may occur.", when I install bower > component. [angular, nvd3, d3, moment, angular-cookies version not consistent] > 2. Some duplicate component: ng-grid & angular-ui-grid, > angular-chosen-localytics & angular-ui-select, font-awesome & > components-font-awesome > 3. angularjs-nvd3-directives is beta version, maybe we can replace it by > angular-nvd3. > 4. some components version are out of date, could we upgrade them such as > angular-bootstrap... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1977) Update front end dependency (bower)
[ https://issues.apache.org/jira/browse/KYLIN-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509053#comment-16509053 ] Pan, Julian commented on KYLIN-1977: Yeah, I think it can be close. > Update front end dependency (bower) > --- > > Key: KYLIN-1977 > URL: https://issues.apache.org/jira/browse/KYLIN-1977 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Zhong,Jason >Priority: Major > > 1. There is warning "Code incompatibilities may occur.", when I install bower > component. [angular, nvd3, d3, moment, angular-cookies version not consistent] > 2. Some duplicate component: ng-grid & angular-ui-grid, > angular-chosen-localytics & angular-ui-select, font-awesome & > components-font-awesome > 3. angularjs-nvd3-directives is beta version, maybe we can replace it by > angular-nvd3. > 4. some components version are out of date, could we upgrade them such as > angular-bootstrap... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[
https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509050#comment-16509050
]
ASF GitHub Bot commented on KYLIN-1948:
---
shaofengshi closed pull request #153: KYLIN-1948, add test to cover the case.
URL: https://github.com/apache/kylin/pull/153
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-metadata/src/test/java/org/apache/kylin/dimension/IntegerDimEncTest.java
b/core-metadata/src/test/java/org/apache/kylin/dimension/IntegerDimEncTest.java
index 039e05637b..d731a87cf8 100644
---
a/core-metadata/src/test/java/org/apache/kylin/dimension/IntegerDimEncTest.java
+++
b/core-metadata/src/test/java/org/apache/kylin/dimension/IntegerDimEncTest.java
@@ -109,6 +109,15 @@ public void testNull() {
}
}
+@Test
+// For JIRA: KYLIN-1948
+public void testEncodeDecodeMinusOne() {
+for (int i = 1; i < 9; i++) {
+IntegerDimEnc enc = new IntegerDimEnc(i);
+testEncodeDecode(enc, -1);
+}
+}
+
@Test
public void testEncodeDecode() {
for (int i = 1; i <= successValue.size(); i++) {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> IntegerDimEnc, does not encode -1 correctly
> ---
>
> Key: KYLIN-1948
> URL: https://issues.apache.org/jira/browse/KYLIN-1948
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.4.0
>
>
> The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1
> is a common value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[ https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509051#comment-16509051 ] ASF subversion and git services commented on KYLIN-1948: Commit fd7cc177f78e4b037d57e7a28b749c747485040e in kylin's branch refs/heads/master from tttMelody [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=fd7cc17 ] KYLIN-1948, add test to cover the case. > IntegerDimEnc, does not encode -1 correctly > --- > > Key: KYLIN-1948 > URL: https://issues.apache.org/jira/browse/KYLIN-1948 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1 > is a common value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3386) the TopN measure validate code refactor to make it more clear
[ https://issues.apache.org/jira/browse/KYLIN-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3386: Fix Version/s: v2.4.0 > the TopN measure validate code refactor to make it more clear > - > > Key: KYLIN-3386 > URL: https://issues.apache.org/jira/browse/KYLIN-3386 > Project: Kylin > Issue Type: Improvement >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Minor > Fix For: v2.4.0 > > Attachments: 0001-KYLIN-3386-adjust-TopN-measure-validation-code.patch > > > _Nowadays, the FunctionRule used for validating measure info, and it should > call measureType.validate() code rather than add the logical in this 2 > places_ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[ https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao resolved KYLIN-1948. --- Resolution: Fixed > IntegerDimEnc, does not encode -1 correctly > --- > > Key: KYLIN-1948 > URL: https://issues.apache.org/jira/browse/KYLIN-1948 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1 > is a common value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3400) wipeCache and createCubeDesc make deadlock happenning
[ https://issues.apache.org/jira/browse/KYLIN-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3400: Fix Version/s: v2.3.2 v2.4.0 > wipeCache and createCubeDesc make deadlock happenning > -- > > Key: KYLIN-3400 > URL: https://issues.apache.org/jira/browse/KYLIN-3400 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v2.3.0 >Reporter: powerinf >Priority: Major > Fix For: v2.4.0, v2.3.2 > > Attachments: kylin_deadlock.log > > > When using the rest API create cube , deadlock occasionally appears，the > kylin web ui hang，and cound not login ，See the [^kylin_deadlock.log] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3397) Kylin UI Timestemp(GMT) issue
[ https://issues.apache.org/jira/browse/KYLIN-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507902#comment-16507902 ] praveenece commented on KYLIN-3397: --- Hi Pan, Could you check this issue? > Kylin UI Timestemp(GMT) issue > - > > Key: KYLIN-3397 > URL: https://issues.apache.org/jira/browse/KYLIN-3397 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: praveenece >Priority: Major > > > Kylin Version 2.3.1 > In Kylin React-the Data has inserted into System Cube table based on GMT > format,but the user > Changed option GMT to GMT+5.30 in kylin UI (Kylin.properties). > when we build segment wise in System Cube(Kylin UI).the system has generated > select statement in first step of build,its > taken internally GMT+5.30 format not for GMT. > below i given generated select query from First step of Create Intermediate > Flat Hive Table(log) > SELECT x,y,z > FROM KYLIN.HIVE_METRICS_QUERY_CUBE_QA as HIVE_METRICS_QUERY_CUBE_QA > WHERE 1=1 AND (((HIVE_METRICS_QUERY_CUBE_QA.KDAY_DATE = '2018-06-01' AND > HIVE_METRICS_QUERY_CUBE_QA.KDAY_TIME >= '17:10:00') > OR (HIVE_METRICS_QUERY_CUBE_QA.KDAY_DATE > '2018-06-01')) > AND ((HIVE_METRICS_QUERY_CUBE_QA.KDAY_DATE = '2018-06-04' AND > HIVE_METRICS_QUERY_CUBE_QA.KDAY_TIME < '16:27:00') > OR (HIVE_METRICS_QUERY_CUBE_QA.KDAY_DATE < '2018-06-04'))) > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[ https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507811#comment-16507811 ] ASF GitHub Bot commented on KYLIN-1948: --- codecov-io commented on issue #153: KYLIN-1948, add test to cover the case. URL: https://github.com/apache/kylin/pull/153#issuecomment-396165401 # [Codecov](https://codecov.io/gh/apache/kylin/pull/153?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@f6b1dfb`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/kylin/pull/153?src=pr=tree) ```diff @@Coverage Diff@@ ## master #153 +/- ## = Coverage ? 22.15% Complexity? 4022 = Files ? 1012 Lines ?61207 Branches ? 8768 = Hits ?13561 Misses?46408 Partials ? 1238 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/153?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/153?src=pr=footer). Last update [f6b1dfb...6140851](https://codecov.io/gh/apache/kylin/pull/153?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IntegerDimEnc, does not encode -1 correctly > --- > > Key: KYLIN-1948 > URL: https://issues.apache.org/jira/browse/KYLIN-1948 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1 > is a common value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[ https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507796#comment-16507796 ] ASF GitHub Bot commented on KYLIN-1948: --- tttMelody opened a new pull request #153: KYLIN-1948, add test to cover the case. URL: https://github.com/apache/kylin/pull/153 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IntegerDimEnc, does not encode -1 correctly > --- > > Key: KYLIN-1948 > URL: https://issues.apache.org/jira/browse/KYLIN-1948 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1 > is a common value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[ https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507797#comment-16507797 ] ASF GitHub Bot commented on KYLIN-1948: --- asfgit commented on issue #153: KYLIN-1948, add test to cover the case. URL: https://github.com/apache/kylin/pull/153#issuecomment-396160212 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > IntegerDimEnc, does not encode -1 correctly > --- > > Key: KYLIN-1948 > URL: https://issues.apache.org/jira/browse/KYLIN-1948 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1 > is a common value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2932) Simplify the thread model for in-memory cubing
[ https://issues.apache.org/jira/browse/KYLIN-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1650#comment-1650 ] ASF subversion and git services commented on KYLIN-2932: Commit 9f64f04f96105b0722b7af4e4e7fd89f66d8905f in kylin's branch refs/heads/KYLIN-2932-master from U-CORPmingmwang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9f64f04 ] APACHE-KYLIN-2932: Simplify the thread model for in-memory cubing Signed-off-by: shaofengshi > Simplify the thread model for in-memory cubing > -- > > Key: KYLIN-2932 > URL: https://issues.apache.org/jira/browse/KYLIN-2932 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang Ken >Assignee: Wang Ken >Priority: Major > Fix For: v2.4.0 > > Attachments: APACHE-KYLIN-2932.patch > > > The current implementation uses split threads, task threads and main thread > to do the cube building, there is complex join and error handling logic. > The new implement leverages the ForkJoinPool from JDK, the event split logic > is handled in > main thread. Cuboid task and sub-tasks are handled in fork join pool, cube > results are collected > async and can be write to output earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3375) Some improvements for lookup table - build change
[ https://issues.apache.org/jira/browse/KYLIN-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507772#comment-16507772 ] ASF subversion and git services commented on KYLIN-3375: Commit 20c66d803d2ce0634078d790b441799a7af29991 in kylin's branch refs/heads/KYLIN-2932-master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=20c66d8 ] KYLIN-3375 Some improvements for lookup table - build change - Fix issue when latest ext snapshot exist > Some improvements for lookup table - build change > - > > Key: KYLIN-3375 > URL: https://issues.apache.org/jira/browse/KYLIN-3375 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Fix For: v2.4.0 > > > build change for new lookup table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3373) Some improvements for lookup table - UI part change
[ https://issues.apache.org/jira/browse/KYLIN-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507771#comment-16507771 ] ASF subversion and git services commented on KYLIN-3373: Commit 71d416adaab0c9c4f7b4b99cdb6ce43625a5c58c in kylin's branch refs/heads/KYLIN-2932-master from liapan [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=71d416a ] KYLIN-3373 Some improvements for lookup table - UI part change - Bug fix for add snapshot desc > Some improvements for lookup table - UI part change > --- > > Key: KYLIN-3373 > URL: https://issues.apache.org/jira/browse/KYLIN-3373 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Pan, Julian >Priority: Major > Fix For: v2.4.0 > > > UI part change -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3375) Some improvements for lookup table - build change
[ https://issues.apache.org/jira/browse/KYLIN-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507768#comment-16507768 ] ASF subversion and git services commented on KYLIN-3375: Commit 777f8fa5d3b1f24f59354909de1ca3d28d5bc137 in kylin's branch refs/heads/KYLIN-2932-master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=777f8fa ] KYLIN-3375 Some improvements for lookup table - build change > Some improvements for lookup table - build change > - > > Key: KYLIN-3375 > URL: https://issues.apache.org/jira/browse/KYLIN-3375 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Fix For: v2.4.0 > > > build change for new lookup table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3377) Some improvements for lookup table - snapshot management
[ https://issues.apache.org/jira/browse/KYLIN-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507770#comment-16507770 ] ASF subversion and git services commented on KYLIN-3377: Commit b7d2cb796ca3e3c9bc25837026a9a8966f3e1cad in kylin's branch refs/heads/KYLIN-2932-master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b7d2cb7 ] KYLIN-3377 Some improvements for lookup table - snapshot management > Some improvements for lookup table - snapshot management > > > Key: KYLIN-3377 > URL: https://issues.apache.org/jira/browse/KYLIN-3377 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine, REST Service >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Fix For: v2.4.0 > > > including build snapshot independently, view snapshot information, etc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Spark cubing without hive-site.xml
[
https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507776#comment-16507776
]
ASF subversion and git services commented on KYLIN-3137:
Commit 8205fac0eb9c7c986e7690166793c5cc08742cd6 in kylin's branch
refs/heads/KYLIN-2932-master from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8205fac ]
KYLIN-3137 update kylin.properties for spark 2.1
> Spark cubing without hive-site.xml
> --
>
> Key: KYLIN-3137
> URL: https://issues.apache.org/jira/browse/KYLIN-3137
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Others, Spark Engine
>Affects Versions: v2.2.0
>Reporter: Ruslan Dautkhanov
>Assignee: Shaofeng SHI
>Priority: Major
> Labels: cdh, cloudera, configuration, hive
> Fix For: v2.4.0
>
>
> Getting following exception while trying to build a cube
> {noformat}
> java.lang.RuntimeException: Cannot find hive-site.xml in
> kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you
> must set kylin.env.hadoop-conf-dir to a dir which contains at least
> core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I am using Kylin binaries for CDH downloaded from kylin.apache.org.
> Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's
> distribution for Hadoop.
> hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf
> The other four files:
> core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml
> can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in
> /etc/hive/conf .
> Would be great to have this adjusted for CDH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3137) Spark cubing without hive-site.xml
[
https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507775#comment-16507775
]
ASF subversion and git services commented on KYLIN-3137:
Commit e0f29e29164d44581c133314ac01c419e4277c35 in kylin's branch
refs/heads/KYLIN-2932-master from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=e0f29e2 ]
KYLIN-3137 Spark cubing refine
> Spark cubing without hive-site.xml
> --
>
> Key: KYLIN-3137
> URL: https://issues.apache.org/jira/browse/KYLIN-3137
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Others, Spark Engine
>Affects Versions: v2.2.0
>Reporter: Ruslan Dautkhanov
>Assignee: Shaofeng SHI
>Priority: Major
> Labels: cdh, cloudera, configuration, hive
> Fix For: v2.4.0
>
>
> Getting following exception while trying to build a cube
> {noformat}
> java.lang.RuntimeException: Cannot find hive-site.xml in
> kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you
> must set kylin.env.hadoop-conf-dir to a dir which contains at least
> core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I am using Kylin binaries for CDH downloaded from kylin.apache.org.
> Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's
> distribution for Hadoop.
> hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf
> The other four files:
> core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml
> can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in
> /etc/hive/conf .
> Would be great to have this adjusted for CDH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3376) Some improvements for lookup table - query change
[ https://issues.apache.org/jira/browse/KYLIN-3376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507769#comment-16507769 ] ASF subversion and git services commented on KYLIN-3376: Commit 5a96f8b46ce73684a19a781a7aa0ce6a587900c7 in kylin's branch refs/heads/KYLIN-2932-master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5a96f8b ] KYLIN-3376 Some improvements for lookup table - query change > Some improvements for lookup table - query change > - > > Key: KYLIN-3376 > URL: https://issues.apache.org/jira/browse/KYLIN-3376 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Fix For: v2.4.0 > > > query part change -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3375) Some improvements for lookup table - build change
[ https://issues.apache.org/jira/browse/KYLIN-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507774#comment-16507774 ] ASF subversion and git services commented on KYLIN-3375: Commit 18316ec54a7b154de3ad3132b4e834efcc0d8d14 in kylin's branch refs/heads/KYLIN-2932-master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=18316ec ] KYLIN-3375 Some improvements for lookup table - build change - Fix rocksdb cache UT issue > Some improvements for lookup table - build change > - > > Key: KYLIN-3375 > URL: https://issues.apache.org/jira/browse/KYLIN-3375 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Fix For: v2.4.0 > > > build change for new lookup table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3283) Support values RelNode
[
https://issues.apache.org/jira/browse/KYLIN-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yiming.xu resolved KYLIN-3283.
--
Resolution: Fixed
> Support values RelNode
> --
>
> Key: KYLIN-3283
> URL: https://issues.apache.org/jira/browse/KYLIN-3283
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: yiming.xu
>Assignee: yiming.xu
>Priority: Major
> Fix For: v2.4.0
>
>
> for queries like:
> select leaf_categ_id, sum(price) as sum_price from test_kylin_fact group by
> leaf_categ_id
> union all
> select 1 as leaf_categ_id2, 2 as sum_price2
> the expected logical plan after volcano is:
> {code:java}
> OLAPToEnumerableConverter
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPUnionRel(all=[true], ctx=[], all=[true])
> OLAPAggregateRel(group=[{0}], SUM_PRICE=[SUM($1)], ctx=[])
> OLAPProjectRel(LEAF_CATEG_ID=[$4], PRICE=[$8], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
> 19, 20, 21, 22, 23, 24, 25, 26]])
> OLAPProjectRel(LEAF_CATEG_ID2=[1], SUM_PRICE2=[2], ctx=[])
> OLAPValuesRel(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
> {code}
> however the actual is:
> {code:java}
> EnumerableLimit(fetch=[5])
> EnumerableUnion(all=[true])
> OLAPToEnumerableConverter
> OLAPAggregateRel(group=[{0}], SUM_PRICE=[SUM($1)], ctx=[])
> OLAPProjectRel(LEAF_CATEG_ID=[$4], PRICE=[$8], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
> 19, 20, 21, 22, 23, 24, 25, 26]])
> EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[2],
> LEAF_CATEG_ID2=[$t1], SUM_PRICE2=[$t2])
> EnumerableValues(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3398.
-
Resolution: Fixed
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: nichunen
>Priority: Major
> Fix For: v2.4.0
>
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3398) Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
[
https://issues.apache.org/jira/browse/KYLIN-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3398:
---
Assignee: nichunen (was: jiatao.tao)
Fix Version/s: v2.4.0
> Inaccurate arithmetic operation in LookupTableToHFileJob#calculateShardNum
> --
>
> Key: KYLIN-3398
> URL: https://issues.apache.org/jira/browse/KYLIN-3398
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: nichunen
>Priority: Major
> Fix For: v2.4.0
>
>
> There're two issues with the following code:
> {code}
> private int calculateShardNum(KylinConfig kylinConfig, long dataSize) {
> long shardSize = kylinConfig.getExtTableSnapshotShardingMB() * 1024 *
> 1024;
> return dataSize < shardSize ? 1 : (int) (Math.ceil(dataSize /
> shardSize));
> {code}
> getExtTableSnapshotShardingMB returns an int. The multiplication is done
> using 32-bit arithmetic, and then used in a context that expects an
> expression of type "long".
> Second, Math.ceil expects a double. The integer division would lose some
> precision.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3378) Support Kafka table join with Hive tables
[ https://issues.apache.org/jira/browse/KYLIN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3378. - Resolution: Fixed > Support Kafka table join with Hive tables > - > > Key: KYLIN-3378 > URL: https://issues.apache.org/jira/browse/KYLIN-3378 > Project: Kylin > Issue Type: Improvement > Components: Streaming >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > At this moment, if the data source is Kafka, only 1 table allowed in the data > model. In some cases, joining the kafka stream with hive lookup tables is > expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3262) Produce jacoco exec files for integration test
[ https://issues.apache.org/jira/browse/KYLIN-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507726#comment-16507726 ] Shaofeng SHI commented on KYLIN-3262: - Yanghong, I'm afrad adding a binary jar in Kylin source is not a good way. Can download it from some script? > Produce jacoco exec files for integration test > -- > > Key: KYLIN-3262 > URL: https://issues.apache.org/jira/browse/KYLIN-3262 > Project: Kylin > Issue Type: Task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.4.0 > > Attachments: APACHE-KYLIN-3262.patch > > > Currently to calculate code coverage, kylin only considers UT. However, IT > also covers lots of code. It's better to include this for code coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3221) Allow externalizing lookup table snapshot
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3221.
-
Resolution: Fixed
The frontend error has been fixed with an update in the PR. And passed the CI,
merged in master branch.
> Allow externalizing lookup table snapshot
> -
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.4.0
>
> Attachments: KYLIN-3221-web-error.png
>
>
> There are two limitations for current look table design:
> # lookup table size is limited, because table snapshot need to be cached in
> Kylin server, too large snapshot table will break the server.
> # lookup table snapshot references are stored in all segments of the cube,
> cannot support global snapshot table, the global snapshot table means when
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the
> existing lookup table design, below is the initial document, any comments and
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
> {{private}} {{List snapshotTableDescList =
> Collections.emptyList();}}|
> SnapshotTableDesc defines how table is stored and whether it is global or
> not, currently we can support two types of store:
> # "metaStore", table snapshot is stored in the metadata store, it is the
> same as current design, and this is the default option.
> # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
> {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>
> @JsonProperty("local_cache_enable")
> private boolean enableLocalCache = true;
>
> {{@JsonProperty}}{{(}}{{"global"}}{{)}}
> {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>
> Add 'snapshots' property in CubeInstance, to store snapshots resource path
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
> {{private}} {{Map snapshots; }}{{// tableName ->
> tableResoucePath mapping}}|
>
> Add new meta model ExtTableSnapshot to describe the extended table snapshot
> information, the information is stored in a new metastore path:
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
> {{private}} {{TableSignature signature;}}
>
> {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
> {{private}} {{String storageLocationIdentifier;}}
>
> @JsonProperty("key_columns")
> private String[] keyColumns; // the key columns of the table
>
> @JsonProperty("storage_type")
> private String storageType;
>
> {{@JsonProperty}}{{(}}{{"size"}}{{)}}
> {{private}} {{long}} {{size;}}
>
> {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
> {{private}} {{long}} {{rowCnt;}}|
>
> Add new section in 'Advance Setting' tab when do cube design, user can set
> table snapshot properties for each table, and by default, it is segment level
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use
> MapReduce job convert the hive source table to hfiles, and then bulk load
> hfiles to HTable. So it will add two job steps to do the lookup table
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
> the first 2 bytes is shard number, HBase table can be pre-split, the shard
> size is configurable through Kylin's properties:
> "kylin.snapshot.ext.shard-mb", default size is 500MB.
> 1 column family c, multiple columns which column name is the index of the
> column in the table definition
> |c|
> |1|2|...|
>
> h2. Query
> For key lookup query, directly call hbase get api to get entire row according
> to key (call local cache if there is local cache enable)
> For queries that need fetch keys according to the derived columns, iterate
> all rows to get related keys. (call local cache if there is local cache
> enable)
> For queries
[jira] [Updated] (KYLIN-3221) Allow externalizing lookup table snapshot
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3221:
Summary: Allow externalizing lookup table snapshot (was: Some improvements
for lookup table )
> Allow externalizing lookup table snapshot
> -
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.4.0
>
> Attachments: KYLIN-3221-web-error.png
>
>
> There are two limitations for current look table design:
> # lookup table size is limited, because table snapshot need to be cached in
> Kylin server, too large snapshot table will break the server.
> # lookup table snapshot references are stored in all segments of the cube,
> cannot support global snapshot table, the global snapshot table means when
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the
> existing lookup table design, below is the initial document, any comments and
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
> {{private}} {{List snapshotTableDescList =
> Collections.emptyList();}}|
> SnapshotTableDesc defines how table is stored and whether it is global or
> not, currently we can support two types of store:
> # "metaStore", table snapshot is stored in the metadata store, it is the
> same as current design, and this is the default option.
> # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
> {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>
> @JsonProperty("local_cache_enable")
> private boolean enableLocalCache = true;
>
> {{@JsonProperty}}{{(}}{{"global"}}{{)}}
> {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>
> Add 'snapshots' property in CubeInstance, to store snapshots resource path
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
> {{private}} {{Map snapshots; }}{{// tableName ->
> tableResoucePath mapping}}|
>
> Add new meta model ExtTableSnapshot to describe the extended table snapshot
> information, the information is stored in a new metastore path:
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
> {{private}} {{TableSignature signature;}}
>
> {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
> {{private}} {{String storageLocationIdentifier;}}
>
> @JsonProperty("key_columns")
> private String[] keyColumns; // the key columns of the table
>
> @JsonProperty("storage_type")
> private String storageType;
>
> {{@JsonProperty}}{{(}}{{"size"}}{{)}}
> {{private}} {{long}} {{size;}}
>
> {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
> {{private}} {{long}} {{rowCnt;}}|
>
> Add new section in 'Advance Setting' tab when do cube design, user can set
> table snapshot properties for each table, and by default, it is segment level
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use
> MapReduce job convert the hive source table to hfiles, and then bulk load
> hfiles to HTable. So it will add two job steps to do the lookup table
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
> the first 2 bytes is shard number, HBase table can be pre-split, the shard
> size is configurable through Kylin's properties:
> "kylin.snapshot.ext.shard-mb", default size is 500MB.
> 1 column family c, multiple columns which column name is the index of the
> column in the table definition
> |c|
> |1|2|...|
>
> h2. Query
> For key lookup query, directly call hbase get api to get entire row according
> to key (call local cache if there is local cache enable)
> For queries that need fetch keys according to the derived columns, iterate
> all rows to get related keys. (call local cache if there is local cache
> enable)
> For queries that only hit the lookup
[jira] [Resolved] (KYLIN-3289) Refactor the storage garbage clean up code
[
https://issues.apache.org/jira/browse/KYLIN-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3289.
-
Resolution: Fixed
> Refactor the storage garbage clean up code
> --
>
> Key: KYLIN-3289
> URL: https://issues.apache.org/jira/browse/KYLIN-3289
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v2.3.0
>Reporter: Guangyao Li
>Assignee: Guangyao Li
>Priority: Minor
> Fix For: v2.4.0
>
>
> Kylin will produce some garbage data in storage when it runs.
> Now, the clean up tool "{{kylin.sh org.apache.kylin.tool.StorageCleanupJob}}"
> can show what is garbage data or clean up the garbage by setting options
> "–delete false" or "delete true".
> But Kylin can't show the size of garbage data for users.
> This reconfiguration adds some member variables and methods recording the
> garbage size in the detection process.
> After clean up job running over, Kylin can get the information about garbage
> size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
