[jira] [Commented] (KYLIN-3295) Unused method SQLDigestUtil#appendTsFilterToExecute
[
https://issues.apache.org/jira/browse/KYLIN-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495340#comment-16495340
]
ASF subversion and git services commented on KYLIN-3295:
Commit 8325ae9abd278d3ce6eae2551403bf41ba9be35a in kylin's branch
refs/heads/master from [~Aron.tao]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8325ae9 ]
KYLIN-3295, clean unused class:SQLDigestUtil
> Unused method SQLDigestUtil#appendTsFilterToExecute
> ---
>
> Key: KYLIN-3295
> URL: https://issues.apache.org/jira/browse/KYLIN-3295
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.4.0
>
>
> SQLDigestUtil#appendTsFilterToExecute is not called anywhere.
> {code}
>T ret = action.apply(null);
> {code}
> Passing null to {{apply}} seems incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3295) Unused method SQLDigestUtil#appendTsFilterToExecute
[
https://issues.apache.org/jira/browse/KYLIN-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495338#comment-16495338
]
ASF GitHub Bot commented on KYLIN-3295:
---
yiming187 commented on issue #145: KYLIN-3295, clean unused class:SQLDigestUtil
URL: https://github.com/apache/kylin/pull/145#issuecomment-393212364
LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Unused method SQLDigestUtil#appendTsFilterToExecute
> ---
>
> Key: KYLIN-3295
> URL: https://issues.apache.org/jira/browse/KYLIN-3295
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.4.0
>
>
> SQLDigestUtil#appendTsFilterToExecute is not called anywhere.
> {code}
>T ret = action.apply(null);
> {code}
> Passing null to {{apply}} seems incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3295) Unused method SQLDigestUtil#appendTsFilterToExecute
[
https://issues.apache.org/jira/browse/KYLIN-3295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495339#comment-16495339
]
ASF GitHub Bot commented on KYLIN-3295:
---
yiming187 closed pull request #145: KYLIN-3295, clean unused class:SQLDigestUtil
URL: https://github.com/apache/kylin/pull/145
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigestUtil.java
b/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigestUtil.java
deleted file mode 100644
index a70a17eef5..00
---
a/core-metadata/src/main/java/org/apache/kylin/metadata/realization/SQLDigestUtil.java
+++ /dev/null
@@ -1,137 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
-*/
-
-package org.apache.kylin.metadata.realization;
-
-import org.apache.kylin.common.util.DateFormat;
-import org.apache.kylin.metadata.datatype.DataType;
-import org.apache.kylin.metadata.filter.ColumnTupleFilter;
-import org.apache.kylin.metadata.filter.CompareTupleFilter;
-import org.apache.kylin.metadata.filter.ConstantTupleFilter;
-import org.apache.kylin.metadata.filter.LogicalTupleFilter;
-import org.apache.kylin.metadata.filter.TupleFilter;
-import org.apache.kylin.metadata.model.TblColRef;
-
-import com.google.common.base.Function;
-import com.google.common.collect.BoundType;
-import com.google.common.collect.Range;
-
-/**
- */
-public class SQLDigestUtil {
-
-public static T appendTsFilterToExecute(SQLDigest sqlDigest,
TblColRef partitionColRef, Range tsRange, Function action) {
-
-// add the boundary condition to query real-time
-TupleFilter originalFilter = sqlDigest.filter;
-sqlDigest.filter = createFilterForRealtime(originalFilter,
partitionColRef, tsRange);
-
-boolean addFilterColumn = false, addAllColumn = false;
-
-if (!sqlDigest.filterColumns.contains(partitionColRef)) {
-sqlDigest.filterColumns.add(partitionColRef);
-addFilterColumn = true;
-}
-
-if (!sqlDigest.allColumns.contains(partitionColRef)) {
-sqlDigest.allColumns.add(partitionColRef);
-addAllColumn = true;
-}
-
-T ret = action.apply(null);
-
-// restore the sqlDigest
-sqlDigest.filter = originalFilter;
-
-if (addFilterColumn)
-sqlDigest.filterColumns.remove(partitionColRef);
-
-if (addAllColumn)
-sqlDigest.allColumns.remove(partitionColRef);
-
-return ret;
-}
-
-//ts column type differentiate
-private static String formatTimeStr(DataType type, long ts) {
-String ret;
-if (type == DataType.getType("date")) {
-ret = DateFormat.formatToDateStr(ts);
-} else if (type == DataType.getType("long")) {
-ret = String.valueOf(ts);
-} else {
-throw new IllegalArgumentException("Illegal type for partition
column " + type);
-}
-return ret;
-}
-
-private static TupleFilter createFilterForRealtime(TupleFilter
originFilter, TblColRef partitionColRef, Range tsRange) {
-DataType type = partitionColRef.getColumnDesc().getType();
-
-String startTimeStr, endTimeStr;
-CompareTupleFilter startFilter = null, endFilter = null;
-if (tsRange.hasLowerBound()) {
-startTimeStr = formatTimeStr(type, tsRange.lowerEndpoint());
-if (tsRange.lowerBoundType() == BoundType.CLOSED) {
-startFilter = new
CompareTupleFilter(TupleFilter.FilterOperatorEnum.GTE);
-} else {
-startFilter = new
CompareTupleFilter(TupleFilter.FilterOperatorEnum.GT);
-}
-ColumnTupleFilter columnTupleFilter = new
ColumnTupleFilter(partitionColRef);
-ConstantTupleFilter constantTupleFilter = new
ConstantTupleFilter(startTimeStr);
-
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495244#comment-16495244 ] wuyingjun commented on KYLIN-3393: -- Yes, I know . But hadoop and hbase are kylin storage components while I think kafka as A datasource should not need to install in the kylin server node. Now kylin can not directly use the kafka data source from another node unless we install kafka in the kylin server node. It is not comfortable. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495142#comment-16495142 ] Shaofeng SHI commented on KYLIN-3393: - hi yingjun, this is by design: Kylin doesn't embed these jars: it expects to use the jar from user's environment. This can avoid compitable issues in various hadoop releases. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Description: You need hava KAFKA environment and set “KAFKA_HOME” environment variable when building streaming cube, or else you can not submit a streaming cube building job, like pic 1； After we change the kafka dependency scope mode described in the KYLIN-3393.patch , we can build streaming cube without a kafka environment in the kylin service node successfully， like pic 2； was: You need hava KAFKA environment and set “KAFKA_HOME” environment variable when building streaming cube, or you can not submit a streaming cube building job, like pic 1； After we change the kafka dependency scope mode described in the KYLIN-3393.patch , we can build streaming cube without a kafka environment successfully， like pic 2； > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495078#comment-16495078 ] wuyingjun commented on KYLIN-3393: -- [~yimingliu] , Please help me validating this bug. If it exsits , I want to submit a patch to kylin. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment > successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Description: You need hava KAFKA environment and set “KAFKA_HOME” environment variable when building streaming cube, or you can not submit a streaming cube building job, like pic 1； After we change the kafka dependency scope mode described in the KYLIN-3393.patch , we can build streaming cube without a kafka environment successfully， like pic 2； was:You need hava KAFKA environment and set “KAFKA_HOME” environment variable when building streaming cube, or you can not submit a streaming cube building job, like pic 1 > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment > successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3137) Update hive-site.xml locating logic for CDH distro
[
https://issues.apache.org/jira/browse/KYLIN-3137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3137:
---
Assignee: Shaofeng SHI
> Update hive-site.xml locating logic for CDH distro
> --
>
> Key: KYLIN-3137
> URL: https://issues.apache.org/jira/browse/KYLIN-3137
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Others, Spark Engine
>Affects Versions: v2.2.0
>Reporter: Ruslan Dautkhanov
>Assignee: Shaofeng SHI
>Priority: Major
> Labels: cdh, cloudera, configuration, hive
>
> Getting following exception while trying to build a cube
> {noformat}
> java.lang.RuntimeException: Cannot find hive-site.xml in
> kylin_hadoop_conf_dir: /etc/hadoop/conf. In order to enable spark cubing, you
> must set kylin.env.hadoop-conf-dir to a dir which contains at least
> core-site.xml, hdfs-site.xml, hive-site.xml, mapred-site.xml, yarn-site.xml
> at
> org.apache.kylin.engine.spark.SparkExecutable.doWork(SparkExecutable.java:117)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> I am using Kylin binaries for CDH downloaded from kylin.apache.org.
> Yes, indeed hive-site.xml is not in /etc/hadoop/conf in Cloudera's
> distribution for Hadoop.
> hive-site.xml is in /etc/hive/conf, not in /etc/hadoop/conf
> The other four files:
> core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml
> can be found in /etc/hadoop/conf but, again, not hive-site.xml which is in
> /etc/hive/conf .
> Would be great to have this adjusted for CDH.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Attachment: 2.png > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Attachment: (was: after_modify.png) > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Attachment: after_modify.png > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, KYLIN-3393.patch, after_modify.png > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Attachment: KYLIN-3393.patch > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Summary: Building streaming cube do not need to set “KAFKA_HOME” environment variable (was: Building streaming cube do not need to set “KAFKA_HOME” environment variable.) > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable.
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Description: You need hava KAFKA environment and set “KAFKA_HOME” environment variable when building streaming cube, or you can not submit a streaming cube building job, like pic 1 (was: You need set “KAFKA_HOME” environment variable when building streaming cube, or you can not submit a streaming cube building job, like pic 1) > Building streaming cube do not need to set “KAFKA_HOME” environment variable. > -- > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or you can not submit a streaming cube building > job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable.
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Attachment: 1.png > Building streaming cube do not need to set “KAFKA_HOME” environment variable. > -- > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png > > > You need set “KAFKA_HOME” environment variable when building streaming cube, > or you can not submit a streaming cube building job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable.
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyingjun updated KYLIN-3393: - Summary: Building streaming cube do not need to set “KAFKA_HOME” environment variable. (was: Building streaming cube do not set “KAFKA_HOME” environment variable.) > Building streaming cube do not need to set “KAFKA_HOME” environment variable. > -- > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > > You need set “KAFKA_HOME” environment variable when building streaming cube, > or you can not submit a streaming cube building job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3393) Building streaming cube do not set “KAFKA_HOME” environment variable.
wuyingjun created KYLIN-3393: Summary: Building streaming cube do not set “KAFKA_HOME” environment variable. Key: KYLIN-3393 URL: https://issues.apache.org/jira/browse/KYLIN-3393 Project: Kylin Issue Type: Improvement Reporter: wuyingjun Assignee: wuyingjun You need set “KAFKA_HOME” environment variable when building streaming cube, or you can not submit a streaming cube building job, like pic 1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-1768) NDCuboidMapper throws ArrayIndexOutOfBoundsException when dimension is fixed length encoded to more than 256 bytes
[
https://issues.apache.org/jira/browse/KYLIN-1768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-1768:
Fix Version/s: v2.4.0
> NDCuboidMapper throws ArrayIndexOutOfBoundsException when dimension is fixed
> length encoded to more than 256 bytes
> --
>
> Key: KYLIN-1768
> URL: https://issues.apache.org/jira/browse/KYLIN-1768
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.5.2
>Reporter: Dayue Gao
>Assignee: Dayue Gao
>Priority: Major
> Fix For: v2.4.0
>
>
> When user defined a dimension which is fixed length encoded to more than 256
> bytes, "Build N-Dimension Cuboid Data" step failed in map phase. The stack
> trace is shown below:
> {noformat}
> Error: java.lang.ArrayIndexOutOfBoundsException at
> java.lang.System.arraycopy(Native Method)
> at org.apache.kylin.cube.common.RowKeySplitter.split(RowKeySplitter.java:103)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.map(NDCuboidMapper.java:125)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.map(NDCuboidMapper.java:49)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
> The reason is that `RowKeySplitter` is hardcoded to 65 splits and 256 bytes
> per split, and trying to put a larger encoded dimension throws
> ArrayIndexOutOfBoundsException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-1750) streaming_cubename_starttimestamp_endtimestamp.log not be deleted on a regular time
[ https://issues.apache.org/jira/browse/KYLIN-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1750. --- Resolution: Won't Fix > streaming_cubename_starttimestamp_endtimestamp.log not be deleted on a > regular time > --- > > Key: KYLIN-1750 > URL: https://issues.apache.org/jira/browse/KYLIN-1750 > Project: Kylin > Issue Type: Bug > Components: Streaming > Environment: Kylin:1.5.2,hadoop:2.4.0.HBase:0.98.8,ZK:3.4.6 >Reporter: dcswinner >Priority: Major > Labels: build > > According to the https://kylin.apache.org/docs15/tutorial/cube_streaming.html > when I add a crontab job for my cube in my crontab: > */5 * * * * sh /root/apache-kylin-1.5.2-bin/bin/streaming_build.sh > STREAMING_CUBE 30 0 > The streaming cube log not be deleted and all of the logs are written at the > same time.Then in the directory, all of the logs like this: > -rw-rw-r-- 1 kylin kylin 4038144 5月 31 10:38 > streaming_STREAMING_CUBE_146438100_146438130.log > -rw-rw-r-- 1 kylin kylin 3954941 5月 31 10:38 > streaming_STREAMING_CUBE_146438700_146438730.log > -rw-rw-r-- 1 kylin kylin 4333603 5月 31 10:38 > streaming_STREAMING_CUBE_146436030_146436060.log > -rw-rw-r-- 1 kylin kylin 3966647 5月 31 10:38 > streaming_STREAMING_CUBE_146438610_146438640.log > -rw-rw-r-- 1 kylin kylin 3899396 5月 31 10:38 > streaming_STREAMING_CUBE_146439090_146439120.log > -rw-rw-r-- 1 kylin kylin 4268748 5月 31 10:38 > streaming_STREAMING_CUBE_146436480_146436510.log > -rw-rw-r-- 1 kylin kylin 4273006 5月 31 10:38 > streaming_STREAMING_CUBE_146436450_146436480.log > -rw-rw-r-- 1 kylin kylin 4372131 5月 31 10:38 > streaming_STREAMING_CUBE_146435760_146435790.log > -rw-rw-r-- 1 kylin kylin 4423483 5月 31 10:38 > streaming_STREAMING_CUBE_146435400_146435430.log > -rw-rw-r-- 1 kylin kylin 3970905 5月 31 10:38 > streaming_STREAMING_CUBE_146438580_146438610.log > -rw-rw-r-- 1 kylin kylin 3975754 5月 31 10:38 > streaming_STREAMING_CUBE_146438550_146438580.log > -rw-rw-r-- 1 kylin kylin 4342175 5月 31 10:38 > streaming_STREAMING_CUBE_146435970_146436000.log > -rw-rw-r-- 1 kylin kylin 4243116 5月 31 10:38 > streaming_STREAMING_CUBE_146436660_146436690.log > -rw-rw-r-- 1 kylin kylin 4302896 5月 31 10:38 > streaming_STREAMING_CUBE_146436240_146436270.log > -rw-rw-r-- 1 kylin kylin 4119274 5月 31 10:38 > streaming_STREAMING_CUBE_146437530_146437560.log > -rw-rw-r-- 1 kylin kylin 4456038 5月 31 10:38 > streaming_STREAMING_CUBE_146435340_146435370.log > -rw-rw-r-- 1 kylin kylin 4337883 5月 31 10:38 > streaming_STREAMING_CUBE_146436000_146436030.log > -rw-rw-r-- 1 kylin kylin 3892620 5月 31 10:38 > streaming_STREAMING_CUBE_146439150_146439180.log > -rw-rw-r-- 1 kylin kylin 4367851 5月 31 10:38 > streaming_STREAMING_CUBE_146435790_146435820.log -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1797) bin/find-hive-dependency.sh may not get correct local classpath when configured hive client is beeline
[
https://issues.apache.org/jira/browse/KYLIN-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-1797.
---
Resolution: Not A Problem
> bin/find-hive-dependency.sh may not get correct local classpath when
> configured hive client is beeline
> --
>
> Key: KYLIN-1797
> URL: https://issues.apache.org/jira/browse/KYLIN-1797
> Project: Kylin
> Issue Type: Bug
> Components: Client - CLI
>Affects Versions: v1.3.0, v1.5.0
>Reporter: Shaofeng SHI
>Priority: Major
> Labels: newbie
>
> in bin/find-hive-dependency.sh, if configured hive client is beeline, it will
> use beeline to execute "set" and then to find the classpath; I noticed the
> result doesn't have "hive-cli.jar", this will cause error in "load hive
> table" action, a NoClassDefFoundError error is thrown;
> If "hive -e set" will return all the needed hive jars.
> {code}
> SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin]
> threw exception [Handler processing failed; nested exception is
> java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliSessionState]
> with root cause
> java.lang.ClassNotFoundException: org.apache.hadoop.hive.cli.CliSessionState
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> at
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> at
> org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:301)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-1784) When Kylin Load HFile to HBase Table,the Hfiles have incorrect permission
[ https://issues.apache.org/jira/browse/KYLIN-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1784. --- Resolution: Incomplete > When Kylin Load HFile to HBase Table,the Hfiles have incorrect permission > -- > > Key: KYLIN-1784 > URL: https://issues.apache.org/jira/browse/KYLIN-1784 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.2 > Environment: hadoop: 2.4.0.5 > hive:0.13.0.5 > hbase:0.98.8 > jdk:1.7.0_60-ea >Reporter: dcswinner >Assignee: Dong Li >Priority: Major > Labels: easyfix, newbie > Fix For: Future > > > when build a cube,the job running to Load HFile to HBase Table'step,I found > that all of these hfiles's owner were belong to user:kylin and group:kylin on > hdfs. > eg: > [bigdata@kylin02-pre ~]$ hadoop fs -ls > /hbase/data/default/KYLIN_556NF0D6BV/03bb84673d43e66c5ac2eeabe6df4020/F1 > Found 2 items > -rwxrwxrwx 2 kylin kylin6965632 2016-06-12 11:06 > /hbase/data/default/KYLIN_556NF0D6BV/03bb84673d43e66c5ac2eeabe6df4020/F1/13f7a13f6f754f2f8a15e0efe60dac1d_SeqId_2_ > -rwxrwxrwx 2 kylin kylin7544515 2016-06-12 11:06 > /hbase/data/default/KYLIN_556NF0D6BV/03bb84673d43e66c5ac2eeabe6df4020/F1/dcf03d3be6ae4dd4a93b7a05cdf00382_SeqId_2_ > the owner were belong to user:kylin and group:kylin like above. > then,when HBase run archive chore in backgroud,will throw an error: > Caused by: > org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): > Permission denied: user=hbase, access=WRITE, > inode="/hbase/.tmp/data/default/KYLIN_9RJA24O > ZEM/4f54f38dc4b96a04a82bdb780fef77c4/F2/3b9adb66f50e42918a9b6c4f67cb0261_SeqId_2_":kylin:kylin:-rw-r--r-- > I have a solution is change these hfiles's permission 777 before bulkload > hfile to hbase: > hadoop fs -chmod 777 > /hbase/data/default/KYLIN_556NF0D6BV/03bb84673d43e66c5ac2eeabe6df4020/F1/dcf03d3be6ae4dd4a93b7a05cdf00382_SeqId_2_ > or change these hfiles's owner and group: > hadoop fs -chown hbase:hbase > /hbase/data/default/KYLIN_556NF0D6BV/03bb84673d43e66c5ac2eeabe6df4020/F1/dcf03d3be6ae4dd4a93b7a05cdf00382_SeqId_2_ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1463) When edit a model, if the old fact table be changed, it will still exist for dimensions selection
[ https://issues.apache.org/jira/browse/KYLIN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1463. --- Resolution: Won't Fix > When edit a model, if the old fact table be changed, it will still exist for > dimensions selection > - > > Key: KYLIN-1463 > URL: https://issues.apache.org/jira/browse/KYLIN-1463 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Dipesh >Priority: Major > > Firstly, I cloned a model. During editing the model, after selecting another > fact table, during the dimension select step, the old fact table still exists > in UI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2038) 'Access denied' occurred if a query hit none of cubes and the user is not in admin role.
[ https://issues.apache.org/jira/browse/KYLIN-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2038. --- Resolution: Won't Fix The ACL has been refactored in 2.x, now this is not needed. > 'Access denied' occurred if a query hit none of cubes and the user is not in > admin role. > > > Key: KYLIN-2038 > URL: https://issues.apache.org/jira/browse/KYLIN-2038 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: all >Reporter: Chaozhong Yang >Assignee: Zhong,Jason >Priority: Major > Fix For: Future > > Attachments: fix_query_quth.patch > > > The method `checkCubeAuthorization` in QueryService will check permission > for every query. If a query has not hit any cube in Kylin, > `checkCubeAuthorization` will throw AccessDeniedException. > "@PreAuthorize(Constant.ACCESS_HAS_ROLE_ADMIN + " or hasPermission(#cube, > 'ADMINISTRATION') or hasPermission(#cube, 'MANAGEMENT')" + " or > hasPermission(#cube, 'OPERATION') or hasPermission(#cube, 'READ')")" will > throw AccessDeniedException if cube is null. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2034) user kylin1.5.3 bulid cube error
[ https://issues.apache.org/jira/browse/KYLIN-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2034. --- Resolution: Incomplete > user kylin1.5.3 bulid cube error > > > Key: KYLIN-2034 > URL: https://issues.apache.org/jira/browse/KYLIN-2034 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.3 > Environment: hbase1.1.6 hadoop2.6,hive2.0.0 >Reporter: panzhiwei >Assignee: Zhong,Jason >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2025) web UI can't build cube Measures,the web console 404 not found the cube add
[ https://issues.apache.org/jira/browse/KYLIN-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2025. --- Resolution: Incomplete > web UI can't build cube Measures,the web console 404 not found the cube add > --- > > Key: KYLIN-2025 > URL: https://issues.apache.org/jira/browse/KYLIN-2025 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.4 >Reporter: Kris >Priority: Major > Labels: build > Attachments: 2016-09-18 23时23分27秒.jpg, 截图 2016-09-18 23时26分37秒.jpg > > > 1.5.4 when build cube the step3 ,can't click the list of all,console print > 404 not found the cube add -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1977) Update front end dependency (bower)
[ https://issues.apache.org/jira/browse/KYLIN-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495022#comment-16495022 ] Shaofeng SHI commented on KYLIN-1977: - [~julianpan] is this issue still valid? > Update front end dependency (bower) > --- > > Key: KYLIN-1977 > URL: https://issues.apache.org/jira/browse/KYLIN-1977 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Zhong,Jason >Priority: Major > > 1. There is warning "Code incompatibilities may occur.", when I install bower > component. [angular, nvd3, d3, moment, angular-cookies version not consistent] > 2. Some duplicate component: ng-grid & angular-ui-grid, > angular-chosen-localytics & angular-ui-select, font-awesome & > components-font-awesome > 3. angularjs-nvd3-directives is beta version, maybe we can replace it by > angular-nvd3. > 4. some components version are out of date, could we upgrade them such as > angular-bootstrap... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1937) Kylin Web not running
[ https://issues.apache.org/jira/browse/KYLIN-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1937. --- Resolution: Incomplete > Kylin Web not running > - > > Key: KYLIN-1937 > URL: https://issues.apache.org/jira/browse/KYLIN-1937 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: Pankaj Shrestha >Assignee: Zhong,Jason >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1948) IntegerDimEnc, does not encode -1 correctly
[ https://issues.apache.org/jira/browse/KYLIN-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495016#comment-16495016 ] Shaofeng SHI commented on KYLIN-1948: - [~Aron.tao] has this been fixed? > IntegerDimEnc, does not encode -1 correctly > --- > > Key: KYLIN-1948 > URL: https://issues.apache.org/jira/browse/KYLIN-1948 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > The code for -1 is all 0xff, which is the code for NULL. Need a fix, since -1 > is a common value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1950) AppendTrieDictionaryTest`and CachedTreeMapTest should not be UT
[ https://issues.apache.org/jira/browse/KYLIN-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1950. --- Resolution: Fixed > AppendTrieDictionaryTest`and CachedTreeMapTest should not be UT > --- > > Key: KYLIN-1950 > URL: https://issues.apache.org/jira/browse/KYLIN-1950 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v1.5.3 >Reporter: KeZaidan >Assignee: hongbin ma >Priority: Minor > > both test cases rely on CachedTreeMap, which in turn relies on hdfs. so they > should be IT instead of UT -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1928) Query elapsed time not shown when query returns 0 result
[ https://issues.apache.org/jira/browse/KYLIN-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1928. --- Resolution: Fixed > Query elapsed time not shown when query returns 0 result > > > Key: KYLIN-1928 > URL: https://issues.apache.org/jira/browse/KYLIN-1928 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: Zhong,Jason >Priority: Major > Attachments: > 0001-KYLIN-1928-Query-elapsed-time-not-shown-when-query-r.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2247) Automatically flush cache after executing "sample.sh" or "metadata.sh restore"
[ https://issues.apache.org/jira/browse/KYLIN-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2247. --- Resolution: Won't Fix > Automatically flush cache after executing "sample.sh" or "metadata.sh restore" > -- > > Key: KYLIN-2247 > URL: https://issues.apache.org/jira/browse/KYLIN-2247 > Project: Kylin > Issue Type: Improvement > Components: Client - CLI >Reporter: Shaofeng SHI >Priority: Major > Fix For: Backlog > > > Today after run "sample.sh" to create the sample cube, or after run > "metadata.sh restore" command to restore the metadata, the user need > explicitly go to Kylin's web UI to "reload metadata"; When it is in a > clustered deployment, user has to do this on each node, this is laborous. > The scripts should call Kylin REST API to flush the caches automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2182) Aggregation Group Joint Dimension generation default rule update
[ https://issues.apache.org/jira/browse/KYLIN-2182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2182. --- Resolution: Won't Fix > Aggregation Group Joint Dimension generation default rule update > > > Key: KYLIN-2182 > URL: https://issues.apache.org/jira/browse/KYLIN-2182 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.4.1 >Reporter: Zhong,Jason >Priority: Major > > when we create cube, if we have multiple dimensions, we will split dimensions > into different Joint, this is not so meaningful -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2267) When FK/PK column type mismatched, get error on query: "NullPointerException: Column ### does not exist in row key desc"
[
https://issues.apache.org/jira/browse/KYLIN-2267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2267.
---
Resolution: Fixed
> When FK/PK column type mismatched, get error on query: "NullPointerException:
> Column ### does not exist in row key desc"
>
>
> Key: KYLIN-2267
> URL: https://issues.apache.org/jira/browse/KYLIN-2267
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Shaofeng SHI
>Priority: Minor
> Labels: newbie
>
> I got this error in a cube which has one fact table and one lookup table; All
> dimensions are normal, which looks be a very simple case (and the FK column
> does exist on rowkey descriptor), but I got this error on every query that
> uses columns on the lookup table; The error log is as below:
> {code}
> 2016-12-08 13:58:25,177 INFO [http-bio-7070-exec-5] routing.QueryRouter:72 :
> The realizations remaining: [his_realtime_cube_v7(CUBE)] And the final chosen
> one is the first one
> 2016-12-08 13:58:25,219 DEBUG [http-bio-7070-exec-5]
> enumerator.OLAPEnumerator:100 : query storage...
> 2016-12-08 13:58:25,220 ERROR [http-bio-7070-exec-5]
> controller.QueryController:208 : Exception when execute sql
> java.sql.SQLException: Error while executing SQL "select count(*) as "m0"
> from "DEFAULT"."HIS_REALTIME" as "HIS_REALTIME" join
> "DEFAULT"."V_STKCODE_INDUSTRYINFO2" as "V_STKCODE_INDUSTRYINFO2" on
> "HIS_REALTIME"."STOCK_CODE" = "V_STKCODE_INDUSTRYINFO2"."STK_CODE"
> LIMIT 5": Column DEFAULT.V_STKCODE_INDUSTRYINFO2.STK_CODE does not exist
> in row key desc
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:147)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:208)
> at
> org.apache.kylin.rest.service.QueryService.execute(QueryService.java:407)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:321)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:128)
> at
> org.apache.kylin.rest.controller.QueryController.doQueryWithCache(QueryController.java:191)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:94)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
> at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
>
[jira] [Closed] (KYLIN-2233) Support HBase 1.0.2
[ https://issues.apache.org/jira/browse/KYLIN-2233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2233. --- Resolution: Won't Fix > Support HBase 1.0.2 > --- > > Key: KYLIN-2233 > URL: https://issues.apache.org/jira/browse/KYLIN-2233 > Project: Kylin > Issue Type: New Feature >Reporter: liyang >Priority: Major > > HBase 0.98, 1.0, 1.1+ all have different coprocessor APIs. WTF. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2303) Step Convert Cuboid Data to HFile always get high CPU
[ https://issues.apache.org/jira/browse/KYLIN-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2303. --- Resolution: Cannot Reproduce Please provide detail info for investigate. Thanks! > Step Convert Cuboid Data to HFile always get high CPU > - > > Key: KYLIN-2303 > URL: https://issues.apache.org/jira/browse/KYLIN-2303 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.6.0 > Environment: CentOS >Reporter: hoangle >Priority: Major > Labels: performance > > At step "Convert Cuboid Data to HFile" in building cubes process. > My nodes always get high load CPU, especially node running Kylin. It always > take 150% - 200% CPU usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2345) Not able to open web interface for Kylin
[ https://issues.apache.org/jira/browse/KYLIN-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2345. --- Resolution: Cannot Reproduce > Not able to open web interface for Kylin > > > Key: KYLIN-2345 > URL: https://issues.apache.org/jira/browse/KYLIN-2345 > Project: Kylin > Issue Type: Bug >Reporter: Anurag >Priority: Blocker > > Not able to open web interface for Kylin > 1. hadoop version is hadoop-2.7.3 > 2. apache kylin version apache-kylin-1.6.0-hbase1.x-bin > 3 .hbase version hbase-1.1.7 > 4. hive version is apache-hive-2.1.0-bin > issues in log > :org.springframework.beans.factory.BeanCurrentlyInCreationException: Error > creating bean with name 'mvcContentNegotiationManager': Requested bean is > currently in creation: Is there an unresolvable circular reference? > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.beforeSingletonCreation(DefaultSingletonBeanRegistry.java:327) > at > org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:217) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2425) Cube query gives exception "java.lang.IllegalStateException: Not sorted!"
[ https://issues.apache.org/jira/browse/KYLIN-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2425. --- Resolution: Fixed > Cube query gives exception "java.lang.IllegalStateException: Not sorted!" > - > > Key: KYLIN-2425 > URL: https://issues.apache.org/jira/browse/KYLIN-2425 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.6.0 >Reporter: Shailesh >Assignee: hongbin ma >Priority: Major > > While performing non aggregate query with limit other than default gives > "java.lang.IllegalStateException: Not sorted!" > gist of kylin log: > https://gist.github.com/shaipraj/780a3dcc80aa2080911b7348c76f5b88 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2478) Lots of warning ‘DFSInputStream has been closed already’ when using HDFSResourceStore for CI
[ https://issues.apache.org/jira/browse/KYLIN-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2478. --- Resolution: Won't Fix > Lots of warning ‘DFSInputStream has been closed already’ when using > HDFSResourceStore for CI > -- > > Key: KYLIN-2478 > URL: https://issues.apache.org/jira/browse/KYLIN-2478 > Project: Kylin > Issue Type: Sub-task >Reporter: XIE FAN >Assignee: XIE FAN >Priority: Minor > > need to find the root cause and try to remove those warnings -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2476) getting error while refreshing cube using REST API
[
https://issues.apache.org/jira/browse/KYLIN-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2476.
---
Resolution: Not A Problem
> getting error while refreshing cube using REST API
> --
>
> Key: KYLIN-2476
> URL: https://issues.apache.org/jira/browse/KYLIN-2476
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
> Environment: Dev region
>Reporter: krishna chaitan malipeddu
>Assignee: Zhong,Jason
>Priority: Major
>
> Hi,
> i'm getting below error while refreshing cube using REST API.
> Request failed:
> {"url":"http://*.*.*.*:7070/kylin/api/cubes/TEST/rebuild","exception":"The
> new refreshing segment TEST[20170228105510_20170228115510] does not match any
> existing segment in cube CUBE[name=TEST]"}
> can you please suggest.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-2480) common.HadoopStatusChecker:93 : error check status
[ https://issues.apache.org/jira/browse/KYLIN-2480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2480. --- Resolution: Cannot Reproduce Close it as long time no update, and we didn't see such behavior in past monthes. Please kindly reopen if it still happen. > common.HadoopStatusChecker:93 : error check status > --- > > Key: KYLIN-2480 > URL: https://issues.apache.org/jira/browse/KYLIN-2480 > Project: Kylin > Issue Type: Bug > Components: Others >Reporter: Gao Xia >Priority: Major > > Hi, > Gus, here is what happend. > 1.successfully run .sample.sh > 2.when buding the sample cube, on step 2: 2017-03-03 23:00:21 PST > #2 Step Name: Extract Fact Table Distinct Columns, error happend. > 3. logs show: > Exception: null > Counters: 0 > 4.from the log file: > ERROR [pool-6-thread-2] common.HadoopStatusChecker:93 : error check status > java.lang.NullPointerException > at > org.apache.kylin.engine.mr.common.HadoopStatusGetter.get(HadoopStatusGetter.java:74) > at > org.apache.kylin.engine.mr.common.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:58) > at > org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:147) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > 2017-03-04 15:00:27,085 DEBUG [pool-6-thread-2] hbase.HBaseResourceStore:260 > : Update row /execute_output/51ed1ad1-6de9-4aa0-9552-f25a4d694f36-01 from > oldTs: 1488610821789, to newTs: 1488610827082, operation result: true > 2017-03-04 15:00:27,092 DEBUG [pool-6-thread-2] common.HadoopCmdOutput:94 : > Counters: 0 > 2017-03-04 15:00:27,126 DEBUG [pool-6-thread-2] hbase.HBaseResourceStore:260 > : Update row /execute_output/51ed1ad1-6de9-4aa0-9552-f25a4d694f36-01 from > oldTs: 1488610827082, to newTs: 1488610827121, operation result: true > 2017-03-04 15:00:27,134 DEBUG [pool-6-thread-2] hbase.HBaseResourceStore:260 > : Update row /execute_output/51ed1ad1-6de9-4aa0-9552-f25a4d694f36-01 from > oldTs: 1488610827121, to newTs: 1488610827129, operation result: true > 2017-03-04 15:00:27,142 DEBUG [pool-6-thread-2] hbase.HBaseResourceStore:260 > : Update row /execute_output/51ed1ad1-6de9-4aa0-9552-f25a4d694f36-01 from > oldTs: 1488610827129, to newTs: 1488610827135, operation result: true > not sure what wen wrong, might be the misconfiguration, hope you will lead up > the right path the correct solution. > Thanks man! you guys did a great job for apache open source ! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2477) Make UT and IT compatible with any kinds of metadata resource store
[ https://issues.apache.org/jira/browse/KYLIN-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2477. --- Resolution: Won't Fix > Make UT and IT compatible with any kinds of metadata resource store > --- > > Key: KYLIN-2477 > URL: https://issues.apache.org/jira/browse/KYLIN-2477 > Project: Kylin > Issue Type: Sub-task >Reporter: XIE FAN >Assignee: XIE FAN >Priority: Major > > Which means that when we replace HBaseResourceStore with another one, our UT > and IT can still works well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2900) Web Interface MR Link BUG
[ https://issues.apache.org/jira/browse/KYLIN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2900. --- Resolution: Won't Fix > Web Interface MR Link BUG > - > > Key: KYLIN-2900 > URL: https://issues.apache.org/jira/browse/KYLIN-2900 > Project: Kylin > Issue Type: Bug > Components: REST Service >Reporter: lynn >Assignee: Zhixiong Chen >Priority: Major > Attachments: 1.png > > > When I notice the detail info of the MR Job, I find the Link starts with > “unsafe” occasionally , as the picture "1.png". > !1.png|thumbnail! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2590) Cube build error on Step 7 Build Base Cuboid Data
[
https://issues.apache.org/jira/browse/KYLIN-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2590.
---
Resolution: Cannot Reproduce
> Cube build error on Step 7 Build Base Cuboid Data
> -
>
> Key: KYLIN-2590
> URL: https://issues.apache.org/jira/browse/KYLIN-2590
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v1.6.0
>Reporter: lufeng
>Priority: Major
>
> The cube build failed on step 7 with following error message. Any help please
> Hadoop 2.7 HBase 1.1.2 Kylin 1.6.0 Tez 0.7.0
> <<<
> 2017-05-01 01:10:38,430 INFO [pool-9-thread-10] common.AbstractHadoopJob:499
> : HDFS meta dir is:
> file:///opt/kylin/bin/../tomcat/temp/kylin_job_meta8686959176405233518/meta
> 2017-05-01 01:10:38,430 INFO [pool-9-thread-10] common.AbstractHadoopJob:372
> : Job 'tmpfiles' updated --
> file:///opt/kylin/bin/../tomcat/temp/kylin_job_meta8686959176405233518/meta
> 2017-05-01 01:10:38,433 INFO [pool-9-thread-10] mapred.FileInputFormat:249 :
> Total input paths to process : 0
> 2017-05-01 01:10:38,433 INFO [pool-9-thread-10] common.AbstractHadoopJob:506
> : tempMetaFileString is :
> file:///opt/kylin/bin/../tomcat/temp/kylin_job_meta8686959176405233518/meta
> 2017-05-01 01:10:38,440 ERROR [pool-9-thread-10]
> common.MapReduceExecutable:127 : error execute
> MapReduceExecutable{id=37f887a1-1f2a-40d0-a4dc-9eca7be28ab7-06, name=Build
> Base Cuboid Data, state=RUNNING}
> java.lang.IllegalArgumentException: Map input splits are 0 bytes, something
> is wrong!
> at
> org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:555)
> at
> org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:175)
> at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:138)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
> at
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> ---
> I sovled this issue by changing execution engine from Tez to MR。 Because Tez
> engine has zero file issue . https://issues.apache.org/jira/browse/HIVE-13988
> I think Kylin should not blocked by empty segment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-2485) Test thread safety of HDFSResourceStore
[ https://issues.apache.org/jira/browse/KYLIN-2485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2485. --- Resolution: Won't Fix HDFSResourceStore is read-only, this task is not needed. > Test thread safety of HDFSResourceStore > --- > > Key: KYLIN-2485 > URL: https://issues.apache.org/jira/browse/KYLIN-2485 > Project: Kylin > Issue Type: Sub-task >Reporter: XIE FAN >Assignee: XIE FAN >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2553) ODBC 1.6 version issue when connecting to Kylin 2.0 beta, Tableau 10.2
[ https://issues.apache.org/jira/browse/KYLIN-2553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2553. --- Resolution: Won't Fix > ODBC 1.6 version issue when connecting to Kylin 2.0 beta, Tableau 10.2 > -- > > Key: KYLIN-2553 > URL: https://issues.apache.org/jira/browse/KYLIN-2553 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 > Environment: Windows 7; ODBC driver(32bit,64bit); >Reporter: Luyuan Zhai >Priority: Major > Attachments: Screen Shot 2017-04-18 at 5.21.26 PM.png > > > In Tableau 10.2, with custom sql, the preview works, but the full query > returns only 1 row with nulls for the following sample query: > select part_dt, sum(price) as total_selled, count(distinct seller_id) as > sellers from kylin_sales group by part_dt order by part_dt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2512) Getting BufferUnderflowException from ConcurrentDiskStore
[
https://issues.apache.org/jira/browse/KYLIN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-2512.
-
Resolution: Cannot Reproduce
Long time no see this error.
> Getting BufferUnderflowException from ConcurrentDiskStore
> -
>
> Key: KYLIN-2512
> URL: https://issues.apache.org/jira/browse/KYLIN-2512
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: Shaofeng SHI
>Assignee: liyang
>Priority: Major
> Attachments: KYLIN-2512_cube_metadata.txt
>
>
> Reported by rahulsingh in mailing list:
> getting this error while building a cube; Attached is the mock of the cube
> descriptor
> {code}
> org.apache.hadoop.mapred.YarnChild: Exception running child :
> java.io.IOException: Failed to build cube in mapper 9
> at
> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.util.concurrent.ExecutionException:
> java.nio.BufferUnderflowException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143)
> ... 8 more
> Caused by: java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:492)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:135)
> at
> org.apache.kylin.common.util.BytesUtil.readVLong(BytesUtil.java:208)
> at org.apache.kylin.common.util.BytesUtil.readVInt(BytesUtil.java:223)
> at
> org.apache.kylin.metadata.datatype.BigDecimalSerializer.peekLength(BigDecimalSerializer.java:83)
> at
> org.apache.kylin.cube.gridtable.CubeCodeSystem.codeLength(CubeCodeSystem.java:91)
> at org.apache.kylin.gridtable.GTRecord.loadColumns(GTRecord.java:294)
> at
> org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Reader$2.hasNext(ConcurrentDiskStore.java:233)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$MergeSlot.fetchNext(DoggedCubeBuilder.java:412)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$Merger.mergeAndOutput(DoggedCubeBuilder.java:328)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:119)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75)
> at
> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-2512) Getting BufferUnderflowException from ConcurrentDiskStore
[
https://issues.apache.org/jira/browse/KYLIN-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2512.
---
> Getting BufferUnderflowException from ConcurrentDiskStore
> -
>
> Key: KYLIN-2512
> URL: https://issues.apache.org/jira/browse/KYLIN-2512
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: Shaofeng SHI
>Assignee: liyang
>Priority: Major
> Attachments: KYLIN-2512_cube_metadata.txt
>
>
> Reported by rahulsingh in mailing list:
> getting this error while building a cube; Attached is the mock of the cube
> descriptor
> {code}
> org.apache.hadoop.mapred.YarnChild: Exception running child :
> java.io.IOException: Failed to build cube in mapper 9
> at
> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:145)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:148)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1714)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.util.concurrent.ExecutionException:
> java.nio.BufferUnderflowException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> at
> org.apache.kylin.engine.mr.steps.InMemCuboidMapper.cleanup(InMemCuboidMapper.java:143)
> ... 8 more
> Caused by: java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:492)
> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:135)
> at
> org.apache.kylin.common.util.BytesUtil.readVLong(BytesUtil.java:208)
> at org.apache.kylin.common.util.BytesUtil.readVInt(BytesUtil.java:223)
> at
> org.apache.kylin.metadata.datatype.BigDecimalSerializer.peekLength(BigDecimalSerializer.java:83)
> at
> org.apache.kylin.cube.gridtable.CubeCodeSystem.codeLength(CubeCodeSystem.java:91)
> at org.apache.kylin.gridtable.GTRecord.loadColumns(GTRecord.java:294)
> at
> org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore$Reader$2.hasNext(ConcurrentDiskStore.java:233)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$MergeSlot.fetchNext(DoggedCubeBuilder.java:412)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$Merger.mergeAndOutput(DoggedCubeBuilder.java:328)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:119)
> at
> org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:75)
> at
> org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:82)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-2484) Spark engine to support source from Kafka
[ https://issues.apache.org/jira/browse/KYLIN-2484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-2484: --- Assignee: Shaofeng SHI Fix Version/s: (was: Backlog) v2.4.0 Issue Type: Improvement (was: Bug) Summary: Spark engine to support source from Kafka (was: Kylin spark engine doesn't support source from Kafka) > Spark engine to support source from Kafka > - > > Key: KYLIN-2484 > URL: https://issues.apache.org/jira/browse/KYLIN-2484 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > The current spark engine uses Spark's HiveConext to read data, so it required > that the source is in Hive; this means the Streaming cube couldn't be built > with Spark engine. > So far will add a validation with a meaningful message when user uses this > combination. Later Kylin need provide solution for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3348) "missing LastBuildJobID" error when building new cube segment
[
https://issues.apache.org/jira/browse/KYLIN-3348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3348.
-
Resolution: Fixed
> "missing LastBuildJobID" error when building new cube segment
> -
>
> Key: KYLIN-3348
> URL: https://issues.apache.org/jira/browse/KYLIN-3348
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.3.0
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> An unstable exception. Likely to happen when there are multiple concurrent
> builds.
> {{2018-04-18 20:11:16,856 ERROR [pool-33-thread-11]
> threadpool.DefaultScheduler : ExecuteException
> job:cc08da19-f53e-4344-a6c5-05e764834924}}
> {{ org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
> \{{ at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307)}}
> \{{ at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
> \{{ at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
> \{{ at java.lang.Thread.run(Thread.java:748)}}
> {{ Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: For cube CUBE[name=cube2], segment
> cube2[2018041423000_2018041423001] missing LastBuildJobID}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
> \{{ at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
> \{{ ... 4 more}}
> {{ Caused by: java.lang.IllegalStateException: For cube CUBE[name=cube2],
> segment cube2[2018041423000_2018041423001] missing LastBuildJobID}}
> \{{ at
> org.apache.kylin.cube.CubeManager$SegmentAssist.promoteNewlyBuiltSegments(CubeManager.java:810)}}
> \{{ at
> org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:535)}}
> \{{ at
> org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:78)}}
> \{{ at
> io.kyligence.kap.engine.mr.steps.KapUpdateCubeInfoAfterBuildStep.doWork(SourceFile:47)}}
> \{{ at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
> \{{ ... 6 more}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3336) java.lang.NoSuchMethodException: org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
[
https://issues.apache.org/jira/browse/KYLIN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494978#comment-16494978
]
ASF GitHub Bot commented on KYLIN-3336:
---
shaofengshi closed pull request #128: KYLIN-3336
URL: https://github.com/apache/kylin/pull/128
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/tool/src/main/java/org/apache/kylin/tool/DiagnosisInfoCLI.java
b/tool/src/main/java/org/apache/kylin/tool/DiagnosisInfoCLI.java
index 6fe63e4ea4..d960f8f6a9 100644
--- a/tool/src/main/java/org/apache/kylin/tool/DiagnosisInfoCLI.java
+++ b/tool/src/main/java/org/apache/kylin/tool/DiagnosisInfoCLI.java
@@ -192,7 +192,7 @@ public void run() {
projectNames, "-compress", "false",
"-submodule", "true" };
logger.info("HBaseUsageExtractor args: " +
Arrays.toString(hbaseArgs));
Object extractor =
ClassUtil.newInstance("org.apache.kylin.tool.HBaseUsageExtractor");
-Method execute =
extractor.getClass().getDeclaredMethod("execute", String[].class);
+Method execute =
extractor.getClass().getMethod("execute", String[].class);
execute.invoke(extractor, (Object) hbaseArgs);
} catch (Throwable e) {
logger.error("Error in export HBase usage.", e);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> java.lang.NoSuchMethodException:
> org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
> ---
>
> Key: KYLIN-3336
> URL: https://issues.apache.org/jira/browse/KYLIN-3336
> Project: Kylin
> Issue Type: Bug
> Components: Tools, Build and Test
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Minor
> Fix For: v2.4.0
>
>
> When using diag.sh, got following exception:
> 2018-04-04 16:14:18,222 ERROR [pool-7-thread-3] tool.DiagnosisInfoCLI:171 :
> Error in export HBase usage.
> java.lang.NoSuchMethodException:
> org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
> at java.lang.Class.getDeclaredMethod(Class.java:2017)
> at
> org.apache.kylin.tool.DiagnosisInfoCLI$3.run(DiagnosisInfoCLI.java:168)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:745)
> Because getDeclaredMethod(String name, Class... parameterTypes) can not
> get the inherited methods
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3336) java.lang.NoSuchMethodException: org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
[ https://issues.apache.org/jira/browse/KYLIN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494979#comment-16494979 ] ASF subversion and git services commented on KYLIN-3336: Commit d1c68f6cf4f6f0f788be65d43964d49e488271e0 in kylin's branch refs/heads/master from [~xingpeng1] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d1c68f6 ] KYLIN-3336 > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > --- > > Key: KYLIN-3336 > URL: https://issues.apache.org/jira/browse/KYLIN-3336 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.4.0 > > > When using diag.sh, got following exception: > 2018-04-04 16:14:18,222 ERROR [pool-7-thread-3] tool.DiagnosisInfoCLI:171 : > Error in export HBase usage. > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > at java.lang.Class.getDeclaredMethod(Class.java:2017) > at > org.apache.kylin.tool.DiagnosisInfoCLI$3.run(DiagnosisInfoCLI.java:168) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > Because getDeclaredMethod(String name, Class... parameterTypes) can not > get the inherited methods -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3336) java.lang.NoSuchMethodException: org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
[ https://issues.apache.org/jira/browse/KYLIN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3336. - Resolution: Fixed PR merged. Thanks Peng xing. > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > --- > > Key: KYLIN-3336 > URL: https://issues.apache.org/jira/browse/KYLIN-3336 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.4.0 > > > When using diag.sh, got following exception: > 2018-04-04 16:14:18,222 ERROR [pool-7-thread-3] tool.DiagnosisInfoCLI:171 : > Error in export HBase usage. > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > at java.lang.Class.getDeclaredMethod(Class.java:2017) > at > org.apache.kylin.tool.DiagnosisInfoCLI$3.run(DiagnosisInfoCLI.java:168) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > Because getDeclaredMethod(String name, Class... parameterTypes) can not > get the inherited methods -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3336) java.lang.NoSuchMethodException: org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;)
[ https://issues.apache.org/jira/browse/KYLIN-3336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3336: Fix Version/s: v2.4.0 > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > --- > > Key: KYLIN-3336 > URL: https://issues.apache.org/jira/browse/KYLIN-3336 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.4.0 > > > When using diag.sh, got following exception: > 2018-04-04 16:14:18,222 ERROR [pool-7-thread-3] tool.DiagnosisInfoCLI:171 : > Error in export HBase usage. > java.lang.NoSuchMethodException: > org.apache.kylin.tool.HBaseUsageExtractor.execute([Ljava.lang.String;) > at java.lang.Class.getDeclaredMethod(Class.java:2017) > at > org.apache.kylin.tool.DiagnosisInfoCLI$3.run(DiagnosisInfoCLI.java:168) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > Because getDeclaredMethod(String name, Class... parameterTypes) can not > get the inherited methods -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2951) Fix couple of NullPointerException in Kylin code base
[ https://issues.apache.org/jira/browse/KYLIN-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2951. --- > Fix couple of NullPointerException in Kylin code base > - > > Key: KYLIN-2951 > URL: https://issues.apache.org/jira/browse/KYLIN-2951 > Project: Kylin > Issue Type: Bug >Reporter: Wang Ken >Assignee: Wang Ken >Priority: Minor > Fix For: v2.3.0 > > > I create this ticket as the umbrella to cover some NPE bugs in Kylin 2.1 & > 2.2 code base. > We see some NPE errors from our server log, some of them are tiny, it is > trivial to create individual ticket to cover them one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3005) getAllDictColumnsOnFact in CubeManager may include fact table's foreign key
[
https://issues.apache.org/jira/browse/KYLIN-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494974#comment-16494974
]
Shaofeng SHI commented on KYLIN-3005:
-
Yanghong, what's the impact of this?
> getAllDictColumnsOnFact in CubeManager may include fact table's foreign key
> ---
>
> Key: KYLIN-3005
> URL: https://issues.apache.org/jira/browse/KYLIN-3005
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Trivial
>
> Suppose there are two tables, FA & FL. In a model, there are two joins: FA
> inner join FL(alias FL1, without snapshot), FA inner join FL(alias FL2, with
> snapshot). The join conditions are the same, FA.colA = FL.colL.
> In this case, when the returns from {{getTable()}} of both FL1.colL &
> FL2.colL are FL which is regarded as fact table. This finally results in the
> return from {{getAllDictColumnsOnFact()}} in {{CubeManager}} include the fact
> table's foreign key, FA.colA.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-2951) Fix couple of NullPointerException in Kylin code base
[ https://issues.apache.org/jira/browse/KYLIN-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2951. - Resolution: Fixed > Fix couple of NullPointerException in Kylin code base > - > > Key: KYLIN-2951 > URL: https://issues.apache.org/jira/browse/KYLIN-2951 > Project: Kylin > Issue Type: Bug >Reporter: Wang Ken >Assignee: Wang Ken >Priority: Minor > Fix For: v2.3.0 > > > I create this ticket as the umbrella to cover some NPE bugs in Kylin 2.1 & > 2.2 code base. > We see some NPE errors from our server log, some of them are tiny, it is > trivial to create individual ticket to cover them one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2951) Fix couple of NullPointerException in Kylin code base
[ https://issues.apache.org/jira/browse/KYLIN-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2951: Fix Version/s: v2.3.0 > Fix couple of NullPointerException in Kylin code base > - > > Key: KYLIN-2951 > URL: https://issues.apache.org/jira/browse/KYLIN-2951 > Project: Kylin > Issue Type: Bug >Reporter: Wang Ken >Assignee: Wang Ken >Priority: Minor > Fix For: v2.3.0 > > > I create this ticket as the umbrella to cover some NPE bugs in Kylin 2.1 & > 2.2 code base. > We see some NPE errors from our server log, some of them are tiny, it is > trivial to create individual ticket to cover them one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3156) Failed to delete meta path in SparkCubingByLayer
[
https://issues.apache.org/jira/browse/KYLIN-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3156.
---
Resolution: Duplicate
> Failed to delete meta path in SparkCubingByLayer
>
>
> Key: KYLIN-3156
> URL: https://issues.apache.org/jira/browse/KYLIN-3156
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Reporter: Liu Shaohui
>Assignee: Yifei Wu
>Priority: Major
>
> After KYLIN-2945, the meta url in SparkCubingByLayer will be a string of
> StorageURL not string with path@hdfs format. This will make the
> deleteHDFSMeta method failed in SparkCubingByLayer.
> {quote}
> 2018-01-08,11:51:50,903 ERROR org.apache.spark.deploy.yarn.ApplicationMaster:
> User class threw exception: java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
> java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653)
> Caused by: java.lang.IllegalArgumentException: Cannot create FileSystem from
> URI: kylin_tst:kylin_metadata
> at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:98)
> at
> org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:78)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.deleteHDFSMeta(SparkCubingByLayer.java:484)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:207)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 6 more
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 5: kylin_tst:kylin_metadata
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3048)
> at java.net.URI.(URI.java:588)
> at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:96)
> ... 10 more
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3156) Failed to delete meta path in SparkCubingByLayer
[
https://issues.apache.org/jira/browse/KYLIN-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494973#comment-16494973
]
Shaofeng SHI commented on KYLIN-3156:
-
Thanks for the reporting, I also found this issue with KYLIN-3368. A hot-fix
has been made on master branch.
> Failed to delete meta path in SparkCubingByLayer
>
>
> Key: KYLIN-3156
> URL: https://issues.apache.org/jira/browse/KYLIN-3156
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Reporter: Liu Shaohui
>Assignee: Yifei Wu
>Priority: Major
>
> After KYLIN-2945, the meta url in SparkCubingByLayer will be a string of
> StorageURL not string with path@hdfs format. This will make the
> deleteHDFSMeta method failed in SparkCubingByLayer.
> {quote}
> 2018-01-08,11:51:50,903 ERROR org.apache.spark.deploy.yarn.ApplicationMaster:
> User class threw exception: java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
> java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653)
> Caused by: java.lang.IllegalArgumentException: Cannot create FileSystem from
> URI: kylin_tst:kylin_metadata
> at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:98)
> at
> org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:78)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.deleteHDFSMeta(SparkCubingByLayer.java:484)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:207)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> ... 6 more
> Caused by: java.net.URISyntaxException: Illegal character in scheme name at
> index 5: kylin_tst:kylin_metadata
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3048)
> at java.net.URI.(URI.java:588)
> at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:96)
> ... 10 more
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3160) More date partition columns for incremental build
[ https://issues.apache.org/jira/browse/KYLIN-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3160. --- Resolution: Not A Problem > More date partition columns for incremental build > - > > Key: KYLIN-3160 > URL: https://issues.apache.org/jira/browse/KYLIN-3160 > Project: Kylin > Issue Type: Task > Components: Web > Environment: Hadoop, Hive >Reporter: Sandeep Reddy >Priority: Major > > I have a table with Partition Columns 1. Year(2017, 2016 etc) and 2. > Week(01,02,03 till 52) in separate columns, please suggest me how can I build > the cube incrementally??? > I found there are only Date and Timestamp columns available during creation > of model, But I need to build the cube on weekly basis from 1st week to 52nd > week of the year. Please suggest me the configuration for incremental build > of cube on weekly basis according to the week column availble in the table. > How can I schedule the incremental cube build on weekly basis??? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3160) More date partition columns for incremental build
[ https://issues.apache.org/jira/browse/KYLIN-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494972#comment-16494972 ] Shaofeng SHI commented on KYLIN-3160: - Please check the IPartitionConditionBuilder interface in PartitionDesc.java, and its sample implementation DefaultPartitionConditionBuilder and YearMonthDayPartitionConditionBuilder as a reference. You can implement a new by your own. > More date partition columns for incremental build > - > > Key: KYLIN-3160 > URL: https://issues.apache.org/jira/browse/KYLIN-3160 > Project: Kylin > Issue Type: Task > Components: Web > Environment: Hadoop, Hive >Reporter: Sandeep Reddy >Priority: Major > > I have a table with Partition Columns 1. Year(2017, 2016 etc) and 2. > Week(01,02,03 till 52) in separate columns, please suggest me how can I build > the cube incrementally??? > I found there are only Date and Timestamp columns available during creation > of model, But I need to build the cube on weekly basis from 1st week to 52nd > week of the year. Please suggest me the configuration for incremental build > of cube on weekly basis according to the week column availble in the table. > How can I schedule the incremental cube build on weekly basis??? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3240) Refine the exception when cannot find JobInstance in JobController
[
https://issues.apache.org/jira/browse/KYLIN-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3240.
-
Resolution: Fixed
Fix Version/s: v2.3.0
> Refine the exception when cannot find JobInstance in JobController
> --
>
> Key: KYLIN-3240
> URL: https://issues.apache.org/jira/browse/KYLIN-3240
> Project: Kylin
> Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.3.0
>
>
> When calling the rest api in {{JobController}}, if cannot initialize the
> related JobInstance based on the job id, BadRequestException should be thrown
> rather than InternalErrorException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3240) Refine the exception when cannot find JobInstance in JobController
[
https://issues.apache.org/jira/browse/KYLIN-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3240.
---
> Refine the exception when cannot find JobInstance in JobController
> --
>
> Key: KYLIN-3240
> URL: https://issues.apache.org/jira/browse/KYLIN-3240
> Project: Kylin
> Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.3.0
>
>
> When calling the rest api in {{JobController}}, if cannot initialize the
> related JobInstance based on the job id, BadRequestException should be thrown
> rather than InternalErrorException.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3250) Upgrade jetty version to 9.2.24.v20180105
[ https://issues.apache.org/jira/browse/KYLIN-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3250: --- Assignee: Chao Long > Upgrade jetty version to 9.2.24.v20180105 > - > > Key: KYLIN-3250 > URL: https://issues.apache.org/jira/browse/KYLIN-3250 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ted Yu >Assignee: Chao Long >Priority: Minor > Fix For: v2.4.0 > > > Currently Kylin depends on 9.2.20.v20161216 which is quite old. > Memory leak was fixed in Jetty 9.4 and later backported however the version > of Jetty 9.2.20 came before this fix. See > [https://github.com/eclipse/jetty.project/issues/1804] and > [https://github.com/apache/hbase/blob/master/pom.xml#L1416.] > This issue is to upgrade to > [9.3.22.v20171030|https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server/9.3.22.v20171030] > which is the latest version of 9.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3329) Step : Convert Cuboid Data to HFile , Call Hbase TimeOut , But status still running
[
https://issues.apache.org/jira/browse/KYLIN-3329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494970#comment-16494970
]
Shaofeng SHI commented on KYLIN-3329:
-
HBase also works as the metastore for Kylin; so once HBase is unhealthy,
Kylin's behavior will also be impacted. This is just the case: Kylin couldn't
update the job's status to fail.
> Step : Convert Cuboid Data to HFile , Call Hbase TimeOut , But status still
> running
> ---
>
> Key: KYLIN-3329
> URL: https://issues.apache.org/jira/browse/KYLIN-3329
> Project: Kylin
> Issue Type: Bug
>Reporter: Keith Chen
>Priority: Major
>
> Step : Convert Cuboid Data to HFile, Call Hbase TimeOut.But status not
> change, it still running.
> 2018-04-02 04:07:49,059 ERROR [Scheduler 107735485 Job
> 09d4c09b-7a53-43f9-9229-f1951c90f6ef-576] dao.ExecutableDao:221 : error
> update job output id:09d4c09b-7a53-43f9-9229-f1951c90f6ef-14
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=1, exceptions:
> Mon Apr 02 04:07:49 GMT+08:00 2018,
> RpcRetryingCaller\{globalStartTime=1522613264057, pause=100, retries=1},
> java.io.IOException: Call to xxx/x:16202 failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Cal
> l id=372918, waitTime=5002, operationTimeout=5000 expired.
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:157)
> at org.apache.hadoop.hbase.client.HTable.checkAndPut(HTable.java:1233)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:311)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:294)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:280)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:260)
> at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:104)
> at
> org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:218)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:470)
> at
> org.apache.kylin.job.execution.ExecutableManager.addJobInfo(ExecutableManager.java:480)
> at
> org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:352)
> at
> org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:364)
> at
> org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:85)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:138)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Call to x/ failed on local exception:
> org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=372918,
> waitTime=5002, operationTimeout=5000 expired.
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:292)
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1271)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.mutate(ClientProtos.java:34082)
> at org.apache.hadoop.hbase.client.HTable$9.call(HTable.java:1226)
> at org.apache.hadoop.hbase.client.HTable$9.call(HTable.java:1216)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
> ... 19 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3332) Support percentile measure in ODBC driver
[ https://issues.apache.org/jira/browse/KYLIN-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494966#comment-16494966 ] Shaofeng SHI commented on KYLIN-3332: - [~lidong_sjtu] is it possible to support percentile in Tableau? > Support percentile measure in ODBC driver > - > > Key: KYLIN-3332 > URL: https://issues.apache.org/jira/browse/KYLIN-3332 > Project: Kylin > Issue Type: Wish >Reporter: Le Anh Vu >Assignee: Dong Li >Priority: Critical > > I'm using Tableau to connect to Kylin via ODBC driver and I see percentile > measure option greyed out and requires extract to be able to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3332) Support percentile measure in ODBC driver
[ https://issues.apache.org/jira/browse/KYLIN-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3332: --- Assignee: Dong Li > Support percentile measure in ODBC driver > - > > Key: KYLIN-3332 > URL: https://issues.apache.org/jira/browse/KYLIN-3332 > Project: Kylin > Issue Type: Wish >Reporter: Le Anh Vu >Assignee: Dong Li >Priority: Critical > > I'm using Tableau to connect to Kylin via ODBC driver and I see percentile > measure option greyed out and requires extract to be able to use it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3378) Support Kafka table join with Hive tables
[ https://issues.apache.org/jira/browse/KYLIN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494936#comment-16494936 ] ASF subversion and git services commented on KYLIN-3378: Commit ddf9432884f14b745c62ad6e74a331f299397733 in kylin's branch refs/heads/KYLIN-3369 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ddf9432 ] KYLIN-3378 Support Kafka table join with Hive tables > Support Kafka table join with Hive tables > - > > Key: KYLIN-3378 > URL: https://issues.apache.org/jira/browse/KYLIN-3378 > Project: Kylin > Issue Type: Improvement > Components: Streaming >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > At this moment, if the data source is Kafka, only 1 table allowed in the data > model. In some cases, joining the kafka stream with hive lookup tables is > expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3369) Reduce the data size sink from Kafka topic to HDFS
[ https://issues.apache.org/jira/browse/KYLIN-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494935#comment-16494935 ] ASF subversion and git services commented on KYLIN-3369: Commit 6d623664ad513bd5a2ad7e9694cdd70f5ef7e6f6 in kylin's branch refs/heads/KYLIN-3369 from [~Na Zhai] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6d62366 ] KYLIN-3369 Kafka join with hive Signed-off-by: shaofengshi > Reduce the data size sink from Kafka topic to HDFS > -- > > Key: KYLIN-3369 > URL: https://issues.apache.org/jira/browse/KYLIN-3369 > Project: Kylin > Issue Type: Improvement > Components: Streaming >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > When building a cube from Kafka topic, the first step is to sink the Kafka > data to HDFS. In today's implementation, it will persist all the fields of a > message to disk. While in many cases, only a couple of fields will be needed > for cubing; Today's behavior wastes network bandwidth and disk space. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3378) Support Kafka table join with Hive tables
[ https://issues.apache.org/jira/browse/KYLIN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3378: Fix Version/s: v2.4.0 > Support Kafka table join with Hive tables > - > > Key: KYLIN-3378 > URL: https://issues.apache.org/jira/browse/KYLIN-3378 > Project: Kylin > Issue Type: Improvement > Components: Streaming >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > At this moment, if the data source is Kafka, only 1 table allowed in the data > model. In some cases, joining the kafka stream with hive lookup tables is > expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3378) Support Kafka table join with Hive tables
[ https://issues.apache.org/jira/browse/KYLIN-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3378: --- Assignee: Shaofeng SHI > Support Kafka table join with Hive tables > - > > Key: KYLIN-3378 > URL: https://issues.apache.org/jira/browse/KYLIN-3378 > Project: Kylin > Issue Type: Improvement > Components: Streaming >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.0 > > > At this moment, if the data source is Kafka, only 1 table allowed in the data > model. In some cases, joining the kafka stream with hive lookup tables is > expected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3392) support NULL value in Sum, Max, Min Aggregation
[ https://issues.apache.org/jira/browse/KYLIN-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494768#comment-16494768 ] Yifei Wu edited comment on KYLIN-3392 at 5/30/18 6:56 AM: -- [~albertoramon] you mean this issue KYLIN_2049 [KYLIN-2049|https://issues.apache.org/jira/browse/KYLIN-2049] was (Author: yifei_wu94): [~albertoramon] you mean this issue KYLIN_2049 > support NULL value in Sum, Max, Min Aggregation > --- > > Key: KYLIN-3392 > URL: https://issues.apache.org/jira/browse/KYLIN-3392 > Project: Kylin > Issue Type: Bug >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > > It is treated as 0 when confronted with NULL value in KYLIN's basic aggregate > measure (like sum, max, min). However, to distinguish the NULL value with 0 > is very necessary. > It should be like this > *sum(null, null) = null* > *sum(null, 1) = 1* > *max(null, null) = null* > *max(null, -1) = -1* > *min(null, -1）= -1* > in accordance with Hive and SparkSQL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3392) support NULL value in Sum, Max, Min Aggregation
[ https://issues.apache.org/jira/browse/KYLIN-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494768#comment-16494768 ] Yifei Wu commented on KYLIN-3392: - [~albertoramon] you mean this issue KYLIN_2049 > support NULL value in Sum, Max, Min Aggregation > --- > > Key: KYLIN-3392 > URL: https://issues.apache.org/jira/browse/KYLIN-3392 > Project: Kylin > Issue Type: Bug >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > > It is treated as 0 when confronted with NULL value in KYLIN's basic aggregate > measure (like sum, max, min). However, to distinguish the NULL value with 0 > is very necessary. > It should be like this > *sum(null, null) = null* > *sum(null, 1) = 1* > *max(null, null) = null* > *max(null, -1) = -1* > *min(null, -1）= -1* > in accordance with Hive and SparkSQL -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
[ https://issues.apache.org/jira/browse/KYLIN-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494733#comment-16494733 ] praveenece commented on KYLIN-3351: --- Could u explain anyone Above my question? > Cube Planner not working in apche kylin 2.3.0(open Source) > -- > > Key: KYLIN-3351 > URL: https://issues.apache.org/jira/browse/KYLIN-3351 > Project: Kylin > Issue Type: Task >Reporter: praveenece >Priority: Major > Attachments: image 1.png, image 2.png > > > Hi Team > i want test Cube planner in apache-kylin(2.3.0),So i created cube with > segment, and i hit query to cube many times more than thousands but the cube > planner there is no change in cuboid level like color changing and not get > COUNT in exactly row count and other items.can u please guide me. > Note: > Could u tell me? > Cube Planner Working only old Cube (like 3 month before) or new Cube also. > Configuration(kylin.Properties) > kylin.cube.cubeplanner.enabled=true > Kylin Version=2.3.0(SINGLE NODE) > MyCube creation =10days before > I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
