[jira] [Updated] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3711: Labels: Refactoring (was: ) > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > Labels: Refactoring > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete > it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3711: Description: I am reading source code and found some unused class, such as +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is created at 2015 and seems do nothing now(it should be replcaed by +org.apache.kylin.rest.controller.CacheController+). So someone should commented it with "duplicated" or simply delete it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] was: I am reading source code and found some unused class, such as +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is created at 2015 and seems do nothing now(it should be replcaed by +org.apache.kylin.rest.controller.CacheController+). So someone should commented it with "duplicated" or simply delete it. > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Critical > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete > it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3711: Priority: Minor (was: Critical) > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete > it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3711: Affects Version/s: Future > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Minor > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete > it.[BroadcasterReceiveServlet|https://github.com/apache/kylin/blob/kylin-2.5.0/server-base/src/main/java/org/apache/kylin/rest/broadcaster/BroadcasterReceiveServlet.java] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711091#comment-16711091 ] ASF subversion and git services commented on KYLIN-2565: Commit f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30 in kylin's branch refs/heads/2.5.x-hadoop3.1 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f4d2405 ] KYLIN-2565 Add cdh60 profile > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711089#comment-16711089 ] ASF subversion and git services commented on KYLIN-2565: Commit a906151cb7c2708351234efcee12bcf74e3e8e77 in kylin's branch refs/heads/2.5.x-hadoop3.1 from [~blue20080] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a906151 ] KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711090#comment-16711090
]
ASF subversion and git services commented on KYLIN-3518:
Commit 1e9305f8844577e3f5f60c85c0a835ae0a79ca92 in kylin's branch
refs/heads/2.5.x-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1e9305f ]
KYLIN-3518 Fix Coprocessor NPE problem on hbase 2
Signed-off-by: shaofengshi
> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes,
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require
> post aggregation, from 0 to 16384 Start:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:332 : Loading kylin-defaults.properties from
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 :
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6,
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692.,
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call exception, tries=7, retries=7,
> started=8436 ms ago, cancelled=false, msg=java.io.IOException
> at
[jira] [Commented] (KYLIN-3187) JDK APIs using the default locale, time zone or character set should be avoided
[
https://issues.apache.org/jira/browse/KYLIN-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711088#comment-16711088
]
ASF subversion and git services commented on KYLIN-3187:
Commit 09851df665c38d4c30d3611808382baf1648d239 in kylin's branch
refs/heads/2.5.x-hadoop3.1 from etherge
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=09851df ]
KYLIN-3187 Forbid calls to JDK APIs that use the default locale
> JDK APIs using the default locale, time zone or character set should be
> avoided
> ---
>
> Key: KYLIN-3187
> URL: https://issues.apache.org/jira/browse/KYLIN-3187
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Major
> Labels: usability
> Fix For: v2.5.2
>
>
> Here are a few examples:
> {code}
> server-base/src/main/java/org/apache/kylin/rest/service/JobService.java:
> Calendar calendar = Calendar.getInstance();
> storage-hbase/src/main/java/org/apache/kylin/storage/hbase/util/HbaseStreamingInput.java:
> Calendar cal = Calendar.getInstance();
> {code}
> Locale should be specified.
> See CALCITE-1667 for related information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3711) Clean unused/duplicated code
[ https://issues.apache.org/jira/browse/KYLIN-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu updated KYLIN-3711: Description: I am reading source code and found some unused class, such as +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is created at 2015 and seems do nothing now(it should be replcaed by +org.apache.kylin.rest.controller.CacheController+). So someone should commented it with "duplicated" or simply delete it. was: I am reading source code and found some unused class, such as org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet , this class is created at 2015 and seems do nothing now(it should be replcaed by org.apache.kylin.rest.controller.CacheController). So someone should commented it with "duplicated" or simply delete it. > Clean unused/duplicated code > > > Key: KYLIN-3711 > URL: https://issues.apache.org/jira/browse/KYLIN-3711 > Project: Kylin > Issue Type: Improvement >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Critical > > I am reading source code and found some unused class, such as > +org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet+ , this class is > created at 2015 and seems do nothing now(it should be replcaed by > +org.apache.kylin.rest.controller.CacheController+). > > So someone should commented it with "duplicated" or simply delete it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3711) Clean unused/duplicated code
XiaoXiang Yu created KYLIN-3711: --- Summary: Clean unused/duplicated code Key: KYLIN-3711 URL: https://issues.apache.org/jira/browse/KYLIN-3711 Project: Kylin Issue Type: Improvement Reporter: XiaoXiang Yu Assignee: XiaoXiang Yu I am reading source code and found some unused class, such as org.apache.kylin.rest.broadcaster.BroadcasterReceiveServlet , this class is created at 2015 and seems do nothing now(it should be replcaed by org.apache.kylin.rest.controller.CacheController). So someone should commented it with "duplicated" or simply delete it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3699) SparkCubingByLayer. Root cause: null
[
https://issues.apache.org/jira/browse/KYLIN-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
风语者 updated KYLIN-3699:
---
Attachment: mapreduce step8.png
> SparkCubingByLayer. Root cause: null
> -
>
> Key: KYLIN-3699
> URL: https://issues.apache.org/jira/browse/KYLIN-3699
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.5.0
> Environment: hdp.version： 2.5.0.0-1245
> kylin: 2.5.0 （apache-kylin-2.5.0-bin-hbase1x）
>Reporter: 风语者
>Assignee: Chao Long
>Priority: Major
> Attachments: Error.png, mapreduce step8.png, 异常信息.png, 异常步骤.png,
> 构建结果.png
>
>
> 你好：
> 我最近在使用 Kylin, 在使用 spark engine 构建 cube 的时候经常会出现一个异常，不知道怎么解决。异常如下：
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: null
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:297)
> at
> org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:108)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:182)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> 我发现，出现这个异常的时候，fact table 与 lookup table 关联后记录条数为0，就会触发这个问题。会在
> {color:#FF}*8 Step Name: Build Cube with Spark*{color} 报错。spark 引擎用的
> kylin 自带的 spark 2.1.2. 集群环境：hdp.version： 2.5.0.0-1245
> 如果用 mapreduce 引擎构建的话，是可以正常构建的，没有数据，流程也可以正常走通。
> 不知道这个问题该怎么解决。
> 希望可以得到回复。
> 。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3699) SparkCubingByLayer. Root cause: null
[
https://issues.apache.org/jira/browse/KYLIN-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711077#comment-16711077
]
风语者 commented on KYLIN-3699:
你好，
1： 第一步生成平表的时候，目录下面确实没有文件。（我单独手动建了一张表，关联后也是没有文件）。
2： hive --version =>Hive 1.2.1000.2.5.0.0-1245
3: mapreduce 构建时的第八步：8 Step Name: Build Base Cuboid：
这一步日志显示的是：skipped
> SparkCubingByLayer. Root cause: null
> -
>
> Key: KYLIN-3699
> URL: https://issues.apache.org/jira/browse/KYLIN-3699
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.5.0
> Environment: hdp.version： 2.5.0.0-1245
> kylin: 2.5.0 （apache-kylin-2.5.0-bin-hbase1x）
>Reporter: 风语者
>Assignee: Chao Long
>Priority: Major
> Attachments: Error.png, 异常信息.png, 异常步骤.png, 构建结果.png
>
>
> 你好：
> 我最近在使用 Kylin, 在使用 spark engine 构建 cube 的时候经常会出现一个异常，不知道怎么解决。异常如下：
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: null
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:297)
> at
> org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:108)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:182)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> 我发现，出现这个异常的时候，fact table 与 lookup table 关联后记录条数为0，就会触发这个问题。会在
> {color:#FF}*8 Step Name: Build Cube with Spark*{color} 报错。spark 引擎用的
> kylin 自带的 spark 2.1.2. 集群环境：hdp.version： 2.5.0.0-1245
> 如果用 mapreduce 引擎构建的话，是可以正常构建的，没有数据，流程也可以正常走通。
> 不知道这个问题该怎么解决。
> 希望可以得到回复。
> 。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3699) SparkCubingByLayer. Root cause: null
[
https://issues.apache.org/jira/browse/KYLIN-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711042#comment-16711042
]
Chao Long commented on KYLIN-3699:
--
我做了一些试验，发现这里存在两个问题：
1，在你的环境，Mapreduce能够成功，Spark会失败。
原因是Factdistinct那一步生成的CubeStats文件中cuboid对应的HLLCounter为空（根本原因是hive平表下的文件为空），Spark构建会报错是因为构建的时候会用CubeStats的数据估计分区的个数，因为没有数据，所以报了空指针异常，Mapreduce没有报错是因为没有数据的话，构建的步骤都被跳过了（你可以去看一下用mr构建的job，确认构建cuboid步骤是否被跳过）。所以Spark构建这里需要修复。
2，在我的环境里，Mapreduce和Spark构建都成功。
原因应该是Hive版本问题。在我的环境里（hdp
2.4.0.0-169)，构建平表没有数据，但是生成平表的目录下会有一个分区文件00_0，在你的环境下应该是没有生成这个文件，所以Spark构建出错（原因见1），你可以去看一下失败的任务，第一步打平表的目录下是否有文件，并且看一下你的hive版本是多少。
> SparkCubingByLayer. Root cause: null
> -
>
> Key: KYLIN-3699
> URL: https://issues.apache.org/jira/browse/KYLIN-3699
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.5.0
> Environment: hdp.version： 2.5.0.0-1245
> kylin: 2.5.0 （apache-kylin-2.5.0-bin-hbase1x）
>Reporter: 风语者
>Assignee: Chao Long
>Priority: Major
> Attachments: Error.png, 异常信息.png, 异常步骤.png, 构建结果.png
>
>
> 你好：
> 我最近在使用 Kylin, 在使用 spark engine 构建 cube 的时候经常会出现一个异常，不知道怎么解决。异常如下：
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: null
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:297)
> at
> org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:108)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:182)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> 我发现，出现这个异常的时候，fact table 与 lookup table 关联后记录条数为0，就会触发这个问题。会在
> {color:#FF}*8 Step Name: Build Cube with Spark*{color} 报错。spark 引擎用的
> kylin 自带的 spark 2.1.2. 集群环境：hdp.version： 2.5.0.0-1245
> 如果用 mapreduce 引擎构建的话，是可以正常构建的，没有数据，流程也可以正常走通。
> 不知道这个问题该怎么解决。
> 希望可以得到回复。
> 。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3699) SparkCubingByLayer. Root cause: null
[
https://issues.apache.org/jira/browse/KYLIN-3699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long reassigned KYLIN-3699:
Assignee: Chao Long
> SparkCubingByLayer. Root cause: null
> -
>
> Key: KYLIN-3699
> URL: https://issues.apache.org/jira/browse/KYLIN-3699
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.5.0
> Environment: hdp.version： 2.5.0.0-1245
> kylin: 2.5.0 （apache-kylin-2.5.0-bin-hbase1x）
>Reporter: 风语者
>Assignee: Chao Long
>Priority: Major
> Attachments: Error.png, 异常信息.png, 异常步骤.png, 构建结果.png
>
>
> 你好：
> 我最近在使用 Kylin, 在使用 spark engine 构建 cube 的时候经常会出现一个异常，不知道怎么解决。异常如下：
> Exception in thread "main" java.lang.RuntimeException: error execute
> org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: null
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:744)
> at
> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:297)
> at
> org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:108)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:182)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> 我发现，出现这个异常的时候，fact table 与 lookup table 关联后记录条数为0，就会触发这个问题。会在
> {color:#FF}*8 Step Name: Build Cube with Spark*{color} 报错。spark 引擎用的
> kylin 自带的 spark 2.1.2. 集群环境：hdp.version： 2.5.0.0-1245
> 如果用 mapreduce 引擎构建的话，是可以正常构建的，没有数据，流程也可以正常走通。
> 不知道这个问题该怎么解决。
> 希望可以得到回复。
> 。
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710913#comment-16710913
]
ASF GitHub Bot commented on KYLIN-3406:
---
hit-lacus commented on issue #373: KYLIN-3406 Ignore execute output file lose
URL: https://github.com/apache/kylin/pull/373#issuecomment-444728036
Local CI pass.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png,
> image-2018-12-05-18-45-32-923.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hit-lacus commented on issue #373: KYLIN-3406 Ignore execute output file lose
hit-lacus commented on issue #373: KYLIN-3406 Ignore execute output file lose URL: https://github.com/apache/kylin/pull/373#issuecomment-444728036 Local CI pass. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[ https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710905#comment-16710905 ] ASF GitHub Bot commented on KYLIN-3406: --- codecov-io commented on issue #373: KYLIN-3406 Ignore execute output file lose URL: https://github.com/apache/kylin/pull/373#issuecomment-444725970 # [Codecov](https://codecov.io/gh/apache/kylin/pull/373?src=pr=h1) Report > Merging [#373](https://codecov.io/gh/apache/kylin/pull/373?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/7f672b074a068d20779d33889f1cb63b0da172a7?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `33.33%`. [](https://codecov.io/gh/apache/kylin/pull/373?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #373 +/- ## + Coverage 23.63% 23.63% +<.01% - Complexity 4837 4843 +6 Files 1131 1132 +1 Lines 6819368233 +40 Branches 9684 9687 +3 + Hits 1611616127 +11 - Misses5050950536 +27 - Partials 1568 1570 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/373?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/kylin/common/persistence/BrokenEntity.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9Ccm9rZW5FbnRpdHkuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/kylin/storage/hbase/HBaseResourceStore.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-c3RvcmFnZS1oYmFzZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc3RvcmFnZS9oYmFzZS9IQmFzZVJlc291cmNlU3RvcmUuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/kylin/job/dao/ExecutableDao.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9kYW8vRXhlY3V0YWJsZURhby5qYXZh) | `39.78% <100%> (ø)` | `25 <0> (ø)` | :arrow_down: | | [...he/kylin/job/dao/ExecutableOutputPOSerializer.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9kYW8vRXhlY3V0YWJsZU91dHB1dFBPU2VyaWFsaXplci5qYXZh) | `56% <56%> (ø)` | `5 <5> (?)` | | | [...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=) | `70.94% <0%> (-1.36%)` | `8% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.3% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `78.37% <0%> (+5.4%)` | `7% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/373?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/373?src=pr=footer). Last update [7f672b0...5037167](https://codecov.io/gh/apache/kylin/pull/373?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > When the /execute_output/ metadata file sinking to HDFS is deleted, the > monitoring page does not display any tasks. > --- > > Key: KYLIN-3406 > URL: https://issues.apache.org/jira/browse/KYLIN-3406 > Project: Kylin > Issue Type: Bug > Components:
[GitHub] codecov-io commented on issue #373: KYLIN-3406 Ignore execute output file lose
codecov-io commented on issue #373: KYLIN-3406 Ignore execute output file lose URL: https://github.com/apache/kylin/pull/373#issuecomment-444725970 # [Codecov](https://codecov.io/gh/apache/kylin/pull/373?src=pr=h1) Report > Merging [#373](https://codecov.io/gh/apache/kylin/pull/373?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/7f672b074a068d20779d33889f1cb63b0da172a7?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `33.33%`. [](https://codecov.io/gh/apache/kylin/pull/373?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #373 +/- ## + Coverage 23.63% 23.63% +<.01% - Complexity 4837 4843 +6 Files 1131 1132 +1 Lines 6819368233 +40 Branches 9684 9687 +3 + Hits 1611616127 +11 - Misses5050950536 +27 - Partials 1568 1570 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/373?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../apache/kylin/common/persistence/BrokenEntity.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9wZXJzaXN0ZW5jZS9Ccm9rZW5FbnRpdHkuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...apache/kylin/storage/hbase/HBaseResourceStore.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-c3RvcmFnZS1oYmFzZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc3RvcmFnZS9oYmFzZS9IQmFzZVJlc291cmNlU3RvcmUuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...n/java/org/apache/kylin/job/dao/ExecutableDao.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9kYW8vRXhlY3V0YWJsZURhby5qYXZh) | `39.78% <100%> (ø)` | `25 <0> (ø)` | :arrow_down: | | [...he/kylin/job/dao/ExecutableOutputPOSerializer.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9kYW8vRXhlY3V0YWJsZU91dHB1dFBPU2VyaWFsaXplci5qYXZh) | `56% <56%> (ø)` | `5 <5> (?)` | | | [...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=) | `70.94% <0%> (-1.36%)` | `8% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `69.3% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/373/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `78.37% <0%> (+5.4%)` | `7% <0%> (+1%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/373?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/373?src=pr=footer). Last update [7f672b0...5037167](https://codecov.io/gh/apache/kylin/pull/373?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710892#comment-16710892
]
ASF GitHub Bot commented on KYLIN-3406:
---
asfgit commented on issue #373: KYLIN-3406 Ignore execute output file lose
URL: https://github.com/apache/kylin/pull/373#issuecomment-444721288
Can one of the admins verify this patch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png,
> image-2018-12-05-18-45-32-923.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710891#comment-16710891
]
ASF GitHub Bot commented on KYLIN-3406:
---
hit-lacus opened a new pull request #373: KYLIN-3406 Ignore execute output file
lose
URL: https://github.com/apache/kylin/pull/373
## Root Cause
When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page
will failed to display all job info and kylin process will throw
FileNotFoundException as above. The page has a warning message.
## Fix solution
The better way should be : miss file should only affect specific task. And
if only execute output miss, task should be success, otherwise that should be a
fatal exeception. You have to fix it manually.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png,
> image-2018-12-05-18-45-32-923.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit commented on issue #373: KYLIN-3406 Ignore execute output file lose
asfgit commented on issue #373: KYLIN-3406 Ignore execute output file lose URL: https://github.com/apache/kylin/pull/373#issuecomment-444721288 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] hit-lacus opened a new pull request #373: KYLIN-3406 Ignore execute output file lose
hit-lacus opened a new pull request #373: KYLIN-3406 Ignore execute output file lose URL: https://github.com/apache/kylin/pull/373 ## Root Cause When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page will failed to display all job info and kylin process will throw FileNotFoundException as above. The page has a warning message. ## Fix solution The better way should be : miss file should only affect specific task. And if only execute output miss, task should be success, otherwise that should be a fatal exeception. You have to fix it manually. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-1111) Ignore unsupported hive column types when sync hive table
[ https://issues.apache.org/jira/browse/KYLIN-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710527#comment-16710527 ] ASF GitHub Bot commented on KYLIN-: --- codecov-io commented on issue #372: KYLIN- Ignore unsupported hive column types when sync hive table URL: https://github.com/apache/kylin/pull/372#issuecomment-444610689 # [Codecov](https://codecov.io/gh/apache/kylin/pull/372?src=pr=h1) Report > Merging [#372](https://codecov.io/gh/apache/kylin/pull/372?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/7f672b074a068d20779d33889f1cb63b0da172a7?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `26.66%`. [](https://codecov.io/gh/apache/kylin/pull/372?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #372 +/- ## - Coverage 23.64% 23.63% -0.01% - Complexity 4838 4840 +2 Files 1131 1131 Lines 6819368199 +6 Branches 9684 9686 +2 - Hits 1612416120 -4 - Misses5050350510 +7 - Partials 1566 1569 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/372?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/kylin/source/hive/HiveMetadataExplorer.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9oaXZlL0hpdmVNZXRhZGF0YUV4cGxvcmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/kylin/metadata/datatype/DataType.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvZGF0YXR5cGUvRGF0YVR5cGUuamF2YQ==) | `70.28% <80%> (+0.28%)` | `44 <3> (+3)` | :arrow_up: | | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (-0.92%)` | `7% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/372?src=pr=footer). Last update [7f672b0...5971b6f](https://codecov.io/gh/apache/kylin/pull/372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Ignore unsupported hive column types when sync hive table > - > > Key: KYLIN- > URL: https://issues.apache.org/jira/browse/KYLIN- > Project: Kylin > Issue Type: Improvement >Reporter: liyang >Assignee: Wu Bin >Priority: Major > Labels: newbie > > Currently unsupported hive column type like map, array will cause failing to > load hive table. Instead, we could just ignore the unsupported column, and > still load the table with remaining columns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #372: KYLIN-1111 Ignore unsupported hive column types when sync hive table
codecov-io commented on issue #372: KYLIN- Ignore unsupported hive column types when sync hive table URL: https://github.com/apache/kylin/pull/372#issuecomment-444610689 # [Codecov](https://codecov.io/gh/apache/kylin/pull/372?src=pr=h1) Report > Merging [#372](https://codecov.io/gh/apache/kylin/pull/372?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/7f672b074a068d20779d33889f1cb63b0da172a7?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `26.66%`. [](https://codecov.io/gh/apache/kylin/pull/372?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #372 +/- ## - Coverage 23.64% 23.63% -0.01% - Complexity 4838 4840 +2 Files 1131 1131 Lines 6819368199 +6 Branches 9684 9686 +2 - Hits 1612416120 -4 - Misses5050350510 +7 - Partials 1566 1569 +3 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/372?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...apache/kylin/source/hive/HiveMetadataExplorer.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9oaXZlL0hpdmVNZXRhZGF0YUV4cGxvcmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/kylin/metadata/datatype/DataType.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvZGF0YXR5cGUvRGF0YVR5cGUuamF2YQ==) | `70.28% <80%> (+0.28%)` | `44 <3> (+3)` | :arrow_up: | | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | | | [.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=) | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/372/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (-0.92%)` | `7% <0%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/372?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/372?src=pr=footer). Last update [7f672b0...5971b6f](https://codecov.io/gh/apache/kylin/pull/372?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-1111) Ignore unsupported hive column types when sync hive table
[ https://issues.apache.org/jira/browse/KYLIN-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710494#comment-16710494 ] ASF GitHub Bot commented on KYLIN-: --- whuwb opened a new pull request #372: KYLIN- Ignore unsupported hive column types when sync hive table URL: https://github.com/apache/kylin/pull/372 Issue : https://issues.apache.org/jira/browse/KYLIN- Solution : Filter in HiveMetadataExplorer#extractColumnFromMeta How to verify: 1. add new UT 2. Verified with IT: [INFO] Apache Kylin - Integration Test SUCCESS [ 02:30 h] [INFO] Apache Kylin - Tomcat Extension SUCCESS [ 2.303 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 02:47 h [INFO] Finished at: 2018-12-05T18:29:37Z [INFO] 3. Manual add a Hive table with map, and try to load into Kylin This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Ignore unsupported hive column types when sync hive table > - > > Key: KYLIN- > URL: https://issues.apache.org/jira/browse/KYLIN- > Project: Kylin > Issue Type: Improvement >Reporter: liyang >Assignee: Wu Bin >Priority: Major > Labels: newbie > > Currently unsupported hive column type like map, array will cause failing to > load hive table. Instead, we could just ignore the unsupported column, and > still load the table with remaining columns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-1111) Ignore unsupported hive column types when sync hive table
[ https://issues.apache.org/jira/browse/KYLIN-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710495#comment-16710495 ] ASF GitHub Bot commented on KYLIN-: --- asfgit commented on issue #372: KYLIN- Ignore unsupported hive column types when sync hive table URL: https://github.com/apache/kylin/pull/372#issuecomment-444600629 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Ignore unsupported hive column types when sync hive table > - > > Key: KYLIN- > URL: https://issues.apache.org/jira/browse/KYLIN- > Project: Kylin > Issue Type: Improvement >Reporter: liyang >Assignee: Wu Bin >Priority: Major > Labels: newbie > > Currently unsupported hive column type like map, array will cause failing to > load hive table. Instead, we could just ignore the unsupported column, and > still load the table with remaining columns. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] whuwb opened a new pull request #372: KYLIN-1111 Ignore unsupported hive column types when sync hive table
whuwb opened a new pull request #372: KYLIN- Ignore unsupported hive column types when sync hive table URL: https://github.com/apache/kylin/pull/372 Issue : https://issues.apache.org/jira/browse/KYLIN- Solution : Filter in HiveMetadataExplorer#extractColumnFromMeta How to verify: 1. add new UT 2. Verified with IT: [INFO] Apache Kylin - Integration Test SUCCESS [ 02:30 h] [INFO] Apache Kylin - Tomcat Extension SUCCESS [ 2.303 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 02:47 h [INFO] Finished at: 2018-12-05T18:29:37Z [INFO] 3. Manual add a Hive table with map, and try to load into Kylin This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #372: KYLIN-1111 Ignore unsupported hive column types when sync hive table
asfgit commented on issue #372: KYLIN- Ignore unsupported hive column types when sync hive table URL: https://github.com/apache/kylin/pull/372#issuecomment-444600629 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (KYLIN-3607) can't build cube with spark in v2.5.0
[
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710462#comment-16710462
]
jon shoberg edited comment on KYLIN-3607 at 12/5/18 6:34 PM:
-
I'm getting the same issue on Kylin-2.5.2 with other services deployed from TAR
(non-HDP)
HBase 1.4.8 is the version currently being used and made sure there are no
other HBase versioned jars which would conflict this. Hadoop Version is 2.8.5.
{code:java}
export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry --conf
spark.executor.instances=40 --conf spark.yarn.queue=default --conf
spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
spark.master=yarn --conf spark.hadoop.yarn.timeline-service.enabled=false
--conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
spark.eventLog.dir=hdfs:///kylin/spark-history --conf
spark.yarn.executor.memoryOverhead=1024 --conf spark.driver.memory=2G --conf
spark.shuffle.service.enabled=true --jars
/opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,
/opt/kylin/lib/kylin-job-2.5.2.jar -className
org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/rowkey_stats/part-r-0_hfile
-counterOutput
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/counter
-cubename HoldingNodeCube -output
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/hfile
-input
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/cuboid/
-segmentId 37ef5ffa-5894-980f-4e20-33ec301e6ecf -metaUrl
kylin_metadata@hdfs,path=hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/metadata
-hbaseConfPath
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/hbase-conf.xml
{code}
was (Author: jshoberg):
I'm getting the same issue on Kylin-2.5.2 with other services deployed from TAR
(non-HDP)
HBase 1.4.8 is the version currently being used and made sure there are no
other HBase versioned jars which would conflict this.
{code:java}
export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry --conf
spark.executor.instances=40 --conf spark.yarn.queue=default --conf
spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
spark.master=yarn --conf spark.hadoop.yarn.timeline-service.enabled=false
--conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
spark.eventLog.dir=hdfs:///kylin/spark-history --conf
spark.yarn.executor.memoryOverhead=1024 --conf spark.driver.memory=2G --conf
spark.shuffle.service.enabled=true --jars
/opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,
/opt/kylin/lib/kylin-job-2.5.2.jar -className
org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/rowkey_stats/part-r-0_hfile
-counterOutput
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/counter
-cubename HoldingNodeCube -output
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/hfile
-input
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/cuboid/
-segmentId 37ef5ffa-5894-980f-4e20-33ec301e6ecf -metaUrl
kylin_metadata@hdfs,path=hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/metadata
-hbaseConfPath
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/hbase-conf.xml
{code}
> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
> Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile,
> the following is the related exception:
>
> ERROR
[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0
[
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710462#comment-16710462
]
jon shoberg commented on KYLIN-3607:
I'm getting the same issue on Kylin-2.5.2 with other services deployed from TAR
(non-HDP)
HBase 1.4.8 is the version currently being used and made sure there are no
other HBase versioned jars which would conflict this.
{code:java}
export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/spark/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry --conf
spark.executor.instances=40 --conf spark.yarn.queue=default --conf
spark.history.fs.logDirectory=hdfs:///kylin/spark-history --conf
spark.master=yarn --conf spark.hadoop.yarn.timeline-service.enabled=false
--conf spark.executor.memory=4G --conf spark.eventLog.enabled=true --conf
spark.eventLog.dir=hdfs:///kylin/spark-history --conf
spark.yarn.executor.memoryOverhead=1024 --conf spark.driver.memory=2G --conf
spark.shuffle.service.enabled=true --jars
/opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,
/opt/kylin/lib/kylin-job-2.5.2.jar -className
org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/rowkey_stats/part-r-0_hfile
-counterOutput
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/counter
-cubename HoldingNodeCube -output
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/hfile
-input
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/cuboid/
-segmentId 37ef5ffa-5894-980f-4e20-33ec301e6ecf -metaUrl
kylin_metadata@hdfs,path=hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/metadata
-hbaseConfPath
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/hbase-conf.xml
{code}
> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
> Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile,
> the following is the related exception:
>
> ERROR yarn.ApplicationMaster: User class threw exception:
> java.lang.RuntimeException: error execute
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted
> due to stage failure: Task 0 in stage 1.0 failed 4 times,
> java.lang.ExceptionInInitializerError
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
> at
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the
> hadoop compatibility jar on the classpath?
> at
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
> at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
> at
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709638#comment-16709638
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 11:53 AM:
---
[5261]I have reproduced error as reporter describe.
h2. Root Cause
When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page will
failed to display all job info and kylin process will throw
FileNotFoundException as above. The page has a warning message.
h2. Fix solution
The better way should be : miss file should only affect specific task. And if
only execute output miss, task should be success, otherwise that should be a
fatal exeception. You have to fix it manually.
was (Author: hit_lacus):
[5261]I have reproduced error as reporter describe.
When any single file of Kylin's metadata miss in HDFS, the "Monitor" Page will
failed to display all job info and kylin process will throw
FileNotFoundException as above.
The better way should be miss file should only affect specific task. And if
only execute output miss, task should be success, otherwise task should be
marked as failed.
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png,
> image-2018-12-05-18-45-32-923.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709899#comment-16709899
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 11:46 AM:
---
h2. Test evidence
* Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
* Update code and restart server.
* Visit the "monitor" page.
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!image-2018-12-05-18-45-32-923.png!
was (Author: hit_lacus):
h2. Test evidence
Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709899#comment-16709899
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 11:41 AM:
---
h2. Test evidence
Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!image-2018-12-05-18-45-32-923.png!
was (Author: hit_lacus):
Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709899#comment-16709899
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 10:45 AM:
---
Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!image-2018-12-05-18-45-32-923.png!
was (Author: hit_lacus):
Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!Step which lost execute output.png!
> When the
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709899#comment-16709899
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 10:44 AM:
---
Remove this file : /execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-10
{quote}⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs 308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs 260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs 254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs 529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs 661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!Step which lost execute output.png!
was (Author: hit_lacus):
{quote} ⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!Step which
[jira] [Comment Edited] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709899#comment-16709899
]
XiaoXiang Yu edited comment on KYLIN-3406 at 12/5/18 10:42 AM:
---
{quote} ⚡ root@sandbox ~/clyne master ● hadoop fs -ls
/kylin/kylin_metadata3406/resources/execute_output
Found 15 items
-rw-r--r-- 3 root hdfs308 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85
-rw-r--r-- 3 root hdfs 3332 2018-12-05 10:51
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-00
-rw-r--r-- 3 root hdfs 2675 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-01
-rw-r--r-- 3 root hdfs260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-02
-rw-r--r-- 3 root hdfs254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-03
-rw-r--r-- 3 root hdfs260 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-04
-rw-r--r-- 3 root hdfs254 2018-12-05 10:52
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-05
-rw-r--r-- 3 root hdfs254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-06
-rw-r--r-- 3 root hdfs254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-07
-rw-r--r-- 3 root hdfs254 2018-12-05 10:53
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-08
-rw-r--r-- 3 root hdfs 2522 2018-12-05 10:54
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-09
-rw-r--r-- 3 root hdfs260 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-11
-rw-r--r-- 3 root hdfs254 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-12
-rw-r--r-- 3 root hdfs529 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-13
-rw-r--r-- 3 root hdfs661 2018-12-05 10:56
/kylin/kylin_metadata3406/resources/execute_output/3b27b7df-aece-ba28-e986-3758fe8b9a85-14
{quote}
!Lost Execute Output.png!
!Step which lost execute output.png!
was (Author: hit_lacus):
!Lost Execute Output.png!
!Step which lost execute output.png!
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3406:
Attachment: Step which lost execute output.png
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709899#comment-16709899
]
XiaoXiang Yu commented on KYLIN-3406:
-
!Lost Execute Output.png!
!Step which lost execute output.png!
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png, Step which lost execute
> output.png, image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3406) When the /execute_output/ metadata file sinking to HDFS is deleted, the monitoring page does not display any tasks.
[
https://issues.apache.org/jira/browse/KYLIN-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiaoXiang Yu updated KYLIN-3406:
Attachment: Lost Execute Output.png
> When the /execute_output/ metadata file sinking to HDFS is deleted, the
> monitoring page does not display any tasks.
> ---
>
> Key: KYLIN-3406
> URL: https://issues.apache.org/jira/browse/KYLIN-3406
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.3.1
>Reporter: yujie.sun
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v2.6.0
>
> Attachments: Lost Execute Output.png,
> image-2018-12-05-13-22-42-456.png
>
>
> *{color:#24292e}Issue description:{color}*
> When a single file of Hbase metadata is greater than a certain value, the
> file is sunk into HDFS for storage. When files in execute_output/ are deleted
> in HDFS, the monitoring page will not display any tasks.
> *{color:#24292e}Reproduction step:{color}*
> {color:#24292e}1.Use Hbase as metadata storage{color}
> {color:#24292e}2.Modify the hbase configuration ,configure
> "hbase.client.keyvalue.maxsize" into a small value, such as 524288.{color}
> {color:#24292e}3.Create a new build task, after the build task finished,find
> the /execute_output hbase metadata path on HDFS,
> /\{working_dir}/\{metadata_url}/resources/execute_output/..., and delete one
> of the files.{color}
> {color:#24292e}4.At this time,{color:#33}the monitoring page will not
> display any tasks.{color}{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709789#comment-16709789
]
Hongtao He commented on KYLIN-3709:
---
Hi Chao, Obtaining the absolute path of the configuration file is a problem.
Rolling back the code may be a good way.
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.6.0
>Reporter: Hongtao He
>Assignee: Chao Long
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example, the value of hdfsConfigFile is
> "hbase.hdfs.xml", so addResource function will not work . The end result is
> that kylin can not load the hbase configuration file hbase.hdfs.xml .
> There are two ways to fix this bug, and I think method 1 is better.
> Method-1.revert the code
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(hdfsConfigFile);{code}
> Method-2.Get the absolute path of the configuration file
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
> hdfsConfigFile;
> hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3710) Spark cubing failed with JDBC data source
Chao Long created KYLIN-3710: Summary: Spark cubing failed with JDBC data source Key: KYLIN-3710 URL: https://issues.apache.org/jira/browse/KYLIN-3710 Project: Kylin Issue Type: Bug Components: RDBMS Source Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.0 Caused by: java.lang.RuntimeException: Cannot adapt to interface org.apache.kylin.engine.spark.ISparkInput at org.apache.kylin.source.jdbc.JdbcSource.adaptToBuildEngine(JdbcSource.java:49) at org.apache.kylin.source.SourceManager.createEngineAdapter(SourceManager.java:148) at org.apache.kylin.engine.spark.SparkUtil.getBatchCubingInputSide(SparkUtil.java:57) at org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:49) at org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44) at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:56) at org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:245) at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:213) at org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:398) ... 77 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709756#comment-16709756
]
Hongtao He commented on KYLIN-3709:
---
Hi shaofeng, you are right, Kylin 2.5.x doesn't have the code change of
KYLIN-3648 .
Thank you!
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.6.0
>Reporter: Hongtao He
>Assignee: Chao Long
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example, the value of hdfsConfigFile is
> "hbase.hdfs.xml", so addResource function will not work . The end result is
> that kylin can not load the hbase configuration file hbase.hdfs.xml .
> There are two ways to fix this bug, and I think method 1 is better.
> Method-1.revert the code
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(hdfsConfigFile);{code}
> Method-2.Get the absolute path of the configuration file
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
> hdfsConfigFile;
> hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3709) Pass the wrong parameter to addResource function, kylin can not load the configuration file hbase.hdfs.xml .
[
https://issues.apache.org/jira/browse/KYLIN-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongtao He updated KYLIN-3709:
--
Affects Version/s: (was: v2.5.2)
(was: v2.5.1)
v2.6.0
> Pass the wrong parameter to addResource function, kylin can not load the
> configuration file hbase.hdfs.xml .
>
>
> Key: KYLIN-3709
> URL: https://issues.apache.org/jira/browse/KYLIN-3709
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.6.0
>Reporter: Hongtao He
>Assignee: Chao Long
>Priority: Critical
> Attachments:
> method-1-Pass-the-wrong-parameter-to-addResource-function-kyl.patch,
> method-2-Pass-the-wrong-parameter-to-addResource-function-kyl.patch
>
>
> When calling Configuration.addResource, KYLIN-3648 use a Path as the
> parameter instead of a string in Kylin's HBaseConnection.
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(new Path(hdfsConfigFile));
> {code}
> Use a Path as the parameter of Configuration.addResource is better.
> Unfortunately, the parameter which passed to the addResource function is
> wrong. The addResource function only accepts absolute paths, but the
> parameter is just a filename. For example, the value of hdfsConfigFile is
> "hbase.hdfs.xml", so addResource function will not work . The end result is
> that kylin can not load the hbase configuration file hbase.hdfs.xml .
> There are two ways to fix this bug, and I think method 1 is better.
> Method-1.revert the code
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> hdfsConf.addResource(hdfsConfigFile);{code}
> Method-2.Get the absolute path of the configuration file
> {code:java}
> String hdfsConfigFile =
> KylinConfig.getInstanceFromEnv().getHBaseClusterHDFSConfigFile();
> if (hdfsConfigFile == null || hdfsConfigFile.isEmpty()) {
> return;
> }
> Configuration hdfsConf = new Configuration(false);
> String hbaseHdfsConfigPath = System.getProperty("user.dir") + "/../conf/" +
> hdfsConfigFile;
> hdfsConf.addResource(new Path(hbaseHdfsConfigPath));
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
