[jira] [Created] (KYLIN-4100) Add overall job number statistics in monitor page
Liu Shaohui created KYLIN-4100: -- Summary: Add overall job number statistics in monitor page Key: KYLIN-4100 URL: https://issues.apache.org/jira/browse/KYLIN-4100 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Currently it's hard to get pending and running job number in mointor page, we can only continue to click more until the end. It's better to have an overall job number statistics in monitor page. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4099) Using no blocking unpersist in spark cubing job
Liu Shaohui created KYLIN-4099:
--
Summary: Using no blocking unpersist in spark cubing job
Key: KYLIN-4099
URL: https://issues.apache.org/jira/browse/KYLIN-4099
Project: Kylin
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
By default, the unpersist operation of RDD in spark is blocking which may cost
a lot time and
some times it may failed for some spark executors lost.
We can set blocking false to improve it.
{code:java}
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:208)
scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
scala.concurrent.Await$.result(package.scala:190)
org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81)
org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:127)
org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1709)
org.apache.spark.rdd.RDD.unpersist(RDD.scala:216)
org.apache.spark.api.java.JavaPairRDD.unpersist(JavaPairRDD.scala:73)
org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:204)
org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4098) Add cube auto merge api
Liu Shaohui created KYLIN-4098: -- Summary: Add cube auto merge api Key: KYLIN-4098 URL: https://issues.apache.org/jira/browse/KYLIN-4098 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Currently the auto merging of cube is triggered by the event of new segment is ready automatically. When the cluster restart, there may be too many merging job. It's better to have a rest api to trigger the merging and make it more controllable. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
Re: 问题求助
Similar to this article: https://zhuanlan.zhihu.com/p/28765127 With Warm regards Billy Liu 张峥 于2019年7月18日周四 下午2:42写道： > Kylin 团队，你们好 > > > > 我的团队在使用中遇到一个问题，求教。 > > 问题举例： > >目前KYLIN只支持在model中定义的度量值，而不能对这个度量值做任何运算。例如learn_kylin demo中的PRICE > ，哪怕我在tableau中新增一个度量值 利润 = PRICE * 0.2，也会报错。 > >基于这个现状，我的思路是，1、把所有的度量都事先通过SQL计算再放入KYLIN建模加速，或者2使用数据提取到tableau。 > >方法一SQL工作量巨大，且浪费了tableau的分析能力。 > >方法二，数据提取后，所有的度量值均变为了null。 > > > > 请问有什么好的解决方法吗？ > > 谢谢。 > > > > > > > > > > --- > 张峥 Frank Zhang > 上海医时信息科技有限公司 > Ethicall Information Technology Co.,ltd > 上海市闵行区万源路2163号新桥商务中心B座501室, 201103 > Room 501,Building B,Xinqiao Business Center,No.2163 Wanyuan Road, Minhang > District, Shanghai > Cell & WeChat:18652024136 > E-mail:zha...@ethicall.cn > www.ethicall.cn > > [image: signature_1707173497] >
[jira] [Created] (KYLIN-4097) Throw exception when too many dict slice eviction in AppendTrieDictionary
Liu Shaohui created KYLIN-4097: -- Summary: Throw exception when too many dict slice eviction in AppendTrieDictionary Key: KYLIN-4097 URL: https://issues.apache.org/jira/browse/KYLIN-4097 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui When the global dict is too large than the spark executor memory, there will be too many dict slice evictions and loads in AppendTrieDictionary, and the build job will be very slow. It's better to throw an exception in advance in this case. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4095) Add RESOURCE_PATH_PREFIX option in ResourceTool
Liu Shaohui created KYLIN-4095: -- Summary: Add RESOURCE_PATH_PREFIX option in ResourceTool Key: KYLIN-4095 URL: https://issues.apache.org/jira/browse/KYLIN-4095 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui ResourceTool is very useful to fix the metadata with overlap segments. But downloading and uploading entire metadata is too heavy. It's better to have a RESOURCE_PATH_PREFIX option for downloading and uploading cmds. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
问题求助
Kylin 团队，你们好 我的团队在使用中遇到一个问题，求教。 问题举例： 目前KYLIN只支持在model中定义的度量值，而不能对这个度量值做任何运算。例如learn_kylin demo中的PRICE，哪怕我在tableau中新增一个度量值 利润 = PRICE * 0.2，也会报错。 基于这个现状，我的思路是，1、把所有的度量都事先通过SQL计算再放入KYLIN建模加速，或者2使用数据提取到tableau。 方法一SQL工作量巨大，且浪费了tableau的分析能力。 方法二，数据提取后，所有的度量值均变为了null。 请问有什么好的解决方法吗？ 谢谢。 --- 张峥 Frank Zhang 上海医时信息科技有限公司 Ethicall Information Technology Co.,ltd 上海市闵行区万源路2163号新桥商务中心B座501室, 201103 Room 501,Building B,Xinqiao Business Center,No.2163 Wanyuan Road, Minhang District, Shanghai Cell & WeChat:18652024136 E-mail:zha...@ethicall.cn www.ethicall.cn
[jira] [Created] (KYLIN-4094) Add script to create system tables and cubes automatically
Liu Shaohui created KYLIN-4094: -- Summary: Add script to create system tables and cubes automatically Key: KYLIN-4094 URL: https://issues.apache.org/jira/browse/KYLIN-4094 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui See: [http://kylin.apache.org/docs/tutorial/setup_systemcube.html] It's a little complex to setup the the system cubes. We can add a scripts to make it easier. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4093) Slow query pages should be open to all users of the project
Liu Shaohui created KYLIN-4093: -- Summary: Slow query pages should be open to all users of the project Key: KYLIN-4093 URL: https://issues.apache.org/jira/browse/KYLIN-4093 Project: Kylin Issue Type: Improvement Reporter: Liu Shaohui Currently the show query page only can been seen for kylin admins. It's very useful for modlers and analysts for this project. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
