RE: How to increase split number for Fact distinct columns when using spark engine?(picture added)

Hi, shaofeng: Thank you for your suggestion! I'll give it a try. -- Best regards, Xi Chen -Original Message- From: ShaoFeng Shi Sent: Monday, November 5, 2018 4:06 PM To: dev Subject: Re: How to increase split number for Fact distinct columns when using spark

Re: How to increase split number for Fact distinct columns when using spark engine?(picture added)

Hi Xi Chen I think your refer to |'kylin.engine.spark.rdd-partition-cut-mb' parameter| (||howto_optimize_build.html was written for 2.1-2.2 version I don't remember, but these concepts are very important yet, I recommended read it) || On 5/11/18 8:52, 陈熹（chenxi07）-技术产品中心 wrote: Hi,

RE: How to increase split number for Fact distinct columns when using spark engine?(picture added)

Hi: Thank you for your suggestion! I checked source code. The ‘kylin.engine.spark.rdd-partition-cut-mb’ parameter is only used for spark cubing job, not fact table distinct count job. Maybe it’s worth adding this parameter to fact table distinct count job! -- Best regards, Xi Chen

Re: How to increase split number for Fact distinct columns when using spark engine?(picture added)

Hi Xi, The core is the same; By default, MR and Spark run a container/task for one file block. If we can split the data to more splits, will get more concurrent tasks: kylin.engine.mr.mapper-input-rows=50 陈熹（chenxi07）-技术产品中心 于2018年11月5日周一 下午3:52写道： > Hi, shaofeng: > Thank you for

[jira] [Created] (KYLIN-3666) MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException

Xi Chen created KYLIN-3666: -- Summary: MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException Key: KYLIN-3666 URL: https://issues.apache.org/jira/browse/KYLIN-3666 Project: Kylin

答复: [VOTE] Release apache-kylin-2.5.1 (RC1)

mvn test passed +1 binding Dong Li 发件人: Zhong, Yanghong 发送时间: 2018年11月5日 0:55 收件人: dev@kylin.apache.org 主题: Re: [VOTE] Release apache-kylin-2.5.1 (RC1) mvn clean test, passed +1 binding My environment: Apache Maven 3.5.4

Understanding about region cut size for base

Hi Team I would like to understand how does the 'kylin.storage.hbase.region-cut-gb’ property works. We are currently using kylin 2.3.1, We are going with the default property value ie; kylin.storage.hbase.region-cut-gb=5 But still we see some segments not adhering to this property; example:

[jira] [Created] (KYLIN-3667) ArrayIndexOutOfBoundsException in NDCuboidBuilder

Hubert STEFANI created KYLIN-3667: - Summary: ArrayIndexOutOfBoundsException in NDCuboidBuilder Key: KYLIN-3667 URL: https://issues.apache.org/jira/browse/KYLIN-3667 Project: Kylin Issue

kylin get error when run step 3

HI, Run Env: hadoop2.6.1，hive1.2.2，hbase1.0.3，kylin2.1.0 Descirbe: when cube run step 3,it can not continue forward,inside the hadoop log show : org.apache.hadoop.mapreduce.v2.app.MRAppMaster:Error starting MRAppMaster

[RESULT][VOTE] Release apache-kylin-2.5.1 (RC1)

Thanks to everyone who has tested the release candidate and given their comments and votes. The tally is as follows. 4 binding +1s: Shaofeng Shi Billy Liu Yanghong Zhong Dong Li 9 non-binding +1s: Chao Long Shaoxiong Zhan Jiatao Tao Cheng Wang Na Zhai Yichen Zhou Xiaoxiang Yu Lijun Cao George

Re: Understanding about region cut size for base

Hi Ketan, Kylin estimates the HBase table size; The estimation might be inaccurate when there are some advanced measures like TopN, Count distinct. The accuracy was improved in v2.5.0 by KYLIN-3453. For previous versions, you may need to manually give smaller value to these parameters:

[jira] [Created] (KYLIN-3668) Performing a cube build error on the Chinese hive table field

wxs created KYLIN-3668: -- Summary: Performing a cube build error on the Chinese hive table field Key: KYLIN-3668 URL: https://issues.apache.org/jira/browse/KYLIN-3668 Project: Kylin Issue Type: Bug