Cube Designer Consultation

Kylin crew: I 'm in the Cube Designer has the following questions need to consult: The first question :Dimensions step: Click Add Dimension, it popups two option: “Normal” and “Derived”, Can explain the effect of Normal and Derived, and in what circumstance,finally have

kylin query problem

Kylin crew: Kyliin source, on Kylin query execution process analysis, there are two questions: 1, kylin is how to use the complete SQL parsing of calcite, obtain the SQL context; 2, kylin, how to get the data from the pre-computed data and calculated. OLAPToEnumerableConverter、OLAPEnumerator

Cube Designer 中Measures问题

您好： 目前kylin版本使用是1.5.2, Cube Designer 中Measures, Expression 为SUM, Return Type 为DECIMAL [cid:image001.jpg@01D1C555.7E9EBF80] 这是按日统计的，第一条是hive查询的结果；第二条是kylin查询的结果，科学计数法二者不一样 Hive里字段为double类型 每天的数据SUM值不同，按维度年来统计导致差距也很大：第一条是kylin查询结果，第二条是hive查询结果。 [cid:image002.png@01D1C555.7E9EBF80] 请问：一、kylin精度是怎么

Questions about kylin page and js modify packaging

您好： 由于根据公司项目使用需要，修改了部分js和html，但是这些修改的js和html要打到生产环境的哪个路径下？ 在/kylin/tomcat/路径下没找到相关文件。 附件是相关js贴图。 [cid:image001.png@01D1C659.8536DFD0]

答复: Questions about kylin page and js modify packaging

- 赵天烁 Kevin Zhao zhaotians...@meizu.com<mailto:zhaotians...@meizu.com> 珠海市魅族科技有限公司 MEIZU Technology Co., Ltd. 广东省珠海市科技创新海岸魅族科技楼 MEIZU Tech Bldg., Technology & Innovation Coast Zhuhai, 519085, Guangdong, China meizu.com 发件人: 仇同心 [mailto:qiutong...@jd

答复: Questions about kylin page and js modify packaging

- 赵天烁 Kevin Zhao zhaotians...@meizu.com<mailto:zhaotians...@meizu.com> 珠海市魅族科技有限公司 MEIZU Technology Co., Ltd. 广东省珠海市科技创新海岸魅族科技楼 MEIZU Tech Bldg., Technology & Innovation Coast Zhuhai, 519085, Guangdong, China meizu.com 发件人: 仇同心 [mailto:qiutong...@jd

Extract Fact Table Distinct Columns problem

Kylin crew: “Extract Fact Table Distinct Columns”, In this step will often happen OOM,I don't know what this step is doing, Can do a simple function description? In this FactDistinctColumnsJob.class , Line 80 ,List columnsNeedDict = cubeMgr.getAllDictColumnsOnFact(cubeDesc) : The method to r

Extract Fact Table Distinct Columns problem

Kylin crew: “Extract Fact Table Distinct Columns”, In this step will often happen OOM,I don't know what this step is doing, Can do a simple function description? In this FactDistinctColumnsJob.class , Line 80 ,List columnsNeedDict = cubeMgr.getAllDictColumnsOnFact(cubeDesc) : The method to r

关于measure预计算

大家好： 在cube构建时，根据cube 设计时，measure可以有不同的聚合函数。我想找到根据不同的聚合函数来做计算的源码，但是在 // Phase 3: Build Cube addLayerCubingSteps(result, jobId, cuboidRootPath); // layer cubing, only selected algorithm will execute result.addTask(createInMemCubingStep(jobId, cuboidRootPath)); // inmem cubing, only selected algori

DISTINCT_COUNT精确计算问题

大家好： Hive字段类型为varchar,字段内容也包含英文字母和中文，对这样的字段能否做DISTINCT_COUNT精确计算？如果不能，有什么好的建议吗？ 谢谢！

答复: 回复：DISTINCT_COUNT精确计算问题

您好： Hash值是否会出现重复呢？ 谢谢！ 发件人: Weatherpop [mailto:623891...@qq.com] 发送时间: 2016年6月21日 15:11 收件人: user@kylin.apache.org; d...@kylin.apache.org 主题: 回复：DISTINCT_COUNT精确计算问题 可以自己做一张映射表，把值hash成int后就可以用精确的count distinct了 我们这边目前实践暂时是这样的 -- 原始邮件 -- 发件人: "仇同心";mail

答复: Rest API for creating API

Hi: Hope this article can help you: http://kylin.apache.org/docs/tutorial/kylin_client_tool.html In 1.3 x branch on a python tool, there are some good scripts, but 1.5 x: not yet, You can refer to this. https://github.com/apache/kylin/tree/1.3.x/tools/kylin_client_tool Thanks 发件人: Su

kylin Remote JDBC Driver problem

大家好： 用kylin的JDBC查询，查询条件是date类型的，但是查询不出数据，以下是代码： driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance(); Properties info = new Properties(); info.put("user", "ADMIN"); info.put("password", "KYLIN"); conn = dr

kylin Remote JDBC Driver problem

大家好： 用kylin的JDBC查询，查询条件是date类型的，但是查询不出数据，以下是代码： driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance(); Properties info = new Properties(); info.put("user", "ADMIN"); info.put("password", "KYLIN"); conn = dr

cube drop problem

大家好： 在cubes list页面drop掉某些cube后，为啥每次loadAllCubeInstance时报错 2016-06-30 10:47:19,773 ERROR [localhost-startStop-1] cube.CubeManager:862 : Error during load cube instance /cube/cube1 .json java.lang.IllegalStateException: CubeInstance desc not found 'CUBE1', at /cube/cube1.json at org.

cube drop problem

大家好： 在cubes list页面drop掉某些cube后，为啥每次loadAllCubeInstance时报错 2016-06-30 10:47:19,773 ERROR [localhost-startStop-1] cube.CubeManager:862 : Error during load cube instance /cube/cube1 .json java.lang.IllegalStateException: CubeInstance desc not found 'CUBE1', at /cube/cube1.json at org.

kylin查询，报超时异常:Timeout visiting cube!

大家好： Kylin查询时报超时异常,sql是： select b.dim_month_name,sum(a.ordr_amt) as 订单金额, sum(a.pay_amt) as 支付金额,count(*) as 订单数, count(distinct a.user_pin)as 用户数,count(distinct a.is_new) as 新用户数 from dmt.dmt_mem_vip_tx_ordr_det_i_d a left join dim.dim_day b on a.pay_time=b.dim_day_txdate left join dim.

kylin查询，报超时异常:Timeout visiting cube!

大家好： Kylin查询时报超时异常,sql是： select b.dim_month_name,sum(a.ordr_amt) as 订单金额, sum(a.pay_amt) as 支付金额,count(*) as 订单数, count(distinct a.user_pin)as 用户数,count(distinct a.is_new) as 新用户数 from dmt.dmt_mem_vip_tx_ordr_det_i_d a left join dim.dim_day b on a.pay_time=b.dim_day_txdate left join dim.

kylin查询，报超时异常:Timeout visiting cube!

大家好： Kylin查询时报超时异常,sql是： select b.dim_month_name,sum(a.ordr_amt) as 订单金额, sum(a.pay_amt) as 支付金额,count(*) as 订单数, count(distinct a.user_pin)as 用户数,count(distinct a.is_new) as 新用户数 from dmt.dmt_mem_vip_tx_ordr_det_i_d a left join dim.dim_day b on a.pay_time=b.dim_day_txdate left join dim.

答复: kylin查询，报超时异常:Timeout visiting cube!

count" measures, are they HyperLogLog counter or Bitmap counter? 2016-06-30 18:09 GMT+08:00 仇同心 : > 大家好： > Kylin查询时报超时异常,sql是： > select b.dim_month_name,sum(a.ordr_amt) as 订单金额, > sum(a.pay_amt) as 支付金额,count(*) as 订单数, > count(distinct a.user_pin)as 用户数,coun

答复: 答复: kylin查询，报超时异常:Timeout visiting cube!

so to reduce the post aggregation. 2016-07-01 11:20 GMT+08:00 仇同心 : > 少峰，您好： > two "distinct count" measures, are HyperLogLog counter。 > >一、 group by b.dim_month_name 这个是derived measure。 > > 测试了下，如果where 条件是月，group by 周 查询时间是66秒， where 条件是周，group by &g

关于cube build和merge的问题

大家好： 目前在kylin的使用过程中遇到以下一个问题： Cube构建时，开始时间设置为：2016-06-27 00:00:00结束时间为：2016-06-27 16:00:00 这样build 任务时，第一步从hive 抽取数据时where 条件是 >=2016-06-27 and <2016-06-27 导致没有抽取到数据，最后保存的hbase里也只是一个空表而已。 为了预计算27号的数据，也build另一个job, 开始时间设置为：2016-06-27 16:00:00结束时间为：2016-06-28 00:00:00 这样build 任务时，第一步从hive 抽

关于cube build和merge的问题

大家好： 目前在kylin的使用过程中遇到以下一个问题： Cube构建时，开始时间设置为：2016-06-27 00:00:00结束时间为：2016-06-27 16:00:00 这样build 任务时，第一步从hive 抽取数据时where 条件是 >=2016-06-27 and <2016-06-27 导致没有抽取到数据，最后保存的hbase里也只是一个空表而已。 为了预计算27号的数据，也build另一个job, 开始时间设置为：2016-06-27 16:00:00结束时间为：2016-06-28 00:00:00 这样build 任务时，第一步从hive 抽

答复: 答复: kylin查询，报超时异常:Timeout visiting cube!

know; from Day to Month, it need aggregate 30 times data in memory to result set; For Quarter it need more; So when the measure is "memory-hungry" measure (like distinct count, raw, top-n), it is likely to get the out of memory error; you can try to define "month" and "

kylin用户角色权限问题

大家好： 关于kylin用户角色权限这块有几点疑问： 一、添加新用户： 官方doc给出解决思路：Kylin是采用Spring security framework做用户认证的，需要配置${KYLIN_HOME}/tomcat/webapps/kylin/WEB-INF/classes/kylinSecurity.xml 的sandbox,testing部分 但是修改完毕后，每次都是需要重启kylin的,如果线上环境是集群部署,这样需要重启所有节点，这样处理不妥当吧？

kylin-1.5.3版本页面登录报错

大家好： 升级到1.5.3版本后，启动kylin，登录时kylin.log文件没生成，在kylin.out文件里报错： INFO: Initializing log4j from [classpath:kylin-server-log4j.properties] Aug 02, 2016 2:26:53 PM org.apache.catalina.core.StandardContext listenerStart SEVERE: Exception sending context initialized event to listener instance of class org.

cube build error in "4 Step Name: Build Dimension Dictionary"

大家好： 在cube 构建的第四步：Build Dimension Dictionary 报错： java.lang.IllegalArgumentException: Value not exists! at org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:162) at org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:167)

答复: cube build error in "4 Step Name: Build Dimension Dictionary"

非常感谢~ 发件人: 赵天烁 [mailto:zhaotians...@meizu.com] 发送时间: 2016年9月12日 19:30 收件人: 仇同心; user; dev 主题: 回复: cube build error in "4 Step Name: Build Dimension Dictionary" maybe is this issue ,already been solved at 1.5.4 https://issues.apache.org/jira/browse/KYLIN-1834 __

答复: which cube to use when u build many cubes under a model

Kylin completed a customized Adapter, completed in Calcite SQL parsing, forming a syntax tree (AST), defined by Kylin syntax tree each node query execution rules.Calcite after traversal syntax tree node generated a Kylin describe the Digest of the query model, Kylin will Digest to judge whe

答复: Re: kylin 1.5.4 measure edit page error？

I also met this problem apache-kylin-1.5.4-bin.tar.gz 发件人: 赵天烁 [mailto:zhaotians...@meizu.com] 发送时间: 2016年9月18日 11:28 收件人: Jian Zhong; user 主题: Re: Re: kylin 1.5.4 measure edit page error？ already did that,also try switch to different browser ,create a new kylin project, same thing again _

答复: Re: kylin 1.5.4 measure edit page error？

•apache-kylin-1.5.4-HBase1.x-bin.tar.gz 用的这个版本 发件人: 仇同心 发送时间: 2016年9月18日 14:17 收件人: user; Jian Zhong 主题: 答复: Re: kylin 1.5.4 measure edit page error？ I also met this problem apache-kylin-1.5.4-bin.tar.gz 发件人: 赵天烁 [mailto:zhaotians...@meizu.com] 发送时间: 2016年9月18日 11:28 收件人: Jian Zhong; user 主题

kylin-1.5.4同步hive元数据报错

大家好： 今天在使用kylin1.5.4版本时，在同步hive元数据时报错： Load Hive Table Metadata From Tree页面一直显示：Loading Databases. 错误信息打印在kylin.out文件 SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin] threw exception [Handler processing failed; nested exception is java.lang.NoClassDefFoundError: org/

cube详细页的SQL标签页报错

大家好： 线上环境kylin 1.5.4.1版本，在cube详细页的SQL标签页报错，不能正常显示cube的sql语句，后台日志错误信息为： ERROR [http-bio-8070-exec-2] controller.BasicController:44 : java.lang.NullPointerException at org.apache.kylin.engine.EngineFactory.batchEngine(EngineFactory.java:44) at org.apache.kylin.engine.EngineFactory.

kylin cube migrating

大家好： 目前在两台服务器上分别搭建了一个kylin servr,想做cube迁移。 能否用案例说明下kylin不同环境下cube迁移工具的参数说明 org.apache.kylin.storage.hbase.util.CubeMigrationCLI 特别是以下两个参数： srcKylinConfigUri: The KylinConfig of the cube’s source dstKylinConfigUri: The KylinConfig of the cube’s new home 谢谢！

Error of migrating cubes

Hi, Throw an exception information when using org.apache.kylin.storage.hbase.util .CubeMigrationCLI migration cube, kylin version is kylin-1.5.4.1-HBase1.x. 2016-10-17 14:57:25,595 INFO [main CubeMigrationCLI:325]: Executing operation: ADD_INTO_PROJECT:CUBE[name=testc], testc, test, 2016-

答复: Most Used BI Tools

Can you detailed introduction of the BI tools？ 发件人: hongbin ma [mailto:mahong...@apache.org] 发送时间: 2016年11月7日 13:39 收件人: user.kylin 主题: Re: Most Used BI Tools nice stuff! On Mon, Nov 7, 2016 at 1:09 AM, Alberto Ramón mailto:a.ramonporto...@gmail.com>> wrote: fyi: https://image-store.slideshare

Cube Merge Error

Hi, The first step in the cube to merge, #1 Step Name: Merge Cuboid Dictionary Error Log info: 2016-11-10 14:08:00,798 DEBUG [pool-7-thread-1] dict.DictionaryGenerator:91 : Dictionary value samples: 10101001120172=> 479, 10101003212212=>480, 10101003812579=>481, 10101005033448=>482, 1010100504

Cube Merge Error

Hi, The first step in the cube to merge, #1 Step Name: Merge Cuboid Dictionary Error Log info: 2016-11-10 14:08:00,798 DEBUG [pool-7-thread-1] dict.DictionaryGenerator:91 : Dictionary value samples: 10101001120172=> 479, 10101003212212=>480, 10101003812579=>481, 10101005033448=>482, 101010050

Cube 构建优化咨询

大家好： 目前在构建cube时遇到问题：cube维度的基数不是很高，但是度量里的字段基数很高，Build Dimension Dictionary就非常的占用本机内存，选取的度量的基数有千万、亿，甚至是十亿左右的，度量大多都是SUM，Count_distinct的精确计算。数据量是10个月的数据，我们是打算一次跑完10个月历史数据，然后在按日增跑作业。 服务器的内存配置为125G，#4 Step Name: Build Dimension Dictionary 会一直在跑很长时间，最后到导致内存溢出。 对于这种度量基数高的问题，有什么好的优化方案吗？ 谢谢

答复: Cube Merge Error

build dictionary to do a string -> integer convertion. To support ultra high cardinality, the "global dictionary" should be selected as it has less memory footprint, which can support up to 2 billion cardinality: https://kylin.apache.org/blog/2016/08/01/count-distinct-in-kylin/ 2016-11-10 1

Auto merge cube error

Hi,all: I met an error when cube was auto merge,I also set -Xmx100g -Xms100g,but this error still appeared. java.lang.OutOfMemoryError: Requested array size exceeds VM limit at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.

答复: Cube 构建优化咨询

@kylin.apache.org 抄送: dev 主题: Re: Cube 构建优化咨询 don't try to run such huge job one time, please run them one by one, for example, run 1 month data and then next one... Best Regards! - Luke Han 2016-11-10 14:54 GMT+08:00 仇同心 : > 大家好： > > 目前在构建cube时遇到问题：cube

About hybrid model of Kylin

Hi ,all When I designed a cube, and have built a period of time, if Iwant to modify the cube dimensions or measurements, It is suggested to use hybrid mode: Create a new cube, and then build back from now;Kylin has a hybrid model, can put the new cube cube and history to make a composite,wh

答复: About hybrid model of Kylin

Thanks! 发件人: Dong Li [mailto:lid...@apache.org] 发送时间: 2016年11月14日 18:13 收件人: d...@kylin.apache.org; user@kylin.apache.org 主题: Re: About hybrid model of Kylin Here’s the document about hybrid: http://kylin.apache.org/blog/2015/09/25/hybrid-model/ Thanks, Dong Li Original Message Sender: 仇同心

Consulting "Define data model"

Hi ,all In this article at http://kylin.apache.org/docs16/tutorial/cube_streaming.html, I have learned that Streaming Cube must be partitioned; If you’re going to build the Cube incrementally at minutes level, select “MINUTE_START” as the cube’s partition date column. If at hours level, sel

答复: Consulting "Define data model"

e, the "MINUTE_START", "HOUR_START" (and other derived date columns) are transformed by Kylin to standard "timestamp" type; Then the date format on "Data model" page is useless in this case. 2016-11-21 11:07 GMT+08:00 仇同心 mailto:qiutong...@jd.com>&g

Cube optimization for help

Hi,all There is a cube optimization for help. Cuhe has 15 dimensions, including 14 normal dimensions and 1 derived dimension, and the cardinality of all dimensions is not high;And this cube also has 10 measures, including 1 count expression,2 sum expressions and 7 COUNT_DISTINCT expressions, and

Consulting "EXTENDED_COLUMN"

Hi ,all I don’t understand the usage scenarios of EXTENDED_COLUMN,although I saw this article “https://issues.apache.org/jira/browse/KYLIN-1313”. What,s the means about parameters of “Host Column” and “Extended Column”? Why use this expression,and what aspects of optimization that this expressio

create dictionary error

Hi,all Building operation error on the of Step Name: Build Dimension Dictionary: java.lang.RuntimeException: Failed to create dictionary on DMT.DMT_KYLIN_JDMALL_ORDR_DTL_I_D.SALE_ORD_ID at org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:325)

关于kylin查询问题咨询

大家好： 目前在使用中遇到一个kylin查询慢的场景: Cube 设计如下： { "uuid": "dfb77a08-f51d-4088-b559-1da67c28a068", "last_modified": 1486997875953, "version": "1.6.0", "name": "insu_t", "model_name": "insu_jdmall_model_test", "description": "", "null_string": null, "dimensions": [ { "name": "BRAND"

java.lang.RuntimeException: Too big dictionary, dictionary cannot be bigger than 2GB

Hi ,all The first step in cube merge, an error : java.lang.RuntimeException: Too big dictionary, dictionary cannot be bigger than 2GB at org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:421) at org.apache.kylin.dict.TrieDictionaryBuilder.b

build cube with spark ERROR

Hi all, When built cube with spark ,I met some errors, Seems to be linked to the hive seems to be linked to the hive,can you help me? javax.jdo.JDOFatalInternalException: Error creating transactional connection factory at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForN

答复: build cube with spark ERROR

tions/33192886/com-mysql-jdbc-driver-not-found-on-classpath-while-starting-spark-sql-and-thrift You can add additional spark jar in kylin.properties, e.g.,: kylin.engine.spark.additional-jars=/path/to/mysql-connector-java-5.1.38-bin.jar 2017-03-10 11:37 GMT+08:00 仇同心 mailto:qiutong...@jd.com>&

build cube with spark

Hi,all Kylin can use external spark cluster？ Thanks~

Build Dimension Dictionary

大家好： 在构建作业的#4 Step Name: Build Dimension Dictionary 这步时,kylin.log里一直打印下面信息： 2017-04-07 17:18:37,699 INFO [pool-7-thread-4] mapreduce.InternalUtil:155 : Initializing org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe with properties {name=DMT.DMT_KYLIN_USR_GROUP_DETAIL_I_D, field.delim=,

kylin 2.0 ODBC Driver

Hi,all After upgraded to kylin 2.0, if the ODBC access, the version of ODBC Driver is still use Kylin ODBC Driver v1.6 ? Thanks~

答复: kylin 2.0 ODBC Driver

Thank you~ 发件人: Billy Liu [mailto:billy...@apache.org] 发送时间: 2017年5月23日 16:12 收件人: user 主题: Re: kylin 2.0 ODBC Driver Yes. There is little changes in ODBC since last release. 2017-05-23 14:43 GMT+08:00 仇同心 mailto:qiutong...@jd.com>>: Hi,all After upgraded to kylin 2.0, if the ODBC acces

kylin明细查询分页

HI,all 在cube设计时，在度量项中，设置了10个字段为RAW，在cube构建完毕后，在查询页面怎么实现分页查询呢？ 谢谢！

How kylin2.0 open distributed build data dictionary

Hi,all Now,I’m using the version 0f kylin 2.0,so how kylin2.0 open distributed build dictionary? Is itthat to configure a parameter in “kylin.properties”? Thanks!

Detailed query of "RAW"

Hi,all There is a business scenario is the detailed query. There are 45 metric for design to “RAW”, one dimension and high base,but in “Rowkey” of “Advanced Setting”,the Encoding is fixed._length,not dict. But when build the cube, there are some errors in “#4 Step Name: Build Dimension Dictio

consultation for kylin2.0 parameters

Hi ,all: I have a question about “kylin.dictionary.growing-enabled”, Can you tell me the function of this parameter? Thanks!