[jira] [Created] (KYLIN-2935) Improve the way to deploy coprocessor
Zhong Yanghong created KYLIN-2935: - Summary: Improve the way to deploy coprocessor Key: KYLIN-2935 URL: https://issues.apache.org/jira/browse/KYLIN-2935 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Zhong Yanghong There are two improvements for deploying coprocessor: # Make thread number configurable # Filter tables by git commit -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2926) DumpMerger return incorrect results
[ https://issues.apache.org/jira/browse/KYLIN-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-2926: -- Attachment: 0001-KYLIN-2926-DumpMerger-return-incorrect-results-creat.patch create codec object for each dump to avoid shared variable. > DumpMerger return incorrect results > --- > > Key: KYLIN-2926 > URL: https://issues.apache.org/jira/browse/KYLIN-2926 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu > Attachments: > 0001-KYLIN-2926-DumpMerger-return-incorrect-results-creat.patch > > > I our scenario, a cube query will get wrong result once coprocessor need to > spill to disk, Our version is 2.0.0 and I find the root cause is that in > DumpMerger.enqueueFromDump > because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It > leading to different elements in dumpCurrentValues share the same object, so > next fill up measure values will change the existing values. > the incorrect measures is HLLC and raw, which use current variable in > deserialize. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2928) PUSH DOWN query cannot use order by function
[ https://issues.apache.org/jira/browse/KYLIN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202926#comment-16202926 ] Yang Hao commented on KYLIN-2928: - [~yimingliu] SparkSQL > PUSH DOWN query cannot use order by function > > > Key: KYLIN-2928 > URL: https://issues.apache.org/jira/browse/KYLIN-2928 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Yang Hao >Assignee: liyang > > SQL : select "DATE",count(1) from table_1 group by "DATE" order by "DATE" > desc; > Exception:org.apache.calcite.sql.SqlOrderBy cannot be cast to > org.apache.calcite.sql.SqlSelect -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202907#comment-16202907 ] Shaofeng SHI commented on KYLIN-2656: - The document is in Kylin's code repository, in the "document" branch. If you fork and check out it, you will see the document structure there. For how to write a document, you can refer to this page[1]. [1] https://kylin.apache.org/development/howto_docs.html > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future, v2.3.0 > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2931: Description: 1,Add project_1 and project_2; 2,Do the following steps: a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy table kylin_sales; b,load table kylin_sales_copy and calculate column cardinality in project_1 and project_2,please refer to 01.png,now each project has the table kylin_sales_copy; c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; d,use 'create table kylin_sales_copy as select TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE from kylin_sales where price < 20;' to create the new kylin_sales_copy table; Now the structure of kylin_sales_copy has changed, the same as the cardinality. 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the cardinality is not change,please refer to 02.png. However,I found that the new cardinality was calculated but not update on the web by debug the code,please refer to 03.png,04.png.It should update correct cardinality like 05.png. was: 1,Add project_1 and project_2,each project has the table kylin_sales_copy; 2,Do the following steps: a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy table kylin_sales; b,load table kylin_sales_copy and calculate column cardinality in project_1 and project_2,please refer to 01.png; c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; d,use 'create table kylin_sales_copy as select TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE from kylin_sales where price < 20;' to create the new kylin_sales_copy table; Now the structure of kylin_sales_copy has changed, the same as the cardinality. 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the cardinality is not change,please refer to 02.png. However,I found that the new cardinality was calculated but not update on the web by debug the code,please refer to 03.png,04.png.It should update correct cardinality like 05.png. > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG, 08.PNG, Screen Shot > 2017-10-12 at 4.32.24 PM.png > > > 1,Add project_1 and project_2; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png,now each project has the table > kylin_sales_copy; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202890#comment-16202890 ] peng.jianhua commented on KYLIN-2656: - Hi [~Shaofengshi], ok. I am writing the user guide. Please refer to KYLIN-2934 sub-task. Can you tell me how to build a document patch? Thanks! > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future, v2.3.0 > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2934) Write user guide for KYLIN-2656
peng.jianhua created KYLIN-2934: --- Summary: Write user guide for KYLIN-2656 Key: KYLIN-2934 URL: https://issues.apache.org/jira/browse/KYLIN-2934 Project: Kylin Issue Type: Sub-task Components: Documentation Reporter: peng.jianhua Assignee: peng.jianhua Write user guide for KYLIN-2656 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua resolved KYLIN-2656. - Resolution: Fixed Fix Version/s: v2.3.0 > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future, v2.3.0 > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2648) kylin.env.hdfs-working-dir should be qualified and absolute path
[
https://issues.apache.org/jira/browse/KYLIN-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202842#comment-16202842
]
Shaofeng SHI commented on KYLIN-2648:
-
Yanghong, when upgrade to 2.1, you need copy the big metadata file from
computing HDFS to query HDFS. I added this in the "how to upgrade" document.
> kylin.env.hdfs-working-dir should be qualified and absolute path
>
>
> Key: KYLIN-2648
> URL: https://issues.apache.org/jira/browse/KYLIN-2648
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
> Environment: hadoop :cdh5.4.0 (both main and hbase env)
> hbase : hbase-1.2.0-cdh5.7.6
> hive: apache-hive-2.1.1
> kylin version: 2.0
>Reporter: suheng.cloud
>Assignee: liyang
> Fix For: v2.1.0
>
>
> I try to deploy kylin on one node of a stand alone hbase
> cluster(hdfs://cdh5-mini/) which seperate from main hive
> cluster(hdfs://cdh5/),
> According to the blog "Deploy Apache Kylin with Standalone HBase Cluster" :
> make sure the configurations of hadoop and hive points to main cluster,
> I clone hadoop dir to another path and modify "fs.defaultFS" in core-site.xml
> to "hdfs://cdh5/" , and in head of kylin.sh, I export HADOOP_HOME to this new
> path.
> So all goes well (include cube build/refresh) until I execute cube merge.
> The merge error occurs at step "#9 Step Name: Garbage Collection on HDFS".
> The stacktrace as follows:
> 2017-05-25 17:28:07,070 INFO [pool-9-thread-1]
> threadpool.DefaultScheduler:114 :
> CubingJob{id=c6709f0b-8858-4e66-a4c2-320ebc70a2e3, name=kylin_sales_cube -
> 2012010100_2014020100 - MERGE - GMT+08:00 2017-05-25 16:51:30,
> state=READY} prepare to schedule
> 2017-05-25 17:28:07,073 INFO [pool-9-thread-1]
> threadpool.DefaultScheduler:117 :
> CubingJob{id=c6709f0b-8858-4e66-a4c2-320ebc70a2e3, name=kylin_sales_cube -
> 2012010100_2014020100 - MERGE - GMT+08:00 2017-05-25 16:51:30,
> state=READY} scheduled
> 2017-05-25 17:28:07,075 INFO [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> execution.AbstractExecutable:110 : Executing AbstractExecutable
> (kylin_sales_cube - 2012010100_2014020100 - MERGE - GMT+08:00
> 2017-05-25 16:51:30)
> 2017-05-25 17:28:07,078 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> dao.ExecutableDao:217 : updating job output, id:
> c6709f0b-8858-4e66-a4c2-320ebc70a2e3
> 2017-05-25 17:28:07,083 INFO [pool-9-thread-1]
> threadpool.DefaultScheduler:124 : Job Fetcher: 0 should running, 1 actual
> running, 0 stopped, 1 ready, 19 already succeed, 0 error, 11 discarded, 0
> others
> 2017-05-25 17:28:07,083 INFO [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> execution.ExecutableManager:389 : job id:c6709f0b-8858-4e66-a4c2-320ebc70a2e3
> from READY to RUNNING
> 2017-05-25 17:28:07,105 INFO [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> execution.AbstractExecutable:110 : Executing AbstractExecutable (Garbage
> Collection on HDFS)
> 2017-05-25 17:28:07,106 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> dao.ExecutableDao:217 : updating job output, id:
> c6709f0b-8858-4e66-a4c2-320ebc70a2e3-08
> 2017-05-25 17:28:07,111 INFO [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> execution.ExecutableManager:389 : job
> id:c6709f0b-8858-4e66-a4c2-320ebc70a2e3-08 from READY to RUNNING
> 2017-05-25 17:28:07,154 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> steps.HDFSPathGarbageCollectionStep:78 : Drop HDFS path on FileSystem:
> hdfs://cdh5
> 2017-05-25 17:28:07,217 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> steps.HDFSPathGarbageCollectionStep:90 : HDFS path
> hdfs:///kylin/kylin_metadata/kylin-a11d510f-d8a5-45c1-b430-bc7def851432 not
> exists.
> 2017-05-25 17:28:07,249 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> steps.HDFSPathGarbageCollectionStep:90 : HDFS path
> hdfs:///kylin/kylin_metadata/kylin-0c1ed2d0-f595-4f58-aaea-2dbe7b41a550 not
> exists.
> 2017-05-25 17:28:07,320 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> steps.HDFSPathGarbageCollectionStep:78 : Drop HDFS path on FileSystem:
> hdfs://cdh5-mini
> 2017-05-25 17:28:07,324 ERROR [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> execution.AbstractExecutable:126 : error running Executable:
> HDFSPathGarbageCollectionStep{id=c6709f0b-8858-4e66-a4c2-320ebc70a2e3-08,
> name=Garbage Collection on HDFS, state=RUNNING}
> 2017-05-25 17:28:07,326 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> dao.ExecutableDao:217 : updating job output, id:
> c6709f0b-8858-4e66-a4c2-320ebc70a2e3-08
> 2017-05-25 17:28:07,331 DEBUG [Job c6709f0b-8858-4e66-a4c2-320ebc70a2e3-128]
> dao.ExecutableDao:217 : updating job output, id:
> c6709f0b-8858-4e66-a4c2-320ebc70a2e3-08
> 2017-05-25 17:28:07,334 INFO
[jira] [Updated] (KYLIN-2806) Empty results from JDBC with Date filter in prepareStatement
[ https://issues.apache.org/jira/browse/KYLIN-2806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2806: --- Fix Version/s: v2.2.0 > Empty results from JDBC with Date filter in prepareStatement > > > Key: KYLIN-2806 > URL: https://issues.apache.org/jira/browse/KYLIN-2806 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC, Query Engine >Affects Versions: v1.6.0, v2.0.0, v2.1.0 > Environment: kylin版本：apache-kylin-1.6.0-cdh5.7-bin > jdbc版本：kylin-jdbc-1.6.0.jar > java版本：java version "1.8.0_92" >Reporter: chengwansheng >Assignee: Wang Cheng > Labels: scope > Fix For: v2.2.0 > > > 用kylin的JDBC查询，查询条件是date类型的，如果通过prepareStatement、setDate是查询不出数据的，然而直接用SQL可以查出数据 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2842) 在升级到KYLIN 2.1之后，我使用JDBC，preparedstatement, 无法查询出任何数据。
[ https://issues.apache.org/jira/browse/KYLIN-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li closed KYLIN-2842. -- > 在升级到KYLIN 2.1之后，我使用JDBC，preparedstatement, 无法查询出任何数据。 > - > > Key: KYLIN-2842 > URL: https://issues.apache.org/jira/browse/KYLIN-2842 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.1.0 >Reporter: Jack >Assignee: liyang > > 你好，我在升级到2.1之后，使用JDBC连接kylin进行查询的时候，使用preparedstatement设定参数查询，无法查询出任何数据，但是在使用普通的statement查询，是可以查询出数据的，请尽快帮忙解决。谢谢。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2815) Empty results with prepareStatement but OK with KylinStatement
[
https://issues.apache.org/jira/browse/KYLIN-2815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li resolved KYLIN-2815.
Resolution: Duplicate
Fix Version/s: v2.2.0
> Empty results with prepareStatement but OK with KylinStatement
> --
>
> Key: KYLIN-2815
> URL: https://issues.apache.org/jira/browse/KYLIN-2815
> Project: Kylin
> Issue Type: Bug
> Components: Driver - JDBC
>Affects Versions: v2.1.0
>Reporter: youcheng.zhang
>Assignee: Wang Cheng
> Labels: scope
> Fix For: v2.2.0
>
>
> Actually, my prepareStatement queries works well with kylin version v2.0.
> After I upgrade kylin to version v2.1, all queries have no results return,
> Then I write test：
> {code:java}
> Driver driver = (Driver)
> Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
> Properties info = new Properties();
> info.put("user", "ADMIN");
> info.put("password", "KYLIN");
> Connection conn = driver.connect("jdbc:kylin://IP:7070/project_name", info);
> PreparedStatement pst = conn.prepareStatement("select XX where XX = ? ")
> pst.setInt(1, 1);
> ResultSet resultSet = pst.executeQuery();
> {code}
> Return no results, and after that I change the query to statement
> {code:java}
> Statement state = conn.createStatement();
> ResultSet resultSet = state.executeQuery("select XX where XX = 1")
> {code}
> Finally, I got return values.
> I have no idea whether it is a bug or I have done something wrong with
> upgrade?
> As I have removed HBase tables “kylin_metadata_user” and
> “kylin_metadata_acl", I think
> I can't come back to version v2.0 any more:(
> I also try changed dependency kylin-jdbc to version 2.1.0 but help nothing.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (KYLIN-2823) Trim TupleFilter after dictionary-based filter optimization
[ https://issues.apache.org/jira/browse/KYLIN-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li resolved KYLIN-2823. Resolution: Fixed Assignee: hongbin ma > Trim TupleFilter after dictionary-based filter optimization > --- > > Key: KYLIN-2823 > URL: https://issues.apache.org/jira/browse/KYLIN-2823 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > Fix For: v2.2.0 > > > with cube's dictionary, kylin will optimize filters like: > ( a = 'value_in_dict' OR a = 'value_not_in_dict') => (a = > 'value_in_dict' OR ConstantTupleFilter.FALSE) > we need to further trim the filter to (a = 'value_in_dict') to avoid too many > children after flatten filter step -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2860) column value with "" is treat as column on query
[
https://issues.apache.org/jira/browse/KYLIN-2860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li closed KYLIN-2860.
--
> column value with "" is treat as column on query
>
>
> Key: KYLIN-2860
> URL: https://issues.apache.org/jira/browse/KYLIN-2860
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Yang Hao
>Assignee: liyang
>
> There is an error when I query a column with "" for it is treated as a
> column, such as this sql, the china will be treated as a column, and it fails
> {code:sql}
> select * from table where column_1 = "china"
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201966#comment-16201966 ] Shaofeng SHI commented on KYLIN-2656: - BTW, could you compose a tip on how to use this feature in Kylin? That would help users to understand the benefit. > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201963#comment-16201963 ] Shaofeng SHI commented on KYLIN-2656: - Hi Jianhua, patch looks good, and UT passed; Merged in master with commit id 8891b1c84ad1fe84199c9c796c0ce57408607435 Thanks for making Kylin better! > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2736) Use multiple threads to calculate HyperLogLogPlusCounter in FactDistinctColumnsMapper
[ https://issues.apache.org/jira/browse/KYLIN-2736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2736: -- Affects Version/s: (was: v2.2.0) > Use multiple threads to calculate HyperLogLogPlusCounter in > FactDistinctColumnsMapper > - > > Key: KYLIN-2736 > URL: https://issues.apache.org/jira/browse/KYLIN-2736 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2735) Introduce an option to make job scheduler consider job priority
[
https://issues.apache.org/jira/browse/KYLIN-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2735:
--
Affects Version/s: (was: v2.1.0)
> Introduce an option to make job scheduler consider job priority
> ---
>
> Key: KYLIN-2735
> URL: https://issues.apache.org/jira/browse/KYLIN-2735
> Project: Kylin
> Issue Type: Sub-task
> Components: Job Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>
> To optimize a cube with *N* ready segments, *N* optimize jobs + *1*
> checkpoint job are needed, which may occupy too much resources and make other
> jobs waiting too much time to be scheduled. Thus, job priority and its
> dynamic changing will be introduced. The idea is as follows:
> * Job Fetcher fetches job metadata from database every *M* seconds.
> * A Running Job List is maintained globally
> * {color:#f79232}A Job Priority Queue is maintained globally{color}
> * Before fetching job metadata,
> ** {color:#f79232}first check the job priority queue and schedule jobs
> meeting condition{color} {color:#f691b2}*C*{color}
> ** then if the size of Running Job List exceeds the max concurrent job limit
> *L*, this fetch round will be skipped
> * During fetch round,
> ** {color:#f79232} the priority of each job in the queue will increase.{color}
> ** {color:#f79232} each new job fetched will be assigned with a default
> priority.{color}
> ** {color:#f79232} the highest *L*{color} jobs will be added to Running Job
> List and be submitted to Job Execute Thread Pool
> * Once a job finishes, it will be removed from Running Job List and notify
> the Job Fetcher to fetch job metadata
> The condition {color:#f691b2}*C*{color} is as follows:
> * job priority should be above a threshold, to filter low priority jobs
> * job should wait at least once, to give chance for low priority jobs to be
> scheduled
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2866) Enlarge the reducer number for hyperloglog statistics calculation at step FactDistinctColumnsJob
[ https://issues.apache.org/jira/browse/KYLIN-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2866: -- Affects Version/s: (was: v2.2.0) > Enlarge the reducer number for hyperloglog statistics calculation at step > FactDistinctColumnsJob > > > Key: KYLIN-2866 > URL: https://issues.apache.org/jira/browse/KYLIN-2866 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > Currently only one reducer is assigned for hll stats calculation, which may > become the bottleneck for slow down this step. Since the stats for different > cuboids will not influence each other, it's better to divide the cuboid set > into several and assign a reduce for each subset. > The strategy of this patch is to assign 100 cuboids into a subset. And > there's a upper limit of reducers for hll stats calculation. Currently it's > 50. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2861) For dictionary building of lookup table columns, reduce the table scan chance
[ https://issues.apache.org/jira/browse/KYLIN-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2861: -- Fix Version/s: (was: v2.2.0) > For dictionary building of lookup table columns, reduce the table scan chance > - > > Key: KYLIN-2861 > URL: https://issues.apache.org/jira/browse/KYLIN-2861 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > In some cases, several columns from the same lookup table are defined as > normal dimensions. We'll build dictionaries for these columns. With current > implementation, for each column, the whole lookup table needs to be scanned > once. Then if there are N columns, kylin needs to scan the lookup table N > times. > For this JIRA, we'll scan only once to build all of the related columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2932) Simplify the thread model for in-memory cubing
[ https://issues.apache.org/jira/browse/KYLIN-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2932: -- Affects Version/s: (was: v2.1.0) > Simplify the thread model for in-memory cubing > -- > > Key: KYLIN-2932 > URL: https://issues.apache.org/jira/browse/KYLIN-2932 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang Ken >Assignee: Dong Li > > The current implementation uses split threads, task threads and main thread > to do the cube building, there is complex join and error handling logic. > The new implement leverages the ForkJoinPool from JDK, the event split logic > is handled in > main thread. Cuboid task and sub-tasks are handled in fork join pool, cube > results are collected > async and can be write to output earlier. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2737) Change the dictionary building process to MR
[ https://issues.apache.org/jira/browse/KYLIN-2737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2737: -- Affects Version/s: (was: v2.2.0) > Change the dictionary building process to MR > > > Key: KYLIN-2737 > URL: https://issues.apache.org/jira/browse/KYLIN-2737 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2867) split large fuzzy Key set
[ https://issues.apache.org/jira/browse/KYLIN-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2867: -- Affects Version/s: (was: v2.1.0) > split large fuzzy Key set > - > > Key: KYLIN-2867 > URL: https://issues.apache.org/jira/browse/KYLIN-2867 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > Too many fuzzy keys in one scan may downgrade the performance. Currently if > the number of fuzzy keys exceeds a threshold, like 200, the scan will not use > fuzzy key. However, this may also downgrade the performance. > There are many cases in eBay that the total number of fuzzy keys is around > 1000. To deal with these cases, it's better to split the large fuzzy key set, > like 5*200. Then for each subset, assign a scan. This strategy will increase > the number of scan to reduce the number of fuzzy keys in each scan to improve > the total performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2881) Improve hbase coprocessor exception handling at kylin server side
[
https://issues.apache.org/jira/browse/KYLIN-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2881:
--
Affects Version/s: (was: v2.1.0)
> Improve hbase coprocessor exception handling at kylin server side
> --
>
> Key: KYLIN-2881
> URL: https://issues.apache.org/jira/browse/KYLIN-2881
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>
> A query may need a bunch of hconnections to visit hbase. Anyone of them
> fails, the whole query should be interrupted, and all of the related
> hconnections should be released for future other queries despite that
> currently we cannot interrupt the threads at hbase coprocessor side.
> To achieve this goal, one property is added to QueryContext to keep
> {{Throwable}}. If any sub-step during the query detects this {{Throwable}},
> it should stop. In {{ExpectedSizeIterator}}, there are two parts where this
> detection is needed.
> * before putting data to the queue
> * during iteration
> If exception happens during a query, it should stop its related threads.
> Thus, {{QueryStopListener}} is added for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2881) Improve hbase coprocessor exception handling at kylin server side
[
https://issues.apache.org/jira/browse/KYLIN-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2881:
--
Fix Version/s: (was: v2.2.0)
> Improve hbase coprocessor exception handling at kylin server side
> --
>
> Key: KYLIN-2881
> URL: https://issues.apache.org/jira/browse/KYLIN-2881
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>
> A query may need a bunch of hconnections to visit hbase. Anyone of them
> fails, the whole query should be interrupted, and all of the related
> hconnections should be released for future other queries despite that
> currently we cannot interrupt the threads at hbase coprocessor side.
> To achieve this goal, one property is added to QueryContext to keep
> {{Throwable}}. If any sub-step during the query detects this {{Throwable}},
> it should stop. In {{ExpectedSizeIterator}}, there are two parts where this
> detection is needed.
> * before putting data to the queue
> * during iteration
> If exception happens during a query, it should stop its related threads.
> Thus, {{QueryStopListener}} is added for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2880) Enhance BadQueryDetector to include query id
[ https://issues.apache.org/jira/browse/KYLIN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2880: -- Fix Version/s: (was: v2.2.0) > Enhance BadQueryDetector to include query id > > > Key: KYLIN-2880 > URL: https://issues.apache.org/jira/browse/KYLIN-2880 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > To better correlate bad queries with the bottom physical execution > information, it's better to let BadQueryDetector include query id -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2746) Separate filter row count & aggregated row count for metrics collection returned by coprocessor
[ https://issues.apache.org/jira/browse/KYLIN-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2746: -- Affects Version/s: (was: v2.2.0) > Separate filter row count & aggregated row count for metrics collection > returned by coprocessor > --- > > Key: KYLIN-2746 > URL: https://issues.apache.org/jira/browse/KYLIN-2746 > Project: Kylin > Issue Type: Sub-task > Components: Storage - HBase >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > Cube Planner used for recommending effective cuboids is based on cost-based > algorithm. The aggregated row count plays an important role of the rollup > cost from target cuboid to source cuboid. Thus we need to separate filter row > count & aggregated row count. > *One important thing is that coprocessor redeployment needs to be done after > this change* -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2880) Enhance BadQueryDetector to include query id
[ https://issues.apache.org/jira/browse/KYLIN-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2880: -- Affects Version/s: (was: v2.1.0) > Enhance BadQueryDetector to include query id > > > Key: KYLIN-2880 > URL: https://issues.apache.org/jira/browse/KYLIN-2880 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > To better correlate bad queries with the bottom physical execution > information, it's better to let BadQueryDetector include query id -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2902) Introduce project-level query number control
[ https://issues.apache.org/jira/browse/KYLIN-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2902: -- Affects Version/s: (was: v2.1.0) > Introduce project-level query number control > > > Key: KYLIN-2902 > URL: https://issues.apache.org/jira/browse/KYLIN-2902 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > One kylin server may contain many projects. To avoid some projects occupying > too much resources, like http connection, project-level query number limit is > useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2897) Improve the query execution for a set of duplicate queries in a short period
[ https://issues.apache.org/jira/browse/KYLIN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2897: -- Fix Version/s: (was: v2.2.0) > Improve the query execution for a set of duplicate queries in a short period > > > Key: KYLIN-2897 > URL: https://issues.apache.org/jira/browse/KYLIN-2897 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2902) Introduce project-level query number control
[ https://issues.apache.org/jira/browse/KYLIN-2902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2902: -- Fix Version/s: (was: v2.2.0) > Introduce project-level query number control > > > Key: KYLIN-2902 > URL: https://issues.apache.org/jira/browse/KYLIN-2902 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > One kylin server may contain many projects. To avoid some projects occupying > too much resources, like http connection, project-level query number limit is > useful. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2897) Improve the query execution for a set of duplicate queries in a short period
[ https://issues.apache.org/jira/browse/KYLIN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2897: -- Affects Version/s: (was: v2.1.0) > Improve the query execution for a set of duplicate queries in a short period > > > Key: KYLIN-2897 > URL: https://issues.apache.org/jira/browse/KYLIN-2897 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2896) Refine query exception cache
[ https://issues.apache.org/jira/browse/KYLIN-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2896: -- Fix Version/s: (was: v2.2.0) > Refine query exception cache > > > Key: KYLIN-2896 > URL: https://issues.apache.org/jira/browse/KYLIN-2896 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > It's not necessary to divide query cache into success and exception. Only one > query cache is enough. And some special exception needs to be cached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2894:
--
Fix Version/s: (was: v2.2.0)
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2896) Refine query exception cache
[ https://issues.apache.org/jira/browse/KYLIN-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2896: -- Affects Version/s: (was: v2.1.0) > Refine query exception cache > > > Key: KYLIN-2896 > URL: https://issues.apache.org/jira/browse/KYLIN-2896 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > It's not necessary to divide query cache into success and exception. Only one > query cache is enough. And some special exception needs to be cached. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2894:
--
Affects Version/s: (was: v2.1.0)
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2908) Add one option for migration tool to indicate whether to migrate segment data
[ https://issues.apache.org/jira/browse/KYLIN-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2908: -- Fix Version/s: (was: v2.2.0) > Add one option for migration tool to indicate whether to migrate segment data > - > > Key: KYLIN-2908 > URL: https://issues.apache.org/jira/browse/KYLIN-2908 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > Current migration tool will migrate segment data to the destination. However, > in some cases, we only want to migrate cube schema without segment data. For > example, when we do some compatibility testing for upgrade. This can also > avoid two env refer the same htable, which may leading to some issue when dst > env do some fresh to delete the htable which src env still refers. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2909) Refine Email Template for notification by freemarker
[ https://issues.apache.org/jira/browse/KYLIN-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2909: -- Fix Version/s: (was: v2.2.0) > Refine Email Template for notification by freemarker > > > Key: KYLIN-2909 > URL: https://issues.apache.org/jira/browse/KYLIN-2909 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2909) Refine Email Template for notification by freemarker
[ https://issues.apache.org/jira/browse/KYLIN-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2909: -- Affects Version/s: (was: v2.1.0) > Refine Email Template for notification by freemarker > > > Key: KYLIN-2909 > URL: https://issues.apache.org/jira/browse/KYLIN-2909 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2738) Add cache for acl
[ https://issues.apache.org/jira/browse/KYLIN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2738: -- Affects Version/s: (was: v2.1.0) > Add cache for acl > - > > Key: KYLIN-2738 > URL: https://issues.apache.org/jira/browse/KYLIN-2738 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > Currently cache is mainly used for query to improve query performance. A > stable cache can also be used to improve the availability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2904) Add compatibility check for model to allow editing model even it's referred by some cubes
[ https://issues.apache.org/jira/browse/KYLIN-2904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2904: -- Affects Version/s: (was: v2.1.0) > Add compatibility check for model to allow editing model even it's referred > by some cubes > - > > Key: KYLIN-2904 > URL: https://issues.apache.org/jira/browse/KYLIN-2904 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > It's a common requirement that user wants to upgrade an existing cube by > adding a column which is not in its referred model. To deal with this > requirement, it's better to add model compatibility check to allow compatible > model changes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2905) Refine the process of submitting a job
[ https://issues.apache.org/jira/browse/KYLIN-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2905: -- Affects Version/s: (was: v2.1.0) > Refine the process of submitting a job > -- > > Key: KYLIN-2905 > URL: https://issues.apache.org/jira/browse/KYLIN-2905 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > In cases hbase is very busy, it cost some time to put all of the job related > metadata into hbase. The job related metadata includes *job info* and *job > output info*. Currently, kylin put *job info* into hbase first, then *job > output info*. If server is down and only part of *job output info* has been > put into hbase during this period, kylin will fail to load job when restarts. > To solve this issue, it's better to adjust the order by adding *job output > info* first, then *job info*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2905) Refine the process of submitting a job
[ https://issues.apache.org/jira/browse/KYLIN-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2905: -- Fix Version/s: (was: v2.2.0) > Refine the process of submitting a job > -- > > Key: KYLIN-2905 > URL: https://issues.apache.org/jira/browse/KYLIN-2905 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > In cases hbase is very busy, it cost some time to put all of the job related > metadata into hbase. The job related metadata includes *job info* and *job > output info*. Currently, kylin put *job info* into hbase first, then *job > output info*. If server is down and only part of *job output info* has been > put into hbase during this period, kylin will fail to load job when restarts. > To solve this issue, it's better to adjust the order by adding *job output > info* first, then *job info*. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2626) Remove config getJobControllerLock() since there is getSchedulerType()
[
https://issues.apache.org/jira/browse/KYLIN-2626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Li updated KYLIN-2626:
---
Fix Version/s: (was: v2.2.0)
Future
> Remove config getJobControllerLock() since there is getSchedulerType()
> --
>
> Key: KYLIN-2626
> URL: https://issues.apache.org/jira/browse/KYLIN-2626
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.1.0
>Reporter: kangkaisen
>Assignee: kangkaisen
>Priority: Minor
> Labels: newbie, scope
> Fix For: Future
>
> Attachments: KYLIN-2626.patch
>
>
> KYLIN-2578 introduced this issue:
> {code:java}
> Caused by: java.lang.RuntimeException: java.lang.InstantiationException:
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock
> at org.apache.kylin.common.util.ClassUtil.newInstance(ClassUtil.java:95)
> at
> org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:110)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1573)
> at
> org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
> ... 38 more
> Caused by: java.lang.InstantiationException:
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock
> at java.lang.Class.newInstance(Class.java:427)
> at org.apache.kylin.common.util.ClassUtil.newInstance(ClassUtil.java:93)
> ... 41 more
> Caused by: java.lang.NoSuchMethodException:
> org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.()
> at java.lang.Class.getConstructor0(Class.java:3082)
> at java.lang.Class.newInstance(Class.java:412)
> ... 42 more
> {code}
> which make the Kylin job server cannot start.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2899) Enable segment level query cache
[ https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2899: --- Fix Version/s: (was: v2.2.0) v2.3.0 > Enable segment level query cache > > > Key: KYLIN-2899 > URL: https://issues.apache.org/jira/browse/KYLIN-2899 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Ma Gang > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2895) Refine query cache
[ https://issues.apache.org/jira/browse/KYLIN-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201875#comment-16201875 ] Dong Li commented on KYLIN-2895: Moved to next version due to subtask unfinished. > Refine query cache > -- > > Key: KYLIN-2895 > URL: https://issues.apache.org/jira/browse/KYLIN-2895 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.2.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201879#comment-16201879
]
Zhong Yanghong commented on KYLIN-2600:
---
Thanks [~lidong_sjtu]
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2913) Enable job retry for configurable exceptions
[ https://issues.apache.org/jira/browse/KYLIN-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2913: --- Fix Version/s: (was: v2.2.0) v2.3.0 > Enable job retry for configurable exceptions > > > Key: KYLIN-2913 > URL: https://issues.apache.org/jira/browse/KYLIN-2913 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.1.0 >Reporter: Wang, Gang >Assignee: Dong Li > Fix For: v2.3.0 > > > In our production environment, we always get some certain exceptions from > Hadoop or HBase, like > "org.apache.kylin.job.exception.NoEnoughReplicationException", > "java.util.ConcurrentModificationException", which results in job failure. > While, these exceptions can be handled by retry actually. So, it will be much > more convenient if we are able to make job retry on some configurable > exceptions. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201867#comment-16201867
]
Dong Li commented on KYLIN-2600:
Yes, now it's double committed.
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201710#comment-16201710 ] peng.jianhua commented on KYLIN-2931: - [~wormholer],please refer to 08.png,it was not separated by projects. > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG, 08.PNG, Screen Shot > 2017-10-12 at 4.32.24 PM.png > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2931: Attachment: 08.PNG > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG, 08.PNG, Screen Shot > 2017-10-12 at 4.32.24 PM.png > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201629#comment-16201629 ] Wang Cheng edited comment on KYLIN-2931 at 10/12/17 8:39 AM: - [~peng.jianhua], can you paste contents under /table_exd from resource store, just like screen shot I attached. PS. use bin/metastore.sh backup was (Author: wormholer): [~peng.jianhua], can you paste contents under /table_ext from resource store, just like screen shot I attached. PS. use bin/metastore.sh backup > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG, Screen Shot 2017-10-12 at > 4.32.24 PM.png > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201629#comment-16201629 ] Wang Cheng commented on KYLIN-2931: --- [~peng.jianhua], can you paste contents under /table_ext from resource store, just like screen shot I attached. PS. use bin/metastore.sh backup > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG, Screen Shot 2017-10-12 at > 4.32.24 PM.png > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Cheng updated KYLIN-2931: -- Attachment: Screen Shot 2017-10-12 at 4.32.24 PM.png > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG, Screen Shot 2017-10-12 at > 4.32.24 PM.png > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201541#comment-16201541 ] peng.jianhua edited comment on KYLIN-2931 at 10/12/17 8:21 AM: --- [~yimingliu] Is table_ext directory means table's extend information? was (Author: peng.jianhua): [~yimingliu] Is table_ext directory means table's extent information? > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201532#comment-16201532 ] peng.jianhua edited comment on KYLIN-2931 at 10/12/17 8:20 AM: --- Hi,[~wormholer]. The table's extend information has been pasted,please refer to 06.png and 07.png above. The bug occurs when one table was loaded in two project. When we drop the table and rebuild the table with different structure,then we reload the table on the web,the cardinality was not update. was (Author: peng.jianhua): Hi,[~wormholer]. The table's extent information has been pasted,please refer to 06.png and 07.png above. The bug occurs when one table was loaded in two project. When we drop the table and rebuild the table with different structure,then we reload the table on the web,the cardinality was not update. > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2794) MultipleDictionaryValueEnumerator should output values in sorted order
[
https://issues.apache.org/jira/browse/KYLIN-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-2794:
-
Fix Version/s: (was: v2.3.0)
v2.2.0
> MultipleDictionaryValueEnumerator should output values in sorted order
> --
>
> Key: KYLIN-2794
> URL: https://issues.apache.org/jira/browse/KYLIN-2794
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
> Environment: hadoop hadoop-2.6.0-cdh5.8.2 hive 2.1 hbase 0.98
>Reporter: 翟玉勇
>Assignee: Yifei Wu
>Priority: Critical
> Labels: scope
> Fix For: v2.2.0
>
>
> Dictionary exception during merge of segments.
> {code}
> 2017-08-18 14:17:48,828 ERROR [pool-11-thread-1]
> threadpool.DistributedScheduler:188 : ExecuteException
> job:8d031b5f-2d3f-445f-a62b-7bc560d919ea in server: **
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.IllegalStateException: Invalid input data. Unordered data cannot be
> split into multi trees
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 4 more
> Caused by: java.lang.IllegalStateException: Invalid input data. Unordered
> data cannot be split into multi trees
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92)
> at
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78)
> at
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.addValue(DictionaryGenerator.java:212)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79)
> at
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64)
> at
> org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104)
> at
> org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:267)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136)
> at
> org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> ... 6 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201547#comment-16201547 ] Billy Liu commented on KYLIN-2931: -- yes. [~peng.jianhua] > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201541#comment-16201541 ] peng.jianhua commented on KYLIN-2931: - [~yimingliu] Is table_ext directory means table's extent information? > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201532#comment-16201532 ] peng.jianhua commented on KYLIN-2931: - Hi,[~wormholer]. The table's extent information has been pasted,please refer to 06.png and 07.png above. The bug occurs when one table was loaded in two project. When we drop the table and rebuild the table with different structure,then we reload the table on the web,the cardinality was not update. > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201528#comment-16201528 ] Billy Liu commented on KYLIN-2931: -- [~peng.jianhua] could you dump the metadata, and check the table_ext directory. That's what [~wormholer] looking for. > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2931: Attachment: 06.PNG 07.PNG > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG, 06.PNG, 07.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
