[jira] [Commented] (KYLIN-2927) Merge Cuboid Dictionary ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201396#comment-16201396 ] songxiangjun commented on KYLIN-2927: - Thanks very much.@Billy Liu > Merge Cuboid Dictionary ERROR > - > > Key: KYLIN-2927 > URL: https://issues.apache.org/jira/browse/KYLIN-2927 > Project: Kylin > Issue Type: Bug >Reporter: songxiangjun > > when i merge the segment encount error, how can i solve it. Log as follows: > 2017-10-11 15:24:56,139 ERROR [pool-9-thread-10] > threadpool.DefaultScheduler:145 : ExecuteException > job:7508dfa0-5a89-4c3c-8685-701226628207 > org.apache.kylin.job.exception.ExecuteException: > org.apache.kylin.job.exception.ExecuteException: > java.lang.IllegalStateException: Invalid input data. Unordered data cannot be > split into multi trees > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:135) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:141) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.kylin.job.exception.ExecuteException: > java.lang.IllegalStateException: Invalid input data. Unordered data cannot be > split into multi trees > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:135) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:65) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > ... 4 more > Caused by: java.lang.IllegalStateException: Invalid input data. Unordered > data cannot be split into multi trees > at > org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92) > at > org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78) > at > org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder.addValue(DictionaryGenerator.java:261) > at > org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79) > at > org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64) > at > org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104) > at > org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:275) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > ... 6 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2656: Attachment: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201373#comment-16201373 ] peng.jianhua commented on KYLIN-2656: - Hi [~Shaofengshi], I had updated the patch. Please help to review again. > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future > > Attachments: 0001-KYLIN-2656-Support-Zookeeper-ACL.patch > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2906) Check model/cube name is duplicated when creating model/cube
[ https://issues.apache.org/jira/browse/KYLIN-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201376#comment-16201376 ] jiatao.tao commented on KYLIN-2906: --- Here's the commit: https://github.com/apache/kylin/commit/b425bb990854918e3dd20f4cf605e360b4d71df7 > Check model/cube name is duplicated when creating model/cube > > > Key: KYLIN-2906 > URL: https://issues.apache.org/jira/browse/KYLIN-2906 > Project: Kylin > Issue Type: Bug >Reporter: jiatao.tao >Assignee: jiatao.tao > Fix For: v2.2.0 > > > Add a API that check the model/cube name is duplicated -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2761) Table Level ACL
[ https://issues.apache.org/jira/browse/KYLIN-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201375#comment-16201375 ] jiatao.tao commented on KYLIN-2761: --- Here's the commits: https://github.com/apache/kylin/commit/a5a8def893644ca2b0bcdd8e34ef251d3c028455 https://github.com/apache/kylin/commit/06138f51b19a9b23b2fbb1665e5bae5f31e8f2b1 https://github.com/apache/kylin/commit/7640c946c7d9fbb5dac49c939b993030e5f90063 https://github.com/apache/kylin/commit/9af6a3f0cd2788b9df412c10c1ac237ff8b6d0ef https://github.com/apache/kylin/commit/555617cb07362dbe80e53fdfc317d641acc25e19 https://github.com/apache/kylin/commit/2e845c21343e4143bfa5a3d6bb54842808c57275 > Table Level ACL > --- > > Key: KYLIN-2761 > URL: https://issues.apache.org/jira/browse/KYLIN-2761 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: Joanna He >Assignee: jiatao.tao >Priority: Blocker > Fix For: v2.2.0 > > > We are planning to introduce table level ACL to kylin. Table level ACL > control whether user can query data from a table. > Table will go under project so that table level ACL will be set at project by > project basis. When an instance starts for the first time or upgrades from > lower version, every user by default has access to all tables that have been > loaded to the project. Admin can modify table level ACL. by removing user > from a table’s access list, admin can disable user’s query access to the > table. > In several cases, a user’s table level ACL will be checked to ensure ACL > integrity > 1.When user visit insight page and browse tables, table that user has no > access to under current project, will not be visible to the user. > 2.When user’s query hit on a cube, the table level ACL on dependent > tables for that user will be checked, if user has no access to any dependent > table, query will be refused on that cube. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2656) Support Zookeeper ACL
[ https://issues.apache.org/jira/browse/KYLIN-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2656: Attachment: (was: 0001-KYLIN-2656-Support-Zookeeper-ACLs.patch) > Support Zookeeper ACL > - > > Key: KYLIN-2656 > URL: https://issues.apache.org/jira/browse/KYLIN-2656 > Project: Kylin > Issue Type: New Feature >Reporter: peng.jianhua >Assignee: peng.jianhua > Fix For: Future > > > Kylin need to support Zookeeper ACL if we want to prevent unauthorized users > to access the znode or reduce risk of bad actions resulting from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2906) Check model/cube name is duplicated when creating model/cube
[ https://issues.apache.org/jira/browse/KYLIN-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-2906: -- Fix Version/s: v2.2.0 > Check model/cube name is duplicated when creating model/cube > > > Key: KYLIN-2906 > URL: https://issues.apache.org/jira/browse/KYLIN-2906 > Project: Kylin > Issue Type: Bug >Reporter: jiatao.tao >Assignee: jiatao.tao > Fix For: v2.2.0 > > > Add a API that check the model/cube name is duplicated -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2761) Table Level ACL
[ https://issues.apache.org/jira/browse/KYLIN-2761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-2761: -- Fix Version/s: v2.2.0 > Table Level ACL > --- > > Key: KYLIN-2761 > URL: https://issues.apache.org/jira/browse/KYLIN-2761 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: Joanna He >Assignee: jiatao.tao >Priority: Blocker > Fix For: v2.2.0 > > > We are planning to introduce table level ACL to kylin. Table level ACL > control whether user can query data from a table. > Table will go under project so that table level ACL will be set at project by > project basis. When an instance starts for the first time or upgrades from > lower version, every user by default has access to all tables that have been > loaded to the project. Admin can modify table level ACL. by removing user > from a table’s access list, admin can disable user’s query access to the > table. > In several cases, a user’s table level ACL will be checked to ensure ACL > integrity > 1.When user visit insight page and browse tables, table that user has no > access to under current project, will not be visible to the user. > 2.When user’s query hit on a cube, the table level ACL on dependent > tables for that user will be checked, if user has no access to any dependent > table, query will be refused on that cube. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201358#comment-16201358 ] Wang Cheng commented on KYLIN-2931: --- Hi Jianhua, can you paste table's name under /table_ext, to verify if it has been separated by projects. > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201313#comment-16201313
]
Billy Liu commented on KYLIN-2600:
--
Hi [~yaho], could you double commit this patch to the 2.2 branch also?
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2931) Update correct cardinality for the modified table
[ https://issues.apache.org/jira/browse/KYLIN-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng.jianhua updated KYLIN-2931: Attachment: 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch > Update correct cardinality for the modified table > - > > Key: KYLIN-2931 > URL: https://issues.apache.org/jira/browse/KYLIN-2931 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Labels: patch > Attachments: > 0001-KYLIN-2931-Update-correct-cardinality-for-the-modifi.patch, 01.PNG, > 02.PNG, 03.PNG, 04.PNG, 05.PNG > > > 1,Add project_1 and project_2,each project has the table kylin_sales_copy; > 2,Do the following steps: > a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy > table kylin_sales; > b,load table kylin_sales_copy and calculate column cardinality in project_1 > and project_2,please refer to 01.png; > c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; > d,use 'create table kylin_sales_copy as select > TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE > from kylin_sales where price < 20;' to create the new kylin_sales_copy table; > Now the structure of kylin_sales_copy has changed, the same as the > cardinality. > 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the > cardinality is not change,please refer to 02.png. > However,I found that the new cardinality was calculated but not update on the > web by debug the code,please refer to 03.png,04.png.It should update correct > cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2931) Update correct cardinality for the modified table
peng.jianhua created KYLIN-2931: --- Summary: Update correct cardinality for the modified table Key: KYLIN-2931 URL: https://issues.apache.org/jira/browse/KYLIN-2931 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v2.0.0 Reporter: peng.jianhua Assignee: peng.jianhua Priority: Minor Attachments: 01.PNG, 02.PNG, 03.PNG, 04.PNG, 05.PNG 1,Add project_1 and project_2,each project has the table kylin_sales_copy; 2,Do the following steps: a,use 'create table kylin_sales_copy as select * from kylin_sales;' to copy table kylin_sales; b,load table kylin_sales_copy and calculate column cardinality in project_1 and project_2,please refer to 01.png; c,use 'drop table kylin_sales_copy;' to drop table kylin_sales_copy; d,use 'create table kylin_sales_copy as select TRANS_ID,PART_DT,LSTG_FORMAT_NAME,LEAF_CATEG_ID,LSTG_SITE_ID,SLR_SEGMENT_CD,PRICE from kylin_sales where price < 20;' to create the new kylin_sales_copy table; Now the structure of kylin_sales_copy has changed, the same as the cardinality. 3,Reload table kylin_sales_copy in project_1 or project_2,we found that the cardinality is not change,please refer to 02.png. However,I found that the new cardinality was calculated but not update on the web by debug the code,please refer to 03.png,04.png.It should update correct cardinality like 05.png. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong resolved KYLIN-2600.
---
Resolution: Fixed
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2600:
--
Attachment: (was: APACHE-KYLIN-2600-master.patch)
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200692#comment-16200692
]
Zhong Yanghong commented on KYLIN-2600:
---
https://github.com/apache/kylin/commit/6e4c5af68edf6d369e6c6739ec31e954111c6812
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
> Attachments: APACHE-KYLIN-2600-master.patch
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-2930) Selecting one column in union causes compile error
Roger Shi created KYLIN-2930:
Summary: Selecting one column in union causes compile error
Key: KYLIN-2930
URL: https://issues.apache.org/jira/browse/KYLIN-2930
Project: Kylin
Issue Type: Bug
Reporter: Roger Shi
Query like
{code:sql}
select count(*) as cnt from TEST_KYLIN_FACT where TRANS_ID < 1000 union select
count(*) as cnt from TEST_KYLIN_FACT where TRANS_ID > 9000
{code}
make kylin throw exception:
{code:java}
Error while executing SQL "select count() from KYLIN_SALES union select count()
from KYLIN_SALES LIMIT 5": Error while compiling generated Java code:
public static class Record1_0 implements java.io.Serializable { public long f0;
public Record1_0() {} public boolean equals(Object o) { if (this == o) { return
true; } if (!(o instanceof Record1_0)) { return false; } return this.f0 ==
((Record1_0) o).f0; } public int hashCode() { int h = 0; h =
org.apache.calcite.runtime.Utilities.hash(h, this.f0); return h; } public int
compareTo(Record1_0 that) { final int c; c =
org.apache.calcite.runtime.Utilities.compare(this.f0, that.f0); if (c != 0) {
return c; } return 0; } public String toString() { return "{f0=" + this.f0 +
"}"; } } org.apache.calcite.DataContext root; public
org.apache.calcite.linq4j.Enumerable bind(final org.apache.calcite.DataContext
root0) { root = root0; final org.apache.calcite.linq4j.Enumerable
_inputEnumerable = ((org.apache.kylin.query.schema.OLAPTable)
root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
1); final org.apache.calcite.linq4j.AbstractEnumerable child = new
org.apache.calcite.linq4j.AbstractEnumerable(){ public
org.apache.calcite.linq4j.Enumerator enumerator() { return new
org.apache.calcite.linq4j.Enumerator(){ public final
org.apache.calcite.linq4j.Enumerator inputEnumerator =
_inputEnumerable.enumerator(); public void reset() { inputEnumerator.reset(); }
public boolean moveNext() { return inputEnumerator.moveNext(); } public void
close() { inputEnumerator.close(); } public Object current() { return
org.apache.calcite.runtime.SqlFunctions.toLong(((Object[])
inputEnumerator.current())[10]); } }; } }; final
org.apache.calcite.linq4j.Enumerable _inputEnumerable0 =
((org.apache.kylin.query.schema.OLAPTable)
root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
2); final org.apache.calcite.linq4j.AbstractEnumerable child1 = new
org.apache.calcite.linq4j.AbstractEnumerable(){ public
org.apache.calcite.linq4j.Enumerator enumerator() { return new
org.apache.calcite.linq4j.Enumerator(){ public final
org.apache.calcite.linq4j.Enumerator inputEnumerator =
inputEnumerable0.enumerator(); public void reset() { inputEnumerator.reset(); }
public boolean moveNext() { return inputEnumerator.moveNext(); } public void
close() { inputEnumerator.close(); } public Object current() { return
((Record12_1) inputEnumerator.current()).KY_COUNT; } }; } }; return
org.apache.calcite.linq4j.Linq4j.singletonEnumerable(child.aggregate(new
org.apache.calcite.linq4j.function.Function0() { public Object apply() { long
a0s0; a0s0 = 0; Record1_0 record0; record0 = new Record1_0(); record0.f0 =
a0s0; return record0; } } .apply(), new
org.apache.calcite.linq4j.function.Function2() { public Record1_0
apply(Record1_0 acc, long in) { acc.f0 = acc.f0 + in; return acc; } public
Record1_0 apply(Record1_0 acc, Long in) { return apply( acc, in.longValue()); }
public Record1_0 apply(Object acc, Object in) { return apply( (Record1_0) acc,
(Long) in); } } , new org.apache.calcite.linq4j.function.Function1() { public
long apply(Record1_0 acc) { return acc.f0; } public Object apply(Object acc) {
return apply( (Record1_0) acc); } }
)).union(org.apache.calcite.linq4j.Linq4j.singletonEnumerable(child1.aggregate(new
org.apache.calcite.linq4j.function.Function0() { public Object apply() { long
a0s0; a0s0 = 0; Record1_0 record0; record0 = new Record1_0(); record0.f0 =
a0s0; return record0; } } .apply(), new
org.apache.calcite.linq4j.function.Function2() { public Record1_0
apply(Record1_0 acc, long in) { acc.f0 = acc.f0 + in; return acc; } public
Record1_0 apply(Record1_0 acc, Long in) { return apply( acc, in.longValue()); }
public Record1_0 apply(Object acc, Object in) { return apply( (Record1_0) acc,
(Long) in); } } , new org.apache.calcite.linq4j.function.Function1() { public
long apply(Record1_0 acc) { return acc.f0; } public Object apply(Object acc) {
return apply( (Record1_0) acc); } } ))).take(5); } public Class
getElementType() { return long.class; }
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2929) speed up Dump file performance
[ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-2929: -- Component/s: Query Engine > speed up Dump file performance > -- > > Key: KYLIN-2929 > URL: https://issues.apache.org/jira/browse/KYLIN-2929 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu > Labels: Performance > > when I work on KYLIN-2926, I find coprocessor will dump to disk once > estimatedMemSize is bigger than spillThreshold, and found that spill data > size is extraordinary smaller that estimatedMemSize, in my case dump file > size is about 8MB and spillThreshold is setting to 3GB. > So, I try to keep the spill data in memory rather than write the file to disk > immediately, and when those in-memory spill data reach the threshold, write > all spill files together. > In my case, the coprocessor process cost time drop from 22s to 16s, it is > about 30% upgrade。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2929) speed up Dump file performance
[ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-2929: -- Affects Version/s: v2.0.0 > speed up Dump file performance > -- > > Key: KYLIN-2929 > URL: https://issues.apache.org/jira/browse/KYLIN-2929 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu > Labels: Performance > > when I work on KYLIN-2926, I find coprocessor will dump to disk once > estimatedMemSize is bigger than spillThreshold, and found that spill data > size is extraordinary smaller that estimatedMemSize, in my case dump file > size is about 8MB and spillThreshold is setting to 3GB. > So, I try to keep the spill data in memory rather than write the file to disk > immediately, and when those in-memory spill data reach the threshold, write > all spill files together. > In my case, the coprocessor process cost time drop from 22s to 16s, it is > about 30% upgrade。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2929) speed up Dump file performance
[ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-2929: -- Labels: Performance (was: ) > speed up Dump file performance > -- > > Key: KYLIN-2929 > URL: https://issues.apache.org/jira/browse/KYLIN-2929 > Project: Kylin > Issue Type: Bug >Reporter: fengYu >Assignee: fengYu > Labels: Performance > > when I work on KYLIN-2926, I find coprocessor will dump to disk once > estimatedMemSize is bigger than spillThreshold, and found that spill data > size is extraordinary smaller that estimatedMemSize, in my case dump file > size is about 8MB and spillThreshold is setting to 3GB. > So, I try to keep the spill data in memory rather than write the file to disk > immediately, and when those in-memory spill data reach the threshold, write > all spill files together. > In my case, the coprocessor process cost time drop from 22s to 16s, it is > about 30% upgrade。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2929) speed up Dump file performance
fengYu created KYLIN-2929: - Summary: speed up Dump file performance Key: KYLIN-2929 URL: https://issues.apache.org/jira/browse/KYLIN-2929 Project: Kylin Issue Type: Bug Reporter: fengYu Assignee: fengYu when I work on KYLIN-2926, I find coprocessor will dump to disk once estimatedMemSize is bigger than spillThreshold, and found that spill data size is extraordinary smaller that estimatedMemSize, in my case dump file size is about 8MB and spillThreshold is setting to 3GB. So, I try to keep the spill data in memory rather than write the file to disk immediately, and when those in-memory spill data reach the threshold, write all spill files together. In my case, the coprocessor process cost time drop from 22s to 16s, it is about 30% upgrade。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2927) Merge Cuboid Dictionary ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1628#comment-1628 ] Billy Liu commented on KYLIN-2927: -- Should have been fixed at https://issues.apache.org/jira/browse/KYLIN-2794 > Merge Cuboid Dictionary ERROR > - > > Key: KYLIN-2927 > URL: https://issues.apache.org/jira/browse/KYLIN-2927 > Project: Kylin > Issue Type: Bug >Reporter: songxiangjun > > when i merge the segment encount error, how can i solve it. Log as follows: > 2017-10-11 15:24:56,139 ERROR [pool-9-thread-10] > threadpool.DefaultScheduler:145 : ExecuteException > job:7508dfa0-5a89-4c3c-8685-701226628207 > org.apache.kylin.job.exception.ExecuteException: > org.apache.kylin.job.exception.ExecuteException: > java.lang.IllegalStateException: Invalid input data. Unordered data cannot be > split into multi trees > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:135) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:141) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.kylin.job.exception.ExecuteException: > java.lang.IllegalStateException: Invalid input data. Unordered data cannot be > split into multi trees > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:135) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:65) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > ... 4 more > Caused by: java.lang.IllegalStateException: Invalid input data. Unordered > data cannot be split into multi trees > at > org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92) > at > org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78) > at > org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder.addValue(DictionaryGenerator.java:261) > at > org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79) > at > org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64) > at > org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104) > at > org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:275) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136) > at > org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) > ... 6 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2727) Introduce cube planner able to select cost-effective cuboids to be built by cost-based algorithms
[ https://issues.apache.org/jira/browse/KYLIN-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2727: -- Affects Version/s: (was: v2.1.0) > Introduce cube planner able to select cost-effective cuboids to be built by > cost-based algorithms > - > > Key: KYLIN-2727 > URL: https://issues.apache.org/jira/browse/KYLIN-2727 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > > There're several disadvantages to create partial cubes only based on static > rules: > * To learn the concept of these rules will bring extra burden to cube admins > * To achieve a goal cuboid set may need a very complicated set of static rules > * Cube designers may be not familiar with business related query patterns, > resulting in few static rules able to applied > * Static rules created at first may not correct or user query patterns are > changing dynamically > The goal of cube planner is to reduce effort for cube admins to design an > effective partial cube. It owns the following advantages: > * No sophisticated combination of static rules is needed > * Allow more dimensions to be included with few static rules > * Able to adjust a non-cost-effective cuboid set to a cost-effective one > based on historical queries -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2727) Introduce cube planner able to select cost-effective cuboids to be built by cost-based algorithms
[ https://issues.apache.org/jira/browse/KYLIN-2727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2727: -- Fix Version/s: (was: v2.2.0) v2.3.0 > Introduce cube planner able to select cost-effective cuboids to be built by > cost-based algorithms > - > > Key: KYLIN-2727 > URL: https://issues.apache.org/jira/browse/KYLIN-2727 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Fix For: v2.3.0 > > > There're several disadvantages to create partial cubes only based on static > rules: > * To learn the concept of these rules will bring extra burden to cube admins > * To achieve a goal cuboid set may need a very complicated set of static rules > * Cube designers may be not familiar with business related query patterns, > resulting in few static rules able to applied > * Static rules created at first may not correct or user query patterns are > changing dynamically > The goal of cube planner is to reduce effort for cube admins to design an > effective partial cube. It owns the following advantages: > * No sophisticated combination of static rules is needed > * Allow more dimensions to be included with few static rules > * Able to adjust a non-cost-effective cuboid set to a cost-effective one > based on historical queries -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2906) Check model/cube name is duplicated when creating model/cube
[ https://issues.apache.org/jira/browse/KYLIN-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao resolved KYLIN-2906. --- Resolution: Fixed > Check model/cube name is duplicated when creating model/cube > > > Key: KYLIN-2906 > URL: https://issues.apache.org/jira/browse/KYLIN-2906 > Project: Kylin > Issue Type: Bug >Reporter: jiatao.tao >Assignee: jiatao.tao > > Add a API that check the model/cube name is duplicated -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2928) PUSH DOWN query cannot use order by function
[ https://issues.apache.org/jira/browse/KYLIN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1626#comment-1626 ] Billy Liu commented on KYLIN-2928: -- Which pushdown engine you are using? > PUSH DOWN query cannot use order by function > > > Key: KYLIN-2928 > URL: https://issues.apache.org/jira/browse/KYLIN-2928 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Yang Hao >Assignee: liyang > > SQL : select "DATE",count(1) from table_1 group by "DATE" order by "DATE" > desc; > Exception:org.apache.calcite.sql.SqlOrderBy cannot be cast to > org.apache.calcite.sql.SqlSelect -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2928) PUSH DOWN query cannot use order by function
Yang Hao created KYLIN-2928: --- Summary: PUSH DOWN query cannot use order by function Key: KYLIN-2928 URL: https://issues.apache.org/jira/browse/KYLIN-2928 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Yang Hao Assignee: liyang SQL : select "DATE",count(1) from table_1 group by "DATE" order by "DATE" desc; Exception:org.apache.calcite.sql.SqlOrderBy cannot be cast to org.apache.calcite.sql.SqlSelect -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2852) Build dict column from lookup table on step "Extract Fact Table Distinct Columns"
[ https://issues.apache.org/jira/browse/KYLIN-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Hao closed KYLIN-2852. --- > Build dict column from lookup table on step "Extract Fact Table Distinct > Columns" > - > > Key: KYLIN-2852 > URL: https://issues.apache.org/jira/browse/KYLIN-2852 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Yang Hao > Attachments: getAllDictColumnsOnFact.png > > > If a dict column is from lookup table, then "Build Dimension Dictionary" > would be very slow, because Step "Extract Fact Table Distinct Columns" only > build dict from fact table. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2910) org.apache.kylin.rest.exception.InternalErrorException: Failed to download file: java.net.SocketException: Broken pipe (Write failed)
[ https://issues.apache.org/jira/browse/KYLIN-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] songxiangjun resolved KYLIN-2910. - Resolution: Resolved > org.apache.kylin.rest.exception.InternalErrorException: Failed to download > file: java.net.SocketException: Broken pipe (Write failed) > - > > Key: KYLIN-2910 > URL: https://issues.apache.org/jira/browse/KYLIN-2910 > Project: Kylin > Issue Type: Bug >Reporter: songxiangjun > > How can i solve this ??Help... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2724) Introduce metrics collector for system metrics with different frequencies
[ https://issues.apache.org/jira/browse/KYLIN-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2724: -- Affects Version/s: (was: v2.0.0) > Introduce metrics collector for system metrics with different frequencies > - > > Key: KYLIN-2724 > URL: https://issues.apache.org/jira/browse/KYLIN-2724 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > For system metrics, we made an extension by introducing collecting frequency. > Since collecting all of system metrics too frequently may bring heavy > workload to servers, and different kind of metrics owns different priority, > we divide system metrics into three groups with different collecting > frequencies as follows: > * high frequency metrics(cpuUsage, jvmCpuUsage, jvmMemAvail, javaFDs, sysFDs) > * medium frequency metrics(jvmMemTotal, jvmVirtualBytes) > * low frequency metrics(avgSystemLoad, serverUpTimeInHrs, physMemAvail, > physMemTotal, jvmMemUsed, jvmPrivateBytes, maxJavaFDs, maxSysFDs) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2723) Introduce metrics collector for query & job metrics
[ https://issues.apache.org/jira/browse/KYLIN-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2723: -- Affects Version/s: (was: v2.2.0) > Introduce metrics collector for query & job metrics > --- > > Key: KYLIN-2723 > URL: https://issues.apache.org/jira/browse/KYLIN-2723 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: APACHE-KYLIN-2722+APACHE-KYLIN-2723.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2722) Introduce a new measure, called active reservoir, for actively pushing metrics to reporters
[
https://issues.apache.org/jira/browse/KYLIN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2722:
--
Affects Version/s: (was: v2.2.0)
> Introduce a new measure, called active reservoir, for actively pushing
> metrics to reporters
> ---
>
> Key: KYLIN-2722
> URL: https://issues.apache.org/jira/browse/KYLIN-2722
> Project: Kylin
> Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Attachments: APACHE-KYLIN-2722.patch
>
>
> For many existing metrics frameworks, they focus on maintaining metrics in
> memory independently for each instance. However, kylin server may consist of
> multiple instances. Thus we extend existing metrics framework by introducing
> *active reservoir* to actively push metrics to reporters which will report
> metrics of its instance to a unified storage.
> Here we introduced two *active reservoirs*. One is called
> {{BlockingReservoir}}, which will buffer the metrics. The other is called
> {{InstantReservoir}}, which owns no buffer and will directly push metrics to
> reporters.
> Generally, one *active reservoir* can push its metrics to multiple reporters
> and one reporter can only listen on one *active reservoir*.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2721) Introduce a new metrics framework based on dropwizard metrics
[ https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2721: -- Affects Version/s: (was: v2.2.0) > Introduce a new metrics framework based on dropwizard metrics > - > > Key: KYLIN-2721 > URL: https://issues.apache.org/jira/browse/KYLIN-2721 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: Metrics Framework.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2600) Incorrectly set the range start when filtering by the minimum value
[
https://issues.apache.org/jira/browse/KYLIN-2600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2600:
--
Fix Version/s: v2.2.0
> Incorrectly set the range start when filtering by the minimum value
> ---
>
> Key: KYLIN-2600
> URL: https://issues.apache.org/jira/browse/KYLIN-2600
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.2.0
>
> Attachments: APACHE-KYLIN-2600-master.patch
>
>
> Before defining a range of a scan, the range start may be not correct in the
> following case:
> {code}
> OR [
> AND [
> a <= 3,
> b = 2
> ],
> AND [
> a = 0, (note that 0 is the minimum of a)
> b = 1
> ]
> ]
> {code}
> In this case, kylin will generate two ranges:
> {code}
> [null,2] - [3, 2]
> [0,1] - [0,1]
> {code}
> There's a rule when merging ranges. If the range start is null, it will be
> ordered before others whose range start is not null. Then the merged range of
> these two ranges will be
> \[null,2\] - \[3, 2\].
> Replacing null with 0, the range sent to coprocessor will be
> \[0,2\] - \[3, 2\].
> However, it should be
> \[0,1\] - \[3, 2\]
> and the rule should be refined.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (KYLIN-2927) Merge Cuboid Dictionary ERROR
songxiangjun created KYLIN-2927: --- Summary: Merge Cuboid Dictionary ERROR Key: KYLIN-2927 URL: https://issues.apache.org/jira/browse/KYLIN-2927 Project: Kylin Issue Type: Bug Reporter: songxiangjun when i merge the segment encount error, how can i solve it. Log as follows: 2017-10-11 15:24:56,139 ERROR [pool-9-thread-10] threadpool.DefaultScheduler:145 : ExecuteException job:7508dfa0-5a89-4c3c-8685-701226628207 org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.IllegalStateException: Invalid input data. Unordered data cannot be split into multi trees at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:135) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:141) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.IllegalStateException: Invalid input data. Unordered data cannot be split into multi trees at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:135) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:65) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) ... 4 more Caused by: java.lang.IllegalStateException: Invalid input data. Unordered data cannot be split into multi trees at org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:92) at org.apache.kylin.dict.TrieDictionaryForestBuilder.addValue(TrieDictionaryForestBuilder.java:78) at org.apache.kylin.dict.DictionaryGenerator$NumberTrieDictForestBuilder.addValue(DictionaryGenerator.java:261) at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:79) at org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:64) at org.apache.kylin.dict.DictionaryGenerator.mergeDictionaries(DictionaryGenerator.java:104) at org.apache.kylin.dict.DictionaryManager.mergeDictionary(DictionaryManager.java:275) at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.mergeDictionaries(MergeDictionaryStep.java:146) at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.makeDictForNewSegment(MergeDictionaryStep.java:136) at org.apache.kylin.engine.mr.steps.MergeDictionaryStep.doWork(MergeDictionaryStep.java:68) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125) ... 6 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2926) DumpMerger return incorrect results
[ https://issues.apache.org/jira/browse/KYLIN-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-2926: -- Description: I our scenario, a cube query will get wrong result once coprocessor need to spill to disk, Our version is 2.0.0 and I find the root cause is that in DumpMerger.enqueueFromDump because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It leading to different elements in dumpCurrentValues share the same object, so next fill up measure values will change the existing values. the incorrect measures is HLLC and raw, which use current variable in deserialize. was: I our scenario, a cube query will get wrong result once coprocessor need to spill to disk, Our version is 2.0.0 and I find the root cause is that in DumpMerger.enqueueFromDump because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It leading to different elements in dumpCurrentValues share the same object, so next fill up measure values will change the existing values. the incorrect measures is HLLC. > DumpMerger return incorrect results > --- > > Key: KYLIN-2926 > URL: https://issues.apache.org/jira/browse/KYLIN-2926 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu > > I our scenario, a cube query will get wrong result once coprocessor need to > spill to disk, Our version is 2.0.0 and I find the root cause is that in > DumpMerger.enqueueFromDump > because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It > leading to different elements in dumpCurrentValues share the same object, so > next fill up measure values will change the existing values. > the incorrect measures is HLLC and raw, which use current variable in > deserialize. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2926) DumpMerger return incorrect results
[
https://issues.apache.org/jira/browse/KYLIN-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16199892#comment-16199892
]
fengYu commented on KYLIN-2926:
---
for simply modify and test, I modify the function like this :
private void enqueueFromDump(int index) {
if (dumpIterators.get(index) != null &&
dumpIterators.get(index).hasNext()) {
Pair pair = dumpIterators.get(index).next();
minHeap.offer(new Pair(pair.getKey(), index));
Object[] metricValues = new Object[metrics.trueBitCount()];
BufferedMeasureCodec codec= request.createMeasureCodec();
codec.decode(ByteBuffer.wrap(pair.getValue()),
metricValues);
dumpCurrentValues.set(index, metricValues);
}
}
the result will be correct.
> DumpMerger return incorrect results
> ---
>
> Key: KYLIN-2926
> URL: https://issues.apache.org/jira/browse/KYLIN-2926
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: fengYu
>Assignee: fengYu
>
> I our scenario, a cube query will get wrong result once coprocessor need to
> spill to disk, Our version is 2.0.0 and I find the root cause is that in
> DumpMerger.enqueueFromDump
> because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It
> leading to different elements in dumpCurrentValues share the same object, so
> next fill up measure values will change the existing values.
> the incorrect measures is HLLC.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-2926) DumpMerger return incorrect results
[ https://issues.apache.org/jira/browse/KYLIN-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fengYu updated KYLIN-2926: -- Description: I our scenario, a cube query will get wrong result once coprocessor need to spill to disk, Our version is 2.0.0 and I find the root cause is that in DumpMerger.enqueueFromDump because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It leading to different elements in dumpCurrentValues share the same object, so next fill up measure values will change the existing values. the incorrect measures is HLLC. was: I our scenario, a cube query will get wrong result once coprocessor need to spill to disk, Our version is 2.0.0 and I find the root cause is that in DumpMerger.enqueueFromDump because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It leading to different elements in dumpCurrentValues share the same object, so next fill up measure values will change the existing values. > DumpMerger return incorrect results > --- > > Key: KYLIN-2926 > URL: https://issues.apache.org/jira/browse/KYLIN-2926 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu > > I our scenario, a cube query will get wrong result once coprocessor need to > spill to disk, Our version is 2.0.0 and I find the root cause is that in > DumpMerger.enqueueFromDump > because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It > leading to different elements in dumpCurrentValues share the same object, so > next fill up measure values will change the existing values. > the incorrect measures is HLLC. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2926) DumpMerger return incorrect results
fengYu created KYLIN-2926: - Summary: DumpMerger return incorrect results Key: KYLIN-2926 URL: https://issues.apache.org/jira/browse/KYLIN-2926 Project: Kylin Issue Type: Bug Affects Versions: v2.0.0 Reporter: fengYu Assignee: fengYu I our scenario, a cube query will get wrong result once coprocessor need to spill to disk, Our version is 2.0.0 and I find the root cause is that in DumpMerger.enqueueFromDump because in DataTypeSerializer kylin use a ThreadLocal variable ‘current’, It leading to different elements in dumpCurrentValues share the same object, so next fill up measure values will change the existing values. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
