[jira] [Commented] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692798#comment-16692798 ] ASF GitHub Bot commented on KYLIN-3290: --- codecov-io commented on issue #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance() URL: https://github.com/apache/kylin/pull/351#issuecomment-440175653 # [Codecov](https://codecov.io/gh/apache/kylin/pull/351?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@dca3ee7`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `50%`. [](https://codecov.io/gh/apache/kylin/pull/351?src=pr=tree) ```diff @@Coverage Diff@@ ## master #351 +/- ## = Coverage ? 23.61% Complexity? 4826 = Files ? 1133 Lines ?68102 Branches ? 9659 = Hits ?16080 Misses?50457 Partials ? 1565 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/351?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/kylin/dict/DictionarySerializer.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L0RpY3Rpb25hcnlTZXJpYWxpemVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...n/java/org/apache/kylin/dict/NumberDictionary.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlckRpY3Rpb25hcnkuamF2YQ==) | `22.22% <0%> (ø)` | `2 <0> (?)` | | | [...g/apache/kylin/query/relnode/OLAPAggregateRel.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3JlbG5vZGUvT0xBUEFnZ3JlZ2F0ZVJlbC5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/kylin/dict/global/GlobalDictHDFSStore.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2dsb2JhbC9HbG9iYWxEaWN0SERGU1N0b3JlLmphdmE=) | `70.44% <100%> (ø)` | `18 <0> (?)` | | | [...n/java/org/apache/kylin/common/util/ClassUtil.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL0NsYXNzVXRpbC5qYXZh) | `51.66% <100%> (ø)` | `9 <1> (?)` | | | [...va/org/apache/kylin/dict/TrieDictionaryForest.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L1RyaWVEaWN0aW9uYXJ5Rm9yZXN0LmphdmE=) | `87.32% <100%> (ø)` | `62 <0> (?)` | | | [...ain/java/org/apache/kylin/dict/TrieDictionary.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L1RyaWVEaWN0aW9uYXJ5LmphdmE=) | `73.02% <100%> (ø)` | `57 <1> (?)` | | | [.../java/org/apache/kylin/dict/NumberDictionary2.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlckRpY3Rpb25hcnkyLmphdmE=) | `71.42% <100%> (ø)` | `2 <0> (?)` | | | [...rg/apache/kylin/dict/DictionaryInfoSerializer.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L0RpY3Rpb25hcnlJbmZvU2VyaWFsaXplci5qYXZh) | `56.66% <20%> (ø)` | `5 <0> (?)` | | | [...n/java/org/apache/kylin/query/QueryConnection.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L1F1ZXJ5Q29ubmVjdGlvbi5qYXZh) | `83.33% <50%> (ø)` | `3 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/351?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/351?src=pr=footer). Last update [dca3ee7...fe29b71](https://codecov.io/gh/apache/kylin/pull/351?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated
[GitHub] codecov-io commented on issue #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance()
codecov-io commented on issue #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance() URL: https://github.com/apache/kylin/pull/351#issuecomment-440175653 # [Codecov](https://codecov.io/gh/apache/kylin/pull/351?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@dca3ee7`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `50%`. [](https://codecov.io/gh/apache/kylin/pull/351?src=pr=tree) ```diff @@Coverage Diff@@ ## master #351 +/- ## = Coverage ? 23.61% Complexity? 4826 = Files ? 1133 Lines ?68102 Branches ? 9659 = Hits ?16080 Misses?50457 Partials ? 1565 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/351?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/kylin/dict/DictionarySerializer.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L0RpY3Rpb25hcnlTZXJpYWxpemVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...n/java/org/apache/kylin/dict/NumberDictionary.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlckRpY3Rpb25hcnkuamF2YQ==) | `22.22% <0%> (ø)` | `2 <0> (?)` | | | [...g/apache/kylin/query/relnode/OLAPAggregateRel.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L3JlbG5vZGUvT0xBUEFnZ3JlZ2F0ZVJlbC5qYXZh) | `0% <0%> (ø)` | `0 <0> (?)` | | | [.../apache/kylin/dict/global/GlobalDictHDFSStore.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2dsb2JhbC9HbG9iYWxEaWN0SERGU1N0b3JlLmphdmE=) | `70.44% <100%> (ø)` | `18 <0> (?)` | | | [...n/java/org/apache/kylin/common/util/ClassUtil.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi91dGlsL0NsYXNzVXRpbC5qYXZh) | `51.66% <100%> (ø)` | `9 <1> (?)` | | | [...va/org/apache/kylin/dict/TrieDictionaryForest.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L1RyaWVEaWN0aW9uYXJ5Rm9yZXN0LmphdmE=) | `87.32% <100%> (ø)` | `62 <0> (?)` | | | [...ain/java/org/apache/kylin/dict/TrieDictionary.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L1RyaWVEaWN0aW9uYXJ5LmphdmE=) | `73.02% <100%> (ø)` | `57 <1> (?)` | | | [.../java/org/apache/kylin/dict/NumberDictionary2.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlckRpY3Rpb25hcnkyLmphdmE=) | `71.42% <100%> (ø)` | `2 <0> (?)` | | | [...rg/apache/kylin/dict/DictionaryInfoSerializer.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L0RpY3Rpb25hcnlJbmZvU2VyaWFsaXplci5qYXZh) | `56.66% <20%> (ø)` | `5 <0> (?)` | | | [...n/java/org/apache/kylin/query/QueryConnection.java](https://codecov.io/gh/apache/kylin/pull/351/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L1F1ZXJ5Q29ubmVjdGlvbi5qYXZh) | `83.33% <50%> (ø)` | `3 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/351?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/351?src=pr=footer). Last update [dca3ee7...fe29b71](https://codecov.io/gh/apache/kylin/pull/351?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With
[jira] [Commented] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692767#comment-16692767 ] ASF GitHub Bot commented on KYLIN-3290: --- asfgit commented on issue #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance() URL: https://github.com/apache/kylin/pull/351#issuecomment-440170481 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692766#comment-16692766 ] ASF GitHub Bot commented on KYLIN-3290: --- whuwb opened a new pull request #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance() URL: https://github.com/apache/kylin/pull/351 Issue : https://issues.apache.org/jira/browse/KYLIN-3290 How to fix : Leverage getDecalredConstructor().newInstance() instead of newInstance() This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] whuwb opened a new pull request #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance()
whuwb opened a new pull request #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance() URL: https://github.com/apache/kylin/pull/351 Issue : https://issues.apache.org/jira/browse/KYLIN-3290 How to fix : Leverage getDecalredConstructor().newInstance() instead of newInstance() This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance()
asfgit commented on issue #351: KYLIN-3290 Leverage getDecalredConstructor().newInstance() instead of newInstance() URL: https://github.com/apache/kylin/pull/351#issuecomment-440170481 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3682) Error while building sample cube with spark ava.lang.NoClassDefFoundError: Could not initialize class org.apache.kylin.metadata.datatype.DataType
[
https://issues.apache.org/jira/browse/KYLIN-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692619#comment-16692619
]
Gaurav Rawat commented on KYLIN-3682:
-
HI [~Shaofengshi] Thanks for the response byt I am using the Kylin spark
version only as per instructions outlined in the documentation . The spark
version is the one that comes with Kylin that is *version 2.1.2* . spark home
is set as
{code:java}
[hadoop@ip-10-81-117-142 ~]$ echo $SPARK_HOME
/usr/local/kylin/spark
[hadoop@ip-10-81-117-142 ~]${code}
> Error while building sample cube with spark ava.lang.NoClassDefFoundError:
> Could not initialize class org.apache.kylin.metadata.datatype.DataType
> -
>
> Key: KYLIN-3682
> URL: https://issues.apache.org/jira/browse/KYLIN-3682
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
> Environment: test
>Reporter: Gaurav Rawat
>Priority: Major
> Labels: starter
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> HI I am having error at step #7 when building with Spark on AWS EMR, with
> mapreduce the same builds fine . The error I am getting is as below :-
> {code:java}
> 18/11/12 21:45:01 INFO yarn.Client:
>client token: N/A
>diagnostics: User class threw exception: java.lang.RuntimeException:
> error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root
> cause: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times,
> most recent failure: Lost task 1.3 in stage 0.0 (TID 8,
> ip-10-81-117-151.wfg1tst.cltest.wellmanage.com, executor 1):
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.kylin.metadata.datatype.DataType
> at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
> at
> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1933)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at java.util.HashSet.readObject(HashSet.java:333)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at
[jira] [Commented] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692608#comment-16692608
]
yangwei commented on KYLIN-3696:
好的我用mr engine
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png, image-2018-11-20-11-27-43-750.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692608#comment-16692608
]
yangwei edited comment on KYLIN-3696 at 11/20/18 3:48 AM:
--
好的我用mr engine试下
was (Author: yangwei):
好的我用mr engine
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png, image-2018-11-20-11-27-43-750.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692597#comment-16692597
]
yangwei commented on KYLIN-3696:
!image-2018-11-20-11-27-43-750.png!
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png, image-2018-11-20-11-27-43-750.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692600#comment-16692600
]
Shaofeng SHI commented on KYLIN-3696:
-
Thank you wei, please switch to MR engine and try again. Please check
KYLIN-3693. Before it be fixed, please use MR engine.
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png, image-2018-11-20-11-27-43-750.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yangwei updated KYLIN-3696:
---
Attachment: image-2018-11-20-11-27-43-750.png
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png, image-2018-11-20-11-27-43-750.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692594#comment-16692594
]
yangwei commented on KYLIN-3696:
spark
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
[
https://issues.apache.org/jira/browse/KYLIN-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692593#comment-16692593
]
Shaofeng SHI commented on KYLIN-3696:
-
Which cube engine are you using, MR or Spark?
> TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
>
>
> Key: KYLIN-3696
> URL: https://issues.apache.org/jira/browse/KYLIN-3696
> Project: Kylin
> Issue Type: Bug
> Components: Measure - TopN
>Affects Versions: v2.5.1
>Reporter: yangwei
>Priority: Major
> Attachments: image-2018-11-20-10-57-28-546.png,
> image-2018-11-20-11-01-25-120.png
>
>
> 我使用的是v2.5.1，度量topN使用上出现不准的总量。
> 问题再现：
> 一，二个cube使用同一个模型就是同一张物理事实表。
> 二，二个cube同时包含相同的topN度量
> 三，二个cube状态都是Ready
> 目前我暂时的解决方法是在其中一个cube去掉一个topN度量
> 同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
> SELECT IP ,
> SUM(ACCESS_COUNT) c
> FROM API_ACCESS
> WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
> group by ip
> ORDER BY
> c DESC
> LIMIT 10;
> 二个cube中的度量:
> cube1:
> !image-2018-11-20-10-57-28-546.png!
> cube2:
> !image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3696) TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
yangwei created KYLIN-3696:
--
Summary: TOPN度量在同一个模型下2个cube同时开启统计值不准与真实值差得较多
Key: KYLIN-3696
URL: https://issues.apache.org/jira/browse/KYLIN-3696
Project: Kylin
Issue Type: Bug
Components: Measure - TopN
Affects Versions: v2.5.1
Reporter: yangwei
Attachments: image-2018-11-20-10-57-28-546.png,
image-2018-11-20-11-01-25-120.png
我使用的是v2.5.1，度量topN使用上出现不准的总量。
问题再现：
一，二个cube使用同一个模型就是同一张物理事实表。
二，二个cube同时包含相同的topN度量
三，二个cube状态都是Ready
目前我暂时的解决方法是在其中一个cube去掉一个topN度量
同一个sql在hive与kylin里查的的结果对不上相差很远,下面给出sql
SELECT IP ,
SUM(ACCESS_COUNT) c
FROM API_ACCESS
WHERE TAG_DATE = CAST('2018-11-19' AS DATE)
group by ip
ORDER BY
c DESC
LIMIT 10;
二个cube中的度量:
cube1:
!image-2018-11-20-10-57-28-546.png!
cube2:
!image-2018-11-20-11-01-25-120.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3694) Kylin On Druid Storage
[ https://issues.apache.org/jira/browse/KYLIN-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692588#comment-16692588 ] Shaofeng SHI commented on KYLIN-3694: - +1 Glad to see an secondary storage engine for Kylin. > Kylin On Druid Storage > -- > > Key: KYLIN-3694 > URL: https://issues.apache.org/jira/browse/KYLIN-3694 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, Metadata, Query Engine >Affects Versions: v2.5.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: Kylin On Druid Storage.pdf > > > Meituan Kylin team has implemented a new storage engine for Kylin: Druid > Storage Engine. > The attach file is the Kylin On Druid Storage Engine architecture design > doc. > We would like to contribute the feature to community, please let us know if > you have any concern. [^Kylin On Druid Storage.pdf] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692579#comment-16692579
]
ASF subversion and git services commented on KYLIN-3689:
Commit 4bcd2f217ffaa83871bdd5cd135abb49db0ea1d0 in kylin's branch
refs/heads/2.3.x from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=4bcd2f2 ]
KYLIN-3689 fix UT
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692577#comment-16692577
]
ASF subversion and git services commented on KYLIN-3689:
Commit 02f19f200be95e0da3e146386f31fe0633e3f6b8 in kylin's branch
refs/heads/2.5.x from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=02f19f2 ]
KYLIN-3689 fix UT
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3666) Mege cube step 2: Update dictionary throws IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3666:
Fix Version/s: (was: v2.5.3)
v2.5.2
> Mege cube step 2: Update dictionary throws IllegalStateException
>
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.0
> Environment: cdh5.11.0
>Reporter: Xi Chen
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.2
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at
> org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3290: --- Assignee: Wu Bin (was: Shaofeng SHI) Bin, I think nobody is working on this, please go ahead. > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692582#comment-16692582 ] ASF subversion and git services commented on KYLIN-3631: Commit dca3ee7e12e238f6c0baf25388c5a63d3298758f in kylin's branch refs/heads/master from 武斌 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=dca3ee7 ] KYLIN-3631 Use Arrays#parallelSort instead of Arrays#sort > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3631. - Resolution: Fixed > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[
https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692581#comment-16692581
]
ASF GitHub Bot commented on KYLIN-3631:
---
shaofengshi closed pull request #350: Fix issue : KYLIN-3631
URL: https://github.com/apache/kylin/pull/350
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
b/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
index 87bf9c3c35..c40e766a67 100644
---
a/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
+++
b/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
@@ -118,8 +118,9 @@ private void testSpanningAndGetParent(CuboidScheduler
scheduler, CubeDesc cube,
}
long[] spanningsArray = Longs.toArray(totalSpanning);
-Arrays.sort(spanningsArray);
-Arrays.sort(expectChildren);
+
+Arrays.parallelSort(spanningsArray);
+Arrays.parallelSort(expectChildren);
assertArrayEquals(expectChildren, spanningsArray);
}
diff --git
a/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
b/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
index bb68299133..6c4c9a3142 100644
--- a/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
+++ b/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
@@ -220,7 +220,7 @@ public SegmentRange autoMergeCubeSegments(boolean
needAutoMerge, String cubeName
// exclude those already under merging segments
readySegs.removeAll(mergingSegs);
-Arrays.sort(timeRanges);
+Arrays.parallelSort(timeRanges);
for (int i = timeRanges.length - 1; i >= 0; i--) {
long toMergeRange = timeRanges[i];
diff --git
a/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
b/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
index 63a78f89e6..6bbc70c96c 100644
--- a/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
+++ b/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
@@ -308,7 +308,7 @@ public void init(KylinConfig config, String project) {
setDatabase(getDatabase().toUpperCase(Locale.ROOT));
if (columns != null) {
-Arrays.sort(columns, new Comparator() {
+Arrays.parallelSort(columns, new Comparator() {
@Override
public int compare(ColumnDesc col1, ColumnDesc col2) {
Integer id1 = Integer.parseInt(col1.getId());
diff --git
a/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
b/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
index e979791df5..30b2e56711 100644
---
a/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
+++
b/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
@@ -67,7 +67,8 @@ public void testRandom() {
for (int i = 0; i < n; i++) {
nums[i] = rand.nextDouble() * 100;
}
-Arrays.sort(nums);
+
+Arrays.parallelSort(nums);
buf.clear();
dds.serialize(nums, buf);
@@ -86,7 +87,8 @@ public void testRandom2() {
for (int i = 0; i < n; i++) {
nums[i] = rand.nextInt();
}
-Arrays.sort(nums);
+
+Arrays.parallelSort(nums);
buf.clear();
dds.serialize(nums, buf);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Utilize Arrays#parallelSort for better performance
> --
>
> Key: KYLIN-3631
> URL: https://issues.apache.org/jira/browse/KYLIN-3631
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Wu Bin
>Priority: Minor
> Fix For: v2.6.0
>
>
> Arrays#parallelSort was introduced since Java 1.8
> We can utilize Arrays#parallelSort to achieve better performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] shaofengshi closed pull request #350: Fix issue : KYLIN-3631
shaofengshi closed pull request #350: Fix issue : KYLIN-3631
URL: https://github.com/apache/kylin/pull/350
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
b/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
index 87bf9c3c35..c40e766a67 100644
---
a/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
+++
b/core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java
@@ -118,8 +118,9 @@ private void testSpanningAndGetParent(CuboidScheduler
scheduler, CubeDesc cube,
}
long[] spanningsArray = Longs.toArray(totalSpanning);
-Arrays.sort(spanningsArray);
-Arrays.sort(expectChildren);
+
+Arrays.parallelSort(spanningsArray);
+Arrays.parallelSort(expectChildren);
assertArrayEquals(expectChildren, spanningsArray);
}
diff --git
a/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
b/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
index bb68299133..6c4c9a3142 100644
--- a/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
+++ b/core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java
@@ -220,7 +220,7 @@ public SegmentRange autoMergeCubeSegments(boolean
needAutoMerge, String cubeName
// exclude those already under merging segments
readySegs.removeAll(mergingSegs);
-Arrays.sort(timeRanges);
+Arrays.parallelSort(timeRanges);
for (int i = timeRanges.length - 1; i >= 0; i--) {
long toMergeRange = timeRanges[i];
diff --git
a/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
b/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
index 63a78f89e6..6bbc70c96c 100644
--- a/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
+++ b/core-metadata/src/main/java/org/apache/kylin/metadata/model/TableDesc.java
@@ -308,7 +308,7 @@ public void init(KylinConfig config, String project) {
setDatabase(getDatabase().toUpperCase(Locale.ROOT));
if (columns != null) {
-Arrays.sort(columns, new Comparator() {
+Arrays.parallelSort(columns, new Comparator() {
@Override
public int compare(ColumnDesc col1, ColumnDesc col2) {
Integer id1 = Integer.parseInt(col1.getId());
diff --git
a/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
b/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
index e979791df5..30b2e56711 100644
---
a/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
+++
b/core-metadata/src/test/java/org/apache/kylin/measure/topn/DoubleDeltaSerializerTest.java
@@ -67,7 +67,8 @@ public void testRandom() {
for (int i = 0; i < n; i++) {
nums[i] = rand.nextDouble() * 100;
}
-Arrays.sort(nums);
+
+Arrays.parallelSort(nums);
buf.clear();
dds.serialize(nums, buf);
@@ -86,7 +87,8 @@ public void testRandom2() {
for (int i = 0; i < n; i++) {
nums[i] = rand.nextInt();
}
-Arrays.sort(nums);
+
+Arrays.parallelSort(nums);
buf.clear();
dds.serialize(nums, buf);
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (KYLIN-3695) Error while creating hive table through Kylin build cube with mysql imported tables
Gaurav Rawat created KYLIN-3695:
---
Summary: Error while creating hive table through Kylin build cube
with mysql imported tables
Key: KYLIN-3695
URL: https://issues.apache.org/jira/browse/KYLIN-3695
Project: Kylin
Issue Type: Bug
Components: Integration
Affects Versions: v2.5.0
Reporter: Gaurav Rawat
HI I am trying to build a cube with Kylin, the data gets souced fine from sqoop
but the next step for creating hive tables fails . Looking at the command being
fired it looks weird as the create statement looks good to me .
I think the issue is with DOUBLE types as when I remove the same the create
statement works fine . Can someone please help .
I am using the stack in AWS EMR, kylin 2.5 hive 2.3.0
The errors logs with commands as as below, the table is a msql table which had
columns with DOUBLE type
Command
{code:java}
hive -e "USE default; DROP TABLE IF EXISTS
kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368; CREATE
EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 (
HOLDINGS_STOCK_INVESTOR_ID string ,STOCK_INVESTORS_CHANNEL string
,STOCK_STOCK_ID string ,STOCK_DOMICILE string ,STOCK_STOCK_NM string
,STOCK_APPROACH string ,STOCK_STOCK_TYP string ,INVESTOR_ID string ,INVESTOR_NM
string ,INVESTOR_DOMICILE_CNTRY string ,CLIENT_NM string
,INVESTOR_HOLDINGS_GROSS_ASSETS_USD double(22)
,INVESTOR_HOLDINGS_NET_ASSETS_USD double(22) ) ROW FORMAT DELIMITED FIELDS
TERMINATED BY '|' STORED AS TEXTFILE LOCATION
's3://wfg1tst-models/kylin/kylin_metadata/kylin-4ae3b18b-831b-da66-eb8c-7318245c4448/kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368';
ALTER TABLE
kylin_intermediate_fm_inv_holdings_8a1c33df_d12b_3609_13ee_39e169169368 SET
TBLPROPERTIES('auto.purge'='true'); " --hiveconf hive.merge.mapredfiles=false
--hiveconf hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf
hive.exec.compress.output=true --hiveconf
hive.auto.convert.join.noconditionaltask=true --hiveconf
mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false
--hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf
hive.stats.autogather=true{code}
Error is as below
{code:java}
OK Time taken: 1.315 seconds OK Time taken: 0.09 seconds
MismatchedTokenException(334!=347) at
org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at
org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:6179)
at
org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:3808)
at
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:2382)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1333)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:204) at
org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:77) at
org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:70) at
org.apache.hadoop.hive.ql.Driver.compile(Driver.java:468) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1316) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1456) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1236) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1226) at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.util.RunJar.run(RunJar.java:221) at
org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException line
15:42 mismatched input '(' expecting ) near 'double' in create table
statement{code}
[hadoop|https://stackoverflow.com/questions/tagged/hadoop]
[hive|https://stackoverflow.com/questions/tagged/hive]
[kylin|https://stackoverflow.com/questions/tagged/kylin]
More details here
[https://stackoverflow.com/questions/53377623/error-while-creating-hive-table-through-kylin-build-cube]
. Though the same got solved when I used DECIMAL type with precision .
Also observer that if you use DECIMAL(10) it does not work unless it has a
precision while building
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692576#comment-16692576
]
ASF subversion and git services commented on KYLIN-3689:
Commit c9d5d0deac6ce587761f3359cdcfe161cb991ec5 in kylin's branch
refs/heads/master from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c9d5d0d ]
KYLIN-3689 fix UT
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692578#comment-16692578
]
ASF subversion and git services commented on KYLIN-3689:
Commit b3df23d876fba2eec30743fd5b86f41a7c6a5de7 in kylin's branch
refs/heads/2.4.x from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b3df23d ]
KYLIN-3689 fix UT
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3684) [Script] find-hive-dependency.sh HIVE_LIB is not set or not resolved correctly
[ https://issues.apache.org/jira/browse/KYLIN-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3684: Fix Version/s: v2.6.0 Component/s: Environment > [Script] find-hive-dependency.sh HIVE_LIB is not set or not resolved > correctly > --- > > Key: KYLIN-3684 > URL: https://issues.apache.org/jira/browse/KYLIN-3684 > Project: Kylin > Issue Type: Bug > Components: Environment >Reporter: PENG Zhengshuai >Priority: Major > Fix For: v2.6.0 > > > There is a bug when executing find-hive-dependency.sh, if $HIVE_CONF has been > set, $hive_exec_path won't be parsed. It will potentially cause $HIVE_LIB not > resolved correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3682) Error while building sample cube with spark ava.lang.NoClassDefFoundError: Could not initialize class org.apache.kylin.metadata.datatype.DataType
[
https://issues.apache.org/jira/browse/KYLIN-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692557#comment-16692557
]
Shaofeng SHI commented on KYLIN-3682:
-
Hi Gaurav, what's your spark version? Can you export SPARK_HOME to
KYLIN_HOME/spark and try again?
> Error while building sample cube with spark ava.lang.NoClassDefFoundError:
> Could not initialize class org.apache.kylin.metadata.datatype.DataType
> -
>
> Key: KYLIN-3682
> URL: https://issues.apache.org/jira/browse/KYLIN-3682
> Project: Kylin
> Issue Type: Bug
> Components: Integration
>Affects Versions: v2.5.0
> Environment: test
>Reporter: Gaurav Rawat
>Priority: Major
> Labels: starter
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> HI I am having error at step #7 when building with Spark on AWS EMR, with
> mapreduce the same builds fine . The error I am getting is as below :-
> {code:java}
> 18/11/12 21:45:01 INFO yarn.Client:
>client token: N/A
>diagnostics: User class threw exception: java.lang.RuntimeException:
> error execute org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root
> cause: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times,
> most recent failure: Lost task 1.3 in stage 0.0 (TID 8,
> ip-10-81-117-151.wfg1tst.cltest.wellmanage.com, executor 1):
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.kylin.metadata.datatype.DataType
> at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
> at
> java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1787)
> at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:72)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:253)
> at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:251)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:250)
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:611)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1933)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1529)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
> at java.util.HashSet.readObject(HashSet.java:333)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2136)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
> at
>
[jira] [Resolved] (KYLIN-3683) package org.apache.commons.lang3 not exists
[ https://issues.apache.org/jira/browse/KYLIN-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3683. - Resolution: Fixed > package org.apache.commons.lang3 not exists > --- > > Key: KYLIN-3683 > URL: https://issues.apache.org/jira/browse/KYLIN-3683 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.5.0 >Reporter: Hongtao He >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.2 > > Attachments: > 0001-fix-bug-package-org.apache.commons.lang3-not-exists.patch > > > When I want run unit test, I got the following error message. > package org.apache.commons.lang3 not exists > at > kylin/atopcalcite/src/main/java/org/apache/calcite/runtime/SqlFunctions.java. > I solved this problem by adding dependencies in the atopcalcite/pom.xml file. > Of course, refactor the SqlFunctions.java is another way to solve the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3683) package org.apache.commons.lang3 not exists
[ https://issues.apache.org/jira/browse/KYLIN-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3683. --- Resolution: Fixed Fix Version/s: v2.5.2 > package org.apache.commons.lang3 not exists > --- > > Key: KYLIN-3683 > URL: https://issues.apache.org/jira/browse/KYLIN-3683 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.5.0 >Reporter: Hongtao He >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.2 > > Attachments: > 0001-fix-bug-package-org.apache.commons.lang3-not-exists.patch > > > When I want run unit test, I got the following error message. > package org.apache.commons.lang3 not exists > at > kylin/atopcalcite/src/main/java/org/apache/calcite/runtime/SqlFunctions.java. > I solved this problem by adding dependencies in the atopcalcite/pom.xml file. > Of course, refactor the SqlFunctions.java is another way to solve the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-3683) package org.apache.commons.lang3 not exists
[ https://issues.apache.org/jira/browse/KYLIN-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reopened KYLIN-3683: - > package org.apache.commons.lang3 not exists > --- > > Key: KYLIN-3683 > URL: https://issues.apache.org/jira/browse/KYLIN-3683 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.5.0 >Reporter: Hongtao He >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.2 > > Attachments: > 0001-fix-bug-package-org.apache.commons.lang3-not-exists.patch > > > When I want run unit test, I got the following error message. > package org.apache.commons.lang3 not exists > at > kylin/atopcalcite/src/main/java/org/apache/calcite/runtime/SqlFunctions.java. > I solved this problem by adding dependencies in the atopcalcite/pom.xml file. > Of course, refactor the SqlFunctions.java is another way to solve the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3684) [Script] find-hive-dependency.sh HIVE_LIB is not set or not resolved correctly
[ https://issues.apache.org/jira/browse/KYLIN-3684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3684: --- Assignee: PENG Zhengshuai > [Script] find-hive-dependency.sh HIVE_LIB is not set or not resolved > correctly > --- > > Key: KYLIN-3684 > URL: https://issues.apache.org/jira/browse/KYLIN-3684 > Project: Kylin > Issue Type: Bug > Components: Environment >Reporter: PENG Zhengshuai >Assignee: PENG Zhengshuai >Priority: Major > Fix For: v2.6.0 > > > There is a bug when executing find-hive-dependency.sh, if $HIVE_CONF has been > set, $hive_exec_path won't be parsed. It will potentially cause $HIVE_LIB not > resolved correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3290: --- Assignee: Shaofeng SHI (was: Jiatao Tao) > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3685) AWS Glue Catalog Not Supported
[ https://issues.apache.org/jira/browse/KYLIN-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692536#comment-16692536 ] Shaofeng SHI commented on KYLIN-3685: - Hi Richard, thanks for letting us know this. Have you find the solution for this issue? Anything we can do from Kylin side to support this case? If want to contribute a patch, that will be great! > AWS Glue Catalog Not Supported > -- > > Key: KYLIN-3685 > URL: https://issues.apache.org/jira/browse/KYLIN-3685 > Project: Kylin > Issue Type: Bug > Components: Integration >Affects Versions: v2.5.0 >Reporter: Richard Jarvis >Priority: Major > > I am trying to use Kylin on AWS (EMR 5.18.0). > I use AWS Glue as the catalog and as a result Kylin can't find the tables. > I am able to see the schemas and tables in the GUI because I have set the AWS > glue properties in hive-site.xml: > > > hive.metastore.client.factory.class > com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory > > However, the job > org.apache.kylin.source.hive.cardinality.HiveColumnCardinalityJob fails to > find the tables (it's looking in the Hive metadata catalog instead of AWS > Glue). > I think this is because Hive 1.2.1 is too old to support the client factory > class. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692533#comment-16692533
]
ASF subversion and git services commented on KYLIN-3689:
Commit 2f9e53f27f5df9876b07ea5cb7a8ee5ac9b46c26 in kylin's branch
refs/heads/2.3.x from 贺红涛
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2f9e53f ]
KYLIN-3689 When the startTime is equal to the endTime in build request, the
segment will build all data.
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692543#comment-16692543 ] ASF GitHub Bot commented on KYLIN-3631: --- coveralls edited a comment on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439873914 ## Pull Request Test Coverage Report for [Build 3860](https://coveralls.io/builds/20193220) * **1** of **2** **(50.0%)** changed or added relevant lines in **2** files are covered. * **7** unchanged lines in **3** files lost coverage. * Overall coverage decreased (**-0.003%**) to **25.913%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java](https://coveralls.io/builds/20193220/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FSegments.java#L223) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/20193220/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/20193220/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/20193220/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 4 | 78.12% | | Totals | [](https://coveralls.io/builds/20193220) | | :-- | --: | | Change from base [Build 3853](https://coveralls.io/builds/20137109): | -0.003% | | Covered Lines: | 17645 | | Relevant Lines: | 68093 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] coveralls edited a comment on issue #350: Fix issue : KYLIN-3631
coveralls edited a comment on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439873914 ## Pull Request Test Coverage Report for [Build 3860](https://coveralls.io/builds/20193220) * **1** of **2** **(50.0%)** changed or added relevant lines in **2** files are covered. * **7** unchanged lines in **3** files lost coverage. * Overall coverage decreased (**-0.003%**) to **25.913%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java](https://coveralls.io/builds/20193220/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FSegments.java#L223) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/20193220/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/20193220/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/20193220/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 4 | 78.12% | | Totals | [](https://coveralls.io/builds/20193220) | | :-- | --: | | Change from base [Build 3853](https://coveralls.io/builds/20137109): | -0.003% | | Covered Lines: | 17645 | | Relevant Lines: | 68093 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (KYLIN-3688) 查询问题
[ https://issues.apache.org/jira/browse/KYLIN-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-3688. --- Resolution: Not A Problem Fix Version/s: (was: v2.5.0) > 查询问题 > > > Key: KYLIN-3688 > URL: https://issues.apache.org/jira/browse/KYLIN-3688 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.5.0 >Reporter: shixiao >Priority: Major > Original Estimate: 96h > Remaining Estimate: 96h > > 1.cube 能否导出到关系型数据库？ > 2.有些可视化报表页面需要查询太多，导致刷新页面总时间很长，有没有建议方案？ > > 谢谢 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692531#comment-16692531
]
ASF subversion and git services commented on KYLIN-3689:
Commit b76ec1f17dbf842a8e79df246f962f3dc24202e7 in kylin's branch
refs/heads/2.5.x from 贺红涛
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b76ec1f ]
KYLIN-3689 When the startTime is equal to the endTime in build request, the
segment will build all data.
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692532#comment-16692532
]
ASF subversion and git services commented on KYLIN-3689:
Commit 34bc87638a66da2b42822ccc66ca3789165c4607 in kylin's branch
refs/heads/2.4.x from 贺红涛
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=34bc876 ]
KYLIN-3689 When the startTime is equal to the endTime in build request, the
segment will build all data.
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692530#comment-16692530
]
ASF subversion and git services commented on KYLIN-3689:
Commit 45fb6a2661a9ae92a5766ace97d8a6a51f188f7c in kylin's branch
refs/heads/master from 贺红涛
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=45fb6a2 ]
KYLIN-3689 When the startTime is equal to the endTime in build request, the
segment will build all data.
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3689) When the startTime is equal to the endTime in build request, the segment will build all data.
[
https://issues.apache.org/jira/browse/KYLIN-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3689:
Fix Version/s: v2.5.2
Good catch! Thank you Hongtao!
> When the startTime is equal to the endTime in build request, the segment will
> build all data.
> -
>
> Key: KYLIN-3689
> URL: https://issues.apache.org/jira/browse/KYLIN-3689
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine, Metadata, Query Engine
>Affects Versions: v2.4.0, v2.5.0
>Reporter: Hongtao He
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.2
>
> Attachments:
> 0001-fix-bug-When-the-startTime-is-equal-to-the-endTime-i.patch
>
>
> Hi all:
> We know that kylin's build job often runs late at night, so calling
> kylin's restful API directly is a very common thing. In this scenario, if
> the startTime is equal to the endTime, something unexpected will happen. The
> job calculates all partition data, so the query result is wrong.
> For example, when I send the following request to kylin.
> curl -X PUT -H "Authorization: Basic XXX" -H 'Content-Type:
> application/json' -d '\{"startTime":'154189440',
> "endTime":'154189440', "buildType":"BUILD"}'
> [http://xx:/kylin/api/cubes/cube_name/build|http://xx/]
> then, the job will calculate all partition data. I find the backend interface
> of build lacks checksum data range, and the sql of generating the flat table
> is wrong.
> I try to fix this bug, and the patch is in the attachment.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3290) Avoid calling Class#newInstance
[ https://issues.apache.org/jira/browse/KYLIN-3290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692518#comment-16692518 ] Wu Bin commented on KYLIN-3290: --- Is there anyone looking at this issue? May I take a try? > Avoid calling Class#newInstance > --- > > Key: KYLIN-3290 > URL: https://issues.apache.org/jira/browse/KYLIN-3290 > Project: Kylin > Issue Type: Task >Reporter: Ted Yu >Assignee: Shaofeng SHI >Priority: Minor > Labels: jdk > Fix For: v2.6.0 > > > Class#newInstance is deprecated starting in Java 9 - > https://bugs.openjdk.java.net/browse/JDK-6850612 - because it may throw > undeclared checked exceptions. > The suggested replacement is getDeclaredConstructor().newInstance(), which > wraps the checked exceptions in InvocationException. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691959#comment-16691959 ] ASF GitHub Bot commented on KYLIN-3631: --- coveralls edited a comment on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439873914 ## Pull Request Test Coverage Report for [Build 3859](https://coveralls.io/builds/20183226) * **1** of **2** **(50.0%)** changed or added relevant lines in **2** files are covered. * **3** unchanged lines in **2** files lost coverage. * Overall coverage decreased (**-0.004%**) to **25.912%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java](https://coveralls.io/builds/20183226/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FSegments.java#L223) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/20183226/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/20183226/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | Totals | [](https://coveralls.io/builds/20183226) | | :-- | --: | | Change from base [Build 3853](https://coveralls.io/builds/20137109): | -0.004% | | Covered Lines: | 17644 | | Relevant Lines: | 68093 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] coveralls edited a comment on issue #350: Fix issue : KYLIN-3631
coveralls edited a comment on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439873914 ## Pull Request Test Coverage Report for [Build 3859](https://coveralls.io/builds/20183226) * **1** of **2** **(50.0%)** changed or added relevant lines in **2** files are covered. * **3** unchanged lines in **2** files lost coverage. * Overall coverage decreased (**-0.004%**) to **25.912%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java](https://coveralls.io/builds/20183226/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FSegments.java#L223) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/20183226/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/20183226/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | Totals | [](https://coveralls.io/builds/20183226) | | :-- | --: | | Change from base [Build 3853](https://coveralls.io/builds/20137109): | -0.004% | | Covered Lines: | 17644 | | Relevant Lines: | 68093 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691919#comment-16691919 ] ASF GitHub Bot commented on KYLIN-3631: --- whuwb commented on a change in pull request #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#discussion_r234681139 ## File path: core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java ## @@ -118,8 +118,13 @@ private void testSpanningAndGetParent(CuboidScheduler scheduler, CubeDesc cube, } long[] spanningsArray = Longs.toArray(totalSpanning); -Arrays.sort(spanningsArray); -Arrays.sort(expectChildren); + +/* +According to issue : https://issues.apache.org/jira/browse/KYLIN-3631 Review comment: Sure This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] whuwb commented on a change in pull request #350: Fix issue : KYLIN-3631
whuwb commented on a change in pull request #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#discussion_r234681139 ## File path: core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java ## @@ -118,8 +118,13 @@ private void testSpanningAndGetParent(CuboidScheduler scheduler, CubeDesc cube, } long[] spanningsArray = Longs.toArray(totalSpanning); -Arrays.sort(spanningsArray); -Arrays.sort(expectChildren); + +/* +According to issue : https://issues.apache.org/jira/browse/KYLIN-3631 Review comment: Sure This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (KYLIN-3666) Mege cube step 2: Update dictionary throws IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3666:
Summary: Mege cube step 2: Update dictionary throws IllegalStateException
(was: MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException)
> Mege cube step 2: Update dictionary throws IllegalStateException
>
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.0
> Environment: cdh5.11.0
>Reporter: Xi Chen
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.3
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at
> org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3666) MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
[
https://issues.apache.org/jira/browse/KYLIN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691769#comment-16691769
]
ASF GitHub Bot commented on KYLIN-3666:
---
shaofengshi commented on a change in pull request #347: KYLIN-3666 HDFS
metadata url not be recognized
URL: https://github.com/apache/kylin/pull/347#discussion_r234639228
##
File path:
engine-mr/src/main/java/org/apache/kylin/engine/mr/common/AbstractHadoopJob.java
##
@@ -494,6 +494,17 @@ public static KylinConfig
loadKylinConfigFromHdfs(SerializableConfiguration conf
}
public static KylinConfig loadKylinConfigFromHdfs(String uri) {
+KylinConfig config = loadKylinConfigFromHdfsNotThreadLocal(uri);
+
+// This is a bad example where the thread local KylinConfig cannot be
auto-closed due to
+// limitation of MR API. It works because MR task runs its own
process. Do not copy.
+@SuppressWarnings("unused")
+SetAndUnsetThreadLocalConfig shouldAutoClose =
KylinConfig.setAndUnsetThreadLocalConfig(config);
+
+return config;
+}
+
+public static KylinConfig loadKylinConfigFromHdfsNotThreadLocal(String
uri) {
Review comment:
The method name is confusing. I think the root cause is the bad sample in
AbstractHadoopJob.loadKylinConfigFromHdfs(), we should fix that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> MERGE CUBE Step 2 Update Dictionary Data Throws IllegalStateException
> -
>
> Key: KYLIN-3666
> URL: https://issues.apache.org/jira/browse/KYLIN-3666
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.5.0
> Environment: cdh5.11.0
>Reporter: Xi Chen
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.5.3
>
>
> MERGE CUBE job failed in Step 2, throwing such exception:
>
> {code:java}
> java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> is not recognized
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:172)
> at
> org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:182)
> at
> org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:70)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.IllegalStateException: Metadata uri :
> kylin_metadata_my_service@hdfs,path=viewfs://hadoop-footest/kylin/kylin_metadata_my_service/kylin-9583f16b-93cb-9461-a58c-43c848007972/kylin_sales_cube_clone_spark/metadata
> looks like a file but it's neither a file nor a directory
> at org.apache.kylin.common.KylinConfig.decideUriType(KylinConfig.java:162)
> ... 9 more
> {code}
> The metadataUrl is on hdfs, but decideUriType() method does not recognize it.
> Thus throwing such exception.
> Both MR and Spark engine are affected.
> Would [~Wayne0101] please have a look? I see it's related to KYLIN-3471
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3636) in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
[
https://issues.apache.org/jira/browse/KYLIN-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691765#comment-16691765
]
KANG-SEN LU commented on KYLIN-3636:
Hi, Shaofeng:
I just created a new cube design, the storage_type was set to 0. I am sending
you the json file of the cube here:
* [Grid|http://192.168.230.62:7070/kylin/]
* [SQL|http://192.168.230.62:7070/kylin/]
* [JSON(Cube)|http://192.168.230.62:7070/kylin/]
* [Notification|http://192.168.230.62:7070/kylin/]
* [Storage|http://192.168.230.62:7070/kylin/]
* [Planner|http://192.168.230.62:7070/kylin/]
{
"uuid": "8b9c51f3-e9b6-45ae-998c-c66fbf87dcad",
"last_modified": 1542636675073,
"version": "2.5.1.20500",
"name": "test",
"is_draft": false,
"model_name": "ma_aggs_model",
"description": "",
"null_string": null,
"dimensions": [
\{
"name": "APPLICATION_NAME",
"table": "A_MA_HOURLY_V",
"column": "APPLICATION_NAME",
"derived": null
},
\{
"name": "BRAND_NAME",
"table": "A_MA_HOURLY_V",
"column": "BRAND_NAME",
"derived": null
}
],
"measures": [
\{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter": {
"type": "constant",
"value": "1"
},
"returntype": "bigint"
}
}
],
"dictionaries": [],
"rowkey": \{
"rowkey_columns": [
{
"column": "A_MA_HOURLY_V.APPLICATION_NAME",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
},
\{
"column": "A_MA_HOURLY_V.BRAND_NAME",
"encoding": "dict",
"encoding_version": 1,
"isShardBy": false
}
]
},
"hbase_mapping": \{
"column_family": [
{
"name": "F1",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"_COUNT_"
]
}
]
}
]
},
"aggregation_groups": [
\{
"includes": [
"A_MA_HOURLY_V.APPLICATION_NAME",
"A_MA_HOURLY_V.BRAND_NAME"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [],
"joint_dims": []
}
}
],
"signature": "iBgKI2sCq9L9zGNctnmryw==",
"notify_list": [],
"status_need_notify": [
"ERROR",
"DISCARDED",
"SUCCEED"
],
"partition_date_start": 0,
"partition_date_end": 31536,
"auto_merge_time_ranges": [
60480,
241920
],
"volatile_range": 0,
"retention_range": 0,
"engine_type": 2,
"storage_type": 0,
"override_kylin_properties": {},
"cuboid_black_list": [],
"parent_forward": 3,
"mandatory_dimension_set_list": [],
"snapshot_table_desc_list": []
}
Here is the java code in our kylin 2.5.1 sandbox:
4af0f33248 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(honma 2015-09-09 10:01:55 +0800 180) @JsonProperty("engine_type")
c56c741a92 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(shaofengshi 2017-11-05 16:57:47 +0800 181) private int engineType =
IEngineAware.ID_MR_V2;
4af0f33248 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(honma 2015-09-09 10:01:55 +0800 182)
@JsonProperty("storage_type")
4af0f33248 core-cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
(honma 2015-09-09 10:01:55 +0800 183) private int storageType =
IStorageAware.ID_HBASE;
> in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
>
>
> Key: KYLIN-3636
> URL: https://issues.apache.org/jira/browse/KYLIN-3636
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.4.1
>Reporter: KANG-SEN LU
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.6.0
>
>
> Hi, ShaoFeng:
>
> Thanks for the reply. I missed this email and not responded earlier, I am
> sorry.
>
> I tried to reproduce this problem with the sample database, and it did not
> happen.
>
> So I am hoping by collecting enough "clue", someone can figure out why this
> problem occurred.
>
> --
> I issued the following query at the sample project to exercise the topn
> aggregation:
>
> select seller_id, SUM(price) as total from kylin_sales group by seller_id
> order by total limit 5;
>
> With my own added debugging, I saw the following log in the kylin.log: (the
> query worked OK).
>
> 2018-10-16 16:18:19,963 INFO [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90]
> model.FunctionDesc:59 : KSL2,
> getRewriteFieldName=_KY_SUM_KYLIN_SALES_PRICE_
> 2018-10-16 16:18:19,963 INFO [kylin-coproc--pool12-t1]
> v2.CubeHBaseEndpointRPC:217 : Query-a747f16f-4b12-cc97-08d2-9b45c27a529f:
>
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691757#comment-16691757 ] ASF GitHub Bot commented on KYLIN-3631: --- shaofengshi commented on a change in pull request #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#discussion_r234635124 ## File path: core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java ## @@ -118,8 +118,13 @@ private void testSpanningAndGetParent(CuboidScheduler scheduler, CubeDesc cube, } long[] spanningsArray = Longs.toArray(totalSpanning); -Arrays.sort(spanningsArray); -Arrays.sort(expectChildren); + +/* +According to issue : https://issues.apache.org/jira/browse/KYLIN-3631 Review comment: This comment is not needed. Can you remove it in all the places? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] shaofengshi commented on a change in pull request #350: Fix issue : KYLIN-3631
shaofengshi commented on a change in pull request #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#discussion_r234635124 ## File path: core-cube/src/test/java/org/apache/kylin/cube/cuboid/CuboidSchedulerTest.java ## @@ -118,8 +118,13 @@ private void testSpanningAndGetParent(CuboidScheduler scheduler, CubeDesc cube, } long[] spanningsArray = Longs.toArray(totalSpanning); -Arrays.sort(spanningsArray); -Arrays.sort(expectChildren); + +/* +According to issue : https://issues.apache.org/jira/browse/KYLIN-3631 Review comment: This comment is not needed. Can you remove it in all the places? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691634#comment-16691634 ] ASF GitHub Bot commented on KYLIN-3631: --- codecov-io commented on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439874742 # [Codecov](https://codecov.io/gh/apache/kylin/pull/350?src=pr=h1) Report > Merging [#350](https://codecov.io/gh/apache/kylin/pull/350?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/634498649f20ce61fcabe0b46fe515bb6ca96af0?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/kylin/pull/350?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #350 +/- ## - Coverage 23.62% 23.61% -0.01% + Complexity 4823 4822 -1 Files 1133 1133 Lines 6809368093 Branches 9657 9657 - Hits 1608716081 -6 - Misses5044550449 +4 - Partials 1561 1563 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/350?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...java/org/apache/kylin/metadata/model/Segments.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvU2VnbWVudHMuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ava/org/apache/kylin/metadata/model/TableDesc.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvVGFibGVEZXNjLmphdmE=) | `47.77% <100%> (ø)` | `27 <0> (ø)` | :arrow_down: | | [...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=) | `70.94% <0%> (-1.36%)` | `8% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/350?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/350?src=pr=footer). Last update [6344986...c47dd9f](https://codecov.io/gh/apache/kylin/pull/350?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #350: Fix issue : KYLIN-3631
codecov-io commented on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439874742 # [Codecov](https://codecov.io/gh/apache/kylin/pull/350?src=pr=h1) Report > Merging [#350](https://codecov.io/gh/apache/kylin/pull/350?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/634498649f20ce61fcabe0b46fe515bb6ca96af0?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `50%`. [](https://codecov.io/gh/apache/kylin/pull/350?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #350 +/- ## - Coverage 23.62% 23.61% -0.01% + Complexity 4823 4822 -1 Files 1133 1133 Lines 6809368093 Branches 9657 9657 - Hits 1608716081 -6 - Misses5044550449 +4 - Partials 1561 1563 +2 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/350?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...java/org/apache/kylin/metadata/model/Segments.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvU2VnbWVudHMuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...ava/org/apache/kylin/metadata/model/TableDesc.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvbW9kZWwvVGFibGVEZXNjLmphdmE=) | `47.77% <100%> (ø)` | `27 <0> (ø)` | :arrow_down: | | [...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=) | `70.94% <0%> (-1.36%)` | `8% <0%> (ø)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (-0.92%)` | `7% <0%> (ø)` | | | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/350/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/350?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/350?src=pr=footer). Last update [6344986...c47dd9f](https://codecov.io/gh/apache/kylin/pull/350?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691627#comment-16691627 ] ASF GitHub Bot commented on KYLIN-3631: --- coveralls commented on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439873914 ## Pull Request Test Coverage Report for [Build 3858](https://coveralls.io/builds/20178446) * **1** of **2** **(50.0%)** changed or added relevant lines in **2** files are covered. * **4** unchanged lines in **2** files lost coverage. * Overall coverage increased (+**0.001%**) to **25.917%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java](https://coveralls.io/builds/20178446/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FSegments.java#L227) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-metadata/src/main/java/org/apache/kylin/source/datagen/ColumnGenerator.java](https://coveralls.io/builds/20178446/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fdatagen%2FColumnGenerator.java#L319) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/20178446/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/20178446) | | :-- | --: | | Change from base [Build 3853](https://coveralls.io/builds/20137109): | 0.001% | | Covered Lines: | 17648 | | Relevant Lines: | 68093 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] coveralls commented on issue #350: Fix issue : KYLIN-3631
coveralls commented on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439873914 ## Pull Request Test Coverage Report for [Build 3858](https://coveralls.io/builds/20178446) * **1** of **2** **(50.0%)** changed or added relevant lines in **2** files are covered. * **4** unchanged lines in **2** files lost coverage. * Overall coverage increased (+**0.001%**) to **25.917%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/metadata/model/Segments.java](https://coveralls.io/builds/20178446/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FSegments.java#L227) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-metadata/src/main/java/org/apache/kylin/source/datagen/ColumnGenerator.java](https://coveralls.io/builds/20178446/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fdatagen%2FColumnGenerator.java#L319) | 1 | 81.08% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/20178446/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/20178446) | | :-- | --: | | Change from base [Build 3853](https://coveralls.io/builds/20137109): | 0.001% | | Covered Lines: | 17648 | | Relevant Lines: | 68093 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691604#comment-16691604 ] ASF GitHub Bot commented on KYLIN-3631: --- asfgit commented on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439868082 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
[ https://issues.apache.org/jira/browse/KYLIN-3631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691603#comment-16691603 ] ASF GitHub Bot commented on KYLIN-3631: --- whuwb opened a new pull request #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350 Issue : https://issues.apache.org/jira/browse/KYLIN-3631 How to fix : 1. Find all the Arrays.sort and replace with Arrays.parallelSort and add comments. 2. Run UT & integration tests to validate the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Utilize Arrays#parallelSort for better performance > -- > > Key: KYLIN-3631 > URL: https://issues.apache.org/jira/browse/KYLIN-3631 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Wu Bin >Priority: Minor > Fix For: v2.6.0 > > > Arrays#parallelSort was introduced since Java 1.8 > We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit commented on issue #350: Fix issue : KYLIN-3631
asfgit commented on issue #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350#issuecomment-439868082 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] whuwb opened a new pull request #350: Fix issue : KYLIN-3631
whuwb opened a new pull request #350: Fix issue : KYLIN-3631 URL: https://github.com/apache/kylin/pull/350 Issue : https://issues.apache.org/jira/browse/KYLIN-3631 How to fix : 1. Find all the Arrays.sort and replace with Arrays.parallelSort and add comments. 2. Run UT & integration tests to validate the fix. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3683) package org.apache.commons.lang3 not exists
[ https://issues.apache.org/jira/browse/KYLIN-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691446#comment-16691446 ] Hongtao He commented on KYLIN-3683: --- Hi Shaofeng, It works well in the latest version of Kylin. Thank you! > package org.apache.commons.lang3 not exists > --- > > Key: KYLIN-3683 > URL: https://issues.apache.org/jira/browse/KYLIN-3683 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.5.0 >Reporter: Hongtao He >Assignee: Shaofeng SHI >Priority: Major > Attachments: > 0001-fix-bug-package-org.apache.commons.lang3-not-exists.patch > > > When I want run unit test, I got the following error message. > package org.apache.commons.lang3 not exists > at > kylin/atopcalcite/src/main/java/org/apache/calcite/runtime/SqlFunctions.java. > I solved this problem by adding dependencies in the atopcalcite/pom.xml file. > Of course, refactor the SqlFunctions.java is another way to solve the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3694) Kylin On Druid Storage
[ https://issues.apache.org/jira/browse/KYLIN-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691440#comment-16691440 ] ASF GitHub Bot commented on KYLIN-3694: --- coveralls commented on issue #349: KYLIN-3694 Kylin On Druid Storage URL: https://github.com/apache/kylin/pull/349#issuecomment-439823023 ## Pull Request Test Coverage Report for [Build 3857](https://coveralls.io/builds/20175448) * **3** of **2886** **(0.1%)** changed or added relevant lines in **86** files are covered. * **17** unchanged lines in **8** files lost coverage. * Overall coverage decreased (**-1.1%**) to **24.816%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/RoaringBitmapCounter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FRoaringBitmapCounter.java#L64) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/RoaringBitmapCounterFactory.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FRoaringBitmapCounterFactory.java#L56) | 1 | 2 | 50.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/CaseTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FCaseTupleFilter.java#L154) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/ColumnTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FColumnTupleFilter.java#L161) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/ConstantTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FConstantTupleFilter.java#L128) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/DynamicTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FDynamicTupleFilter.java#L83) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/ExtractTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FExtractTupleFilter.java#L127) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/FunctionTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FFunctionTupleFilter.java#L31) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FLogicalTupleFilter.java#L201) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/UnsupportedTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FUnsupportedTupleFilter.java#L62) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [query/src/main/java/org/apache/kylin/query/relnode/OLAPProjectRel.java](https://coveralls.io/builds/20175448/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Frelnode%2FOLAPProjectRel.java#L204) | 1 | 0.0% | | [core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java](https://coveralls.io/builds/20175448/source?filename=core-storage%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2FStorageContext.java#L77) | 1 | 0.0% | | [server-base/src/main/java/org/apache/kylin/rest/service/CubeService.java](https://coveralls.io/builds/20175448/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FCubeService.java#L222) | 1 | 0.0% | | [core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2FMeasureCodec.java#L94) | 1 | 0.0% | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/20175448/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | |
[GitHub] coveralls commented on issue #349: KYLIN-3694 Kylin On Druid Storage
coveralls commented on issue #349: KYLIN-3694 Kylin On Druid Storage URL: https://github.com/apache/kylin/pull/349#issuecomment-439823023 ## Pull Request Test Coverage Report for [Build 3857](https://coveralls.io/builds/20175448) * **3** of **2886** **(0.1%)** changed or added relevant lines in **86** files are covered. * **17** unchanged lines in **8** files lost coverage. * Overall coverage decreased (**-1.1%**) to **24.816%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/RoaringBitmapCounter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FRoaringBitmapCounter.java#L64) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/measure/bitmap/RoaringBitmapCounterFactory.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbitmap%2FRoaringBitmapCounterFactory.java#L56) | 1 | 2 | 50.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/CaseTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FCaseTupleFilter.java#L154) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/ColumnTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FColumnTupleFilter.java#L161) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/ConstantTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FConstantTupleFilter.java#L128) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/DynamicTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FDynamicTupleFilter.java#L83) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/ExtractTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FExtractTupleFilter.java#L127) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/FunctionTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FFunctionTupleFilter.java#L31) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/LogicalTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FLogicalTupleFilter.java#L201) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/metadata/filter/UnsupportedTupleFilter.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Ffilter%2FUnsupportedTupleFilter.java#L62) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [query/src/main/java/org/apache/kylin/query/relnode/OLAPProjectRel.java](https://coveralls.io/builds/20175448/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Frelnode%2FOLAPProjectRel.java#L204) | 1 | 0.0% | | [core-storage/src/main/java/org/apache/kylin/storage/StorageContext.java](https://coveralls.io/builds/20175448/source?filename=core-storage%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2FStorageContext.java#L77) | 1 | 0.0% | | [server-base/src/main/java/org/apache/kylin/rest/service/CubeService.java](https://coveralls.io/builds/20175448/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FCubeService.java#L222) | 1 | 0.0% | | [core-metadata/src/main/java/org/apache/kylin/measure/MeasureCodec.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2FMeasureCodec.java#L94) | 1 | 0.0% | | [core-dictionary/src/main/java/org/apache/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://coveralls.io/builds/20175448/source?filename=core-dictionary%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fdict%2Flookup%2Fcache%2FRocksDBLookupTable.java#L62) | 1 | 81.08% | | [core-metadata/src/main/java/org/apache/kylin/source/datagen/ColumnGenerator.java](https://coveralls.io/builds/20175448/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fsource%2Fdatagen%2FColumnGenerator.java#L319) | 1 | 81.08% | |
[jira] [Commented] (KYLIN-3694) Kylin On Druid Storage
[ https://issues.apache.org/jira/browse/KYLIN-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691415#comment-16691415 ] ASF GitHub Bot commented on KYLIN-3694: --- hit-lacus opened a new pull request #349: KYLIN-3694 Kylin On Druid Storage URL: https://github.com/apache/kylin/pull/349 **Meituan** Kylin team has implemented a new storage engine for Kylin: Druid Storage Engine This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kylin On Druid Storage > -- > > Key: KYLIN-3694 > URL: https://issues.apache.org/jira/browse/KYLIN-3694 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, Metadata, Query Engine >Affects Versions: v2.5.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: Kylin On Druid Storage.pdf > > > Meituan Kylin team has implemented a new storage engine for Kylin: Druid > Storage Engine. > The attach file is the Kylin On Druid Storage Engine architecture design > doc. > We would like to contribute the feature to community, please let us know if > you have any concern. [^Kylin On Druid Storage.pdf] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3694) Kylin On Druid Storage
[ https://issues.apache.org/jira/browse/KYLIN-3694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691416#comment-16691416 ] ASF GitHub Bot commented on KYLIN-3694: --- asfgit commented on issue #349: KYLIN-3694 Kylin On Druid Storage URL: https://github.com/apache/kylin/pull/349#issuecomment-439816566 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Kylin On Druid Storage > -- > > Key: KYLIN-3694 > URL: https://issues.apache.org/jira/browse/KYLIN-3694 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, Metadata, Query Engine >Affects Versions: v2.5.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: Kylin On Druid Storage.pdf > > > Meituan Kylin team has implemented a new storage engine for Kylin: Druid > Storage Engine. > The attach file is the Kylin On Druid Storage Engine architecture design > doc. > We would like to contribute the feature to community, please let us know if > you have any concern. [^Kylin On Druid Storage.pdf] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] hit-lacus opened a new pull request #349: KYLIN-3694 Kylin On Druid Storage
hit-lacus opened a new pull request #349: KYLIN-3694 Kylin On Druid Storage URL: https://github.com/apache/kylin/pull/349 **Meituan** Kylin team has implemented a new storage engine for Kylin: Druid Storage Engine This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #349: KYLIN-3694 Kylin On Druid Storage
asfgit commented on issue #349: KYLIN-3694 Kylin On Druid Storage URL: https://github.com/apache/kylin/pull/349#issuecomment-439816566 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3683) package org.apache.commons.lang3 not exists
[ https://issues.apache.org/jira/browse/KYLIN-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691403#comment-16691403 ] Shaofeng SHI commented on KYLIN-3683: - Hi Hongtao, thanks for the reporting! In the latest version of Kylin, the "atopcalcite" module is removed. So this patch couldn't be applied now, could you please check whether you have the similar problem with the latest master code base? Thank you! > package org.apache.commons.lang3 not exists > --- > > Key: KYLIN-3683 > URL: https://issues.apache.org/jira/browse/KYLIN-3683 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.5.0 >Reporter: Hongtao He >Assignee: Shaofeng SHI >Priority: Major > Attachments: > 0001-fix-bug-package-org.apache.commons.lang3-not-exists.patch > > > When I want run unit test, I got the following error message. > package org.apache.commons.lang3 not exists > at > kylin/atopcalcite/src/main/java/org/apache/calcite/runtime/SqlFunctions.java. > I solved this problem by adding dependencies in the atopcalcite/pom.xml file. > Of course, refactor the SqlFunctions.java is another way to solve the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3636) in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
[
https://issues.apache.org/jira/browse/KYLIN-3636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691399#comment-16691399
]
Shaofeng SHI commented on KYLIN-3636:
-
Hi [~kangsenlu], can you reproduce the case "the default storage-type was set
to 0"? We didn't have this issue, but there were some reporting in the
community, I'm curious the root cause. Thank you!
> in kylin 2.4.1 and 2.5.1 topn aggregation query caused exception
>
>
> Key: KYLIN-3636
> URL: https://issues.apache.org/jira/browse/KYLIN-3636
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.4.1
>Reporter: KANG-SEN LU
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.6.0
>
>
> Hi, ShaoFeng:
>
> Thanks for the reply. I missed this email and not responded earlier, I am
> sorry.
>
> I tried to reproduce this problem with the sample database, and it did not
> happen.
>
> So I am hoping by collecting enough "clue", someone can figure out why this
> problem occurred.
>
> --
> I issued the following query at the sample project to exercise the topn
> aggregation:
>
> select seller_id, SUM(price) as total from kylin_sales group by seller_id
> order by total limit 5;
>
> With my own added debugging, I saw the following log in the kylin.log: (the
> query worked OK).
>
> 2018-10-16 16:18:19,963 INFO [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90]
> model.FunctionDesc:59 : KSL2,
> getRewriteFieldName=_KY_SUM_KYLIN_SALES_PRICE_
> 2018-10-16 16:18:19,963 INFO [kylin-coproc--pool12-t1]
> v2.CubeHBaseEndpointRPC:217 : Query-a747f16f-4b12-cc97-08d2-9b45c27a529f:
> send request to the init region server anovadata4.anovadata.local on table
> ANOVA_KYLIN_25X_K758MEAWJG
> 2018-10-16 16:18:19,963 INFO [Query a747f16f-4b12-cc97-08d2-9b45c27a529f-90]
> topn.TopNMeasureType:399 : KSL888: in TopNMeasureType.java, sumFieldName=
> _KY_SUM_KYLIN_SALES_PRICE_
>
>
> When I was executing my project query, I issued the following select
> statement:
>
> SELECT ZETTICSDW.A_VL_HOURLY_V.IMSIID \"ZETTICSDW_A_VL_HOURLY_V_IMSIID\",
> SUM(ZETTICSDW.A_VL_HOURLY_V.SIG_EVENT_COUNT)
> \"vl_aggs_model___USERS_BY_ERROR_3XX\" FROM ZETTICSDW.A_VL_HOURLY_V inner
> JOIN ZETTICSDW.T_VL_TRANSACTION_RULE_V ON (
> ZETTICSDW.A_VL_HOURLY_V.CAUSE_CODE_KEY =
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.CAUSE_CODE_KEY AND
> ZETTICSDW.A_VL_HOURLY_V.REASON_CODE_KEY =
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.REASON_CODE_KEY AND
> ZETTICSDW.A_VL_HOURLY_V.TRANSACTION_TYPE_KEY =
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.TRANSACTION_TYPE_KEY)
> WHERE ((ZETTICSDW.A_VL_HOURLY_V.THEDATE = '20180209') AND
> ((ZETTICSDW.A_VL_HOURLY_V.THEHOUR >= '02') AND
> (ZETTICSDW.A_VL_HOURLY_V.THEHOUR <= '03'))) AND
> ZETTICSDW.T_VL_TRANSACTION_RULE_V.DISPLAY_STRING LIKE '%+3%'
> GROUP BY ZETTICSDW.A_VL_HOURLY_V.IMSIID
> ORDER BY \"vl_aggs_model___USERS_BY_ERROR_3XX\"
> LIMIT 25
>
> An exception occurred within the method "private ColumnRowType
> buildColumnRowType()" of
> "query/src/main/java/org/apache/kylin/query/relnode/OLAPTableScan.java".
>
> if (columns.size() != rowType.getFieldCount()) {
> throw new IllegalStateException("RowType=" +
> rowType.getFieldCount() + ", ColumnRowType=" + columns.size());
> }
>
> It printed "RowType=133, ColumnRowType=132".
> The RowType list contains one extra column name: "ANY
> _KY_SUM_1_3a1aedef_SIG_EVENT_COUNT_".
>
>
> I think this has something to do with the bug fix "KYLIN-3359 Support
> sum(expression) if possible".
>
> After this bug fix was submitted, I noticed that a lot of column name was
> added into rowType like "_KY_SUM_XXX".
>
> This strange column name "_KY_SUM_1_3a1aedef_SIG_EVENT_COUNT_" is very
> similar.
>
> I also found that this extra column name, only existed in RowType but not in
> ColumnRowType, was added in the method "public void
> implementRewrite(RewriteImplementor implementor)" within "OLAPJoinRel.java".
>
> With my own debug statement, I saw this debug text in kylin.log:
>
> relnode.OLAPJoinRel:362 : KSL54: newField= #132:
> _KY_SUM_1_6735969a_SIG_EVENT_COUNT_ ANY
>
>
> I hope someone with a deeper technical knowledge in kylin query engine can
> figure out what was causing the problem I have seen.
>
> Thanks again.
>
> Kang-sen
>
> *From:* ShaoFeng Shi [[mailto:shaofeng...@apache.org]]
> *Sent:* Friday, October 05, 2018 9:59 PM
> *To:* user <[u...@kylin.apache.org|mailto:u...@kylin.apache.org]>
> *Subject:* Re: any body see topn in kylin 2.5.1 working?
>
> Hi Kang-Sen,
>
> Didn't see this; Can you reproduce the problem with the sample cube?
>
> Kang-Sen Lu <[k...@anovadata.com|mailto:k...@anovadata.com]> 于2018年10月6日周六
>
