[jira] [Created] (KYLIN-4785) Unable to have multiple AD groups access restriction

2020-10-10 Thread BKRV (Jira) 
BKRV created KYLIN-4785:
---

 Summary: Unable to have multiple AD groups access restriction
 Key: KYLIN-4785
 URL: https://issues.apache.org/jira/browse/KYLIN-4785
 Project: Kylin
  Issue Type: Bug
  Components: Others
Affects Versions: v3.0.1
 Environment: Production - Edge Node of AWS EMR 5.26
Reporter: BKRV


I have three different user groups in AD requiring different user privileges.

Readonly

Create

Admin

With the current AD integration, I am unable to have access setting privileges.

How to fixthis? Is this is a feature issue?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4775) Use docker-compose to deploy Hadoop and Kylin

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211711#comment-17211711
 ] 

ASF GitHub Bot commented on KYLIN-4775:
---

coveralls edited a comment on pull request #1424:
URL: https://github.com/apache/kylin/pull/1424#issuecomment-698783979


   ## Pull Request Test Coverage Report for [Build 
6441](https://coveralls.io/builds/34077881)
   
   * **0** of **0**   changed or added relevant lines in **0** files are 
covered.
   * **455** unchanged lines in **19** files lost coverage.
   * Overall coverage decreased (**-0.005%**) to **28.051%**
   
   ---
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-|--|--: |
   | 
[core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java](https://coveralls.io/builds/34077881/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FDataModelDesc.java#L832)
 | 1 | 60.58% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/security/KylinUserManager.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinUserManager.java#L144)
 | 1 | 0% |
   | 
[tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/34077881/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L42)
 | 1 | 81.58% |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/34077881/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124)
 | 2 | 68.46% |
   | 
[core-common/src/main/java/org/apache/kylin/common/util/ServerMode.java](https://coveralls.io/builds/34077881/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FServerMode.java#L71)
 | 3 | 0% |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/34077881/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449)
 | 3 | 78.42% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserService.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserService.java#L219)
 | 7 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserGroupService.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserGroupService.java#L159)
 | 9 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/controller/ModelController.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FModelController.java#L259)
 | 10 | 0% |
   | 
[core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/34077881/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L67)
 | 11 | 81.4% |
   
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/34077881/badge)](https://coveralls.io/builds/34077881)
 |
   | :-- | --: |
   | Change from base [Build 6388](https://coveralls.io/builds/33669186): |  
-0.005% |
   | Covered Lines: | 26255 |
   | Relevant Lines: | 93598 |
   
   ---
   #   - [Coveralls](https://coveralls.io)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Use docker-compose to deploy Hadoop and Kylin
> -
>
> Key: KYLIN-4775
> URL: https://issues.apache.org/jira/browse/KYLIN-4775
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Tools, Build and Test
>Reporter: Xiaoxiang Yu
>Assignee: liuyongheng
>Priority: Major
>  Labels: Testing
> Fix For: v3.1.2
>
>
> We know that Kylin has *kylin-it* for intergration test, which cover build, 
> query, tools etc. But some complex features, such as system cube, read write 
> separation deployment, jdbc source can not be covered by intergration test. I 
> wonder if we could automate system level testing to cover these important 
> feature.
> So the first step is to enable automatic deployment for Kylin and Hadoop. I 
> propose to use docker-compose to make it happen.
>  



--
This message was sent by Atlassian Jira

[GitHub] [kylin] coveralls edited a comment on pull request #1424: KYLIN-4775 Use docker-compose to deploy Hadoop and Kylin

2020-10-10 Thread GitBox 


coveralls edited a comment on pull request #1424:
URL: https://github.com/apache/kylin/pull/1424#issuecomment-698783979


   ## Pull Request Test Coverage Report for [Build 
6441](https://coveralls.io/builds/34077881)
   
   * **0** of **0**   changed or added relevant lines in **0** files are 
covered.
   * **455** unchanged lines in **19** files lost coverage.
   * Overall coverage decreased (**-0.005%**) to **28.051%**
   
   ---
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-|--|--: |
   | 
[core-metadata/src/main/java/org/apache/kylin/metadata/model/DataModelDesc.java](https://coveralls.io/builds/34077881/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmetadata%2Fmodel%2FDataModelDesc.java#L832)
 | 1 | 60.58% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/security/KylinUserManager.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinUserManager.java#L144)
 | 1 | 0% |
   | 
[tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/34077881/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L42)
 | 1 | 81.58% |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/34077881/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124)
 | 2 | 68.46% |
   | 
[core-common/src/main/java/org/apache/kylin/common/util/ServerMode.java](https://coveralls.io/builds/34077881/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FServerMode.java#L71)
 | 3 | 0% |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/34077881/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449)
 | 3 | 78.42% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserService.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserService.java#L219)
 | 7 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserGroupService.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserGroupService.java#L159)
 | 9 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/controller/ModelController.java](https://coveralls.io/builds/34077881/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FModelController.java#L259)
 | 10 | 0% |
   | 
[core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/34077881/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L67)
 | 11 | 81.4% |
   
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/34077881/badge)](https://coveralls.io/builds/34077881)
 |
   | :-- | --: |
   | Change from base [Build 6388](https://coveralls.io/builds/33669186): |  
-0.005% |
   | Covered Lines: | 26255 |
   | Relevant Lines: | 93598 |
   
   ---
   #   - [Coveralls](https://coveralls.io)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kylin] hit-lacus commented on pull request #1438: KYLIN-4771 Clear the recordCachePool when the deadline has reached

2020-10-10 Thread GitBox 


hit-lacus commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706552419


   Thank you, that's great.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (KYLIN-4771) Query streaming cube - Thread pool of MultiThreadsResultCollector be blocked.

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211709#comment-17211709
 ] 

ASF GitHub Bot commented on KYLIN-4771:
---

hit-lacus commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706552419


   Thank you, that's great.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Query streaming cube - Thread pool of MultiThreadsResultCollector be blocked.
> -
>
> Key: KYLIN-4771
> URL: https://issues.apache.org/jira/browse/KYLIN-4771
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine, Real-time Streaming
>Affects Versions: v3.1.0
> Environment: Centos 7.4.1
> hbase 1.2.4
> hive 2.0.1
> hadoop 2.7.2
>Reporter: GuKe
>Assignee: GuKe
>Priority: Major
> Fix For: Future
>
> Attachments: image-2020-09-23-11-57-15-795.png, 
> image-2020-09-23-11-57-55-413.png
>
>
> When the receiver query streaming-cube on local it will be blocked for 
> unknown reason.
>  This problem can lead to it the receiver can't response query to request.
> !image-2020-09-23-11-57-15-795.png!
> !image-2020-09-23-11-57-55-413.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4771) Query streaming cube - Thread pool of MultiThreadsResultCollector be blocked.

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211708#comment-17211708
 ] 

ASF GitHub Bot commented on KYLIN-4771:
---

ggKe commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706552143


   > Hello @ggKe , I have checked your patch, your patch look good to me. But 
there are some UT failure, could you please help to fix them?
   > 
   > Following is part of log:
   > 
   > ```
   > [�[1;34mINFO�[m] �[1;32mTests run: �[0;1;32m1�[m, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.242 s - in 
org.apache.kylin.stream.core.storage.�[1mRecordsSerDeTest�[m
   > [�[1;34mINFO�[m] Running 
org.apache.kylin.stream.core.storage.�[1mCheckPointStoreTest�[m
   > client:0 round266 scan finished
   > client:1 round269 scan finished
   > CheckPoint{sourceConsumePosition='{"1":3,"2":4,"3":5}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1602242847871, totalCount=100, checkPointCount=5}
   > going to delete checkpoint file CP-160220160
   > going to delete checkpoint file CP-160228800
   > going to delete checkpoint file CP-160237440
   > going to delete checkpoint file CP-160246080
   > going to delete checkpoint file CP-160254720
   > CheckPoint{sourceConsumePosition='{"1":1,"2":2,"3":3}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1603020447894, totalCount=100, checkPointCount=5}
   > [�[1;34mINFO�[m] �[1;32mTests run: �[0;1;32m3�[m, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.028 s - in 
org.apache.kylin.stream.core.storage.�[1mCheckPointStoreTest�[m
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] Results:
   > [�[1;34mINFO�[m] 
   > [�[1;31mERROR�[m] �[1;31mFailures: �[m
   > [�[1;31mERROR�[m] �[1;31m  
StreamingSegmentManagerTest.testAddEventAndScan:120 expected:<8> but 
was:<30007>�[m
   > [�[1;31mERROR�[m] �[1;31m  StreamingSegmentManagerTest.testIndexFilter:182 
expected:<153298[080]> but was:<153298[104]>�[m
   > [�[1;34mINFO�[m] 
   > [�[1;31mERROR�[m] �[1;31mTests run: 29, Failures: 2, Errors: 0, Skipped: 
0�[m
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] 
�[1m�[m
   > [�[1;34mINFO�[m] �[1mSkipping Apache Kylin�[m
   > [�[1;34mINFO�[m] This project has been banned from the build due to 
previous failures.
   > [�[1;34mINFO�[m] 
�[1m�[m
   > [�[1;34mINFO�[m] 
�[1m�[m
   > [�[1;34mINFO�[m] �[1mReactor Summary:�[m
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] Apache Kylin 3.1.1-SNAPSHOT  
�[1;32mSUCCESS�[m [  3.516 s]
   > [�[1;34mINFO�[m] Apache Kylin - kylin External .. 
�[1;32mSUCCESS�[m [  9.946 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Common . 
�[1;32mSUCCESS�[m [ 22.354 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Metadata ... 
�[1;32mSUCCESS�[m [ 48.745 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Dictionary . 
�[1;32mSUCCESS�[m [03:16 min]
   > [�[1;34mINFO�[m] Apache Kylin - Core Cube ... 
�[1;32mSUCCESS�[m [02:19 min]
   > [�[1;34mINFO�[m] Apache Kylin - Core Metrics  
�[1;32mSUCCESS�[m [  5.364 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Job  
�[1;32mSUCCESS�[m [01:08 min]
   > [�[1;34mINFO�[m] Apache Kylin - Core Storage  
�[1;32mSUCCESS�[m [  9.280 s]
   > [�[1;34mINFO�[m] Apache Kylin - Stream Core . 
�[1;31mFAILURE�[m [01:02 min]
   > [�[1;34mINFO�[m] Apache Kylin - MapReduce Engine  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Spark Engine  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Flink Engine  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Hive Source . 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - DataSource SDK .. 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Jdbc Source . 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Kafka Source  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Cache ... 
�[1;33mSKIPPED�[m
   >

[GitHub] [kylin] ggKe commented on pull request #1438: KYLIN-4771 Clear the recordCachePool when the deadline has reached

2020-10-10 Thread GitBox 


ggKe commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706552143


   > Hello @ggKe , I have checked your patch, your patch look good to me. But 
there are some UT failure, could you please help to fix them?
   > 
   > Following is part of log:
   > 
   > ```
   > [�[1;34mINFO�[m] �[1;32mTests run: �[0;1;32m1�[m, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.242 s - in 
org.apache.kylin.stream.core.storage.�[1mRecordsSerDeTest�[m
   > [�[1;34mINFO�[m] Running 
org.apache.kylin.stream.core.storage.�[1mCheckPointStoreTest�[m
   > client:0 round266 scan finished
   > client:1 round269 scan finished
   > CheckPoint{sourceConsumePosition='{"1":3,"2":4,"3":5}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1602242847871, totalCount=100, checkPointCount=5}
   > going to delete checkpoint file CP-160220160
   > going to delete checkpoint file CP-160228800
   > going to delete checkpoint file CP-160237440
   > going to delete checkpoint file CP-160246080
   > going to delete checkpoint file CP-160254720
   > CheckPoint{sourceConsumePosition='{"1":1,"2":2,"3":3}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1603020447894, totalCount=100, checkPointCount=5}
   > [�[1;34mINFO�[m] �[1;32mTests run: �[0;1;32m3�[m, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.028 s - in 
org.apache.kylin.stream.core.storage.�[1mCheckPointStoreTest�[m
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] Results:
   > [�[1;34mINFO�[m] 
   > [�[1;31mERROR�[m] �[1;31mFailures: �[m
   > [�[1;31mERROR�[m] �[1;31m  
StreamingSegmentManagerTest.testAddEventAndScan:120 expected:<8> but 
was:<30007>�[m
   > [�[1;31mERROR�[m] �[1;31m  StreamingSegmentManagerTest.testIndexFilter:182 
expected:<153298[080]> but was:<153298[104]>�[m
   > [�[1;34mINFO�[m] 
   > [�[1;31mERROR�[m] �[1;31mTests run: 29, Failures: 2, Errors: 0, Skipped: 
0�[m
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] 
�[1m�[m
   > [�[1;34mINFO�[m] �[1mSkipping Apache Kylin�[m
   > [�[1;34mINFO�[m] This project has been banned from the build due to 
previous failures.
   > [�[1;34mINFO�[m] 
�[1m�[m
   > [�[1;34mINFO�[m] 
�[1m�[m
   > [�[1;34mINFO�[m] �[1mReactor Summary:�[m
   > [�[1;34mINFO�[m] 
   > [�[1;34mINFO�[m] Apache Kylin 3.1.1-SNAPSHOT  
�[1;32mSUCCESS�[m [  3.516 s]
   > [�[1;34mINFO�[m] Apache Kylin - kylin External .. 
�[1;32mSUCCESS�[m [  9.946 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Common . 
�[1;32mSUCCESS�[m [ 22.354 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Metadata ... 
�[1;32mSUCCESS�[m [ 48.745 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Dictionary . 
�[1;32mSUCCESS�[m [03:16 min]
   > [�[1;34mINFO�[m] Apache Kylin - Core Cube ... 
�[1;32mSUCCESS�[m [02:19 min]
   > [�[1;34mINFO�[m] Apache Kylin - Core Metrics  
�[1;32mSUCCESS�[m [  5.364 s]
   > [�[1;34mINFO�[m] Apache Kylin - Core Job  
�[1;32mSUCCESS�[m [01:08 min]
   > [�[1;34mINFO�[m] Apache Kylin - Core Storage  
�[1;32mSUCCESS�[m [  9.280 s]
   > [�[1;34mINFO�[m] Apache Kylin - Stream Core . 
�[1;31mFAILURE�[m [01:02 min]
   > [�[1;34mINFO�[m] Apache Kylin - MapReduce Engine  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Spark Engine  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Flink Engine  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Hive Source . 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - DataSource SDK .. 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Jdbc Source . 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Kafka Source  
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Cache ... 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - HBase Storage ... 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Query ... 
�[1;33mSKIPPED�[m
   > [�[1;34mINFO�[m] Apache Kylin - Metrics Reporter Hive

[jira] [Commented] (KYLIN-4771) Query streaming cube - Thread pool of MultiThreadsResultCollector be blocked.

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211701#comment-17211701
 ] 

ASF GitHub Bot commented on KYLIN-4771:
---

coveralls commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706548304


   ## Pull Request Test Coverage Report for [Build 
6440](https://coveralls.io/builds/34077445)
   
   * **3** of **7**   **(42.86%)**  changed or added relevant lines in **1** 
file are covered.
   * **255** unchanged lines in **12** files lost coverage.
   * Overall coverage decreased (**-0.0006%**) to **28.05%**
   
   ---
   
   |  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
   | :-|--||---: |
   | 
[stream-core/src/main/java/org/apache/kylin/stream/core/query/MultiThreadsResultCollector.java](https://coveralls.io/builds/34077445/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fquery%2FMultiThreadsResultCollector.java#L92)
 | 3 | 7 | 42.86%
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-|--|--: |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/34077445/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553)
 | 1 | 78.12% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/security/KylinUserManager.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinUserManager.java#L144)
 | 1 | 0% |
   | 
[tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/34077445/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L42)
 | 1 | 81.58% |
   | 
[core-common/src/main/java/org/apache/kylin/common/util/ServerMode.java](https://coveralls.io/builds/34077445/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FServerMode.java#L71)
 | 3 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserService.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserService.java#L219)
 | 7 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserGroupService.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserGroupService.java#L159)
 | 9 | 0% |
   | 
[core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/34077445/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L67)
 | 13 | 79.07% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinAuthenticationProvider.java#L56)
 | 30 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/controller/UserController.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FUserController.java#L223)
 | 34 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FQueryService.java#L152)
 | 45 | 0% |
   
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/34077445/badge)](https://coveralls.io/builds/34077445)
 |
   | :-- | --: |
   | Change from base [Build 6416](https://coveralls.io/builds/33804340): |  
-0.0006% |
   | Covered Lines: | 26256 |
   | Relevant Lines: | 93604 |
   
   ---
   #   - [Coveralls](https://coveralls.io)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Query streaming cube - Thread pool of MultiThreadsResultCollector be blocked.
> -
>
> Key: KYLIN-4771
> URL: https://issues.apache.org/jira/browse/KYLIN-4771
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine, Real-time Streaming
>Affects Versions: v3.1.0
> Environment: Centos 7.4.1
> hbase 1.2.4
> hive 2.0.1
> hadoop 2.7.2
>Reporter: GuKe
>Assignee: GuKe
>

[GitHub] [kylin] coveralls commented on pull request #1438: KYLIN-4771 Clear the recordCachePool when the deadline has reached

2020-10-10 Thread GitBox 


coveralls commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706548304


   ## Pull Request Test Coverage Report for [Build 
6440](https://coveralls.io/builds/34077445)
   
   * **3** of **7**   **(42.86%)**  changed or added relevant lines in **1** 
file are covered.
   * **255** unchanged lines in **12** files lost coverage.
   * Overall coverage decreased (**-0.0006%**) to **28.05%**
   
   ---
   
   |  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
   | :-|--||---: |
   | 
[stream-core/src/main/java/org/apache/kylin/stream/core/query/MultiThreadsResultCollector.java](https://coveralls.io/builds/34077445/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fquery%2FMultiThreadsResultCollector.java#L92)
 | 3 | 7 | 42.86%
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-|--|--: |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/34077445/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L553)
 | 1 | 78.12% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/security/KylinUserManager.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinUserManager.java#L144)
 | 1 | 0% |
   | 
[tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/34077445/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L42)
 | 1 | 81.58% |
   | 
[core-common/src/main/java/org/apache/kylin/common/util/ServerMode.java](https://coveralls.io/builds/34077445/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2Futil%2FServerMode.java#L71)
 | 3 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserService.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserService.java#L219)
 | 7 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/KylinUserGroupService.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FKylinUserGroupService.java#L159)
 | 9 | 0% |
   | 
[core-job/src/main/java/org/apache/kylin/job/impl/threadpool/DefaultScheduler.java](https://coveralls.io/builds/34077445/source?filename=core-job%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fjob%2Fimpl%2Fthreadpool%2FDefaultScheduler.java#L67)
 | 13 | 79.07% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fsecurity%2FKylinAuthenticationProvider.java#L56)
 | 30 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/controller/UserController.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fcontroller%2FUserController.java#L223)
 | 34 | 0% |
   | 
[server-base/src/main/java/org/apache/kylin/rest/service/QueryService.java](https://coveralls.io/builds/34077445/source?filename=server-base%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Frest%2Fservice%2FQueryService.java#L152)
 | 45 | 0% |
   
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/34077445/badge)](https://coveralls.io/builds/34077445)
 |
   | :-- | --: |
   | Change from base [Build 6416](https://coveralls.io/builds/33804340): |  
-0.0006% |
   | Covered Lines: | 26256 |
   | Relevant Lines: | 93604 |
   
   ---
   #   - [Coveralls](https://coveralls.io)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (KYLIN-4545) unable to load HiveMetaStore

2020-10-10 Thread Pengfei Wang (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211643#comment-17211643
 ] 

Pengfei Wang edited comment on KYLIN-4545 at 10/10/20, 10:04 AM:
-

I got it! thank you!


was (Author: gobird):
How did you modify it ?

> unable to load HiveMetaStore
> 
>
> Key: KYLIN-4545
> URL: https://issues.apache.org/jira/browse/KYLIN-4545
> Project: Kylin
>  Issue Type: Test
>  Components: Environment 
>Affects Versions: v3.0.2
> Environment: hadoop-3.2.0
> hbase-2.2.0
> hive-3.1.0
> kylin-3.0.2-bin-hadoop3
>Reporter: praveenece
>Priority: Critical
>
> We are using the below software to setup kylin environment.
> hadoop-3.2.0
> hbase-2.2.0
> hive-3.1.0
> kylin-3.0.2-bin-hadoop3
> After configuring everything getting *message: Caused by: 
> javax.jdo.JDOException: Exception thrown when executing query : SELECT 
> `A0`.`NAME` FROM `DBS` `A0` WHERE `A0`.`CATALOG_NAME` = ?* this error message.
> i have executed below query against hive Metastore to fix the issues.
>   USE metastore;
>ALTER TABLE TBLS ADD REWRITE_ENABLED BIT(1) NOT NULL;
>UPDATE TBLS SET REWRITE_ENABLED=IS_REWRITE_ENABLED;
>ALTER TABLE DBS ADD CATALOG_NAME varchar(256) not Null;
>UPDATE DBS SET CATALOG_NAME=CTLG_NAME;
> After that i am able to see the hive tables in kylin model, but i am not able 
> to Sync the tables. Getting the "can not get Hive TableMeta" error message.
> Please help us to fix the issues ASAP. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4545) unable to load HiveMetaStore

2020-10-10 Thread Pengfei Wang (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211643#comment-17211643
 ] 

Pengfei Wang commented on KYLIN-4545:
-

How did you modify it ?

> unable to load HiveMetaStore
> 
>
> Key: KYLIN-4545
> URL: https://issues.apache.org/jira/browse/KYLIN-4545
> Project: Kylin
>  Issue Type: Test
>  Components: Environment 
>Affects Versions: v3.0.2
> Environment: hadoop-3.2.0
> hbase-2.2.0
> hive-3.1.0
> kylin-3.0.2-bin-hadoop3
>Reporter: praveenece
>Priority: Critical
>
> We are using the below software to setup kylin environment.
> hadoop-3.2.0
> hbase-2.2.0
> hive-3.1.0
> kylin-3.0.2-bin-hadoop3
> After configuring everything getting *message: Caused by: 
> javax.jdo.JDOException: Exception thrown when executing query : SELECT 
> `A0`.`NAME` FROM `DBS` `A0` WHERE `A0`.`CATALOG_NAME` = ?* this error message.
> i have executed below query against hive Metastore to fix the issues.
>   USE metastore;
>ALTER TABLE TBLS ADD REWRITE_ENABLED BIT(1) NOT NULL;
>UPDATE TBLS SET REWRITE_ENABLED=IS_REWRITE_ENABLED;
>ALTER TABLE DBS ADD CATALOG_NAME varchar(256) not Null;
>UPDATE DBS SET CATALOG_NAME=CTLG_NAME;
> After that i am able to see the hive tables in kylin model, but i am not able 
> to Sync the tables. Getting the "can not get Hive TableMeta" error message.
> Please help us to fix the issues ASAP. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4782) Verify if the query hit the true cuboid in IT

2020-10-10 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211632#comment-17211632
 ] 

ASF subversion and git services commented on KYLIN-4782:


Commit 9b2a12bc334edf4f8756a1a758c941a4571efa78 in kylin's branch 
refs/heads/kylin-on-parquet-v2 from rupengwang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9b2a12b ]

KYLIN-4782 Verify if the query hit the true cuboid in IT


> Verify if the query hit the true cuboid in IT
> -
>
> Key: KYLIN-4782
> URL: https://issues.apache.org/jira/browse/KYLIN-4782
> Project: Kylin
>  Issue Type: Improvement
>  Components: Integration
>Affects Versions: v4.0.0-alpha
>Reporter: wangrupeng
>Assignee: wangrupeng
>Priority: Minor
> Fix For: v4.0.0-beta
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4782) Verify if the query hit the true cuboid in IT

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211633#comment-17211633
 ] 

ASF GitHub Bot commented on KYLIN-4782:
---

hit-lacus merged pull request #1436:
URL: https://github.com/apache/kylin/pull/1436


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Verify if the query hit the true cuboid in IT
> -
>
> Key: KYLIN-4782
> URL: https://issues.apache.org/jira/browse/KYLIN-4782
> Project: Kylin
>  Issue Type: Improvement
>  Components: Integration
>Affects Versions: v4.0.0-alpha
>Reporter: wangrupeng
>Assignee: wangrupeng
>Priority: Minor
> Fix For: v4.0.0-beta
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [kylin] hit-lacus merged pull request #1436: KYLIN-4782 Verify if the query hit the true cuboid in IT

2020-10-10 Thread GitBox 


hit-lacus merged pull request #1436:
URL: https://github.com/apache/kylin/pull/1436


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (KYLIN-4771) Query streaming cube - Thread pool of MultiThreadsResultCollector be blocked.

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211613#comment-17211613
 ] 

ASF GitHub Bot commented on KYLIN-4771:
---

hit-lacus commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706512995


   Hello @ggKe , I have checked your patch, your patch look good to me. But 
there are some UT failure, could you please help to fix them? 
   
   Following is part of log:
   
   
   ```sh
   [INFO] Tests run: 1, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.242 s - in 
org.apache.kylin.stream.core.storage.RecordsSerDeTest
   [INFO] Running 
org.apache.kylin.stream.core.storage.CheckPointStoreTest
   client:0 round266 scan finished
   client:1 round269 scan finished
   CheckPoint{sourceConsumePosition='{"1":3,"2":4,"3":5}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1602242847871, totalCount=100, checkPointCount=5}
   going to delete checkpoint file CP-160220160
   going to delete checkpoint file CP-160228800
   going to delete checkpoint file CP-160237440
   going to delete checkpoint file CP-160246080
   going to delete checkpoint file CP-160254720
   CheckPoint{sourceConsumePosition='{"1":1,"2":2,"3":3}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1603020447894, totalCount=100, checkPointCount=5}
   [INFO] Tests run: 3, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.028 s - in 
org.apache.kylin.stream.core.storage.CheckPointStoreTest
   [INFO] 
   [INFO] Results:
   [INFO] 
   [ERROR] Failures: 
   [ERROR]   
StreamingSegmentManagerTest.testAddEventAndScan:120 expected:<8> but 
was:<30007>
   [ERROR]   StreamingSegmentManagerTest.testIndexFilter:182 
expected:<153298[080]> but was:<153298[104]>
   [INFO] 
   [ERROR] Tests run: 29, Failures: 2, Errors: 0, Skipped: 0
   [INFO] 
   [INFO] 
   [INFO] 

   [INFO] Skipping Apache Kylin
   [INFO] This project has been banned from the build due to previous 
failures.
   [INFO] 

   [INFO] 

   [INFO] Reactor Summary:
   [INFO] 
   [INFO] Apache Kylin 3.1.1-SNAPSHOT  
SUCCESS [  3.516 s]
   [INFO] Apache Kylin - kylin External .. 
SUCCESS [  9.946 s]
   [INFO] Apache Kylin - Core Common . 
SUCCESS [ 22.354 s]
   [INFO] Apache Kylin - Core Metadata ... 
SUCCESS [ 48.745 s]
   [INFO] Apache Kylin - Core Dictionary . 
SUCCESS [03:16 min]
   [INFO] Apache Kylin - Core Cube ... 
SUCCESS [02:19 min]
   [INFO] Apache Kylin - Core Metrics  
SUCCESS [  5.364 s]
   [INFO] Apache Kylin - Core Job  
SUCCESS [01:08 min]
   [INFO] Apache Kylin - Core Storage  
SUCCESS [  9.280 s]
   [INFO] Apache Kylin - Stream Core . 
FAILURE [01:02 min]
   [INFO] Apache Kylin - MapReduce Engine  
SKIPPED
   [INFO] Apache Kylin - Spark Engine  
SKIPPED
   [INFO] Apache Kylin - Flink Engine  
SKIPPED
   [INFO] Apache Kylin - Hive Source . 
SKIPPED
   [INFO] Apache Kylin - DataSource SDK .. 
SKIPPED
   [INFO] Apache Kylin - Jdbc Source . 
SKIPPED
   [INFO] Apache Kylin - Kafka Source  
SKIPPED
   [INFO] Apache Kylin - Cache ... 
SKIPPED
   [INFO] Apache Kylin - HBase Storage ... 
SKIPPED
   [INFO]

[GitHub] [kylin] hit-lacus commented on pull request #1438: KYLIN-4771 Clear the recordCachePool when the deadline has reached

2020-10-10 Thread GitBox 


hit-lacus commented on pull request #1438:
URL: https://github.com/apache/kylin/pull/1438#issuecomment-706512995


   Hello @ggKe , I have checked your patch, your patch look good to me. But 
there are some UT failure, could you please help to fix them? 
   
   Following is part of log:
   
   
   ```sh
   [INFO] Tests run: 1, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.242 s - in 
org.apache.kylin.stream.core.storage.RecordsSerDeTest
   [INFO] Running 
org.apache.kylin.stream.core.storage.CheckPointStoreTest
   client:0 round266 scan finished
   client:1 round269 scan finished
   CheckPoint{sourceConsumePosition='{"1":3,"2":4,"3":5}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1602242847871, totalCount=100, checkPointCount=5}
   going to delete checkpoint file CP-160220160
   going to delete checkpoint file CP-160228800
   going to delete checkpoint file CP-160237440
   going to delete checkpoint file CP-160246080
   going to delete checkpoint file CP-160254720
   CheckPoint{sourceConsumePosition='{"1":1,"2":2,"3":3}', 
persistedIndexes={160224120=1, 160224119=2, 160224118=3}, 
longLatencyInfo=LongLatencyInfo{longLatencyEventCnts={}, 
totalLongLatencyEventCnt=0}, 
segmentSourceStartPosition={160224120={"1":1,"2":2,"3":3}}, 
checkPointTime=1603020447894, totalCount=100, checkPointCount=5}
   [INFO] Tests run: 3, Failures: 0, Errors: 0, 
Skipped: 0, Time elapsed: 0.028 s - in 
org.apache.kylin.stream.core.storage.CheckPointStoreTest
   [INFO] 
   [INFO] Results:
   [INFO] 
   [ERROR] Failures: 
   [ERROR]   
StreamingSegmentManagerTest.testAddEventAndScan:120 expected:<8> but 
was:<30007>
   [ERROR]   StreamingSegmentManagerTest.testIndexFilter:182 
expected:<153298[080]> but was:<153298[104]>
   [INFO] 
   [ERROR] Tests run: 29, Failures: 2, Errors: 0, Skipped: 0
   [INFO] 
   [INFO] 
   [INFO] 

   [INFO] Skipping Apache Kylin
   [INFO] This project has been banned from the build due to previous 
failures.
   [INFO] 

   [INFO] 

   [INFO] Reactor Summary:
   [INFO] 
   [INFO] Apache Kylin 3.1.1-SNAPSHOT  
SUCCESS [  3.516 s]
   [INFO] Apache Kylin - kylin External .. 
SUCCESS [  9.946 s]
   [INFO] Apache Kylin - Core Common . 
SUCCESS [ 22.354 s]
   [INFO] Apache Kylin - Core Metadata ... 
SUCCESS [ 48.745 s]
   [INFO] Apache Kylin - Core Dictionary . 
SUCCESS [03:16 min]
   [INFO] Apache Kylin - Core Cube ... 
SUCCESS [02:19 min]
   [INFO] Apache Kylin - Core Metrics  
SUCCESS [  5.364 s]
   [INFO] Apache Kylin - Core Job  
SUCCESS [01:08 min]
   [INFO] Apache Kylin - Core Storage  
SUCCESS [  9.280 s]
   [INFO] Apache Kylin - Stream Core . 
FAILURE [01:02 min]
   [INFO] Apache Kylin - MapReduce Engine  
SKIPPED
   [INFO] Apache Kylin - Spark Engine  
SKIPPED
   [INFO] Apache Kylin - Flink Engine  
SKIPPED
   [INFO] Apache Kylin - Hive Source . 
SKIPPED
   [INFO] Apache Kylin - DataSource SDK .. 
SKIPPED
   [INFO] Apache Kylin - Jdbc Source . 
SKIPPED
   [INFO] Apache Kylin - Kafka Source  
SKIPPED
   [INFO] Apache Kylin - Cache ... 
SKIPPED
   [INFO] Apache Kylin - HBase Storage ... 
SKIPPED
   [INFO] Apache Kylin - Query ... 
SKIPPED
   [INFO] Apache Kylin - Metrics Reporter Hive ... 
SKIPPED
   [INFO] Apache Kylin - Metrics Reporter Kafka ..

[jira] [Commented] (KYLIN-4545) unable to load HiveMetaStore

2020-10-10 Thread Liubo (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211601#comment-17211601
 ] 

Liubo commented on KYLIN-4545:
--

I also had the same problem. My problem was caused by the spark3.0 deployed in 
the cluster, because spark contains hive-metastore-2.3.7.jar, which conflicts 
with hive-3.* in the cluster. So I modified the find-spark-dependency.sh script 
in the bin directory of kylin, tried to exclude hive-metastore-2.3.7.jar, and 
the problem was solved.

> unable to load HiveMetaStore
> 
>
> Key: KYLIN-4545
> URL: https://issues.apache.org/jira/browse/KYLIN-4545
> Project: Kylin
>  Issue Type: Test
>  Components: Environment 
>Affects Versions: v3.0.2
> Environment: hadoop-3.2.0
> hbase-2.2.0
> hive-3.1.0
> kylin-3.0.2-bin-hadoop3
>Reporter: praveenece
>Priority: Critical
>
> We are using the below software to setup kylin environment.
> hadoop-3.2.0
> hbase-2.2.0
> hive-3.1.0
> kylin-3.0.2-bin-hadoop3
> After configuring everything getting *message: Caused by: 
> javax.jdo.JDOException: Exception thrown when executing query : SELECT 
> `A0`.`NAME` FROM `DBS` `A0` WHERE `A0`.`CATALOG_NAME` = ?* this error message.
> i have executed below query against hive Metastore to fix the issues.
>   USE metastore;
>ALTER TABLE TBLS ADD REWRITE_ENABLED BIT(1) NOT NULL;
>UPDATE TBLS SET REWRITE_ENABLED=IS_REWRITE_ENABLED;
>ALTER TABLE DBS ADD CATALOG_NAME varchar(256) not Null;
>UPDATE DBS SET CATALOG_NAME=CTLG_NAME;
> After that i am able to see the hive tables in kylin model, but i am not able 
> to Sync the tables. Getting the "can not get Hive TableMeta" error message.
> Please help us to fix the issues ASAP. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KYLIN-4626) add set kylin home sh

2020-10-10 Thread Xiaoxiang Yu (Jira) 


 [ 
https://issues.apache.org/jira/browse/KYLIN-4626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiaoxiang Yu resolved KYLIN-4626.
-
Fix Version/s: v3.1.1
Reviewers: Xiaoxiang Yu
   Resolution: Fixed

Merged

> add set kylin home sh
> -
>
> Key: KYLIN-4626
> URL: https://issues.apache.org/jira/browse/KYLIN-4626
> Project: Kylin
>  Issue Type: Improvement
>Reporter: chuxiao
>Assignee: chuxiao
>Priority: Major
> Fix For: v3.1.1
>
>
> KYLIN_HOME 是重要的，几乎每个脚本都离不开它。但设置环境变量并不是一个最佳行为，比如安装了多套实例。增加set-kylin-home.sh, 
> kylin实例可以自动设置自己的安装目录。



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4421) Allow to update table & database name

2020-10-10 Thread Zhong Yanghong (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211583#comment-17211583
 ] 

Zhong Yanghong commented on KYLIN-4421:
---

The usage is as follows:
{code:java}
curl -kXPOST 'http://localhost:7070/kylin/api/tables/default/update' \
-H 'Authorization: Basic XX' \
-H 'Content-Type: application/json' \
-d '{  
   "mapping":{
  "DEFAULT.KYLIN_SALES": {
 "database": "TEST",
 "tableName": "KYLIN_FACT"
  },
  "DEFAULT.KYLIN_CAL_DT": {
 "tableName": "CAL_DT"
  },
  "DEFAULT.KYLIN_CATEGORY_GROUPINGS": {
 "database": "TEST"
  }
   },
   "isUseExisting":true
}'
{code}

> Allow to update table & database name 
> --
>
> Key: KYLIN-4421
> URL: https://issues.apache.org/jira/browse/KYLIN-4421
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Minor
> Fix For: v3.1.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KYLIN-4784) Unknown column CATALOG_NAME in table DBS

2020-10-10 Thread Xiaoxiang Yu (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211582#comment-17211582
 ] 

Xiaoxiang Yu commented on KYLIN-4784:
-

>From my experience, there are some incompatible issue when deploy Kylin on 
>some custom Hadoop combination. I suggested you to move Kylin to a popular 
>Hadoop distribution like CDH or HDP.

> Unknown column CATALOG_NAME in table DBS
> 
>
> Key: KYLIN-4784
> URL: https://issues.apache.org/jira/browse/KYLIN-4784
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v3.1.0
>Reporter: Pengfei Wang
>Priority: Major
>
> Mysql: 5.27
> Hadoop: 3.2.1
> HBase: 2.3.0
> Kylin: 3.1.0
>  
>  
>  
> 2020-10-09 03:52:44,128 ERROR [http-nio-7070-exec-7] 
> controller.TableController:215 : Exception thrown when executing query : 
> SELECT `A0`.`NAME` FROM `DBS` `A0` WHERE `A0`.`CATALOG_NAME` = ?2020-10-09 
> 03:52:44,128 ERROR [http-nio-7070-exec-7] controller.TableController:215 : 
> Exception thrown when executing query : SELECT `A0`.`NAME` FROM `DBS` `A0` 
> WHERE `A0`.`CATALOG_NAME` = ?MetaException(message:Exception thrown when 
> executing query : SELECT `A0`.`NAME` FROM `DBS` `A0` WHERE 
> `A0`.`CATALOG_NAME` = ?) at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
>  at 
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
>  at com.sun.proxy.$Proxy73.get_databases(Unknown Source) at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1356)
>  at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllDatabases(HiveMetaStoreClient.java:1351)
>  at 
> org.apache.kylin.source.hive.CLIHiveClient.getHiveDbNames(CLIHiveClient.java:117)
>  at 
> org.apache.kylin.source.hive.HiveMetadataExplorer.listDatabases(HiveMetadataExplorer.java:51)
>  at 
> org.apache.kylin.rest.service.TableService.getSourceDbNames(TableService.java:329)
>  at 
> org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:213)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
>  at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
>  at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>  at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
>  at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
>  at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>  at 
> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
>  at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
>  at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
>  at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
>  at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at 
>

[jira] [Commented] (KYLIN-4689) Deadlock in Kylin job execution

2020-10-10 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211580#comment-17211580
 ] 

ASF GitHub Bot commented on KYLIN-4689:
---

hit-lacus commented on a change in pull request #1440:
URL: https://github.com/apache/kylin/pull/1440#discussion_r502754031



##
File path: 
stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/coordinate/BuildJobSubmitter.java
##
@@ -165,6 +166,11 @@ void doRun() {
 for (StreamingCubeInfo cubeInfo : 
coordinator.getEnableStreamingCubes()) {
 List segmentList = 
checkSegmentBuildJobFromMetadata(cubeInfo.getCubeName());
 for (String segmentName : segmentList) {
+boolean isFull = 
DefaultScheduler.getInstance().getFetcherRunner().isJobPoolFull();
+if (isFull) {
+logger.info("Hit MaxConcurrentJobLimit, hold {}.", 
segmentName);
+break;
+}

Review comment:
   OK, it make sense.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Deadlock in Kylin job execution
> ---
>
> Key: KYLIN-4689
> URL: https://issues.apache.org/jira/browse/KYLIN-4689
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v3.0.0, v3.1.0, v3.0.1, v3.0.2
>Reporter: Gabor Arki
>Assignee: Xiaoxiang Yu
>Priority: Critical
> Fix For: v3.1.1
>
>
> h4. Reproduction steps
>  * Install Kylin 3.1.0
>  * Deploy a streaming cube
>  * Enable the cube having historical data present in the Kafka topic
>  * Note: in our case, we had 3 cubes deployed, each consuming ~20-20 hourly 
> segments from Kafka when the cubes were enabled
> h4. Expected result
>  * Kylin is starting to process stream segments with stream jobs, eventually 
> processing the older segments and catching up with the stream
> h4. Actual result
>  * A short time time after the stream jobs have started (37 successful stream 
> jobs), all jobs are completely stuck without any progress. Some in running 
> state, some in pending state.
>  * The following logs are continuously written:
> {code:java}
> 2020-08-06 06:16:22 INFO  [Scheduler 116797841 Job 
> 12750aea-3b96-c817-64e8-bf893d8c120f-254] MapReduceExecutable:409 - 
> 12750aea-3b96-c817-64e8-bf893d8c120f-00, parent lock 
> path(/cube_job_lock/cube_vm) is locked by other job result is true ,ephemeral 
> lock path :/cube_job_ephemeral_lock/cube_vm is locked by other job result is 
> true,will try after one minute
> 2020-08-06 06:16:33 WARN  [FetcherRunner 787667774-43] FetcherRunner:56 - 
> There are too many jobs running, Job Fetch will wait until next schedule time
> {code}
>  * Zookeeper indicates the following locks are in place:
> {code:java}
> ls /kylin/kylin_metadata/cube_job_ephemeral_lock
> [cube_cm, cube_vm, cube_jm]
> ls /kylin/kylin_metadata/cube_job_ephemeral_lock/cube_cm 
> []
> ls /kylin/kylin_metadata/cube_job_ephemeral_lock/cube_vm
> []
> ls /kylin/kylin_metadata/cube_job_ephemeral_lock/cube_jm
> []
> ls /kylin/kylin_metadata/cube_job_lock
> [cube_cm, cube_vm, cube_jm]
> ls /kylin/kylin_metadata/cube_job_lock/cube_cm
> [f888380e-9ff4-98f5-2df4-1ae71e045f93]
> ls /kylin/kylin_metadata/cube_job_lock/cube_vm
> [fc186bd9-1186-6ed4-e58c-bbbf6dd8ef74]
> ls /kylin/kylin_metadata/cube_job_lock/cube_jm
> [d1a6475a-9ab2-5ee4-6714-f395e20cfc01]
> {code}
>  * The job IDs for the running jobs:
>  ** 169f75fa-a02f-221b-fc48-037bc7a842d0
>  ** 0b5dae1b-6faf-66c5-71dc-86f5b820f1c4
>  ** 00924699-8b51-8091-6e71-34ccfeba3a98
>  ** 4620192a-71e1-16dd-3b05-44d7f9144ad4
>  ** 416355c2-a3d7-57eb-55c6-c042aa256510
>  ** 12750aea-3b96-c817-64e8-bf893d8c120f
>  ** 42819dde-5857-fd6b-b075-439952f47140
>  ** 00128937-bd4a-d6c1-7a4e-744dee946f67
>  ** 46a0233f-217e-9155-725b-c815ad77ba2c
>  ** 062150ba-bacd-6644-4801-3a51b260d1c5
> As you can see, the 10 jobs that are actually running do not possess the 
> locks thus cannot actually do anything (these all were stuck at step Build 
> Dimension Dictionaries For Steaming Job). On the other hand, the 3 jobs 
> possessing the locks cannot resume running because there are already 10 jobs 
> in running state, thus cannot proceed and release the locks. This is a 
> deadlock and the cluster is completely stuck.
> We have been observing this behavior in 3.0.0 (where rolling back 
> https://issues.apache.org/jira/browse/KYLIN-4165 resolved the issue), and now 
> in 3.1.0 as well. It has been originally reported in the comments of 
>

[GitHub] [kylin] hit-lacus commented on a change in pull request #1440: KYLIN-4689 BuildJobSubmitter should obey MaxJobLimit

2020-10-10 Thread GitBox 


hit-lacus commented on a change in pull request #1440:
URL: https://github.com/apache/kylin/pull/1440#discussion_r502754031



##
File path: 
stream-coordinator/src/main/java/org/apache/kylin/stream/coordinator/coordinate/BuildJobSubmitter.java
##
@@ -165,6 +166,11 @@ void doRun() {
 for (StreamingCubeInfo cubeInfo : 
coordinator.getEnableStreamingCubes()) {
 List segmentList = 
checkSegmentBuildJobFromMetadata(cubeInfo.getCubeName());
 for (String segmentName : segmentList) {
+boolean isFull = 
DefaultScheduler.getInstance().getFetcherRunner().isJobPoolFull();
+if (isFull) {
+logger.info("Hit MaxConcurrentJobLimit, hold {}.", 
segmentName);
+break;
+}

Review comment:
   OK, it make sense.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (KYLIN-4772) BadQueryDetector detect scan too many bytes query

2020-10-10 Thread Xiaoxiang Yu (Jira) 


[ 
https://issues.apache.org/jira/browse/KYLIN-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211576#comment-17211576
 ] 

Xiaoxiang Yu commented on KYLIN-4772:
-

Good proposal.

> BadQueryDetector detect scan too many bytes query
> -
>
> Key: KYLIN-4772
> URL: https://issues.apache.org/jira/browse/KYLIN-4772
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine, Tools, Build and Test
>Affects Versions: v3.1.0
>Reporter: kaishen
>Priority: Minor
>
> now BadQueryDetector only detect long runing query , if it can also detect 
> huge bytes scan  query will be useful



--
This message was sent by Atlassian Jira
(v8.3.4#803005)