[GitHub] [kylin] hit-lacus edited a comment on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
hit-lacus edited a comment on issue #797: KYLIN-4114 Provided a self-contained 
docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-521892892
 
 
   ## Test Env 
   
   - MacOS 10.14.4 
   - Docker Desktop for Mac (Docker Engine: 18.09.2)
   
    Start standalone docker container 
   ```sh
   docker run -d --rm \
   -p 7070:7070 \
   -p 8088:8088 \
   -p 50070:50070 \
   -p 8032:8032 \
   -p 8042:8042 \
   -p 60010:60010 \
   codingforfun/apache-kylin-standalone
   ```
   
    Attach to container
   ``` sh
   docker exec -it ${CONTAINER_ID} /bin/bash
   ```
   
   
    Building Cube (Build/Refresh/Merge)
   
   https://user-images.githubusercontent.com/14030549/63145830-7cb6eb00-c02b-11e9-990e-1fa61c8fce39.png;>
   
   
   https://user-images.githubusercontent.com/14030549/63141178-5c7d3100-c017-11e9-9786-77e2f7b79460.png;>
   
   
    Near Realtime Streaming
   
   https://user-images.githubusercontent.com/14030549/63141276-bf6ec800-c017-11e9-959a-a1eb66adc689.png;>
   
    Realtime OLAP
   
   https://user-images.githubusercontent.com/14030549/63141141-348dcd80-c017-11e9-90c2-33eb4b50bc03.png;>
   
   https://user-images.githubusercontent.com/14030549/63141128-22139400-c017-11e9-91a4-cfda28f8a547.png;>
   
   ### Known issue
   
![image](https://user-images.githubusercontent.com/14030549/63145905-c99ac180-c02b-11e9-9d75-a6386387ccf2.png)
   https://user-images.githubusercontent.com/14030549/63146599-0a93d580-c02e-11e9-8500-eb1d94775cbf.png;>
   
   
   1. Nodemanager may be unhealthy because of lack of disk space.
   This may be fixed by `docker image prune -h` to reduce needless disk space. 
   
   2. Container cannot restart successfully.
   
   3. Realtime OLAP building job cubing job throw NPE for some reasons(to be 
checked).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] hit-lacus edited a comment on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
hit-lacus edited a comment on issue #797: KYLIN-4114 Provided a self-contained 
docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-521892892
 
 
   ## Test Env 
   
   - MacOS 10.14.4 
   - Docker Desktop for Mac (Docker Engine: 18.09.2)
   
    Start standalone docker container 
   ```sh
   docker run -d --rm \
   -p 7070:7070 \
   -p 8088:8088 \
   -p 50070:50070 \
   -p 8032:8032 \
   -p 8042:8042 \
   -p 60010:60010 \
   codingforfun/apache-kylin-standalone
   ```
   
    Attach to container
   ``` sh
   docker exec -it ${CONTAINER_ID} /bin/bash
   ```
   
   
    Building Cube (Build/Refresh/Merge)
   
   https://user-images.githubusercontent.com/14030549/63145830-7cb6eb00-c02b-11e9-990e-1fa61c8fce39.png;>
   
   
   https://user-images.githubusercontent.com/14030549/63141178-5c7d3100-c017-11e9-9786-77e2f7b79460.png;>
   
   
    Near Realtime Streaming
   
   https://user-images.githubusercontent.com/14030549/63141276-bf6ec800-c017-11e9-959a-a1eb66adc689.png;>
   
    Realtime OLAP
   
   https://user-images.githubusercontent.com/14030549/63141141-348dcd80-c017-11e9-90c2-33eb4b50bc03.png;>
   
   https://user-images.githubusercontent.com/14030549/63141128-22139400-c017-11e9-91a4-cfda28f8a547.png;>
   
   ### Known issue
   
![image](https://user-images.githubusercontent.com/14030549/63145905-c99ac180-c02b-11e9-9d75-a6386387ccf2.png)
   
   1. Nodemanager may be unhealthy because of lack of disk space.
   This may be fixed by `docker image prune -h` to reduce needless disk space. 
   
   2. Container cannot restart successfully.
   
   3. Realtime OLAP building job cubing job throw NPE for some reasons(to be 
checked).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] hit-lacus commented on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
hit-lacus commented on issue #797: KYLIN-4114 Provided a self-contained docker 
image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-521892892
 
 
   ## Test Env 
   
   - MacOS 10.14.4 
   - Docker Desktop for Mac (Docker Engine: 18.09.2)
   
    Start standalone docker container 
   ```sh
   docker run -d --rm \
   -p 7070:7070 \
   -p 8088:8088 \
   -p 50070:50070 \
   -p 8032:8032 \
   -p 8042:8042 \
   -p 60010:60010 \
   codingforfun/apache-kylin-standalone
   ```
   
    Attach to container
   ``` sh
   docker exec -it ${CONTAINER_ID} /bin/bash
   ```
   
   
    Building Cube 
   
   https://user-images.githubusercontent.com/14030549/63145830-7cb6eb00-c02b-11e9-990e-1fa61c8fce39.png;>
   
   
   https://user-images.githubusercontent.com/14030549/63141178-5c7d3100-c017-11e9-9786-77e2f7b79460.png;>
   
   
    Near Realtime Streaming
   
   https://user-images.githubusercontent.com/14030549/63141276-bf6ec800-c017-11e9-959a-a1eb66adc689.png;>
   
    Realtime OLAP
   
   https://user-images.githubusercontent.com/14030549/63141141-348dcd80-c017-11e9-90c2-33eb4b50bc03.png;>
   
   https://user-images.githubusercontent.com/14030549/63141128-22139400-c017-11e9-91a4-cfda28f8a547.png;>
   
   ### Known issue
   
![image](https://user-images.githubusercontent.com/14030549/63145905-c99ac180-c02b-11e9-9d75-a6386387ccf2.png)
   
   Nodemanager may be unhealthy because of lack of disk space.
   This may be fixed by `docker image prune -h` to reduce needless disk space. 
   
   Container cannot restart successfully because `entrypoint.sh`.
   
   Realtime OLAP building job cubing job throw NPE for some reasons(to be 
analysed).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (KYLIN-4137) Accelerate metadata reloading

2019-08-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908735#comment-16908735
 ] 

ASF GitHub Bot commented on KYLIN-4137:
---

nichunen commented on pull request #802: KYLIN-4137 Accelerate metadata 
reloading
URL: https://github.com/apache/kylin/pull/802
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Accelerate metadata reloading
> -
>
> Key: KYLIN-4137
> URL: https://issues.apache.org/jira/browse/KYLIN-4137
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Affects Versions: v2.6.2
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
> Fix For: v3.0.0-beta
>
>
> Now, org.apache.kylin.metadata.cachesync.CachedCrudAssist#reloadAll is using 
> an inappropriate method to deal with MySQLJdbcMetadata.
> As the every call of reloadAt(path) will access the database, which is almost 
> fine to HBase but really unfriendly to the MySQL(RDBMS).
> I think we should get all of the resource with single request instead of the 
> separate request to get every resource.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[GitHub] [kylin] nichunen merged pull request #802: KYLIN-4137 Accelerate metadata reloading

2019-08-15 Thread GitBox
nichunen merged pull request #802: KYLIN-4137 Accelerate metadata reloading
URL: https://github.com/apache/kylin/pull/802
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (KYLIN-4137) Accelerate metadata reloading

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908737#comment-16908737
 ] 

ASF subversion and git services commented on KYLIN-4137:


Commit dfb848844f9e6c9e9984f99a347d4e70f44c4249 in kylin's branch 
refs/heads/master from Temple Zhou
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=dfb8488 ]

KYLIN-4137 Accelerate metadata reloading


> Accelerate metadata reloading
> -
>
> Key: KYLIN-4137
> URL: https://issues.apache.org/jira/browse/KYLIN-4137
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Affects Versions: v2.6.2
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
> Fix For: v3.0.0-beta
>
>
> Now, org.apache.kylin.metadata.cachesync.CachedCrudAssist#reloadAll is using 
> an inappropriate method to deal with MySQLJdbcMetadata.
> As the every call of reloadAt(path) will access the database, which is almost 
> fine to HBase but really unfriendly to the MySQL(RDBMS).
> I think we should get all of the resource with single request instead of the 
> separate request to get every resource.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[GitHub] [kylin] codecov-io edited a comment on issue #800: 添加当天作业过滤

2019-08-15 Thread GitBox
codecov-io edited a comment on issue #800: 添加当天作业过滤
URL: https://github.com/apache/kylin/pull/800#issuecomment-521109237
 
 
   # [Codecov](https://codecov.io/gh/apache/kylin/pull/800?src=pr=h1) Report
   > Merging [#800](https://codecov.io/gh/apache/kylin/pull/800?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/kylin/commit/abc0ab80a04f1bab95f93901f1b802aecaf72382?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/kylin/pull/800/graphs/tree.svg?width=650=JawVgbgsVo=150=pr)](https://codecov.io/gh/apache/kylin/pull/800?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master #800  +/-   ##
   
   + Coverage 25.62%   25.63%   +0.01% 
   - Complexity 6048 6049   +1 
   
 Files  1398 1398  
 Lines 8341183428  +17 
 Branches  1168011681   +1 
   
   + Hits  2137121387  +16 
   - Misses5998559991   +6 
   + Partials   2055 2050   -5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/kylin/pull/800?src=pr=tree) | Coverage Δ 
| Complexity Δ | |
   |---|---|---|---|
   | 
[...g/apache/kylin/job/constant/JobTimeFilterEnum.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9jb25zdGFudC9Kb2JUaW1lRmlsdGVyRW51bS5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...java/org/apache/kylin/rest/service/JobService.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9Kb2JTZXJ2aWNlLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...main/java/org/apache/kylin/common/KylinConfig.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZy5qYXZh)
 | `39.53% <0%> (-1.6%)` | `27% <0%> (ø)` | |
   | 
[...g/apache/kylin/metadata/cachesync/Broadcaster.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvY2FjaGVzeW5jL0Jyb2FkY2FzdGVyLmphdmE=)
 | `52% <0%> (+0.48%)` | `23% <0%> (ø)` | :arrow_down: |
   | 
[...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh)
 | `76.74% <0%> (+2.32%)` | `12% <0%> (ø)` | :arrow_down: |
   | 
[...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh)
 | `69.6% <0%> (+3.03%)` | `7% <0%> (ø)` | :arrow_down: |
   | 
[...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==)
 | `78.37% <0%> (+5.4%)` | `7% <0%> (+1%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/kylin/pull/800?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/kylin/pull/800?src=pr=footer). Last 
update 
[abc0ab8...5d2cd07](https://codecov.io/gh/apache/kylin/pull/800?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (KYLIN-4139) Compatible old user security xml config when user upgrate new kylin version

2019-08-15 Thread luguosheng (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

luguosheng updated KYLIN-4139:
--
Description: 
           Forward compatible
 # when user config 'spring.profiles.active=testing', sync old version user 
metadata with user config in kylinSecurity.xml

Other improvement

 # Remove the logic of auto trans new user name to uppercase

  was:
* Forward compatible

 # when user config 'spring.profiles.active=testing', sync old version user 
metadata with user config in kylinSecurity.xml

 * Other improvement

 # Remove the logic of auto trans new user name to uppercase


>  Compatible old user security xml config when user upgrate new kylin version
> 
>
> Key: KYLIN-4139
> URL: https://issues.apache.org/jira/browse/KYLIN-4139
> Project: Kylin
>  Issue Type: Improvement
>Reporter: luguosheng
>Assignee: luguosheng
>Priority: Major
>
>            Forward compatible
>  # when user config 'spring.profiles.active=testing', sync old version user 
> metadata with user config in kylinSecurity.xml
> Other improvement
>  # Remove the logic of auto trans new user name to uppercase



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4139) Compatible old user security xml config when user upgrate new kylin version

2019-08-15 Thread luguosheng (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

luguosheng updated KYLIN-4139:
--
Description: 
         

 Forward compatible
 # when user config 'spring.profiles.active=testing', sync old version user 
metadata with user config in kylinSecurity.xml

Other improvement
 # Remove the logic of auto trans new user name to uppercase

  was:
           Forward compatible
 # when user config 'spring.profiles.active=testing', sync old version user 
metadata with user config in kylinSecurity.xml

Other improvement

 # Remove the logic of auto trans new user name to uppercase


>  Compatible old user security xml config when user upgrate new kylin version
> 
>
> Key: KYLIN-4139
> URL: https://issues.apache.org/jira/browse/KYLIN-4139
> Project: Kylin
>  Issue Type: Improvement
>Reporter: luguosheng
>Assignee: luguosheng
>Priority: Major
>
>          
>  Forward compatible
>  # when user config 'spring.profiles.active=testing', sync old version user 
> metadata with user config in kylinSecurity.xml
> Other improvement
>  # Remove the logic of auto trans new user name to uppercase



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4139) Compatible old user security xml config when user upgrate new kylin version

2019-08-15 Thread luguosheng (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

luguosheng updated KYLIN-4139:
--
Description: 
* Forward compatible

 # when user config 'spring.profiles.active=testing', sync old version user 
metadata with user config in kylinSecurity.xml

 * Other improvement

 # Remove the logic of auto trans new user name to uppercase

>  Compatible old user security xml config when user upgrate new kylin version
> 
>
> Key: KYLIN-4139
> URL: https://issues.apache.org/jira/browse/KYLIN-4139
> Project: Kylin
>  Issue Type: Improvement
>Reporter: luguosheng
>Assignee: luguosheng
>Priority: Major
>
> * Forward compatible
>  # when user config 'spring.profiles.active=testing', sync old version user 
> metadata with user config in kylinSecurity.xml
>  * Other improvement
>  # Remove the logic of auto trans new user name to uppercase



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (KYLIN-4139) Compatible old user security xml config when user upgrate new kylin version

2019-08-15 Thread luguosheng (JIRA)
luguosheng created KYLIN-4139:
-

 Summary:  Compatible old user security xml config when user 
upgrate new kylin version
 Key: KYLIN-4139
 URL: https://issues.apache.org/jira/browse/KYLIN-4139
 Project: Kylin
  Issue Type: Improvement
Reporter: luguosheng
Assignee: luguosheng






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Closed] (KYLIN-4138) kylin2.6.3使用Spark构建Cube,找不到Spark的Class

2019-08-15 Thread shaozhipeng (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shaozhipeng closed KYLIN-4138.
--
Resolution: Done

2.6.3 和 2.5.2版本,同样的方法解决即可,kylin 
server依赖的相关jar包无法从CLASSPATH中找到,需要设置或者拷贝到tomcat/lib下面,拷贝到tomcat/webapps/kylin/WEB-INF/lib下效果是一样的,能加载到即可。

> kylin2.6.3使用Spark构建Cube,找不到Spark的Class
> --
>
> Key: KYLIN-4138
> URL: https://issues.apache.org/jira/browse/KYLIN-4138
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.3
>Reporter: shaozhipeng
>Priority: Major
> Attachments: kylin-spark0.png, kylin-spark1.png
>
>
> apache-kylin-2.6.3-bin-hbase1x.tar.gz
> $KYLIN_HOME/bin/download-spark.sh
> kylin.engine.spark-conf.spark.yarn.archive=hdfs://master:9000/kylin/spark/spark-libs.jar
>  
> 报错:
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/spark/api/java/function/Function
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
>  ~[kylin-core-job-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) 
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]
>  ... 77 more
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.api.java.function.Function
>  at 
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
>  ~[catalina.jar:7.0.91]
>  at 
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
>  ~[catalina.jar:7.0.91]
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
>  ~[kylin-core-job-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) 
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4138) kylin2.6.3使用Spark构建Cube,找不到Spark的Class

2019-08-15 Thread shaozhipeng (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908627#comment-16908627
 ] 

shaozhipeng commented on KYLIN-4138:


cp spark/jars/spark-core_2.11-2.3.2.jar tomcat/lib/

cp spark/jars/scala-library-2.11.8.jar tomcat/lib/

 

in bin/kylin.sh add:

export 
HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX

 

download spark source code and build out 

spark--yarn-shuffle.jar transfer to $HADOOP_HOME/share/hadoop/yarn/lib

 

update yarn-site.xml add

spark_shuffle and YarnShuffleService

> kylin2.6.3使用Spark构建Cube,找不到Spark的Class
> --
>
> Key: KYLIN-4138
> URL: https://issues.apache.org/jira/browse/KYLIN-4138
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.3
>Reporter: shaozhipeng
>Priority: Major
> Attachments: kylin-spark0.png, kylin-spark1.png
>
>
> apache-kylin-2.6.3-bin-hbase1x.tar.gz
> $KYLIN_HOME/bin/download-spark.sh
> kylin.engine.spark-conf.spark.yarn.archive=hdfs://master:9000/kylin/spark/spark-libs.jar
>  
> 报错:
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/spark/api/java/function/Function
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
>  ~[kylin-core-job-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) 
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]
>  ... 77 more
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.spark.api.java.function.Function
>  at 
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
>  ~[catalina.jar:7.0.91]
>  at 
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
>  ~[catalina.jar:7.0.91]
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
>  ~[kylin-engine-spark-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
>  ~[kylin-core-job-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202) 
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>  at 
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
>  ~[kylin-server-base-2.6.3.jar:2.6.3]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-1856) Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908084#comment-16908084
 ] 

ASF subversion and git services commented on KYLIN-1856:


Commit 13e8e9be10b9699cdb6fa3b9e37c72d14e3332c0 in kylin's branch 
refs/heads/2.6.x from yaqian.zhang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=13e8e9b ]

KYLIN-1856 Clean up old error in step output immediately after resume job


> Kylin shows old error in job step output after resume - specifically in #4 
> Step Name: Build Dimension Dictionary
> 
>
> Key: KYLIN-1856
> URL: https://issues.apache.org/jira/browse/KYLIN-1856
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v1.5.2, v1.5.2.1
>Reporter: Richard Calaba
>Assignee: Yaqian Zhang
>Priority: Minor
> Fix For: v2.6.4
>
>
> I have realized that if my job stops with error and I try to recover the 
> error and resume the job - then the latest step starts again from scratch. 
> This is fine but in my opinion the log of the Step should clear as well - now 
> it is showing the error from my previous attempt.
> Specifically observed in #4 Step Name: Build Dimension Dictionary - but is 
> probbaly generic issue.
> To correct this: clear the log of the Build Step after the job Step is 
> resumed. Already when the job step is restarted, not after it is completed.
> (if Kylin fails i.e. for out of memory - it silently dies and analyzing the 
> step log shows wrong error (from previous run) - if it would be empty -> I 
> would know that most probable cause was that Kylin died)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4047) Use push-down query when division dynamic column cube query is not supported

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908078#comment-16908078
 ] 

ASF subversion and git services commented on KYLIN-4047:


Commit 9650c8d597a57472677a283d51956f1a69a1e735 in kylin's branch 
refs/heads/2.6.x from wangxiaojing123
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9650c8d ]

KYLIN-4047 Use push-down query when division dynamic column cube query is not 
supported (#689)

* KYLIN-4047 Use push-down query when division dynamic column cube query is not 
supported

* Minor, Code format


> Use push-down query when division dynamic column cube query is not supported
> 
>
> Key: KYLIN-4047
> URL: https://issues.apache.org/jira/browse/KYLIN-4047
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Affects Versions: Future, v2.6.2, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Minor
> Fix For: v2.6.4
>
>
> Hello all ,
>        At present, if a table has created cube, the query with division 
> dynamic column will throw an exception without pushdown query results, but if 
> switching an project, pushdown query will be carried out, which will give 
> users inconsistent experience. Can we also use pushdown query when query 
> division, multiplication dynamic column throw exceptions ?
> Scene:
> SQL:select round (car_dismatch_cnt*100/answer_order_cnt, 4) from db.table1.
> 1、Executes the SQL on project where has create a cube has db.table1,it will 
> throws the right side of the BinaryTuple Expressions columns is not supported 
> for / exception and is not queried through pushdown;
> 2、When change a project where not create the table cube , the same query  can 
> be executed to push down and get the right result.
> 3. Expect that no matter whether a table cube is created or not, when the 
> cube query is not success, the result can still be obtained by pushdown 
> query, which will give users a consistent experience.
> {code:java}
> // exception
> 2019-06-06 19:04:55,902 ERROR [http-bio-8081-exec-1] 
> controller.BasicController:63 :
> org.apache.kylin.rest.exception.InternalErrorException: That the right side 
> of the BinaryTupleExpression owns columns is not supported for /
> while executing SQL: "select DT,ORD_AMT/ORD_AMT from db.table1 LIMIT 5"
> at 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:452)
> at 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:373)
> at 
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at 
> 

[jira] [Commented] (KYLIN-4057) autoMerge job can not stop

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908085#comment-16908085
 ] 

ASF subversion and git services commented on KYLIN-4057:


Commit b767f4c08682155a3d6818aa6619a3380bb846aa in kylin's branch 
refs/heads/2.6.x from rupengwang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b767f4c ]

KYLIN-4057 Don't merge the job that has been discarded manually before.


> autoMerge job can not stop
> --
>
> Key: KYLIN-4057
> URL: https://issues.apache.org/jira/browse/KYLIN-4057
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.2
>Reporter: chenchen
>Assignee: wangrupeng
>Priority: Major
> Fix For: v2.6.4
>
> Attachments: 5031561517158_.pic_hd.jpg
>
>
> In this version,I used DistributedScheduler。 
> In my case, A cube config autoMerge,   But the job that automatically merges 
> segments is wrong  ,I can't fix this error, I can only discard and drop the 
> job, but Kylin's monitoring mechanism creates the same new job.
> I want to ask if there is any way to fix this problem without banning cube. 
> The error is reported as follows.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-3628) Query with lookup table always use latest snapshot

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908081#comment-16908081
 ] 

ASF subversion and git services commented on KYLIN-3628:


Commit 6703c7c5ec2eabc5a6f5f264f44082be332ff881 in kylin's branch 
refs/heads/2.6.x from XiaoxiangYu
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6703c7c ]

KYLIN-3628 Fix unexpected exception for select from Lookup Table

In some case, cube returned by findLatestSnapshot didn't contains the snapshot 
table
for what you need. Because some dimension which exists in model but was removed 
in cube design phase, query such as "select * from LookupTable" will throw 
unexpected exception and confused user.


> Query with lookup table always use latest snapshot
> --
>
> Key: KYLIN-3628
> URL: https://issues.apache.org/jira/browse/KYLIN-3628
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> If user queries a lookup table, Kylin will randomly selects a Cube (which has 
> the snapshot of this lookup table) to answer it. This causes uncertainty when 
> there are multiple cubes (share the same lookup): some cubes are newly built, 
> some not. If Kylin picks an old cube, the query result is old.
> To remove this uncertainty, for such queries, either always use latest 
> snapshot, or use earlist snapshot. We believe the "latest" version is better.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4115) Always load KafkaConsumerProperties

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908083#comment-16908083
 ] 

ASF subversion and git services commented on KYLIN-4115:


Commit 063456b4c8ede083806a4fc79df68cbc4623c5fb in kylin's branch 
refs/heads/2.6.x from Zhixiong Chen
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=063456b ]

KYLIN-4115 Always  load kafkaConsumerProperties


> Always load KafkaConsumerProperties
> ---
>
> Key: KYLIN-4115
> URL: https://issues.apache.org/jira/browse/KYLIN-4115
> Project: Kylin
>  Issue Type: Bug
>  Components: NRT Streaming
>Reporter: Zhixiong Chen
>Assignee: Zhixiong Chen
>Priority: Major
> Fix For: v2.6.4
>
>
> KafkaConsumerProperties can't override by conf/kylin-kafka-consumer.xml or 
> “kylin.source.kafka.config-override.”



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4093) Slow query pages should be open to all users of the project

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908086#comment-16908086
 ] 

ASF subversion and git services commented on KYLIN-4093:


Commit 93da88dbb7f2393f9728927bce6ea68c2f2e8483 in kylin's branch 
refs/heads/2.6.x from Liu Shaohui
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=93da88d ]

KYLIN-4093: Slow query pages should be open to all users of the project


> Slow query pages should be open to all users of the project
> ---
>
> Key: KYLIN-4093
> URL: https://issues.apache.org/jira/browse/KYLIN-4093
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
> Fix For: v2.6.4
>
>
> Currently the show query page only can been seen for kylin admins. It's very 
> useful for modlers and analysts for this project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4121) Cleanup hive view intermediate tables after job be finished

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908087#comment-16908087
 ] 

ASF subversion and git services commented on KYLIN-4121:


Commit 75179ab5d96a574e32807491edcf06d0a0292b07 in kylin's branch 
refs/heads/2.6.x from rupengwang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=75179ab ]

KYLIN-4121 Cleanup hive view intermediate tables after job be finished


> Cleanup hive view intermediate tables after job be finished
> ---
>
> Key: KYLIN-4121
> URL: https://issues.apache.org/jira/browse/KYLIN-4121
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Reporter: Shaofeng SHI
>Assignee: wangrupeng
>Priority: Major
>
> Reported by community user:
> I have a cube with a fact table join a lookup table  in hive, and both are 
> hive view. I submit a job one time per hour.
>  
> Kylin can drop the intermediate fact table , but doesn't drop the 
> intermediate lookup table .
>  
> I check the source code , and find out that , at '13 step: Hive Cleanup' ,  
> the source code has been commented.
>  
>  
> It is a legacy issue. Now KYLIN-3515 has fixed that, so the cleanup can be 
> enabled.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4013) Only show the cubes under one model

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908080#comment-16908080
 ] 

ASF subversion and git services commented on KYLIN-4013:


Commit d192bcd6b2bc9b04bedb8db42470e92c5131b715 in kylin's branch 
refs/heads/2.6.x from yuzhang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d192bcd ]

KYLIN-4013 Only show the cubes under one model


> Only show the cubes under one model
> ---
>
> Key: KYLIN-4013
> URL: https://issues.apache.org/jira/browse/KYLIN-4013
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> Some improvement for UI。
> User may want to see the cubes under specified model. Add an extra action 
> 'Cubes' in the drop-down list of model to filter cube list



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4137) Accelerate metadata reloading

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4137:

Fix Version/s: (was: v2.6.4)
   v3.0.0-beta

> Accelerate metadata reloading
> -
>
> Key: KYLIN-4137
> URL: https://issues.apache.org/jira/browse/KYLIN-4137
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Affects Versions: v2.6.2
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
> Fix For: v3.0.0-beta
>
>
> Now, org.apache.kylin.metadata.cachesync.CachedCrudAssist#reloadAll is using 
> an inappropriate method to deal with MySQLJdbcMetadata.
> As the every call of reloadAt(path) will access the database, which is almost 
> fine to HBase but really unfriendly to the MySQL(RDBMS).
> I think we should get all of the resource with single request instead of the 
> separate request to get every resource.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4066) No planner for not ROLE_ADMIN user on WebSite

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908082#comment-16908082
 ] 

ASF subversion and git services commented on KYLIN-4066:


Commit 4a30ba4f373ee13286e11fb1ef5e91002866caf1 in kylin's branch 
refs/heads/2.6.x from langdamao
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=4a30ba4 ]

[KYLIN-4066] Fix No planner for not ROLE_ADMIN user

Signed-off-by: langdamao 


> No planner for not ROLE_ADMIN user on WebSite
> -
>
> Key: KYLIN-4066
> URL: https://issues.apache.org/jira/browse/KYLIN-4066
> Project: Kylin
>  Issue Type: Bug
>  Components: Website
>Affects Versions: v2.6.1
> Environment: macOs 
>Reporter: langdamao
>Priority: Major
>  Labels: easyfix
> Fix For: v2.6.4
>
>
> There is no planner page for none ROLE_ADMIN user ,even though he has ADMIN 
> access for the project.
> I found in webapp/app/partials/cubes/cube_detail.html line44
> {code:java}
>  ng-if="(userService.hasRole('ROLE_ADMIN') || hasPermission(cube, 
> permissions.ADMINISTRATION.mask)) && isShowCubeplanner">  ng-click="cube.visiblePage='planner';getCubePlanner(cube);">Planner 
> {code}
> but other place the hasPermission function use as
> {code:java}
> hasPermission('cube',cube, permissions.ADMINISTRATION.mask) 
> {code}
> the function hasPermission is defined in webapp/app/js/controllers/page.js 
> line100
> {code:java}
> $scope.hasPermission = function (accessType, entity){
> ...
> if (accessType === 'cube') {
>   project = entity.project
> } else if (accessType === 'project') {
>   project = entity && entity.name || entity.selectedProject
> } else if (accessType === 'model') {
>   project =  ProjectModel.getProjectByCubeModel(entity.name)
> }
> ...
> }{code}
> I guess whether it's cause by no param of accessType.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4093) Slow query pages should be open to all users of the project

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4093:

Fix Version/s: v2.6.4

> Slow query pages should be open to all users of the project
> ---
>
> Key: KYLIN-4093
> URL: https://issues.apache.org/jira/browse/KYLIN-4093
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
> Fix For: v2.6.4
>
>
> Currently the show query page only can been seen for kylin admins. It's very 
> useful for modlers and analysts for this project.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4057) autoMerge job can not stop

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4057:

Fix Version/s: v2.6.4

> autoMerge job can not stop
> --
>
> Key: KYLIN-4057
> URL: https://issues.apache.org/jira/browse/KYLIN-4057
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.2
>Reporter: chenchen
>Assignee: wangrupeng
>Priority: Major
> Fix For: v2.6.4
>
> Attachments: 5031561517158_.pic_hd.jpg
>
>
> In this version,I used DistributedScheduler。 
> In my case, A cube config autoMerge,   But the job that automatically merges 
> segments is wrong  ,I can't fix this error, I can only discard and drop the 
> job, but Kylin's monitoring mechanism creates the same new job.
> I want to ask if there is any way to fix this problem without banning cube. 
> The error is reported as follows.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4092) Support setting seperate jvm params for kylin backgroud tools

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4092:

Fix Version/s: v3.0.0-beta

> Support setting seperate jvm params for kylin backgroud tools
> -
>
> Key: KYLIN-4092
> URL: https://issues.apache.org/jira/browse/KYLIN-4092
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
> Fix For: v3.0.0-beta
>
>
> Usually, the memory set in setenv.sh for query server is larger then 8G, 
> which is not suitable for kylin background tools (meta cleaup, storage 
> cleanup, health check) 
> So It's better to have a seperate env for kylin tools



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4067) Speed up response of kylin cube page

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4067:

Fix Version/s: (was: v2.6.4)
   v3.0.0-beta

> Speed up response of kylin cube page
> 
>
> Key: KYLIN-4067
> URL: https://issues.apache.org/jira/browse/KYLIN-4067
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v3.0.0-alpha, v2.6.3
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: v3.0.0-beta
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hi Team:
> My Kylin project has more than 100 cubes. It is very slowly when open kylin 
> web page.
> I find that kylin load all information of cubes  when open cube page. It load 
> all information of all cubes at every cube in kylin project. The url of 
> request is "http://ip:port/kylin/api/cubes?limit=65535=0;. For 
> example, if one project has 10 cubes, this request will be called by 10 
> times.  But this information is only be used to determine whether the name is 
> duplicated when adding a new cube. 
> This page loading mechanism can be optimized. Getting all the information of 
> all cube only needs to be called when adding a new cube.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-1856) Kylin shows old error in job step output after resume - specifically in #4 Step Name: Build Dimension Dictionary

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-1856:

Fix Version/s: v2.6.4

> Kylin shows old error in job step output after resume - specifically in #4 
> Step Name: Build Dimension Dictionary
> 
>
> Key: KYLIN-1856
> URL: https://issues.apache.org/jira/browse/KYLIN-1856
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v1.5.2, v1.5.2.1
>Reporter: Richard Calaba
>Assignee: Yaqian Zhang
>Priority: Minor
> Fix For: v2.6.4
>
>
> I have realized that if my job stops with error and I try to recover the 
> error and resume the job - then the latest step starts again from scratch. 
> This is fine but in my opinion the log of the Step should clear as well - now 
> it is showing the error from my previous attempt.
> Specifically observed in #4 Step Name: Build Dimension Dictionary - but is 
> probbaly generic issue.
> To correct this: clear the log of the Build Step after the job Step is 
> resumed. Already when the job step is restarted, not after it is completed.
> (if Kylin fails i.e. for out of memory - it silently dies and analyzing the 
> step log shows wrong error (from previous run) - if it would be empty -> I 
> would know that most probable cause was that Kylin died)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4067) Speed up response of kylin cube page

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4067:

Fix Version/s: (was: Future)
   v2.6.4

> Speed up response of kylin cube page
> 
>
> Key: KYLIN-4067
> URL: https://issues.apache.org/jira/browse/KYLIN-4067
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v3.0.0-alpha, v2.6.3
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: v2.6.4
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hi Team:
> My Kylin project has more than 100 cubes. It is very slowly when open kylin 
> web page.
> I find that kylin load all information of cubes  when open cube page. It load 
> all information of all cubes at every cube in kylin project. The url of 
> request is "http://ip:port/kylin/api/cubes?limit=65535=0;. For 
> example, if one project has 10 cubes, this request will be called by 10 
> times.  But this information is only be used to determine whether the name is 
> duplicated when adding a new cube. 
> This page loading mechanism can be optimized. Getting all the information of 
> all cube only needs to be called when adding a new cube.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4067) Speed up response of kylin cube page

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4067:

Affects Version/s: v2.6.3

> Speed up response of kylin cube page
> 
>
> Key: KYLIN-4067
> URL: https://issues.apache.org/jira/browse/KYLIN-4067
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v3.0.0-alpha, v2.6.3
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: Future
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hi Team:
> My Kylin project has more than 100 cubes. It is very slowly when open kylin 
> web page.
> I find that kylin load all information of cubes  when open cube page. It load 
> all information of all cubes at every cube in kylin project. The url of 
> request is "http://ip:port/kylin/api/cubes?limit=65535=0;. For 
> example, if one project has 10 cubes, this request will be called by 10 
> times.  But this information is only be used to determine whether the name is 
> duplicated when adding a new cube. 
> This page loading mechanism can be optimized. Getting all the information of 
> all cube only needs to be called when adding a new cube.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4067) Speed up response of kylin cube page

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4067:

Affects Version/s: (was: v3.0.0-beta)
   v3.0.0-alpha

> Speed up response of kylin cube page
> 
>
> Key: KYLIN-4067
> URL: https://issues.apache.org/jira/browse/KYLIN-4067
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v3.0.0-alpha
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: Future
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hi Team:
> My Kylin project has more than 100 cubes. It is very slowly when open kylin 
> web page.
> I find that kylin load all information of cubes  when open cube page. It load 
> all information of all cubes at every cube in kylin project. The url of 
> request is "http://ip:port/kylin/api/cubes?limit=65535=0;. For 
> example, if one project has 10 cubes, this request will be called by 10 
> times.  But this information is only be used to determine whether the name is 
> duplicated when adding a new cube. 
> This page loading mechanism can be optimized. Getting all the information of 
> all cube only needs to be called when adding a new cube.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Reopened] (KYLIN-4067) Speed up response of kylin cube page

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen reopened KYLIN-4067:
-

> Speed up response of kylin cube page
> 
>
> Key: KYLIN-4067
> URL: https://issues.apache.org/jira/browse/KYLIN-4067
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v3.0.0-beta
>Reporter: zhao jintao
>Assignee: zhao jintao
>Priority: Minor
> Fix For: Future
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hi Team:
> My Kylin project has more than 100 cubes. It is very slowly when open kylin 
> web page.
> I find that kylin load all information of cubes  when open cube page. It load 
> all information of all cubes at every cube in kylin project. The url of 
> request is "http://ip:port/kylin/api/cubes?limit=65535=0;. For 
> example, if one project has 10 cubes, this request will be called by 10 
> times.  But this information is only be used to determine whether the name is 
> duplicated when adding a new cube. 
> This page loading mechanism can be optimized. Getting all the information of 
> all cube only needs to be called when adding a new cube.
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Closed] (KYLIN-4039) ZookeeperDistributedLock may not release lock when unlock operation was interrupted

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen closed KYLIN-4039.
---
Resolution: Fixed

> ZookeeperDistributedLock may not release lock when unlock operation was 
> interrupted
> ---
>
> Key: KYLIN-4039
> URL: https://issues.apache.org/jira/browse/KYLIN-4039
> Project: Kylin
>  Issue Type: Bug
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> ZookeeperDistributedLock may hold the lock and not release it when the unlock 
> operation was interrupted.
> Because the unlock operation contains two steps: 
> 1. peekLock: get the owner of the lock
> 2. purgeLock: purge the lock if the owner of the lock is the current client.
> If the peekLock step is interrupted, the purgeLock step won't be executed. 
> Thus the lock won't be released.
> Meanwhile, the lock operation should also consider the interrupt cases.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4039) ZookeeperDistributedLock may not release lock when unlock operation was interrupted

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4039:

Fix Version/s: (was: v2.6.4)

> ZookeeperDistributedLock may not release lock when unlock operation was 
> interrupted
> ---
>
> Key: KYLIN-4039
> URL: https://issues.apache.org/jira/browse/KYLIN-4039
> Project: Kylin
>  Issue Type: Bug
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> ZookeeperDistributedLock may hold the lock and not release it when the unlock 
> operation was interrupted.
> Because the unlock operation contains two steps: 
> 1. peekLock: get the owner of the lock
> 2. purgeLock: purge the lock if the owner of the lock is the current client.
> If the peekLock step is interrupted, the purgeLock step won't be executed. 
> Thus the lock won't be released.
> Meanwhile, the lock operation should also consider the interrupt cases.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4066) No planner for not ROLE_ADMIN user on WebSite

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4066:

Fix Version/s: (was: Future)
   v2.6.4

> No planner for not ROLE_ADMIN user on WebSite
> -
>
> Key: KYLIN-4066
> URL: https://issues.apache.org/jira/browse/KYLIN-4066
> Project: Kylin
>  Issue Type: Bug
>  Components: Website
>Affects Versions: v2.6.1
> Environment: macOs 
>Reporter: langdamao
>Priority: Major
>  Labels: easyfix
> Fix For: v2.6.4
>
>
> There is no planner page for none ROLE_ADMIN user ,even though he has ADMIN 
> access for the project.
> I found in webapp/app/partials/cubes/cube_detail.html line44
> {code:java}
>  ng-if="(userService.hasRole('ROLE_ADMIN') || hasPermission(cube, 
> permissions.ADMINISTRATION.mask)) && isShowCubeplanner">  ng-click="cube.visiblePage='planner';getCubePlanner(cube);">Planner 
> {code}
> but other place the hasPermission function use as
> {code:java}
> hasPermission('cube',cube, permissions.ADMINISTRATION.mask) 
> {code}
> the function hasPermission is defined in webapp/app/js/controllers/page.js 
> line100
> {code:java}
> $scope.hasPermission = function (accessType, entity){
> ...
> if (accessType === 'cube') {
>   project = entity.project
> } else if (accessType === 'project') {
>   project = entity && entity.name || entity.selectedProject
> } else if (accessType === 'model') {
>   project =  ProjectModel.getProjectByCubeModel(entity.name)
> }
> ...
> }{code}
> I guess whether it's cause by no param of accessType.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4099) Using no blocking RDD unpersist in spark cubing job

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4099:

Fix Version/s: (was: v3.0.0)
   v2.6.4

> Using no blocking RDD unpersist in spark cubing job 
> 
>
> Key: KYLIN-4099
> URL: https://issues.apache.org/jira/browse/KYLIN-4099
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.4
>
>
> By default, the unpersist operation of RDD in spark is blocking which may 
> cost a lot time and
> some times it may failed for some spark executors lost. 
> We can set blocking false to improve it.
> {code:java}
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:208)
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> scala.concurrent.Await$.result(package.scala:190)
> org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81)
> org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:127)
> org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1709)
> org.apache.spark.rdd.RDD.unpersist(RDD.scala:216)
> org.apache.spark.api.java.JavaPairRDD.unpersist(JavaPairRDD.scala:73)
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:204)
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653){code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-3628) Query with lookup table always use latest snapshot

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-3628:

Fix Version/s: v2.6.4

> Query with lookup table always use latest snapshot
> --
>
> Key: KYLIN-3628
> URL: https://issues.apache.org/jira/browse/KYLIN-3628
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> If user queries a lookup table, Kylin will randomly selects a Cube (which has 
> the snapshot of this lookup table) to answer it. This causes uncertainty when 
> there are multiple cubes (share the same lookup): some cubes are newly built, 
> some not. If Kylin picks an old cube, the query result is old.
> To remove this uncertainty, for such queries, either always use latest 
> snapshot, or use earlist snapshot. We believe the "latest" version is better.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4037) Can't Cleanup Data in Hbase's HDFS Storage When Deploy Apache Kylin with Standalone HBase Cluster

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4037:

Fix Version/s: v2.6.4

> Can't Cleanup Data in Hbase's HDFS Storage When Deploy Apache Kylin with 
> Standalone HBase Cluster
> -
>
> Key: KYLIN-4037
> URL: https://issues.apache.org/jira/browse/KYLIN-4037
> Project: Kylin
>  Issue Type: Bug
>  Components: Others, Tools, Build and Test
>Affects Versions: Future, all, v2.6.1, v2.6.2
>Reporter: wangxiaojing
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
> Attachments: image-2019-06-10-15-56-58-124.png
>
>
> Hello,
>      When Deploy Apache Kylin with Standalone HBase Cluster and use kylin 
> cleanup storage tools to cleanup dirty data ,but it seems only cleanup the 
> data In computer cluster HDFS/HIVE and table in hbase cluster, not clean the 
> HDFS data in hbase,also have many dirdy data in hbase HDFS .
>     Tools: [http://kylin.apache.org/docs/howto/howto_cleanup_storage.html]
>      Version:master as well as 2.6.2 release.
>      Data that should be cleaned up in the HBase HDFS cluster ,which cube or 
> segment has been droped.
> !image-2019-06-10-15-56-58-124.png!



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Reopened] (KYLIN-4013) Only show the cubes under one model

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen reopened KYLIN-4013:
-

Reopen to set fix version

> Only show the cubes under one model
> ---
>
> Key: KYLIN-4013
> URL: https://issues.apache.org/jira/browse/KYLIN-4013
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> Some improvement for UI。
> User may want to see the cubes under specified model. Add an extra action 
> 'Cubes' in the drop-down list of model to filter cube list



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4013) Only show the cubes under one model

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4013:

Fix Version/s: v2.6.4

> Only show the cubes under one model
> ---
>
> Key: KYLIN-4013
> URL: https://issues.apache.org/jira/browse/KYLIN-4013
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> Some improvement for UI。
> User may want to see the cubes under specified model. Add an extra action 
> 'Cubes' in the drop-down list of model to filter cube list



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[GitHub] [kylin] hit-lacus commented on a change in pull request #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
hit-lacus commented on a change in pull request #797: KYLIN-4114 Provided a 
self-contained docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#discussion_r314266740
 
 

 ##
 File path: docker/run_container.sh
 ##
 @@ -0,0 +1,9 @@
+docker run -d \
+-m 8G \
+-p 7070:7070 \
+-p 8088:8088 \
+-p 50070:50070 \
+-p 8032:8032 \
+-p 8042:8042 \
+-p 60010:60010 \
+apache-kylin-standalone
 
 Review comment:
   Should be `codingforfun/apache-kylin-standalone` instead.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (KYLIN-4047) Use push-down query when division dynamic column cube query is not supported

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4047:

Fix Version/s: v2.6.4

> Use push-down query when division dynamic column cube query is not supported
> 
>
> Key: KYLIN-4047
> URL: https://issues.apache.org/jira/browse/KYLIN-4047
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Affects Versions: Future, v2.6.2, v3.0.0-alpha
>Reporter: wangxiaojing
>Priority: Minor
> Fix For: v2.6.4
>
>
> Hello all ,
>        At present, if a table has created cube, the query with division 
> dynamic column will throw an exception without pushdown query results, but if 
> switching an project, pushdown query will be carried out, which will give 
> users inconsistent experience. Can we also use pushdown query when query 
> division, multiplication dynamic column throw exceptions ?
> Scene:
> SQL:select round (car_dismatch_cnt*100/answer_order_cnt, 4) from db.table1.
> 1、Executes the SQL on project where has create a cube has db.table1,it will 
> throws the right side of the BinaryTuple Expressions columns is not supported 
> for / exception and is not queried through pushdown;
> 2、When change a project where not create the table cube , the same query  can 
> be executed to push down and get the right result.
> 3. Expect that no matter whether a table cube is created or not, when the 
> cube query is not success, the result can still be obtained by pushdown 
> query, which will give users a consistent experience.
> {code:java}
> // exception
> 2019-06-06 19:04:55,902 ERROR [http-bio-8081-exec-1] 
> controller.BasicController:63 :
> org.apache.kylin.rest.exception.InternalErrorException: That the right side 
> of the BinaryTupleExpression owns columns is not supported for /
> while executing SQL: "select DT,ORD_AMT/ORD_AMT from db.table1 LIMIT 5"
> at 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:452)
> at 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:373)
> at 
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at 
> 

[GitHub] [kylin] hit-lacus commented on a change in pull request #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
hit-lacus commented on a change in pull request #797: KYLIN-4114 Provided a 
self-contained docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#discussion_r314266740
 
 

 ##
 File path: docker/run_container.sh
 ##
 @@ -0,0 +1,9 @@
+docker run -d \
+-m 8G \
+-p 7070:7070 \
+-p 8088:8088 \
+-p 50070:50070 \
+-p 8032:8032 \
+-p 8042:8042 \
+-p 60010:60010 \
+apache-kylin-standalone
 
 Review comment:
   Should be `codingforfun/apache-kylin-standalone` instead.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] hit-lacus commented on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
hit-lacus commented on issue #797: KYLIN-4114 Provided a self-contained docker 
image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-521607101
 
 
   Hi, @weibin0516 
   
   Did you update your PR? I cannot find `kylin.sh start` & `kylin.sh streaming 
start` anywhere.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Reopened] (KYLIN-4074) Exception in thread "Memcached IO over {MemcachedConnection to ..." java.lang.NullPointerException

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen reopened KYLIN-4074:
-

Reopen to set fix version

> Exception in thread "Memcached IO over {MemcachedConnection to ..." 
> java.lang.NullPointerException
> --
>
> Key: KYLIN-4074
> URL: https://issues.apache.org/jira/browse/KYLIN-4074
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.4.0, v2.5.0, v2.6.2
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> This exception will lead to the failure of thread "Memcached IO over  
> \{MemcachedConnection to ...\}",
> so, the inputQueue of memcache client will be blocked, and every request will 
> take 2000ms (timeout).
> The log is as follows:
> {code:java}
> 2019-07-01 15:21:24,039 INFO  [Query 
> 3f75c477-9c38-35bf-24a0-656459d9ebb1-212641] v2.CubeHBaseEndpointRPC:197 : 
> Query-3f75c477-9c38-35bf-24a0-656459d9ebb1: no segment result is cached for 
> segment:1102_pv_uv_users_di[2016070100_2017010100], take time:2000ms
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4074) Exception in thread "Memcached IO over {MemcachedConnection to ..." java.lang.NullPointerException

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4074:

Fix Version/s: v2.6.4

> Exception in thread "Memcached IO over {MemcachedConnection to ..." 
> java.lang.NullPointerException
> --
>
> Key: KYLIN-4074
> URL: https://issues.apache.org/jira/browse/KYLIN-4074
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.4.0, v2.5.0, v2.6.2
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> This exception will lead to the failure of thread "Memcached IO over  
> \{MemcachedConnection to ...\}",
> so, the inputQueue of memcache client will be blocked, and every request will 
> take 2000ms (timeout).
> The log is as follows:
> {code:java}
> 2019-07-01 15:21:24,039 INFO  [Query 
> 3f75c477-9c38-35bf-24a0-656459d9ebb1-212641] v2.CubeHBaseEndpointRPC:197 : 
> Query-3f75c477-9c38-35bf-24a0-656459d9ebb1: no segment result is cached for 
> segment:1102_pv_uv_users_di[2016070100_2017010100], take time:2000ms
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (KYLIN-4039) ZookeeperDistributedLock may not release lock when unlock operation was interrupted

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen updated KYLIN-4039:

Fix Version/s: v2.6.4

> ZookeeperDistributedLock may not release lock when unlock operation was 
> interrupted
> ---
>
> Key: KYLIN-4039
> URL: https://issues.apache.org/jira/browse/KYLIN-4039
> Project: Kylin
>  Issue Type: Bug
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-alpha2, v2.6.4
>
>
> ZookeeperDistributedLock may hold the lock and not release it when the unlock 
> operation was interrupted.
> Because the unlock operation contains two steps: 
> 1. peekLock: get the owner of the lock
> 2. purgeLock: purge the lock if the owner of the lock is the current client.
> If the peekLock step is interrupted, the purgeLock step won't be executed. 
> Thus the lock won't be released.
> Meanwhile, the lock operation should also consider the interrupt cases.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Reopened] (KYLIN-4039) ZookeeperDistributedLock may not release lock when unlock operation was interrupted

2019-08-15 Thread nichunen (JIRA)


 [ 
https://issues.apache.org/jira/browse/KYLIN-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nichunen reopened KYLIN-4039:
-

Reopen to add fix version

> ZookeeperDistributedLock may not release lock when unlock operation was 
> interrupted
> ---
>
> Key: KYLIN-4039
> URL: https://issues.apache.org/jira/browse/KYLIN-4039
> Project: Kylin
>  Issue Type: Bug
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-alpha2
>
>
> ZookeeperDistributedLock may hold the lock and not release it when the unlock 
> operation was interrupted.
> Because the unlock operation contains two steps: 
> 1. peekLock: get the owner of the lock
> 2. purgeLock: purge the lock if the owner of the lock is the current client.
> If the peekLock step is interrupted, the purgeLock step won't be executed. 
> Thus the lock won't be released.
> Meanwhile, the lock operation should also consider the interrupt cases.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[GitHub] [kylin] codecov-io edited a comment on issue #800: 添加当天作业过滤

2019-08-15 Thread GitBox
codecov-io edited a comment on issue #800: 添加当天作业过滤
URL: https://github.com/apache/kylin/pull/800#issuecomment-521109237
 
 
   # [Codecov](https://codecov.io/gh/apache/kylin/pull/800?src=pr=h1) Report
   > Merging [#800](https://codecov.io/gh/apache/kylin/pull/800?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/kylin/commit/abc0ab80a04f1bab95f93901f1b802aecaf72382?src=pr=desc)
 will **increase** coverage by `0.01%`.
   > The diff coverage is `0%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/kylin/pull/800/graphs/tree.svg?width=650=JawVgbgsVo=150=pr)](https://codecov.io/gh/apache/kylin/pull/800?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master #800  +/-   ##
   
   + Coverage 25.62%   25.63%   +0.01% 
 Complexity 6048 6048  
   
 Files  1398 1398  
 Lines 8341183428  +17 
 Branches  1168011681   +1 
   
   + Hits  2137121387  +16 
   - Misses5998559991   +6 
   + Partials   2055 2050   -5
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/kylin/pull/800?src=pr=tree) | Coverage Δ 
| Complexity Δ | |
   |---|---|---|---|
   | 
[...g/apache/kylin/job/constant/JobTimeFilterEnum.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9jb25zdGFudC9Kb2JUaW1lRmlsdGVyRW51bS5qYXZh)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...java/org/apache/kylin/rest/service/JobService.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9Kb2JTZXJ2aWNlLmphdmE=)
 | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: |
   | 
[...main/java/org/apache/kylin/common/KylinConfig.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZy5qYXZh)
 | `39.53% <0%> (-1.6%)` | `27% <0%> (ø)` | |
   | 
[...g/apache/kylin/metadata/cachesync/Broadcaster.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWV0YWRhdGEvY2FjaGVzeW5jL0Jyb2FkY2FzdGVyLmphdmE=)
 | `52% <0%> (+0.48%)` | `23% <0%> (ø)` | :arrow_down: |
   | 
[...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh)
 | `76.74% <0%> (+2.32%)` | `12% <0%> (ø)` | :arrow_down: |
   | 
[...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/800/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh)
 | `70.21% <0%> (+3.64%)` | `7% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/kylin/pull/800?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/kylin/pull/800?src=pr=footer). Last 
update 
[abc0ab8...16e20c7](https://codecov.io/gh/apache/kylin/pull/800?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] weibin0516 commented on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
weibin0516 commented on issue #797: KYLIN-4114 Provided a self-contained docker 
image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-521568933
 
 
   @hit-lacus , thanks for review, all functions except `Integration test` work 
fine work fine. In my opinion, Integration test is not necessarily needed, may 
be i will solve it in next pr.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] weibin0516 commented on a change in pull request #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
weibin0516 commented on a change in pull request #797: KYLIN-4114 Provided a 
self-contained docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#discussion_r314226056
 
 

 ##
 File path: docker/Dockerfile
 ##
 @@ -0,0 +1,80 @@
+# Docker image for apache kylin
+FROM centos:6.9
+
+MAINTAINER weibin0516 
+
+ENV HIVE_VERSION 1.2.1
+ENV HADOOP_VERSION 2.7.0
+ENV HBASE_VERSION 1.1.2
+ENV SPARK_VERSION spark-2.3.1-bin-hadoop2.6
+ENV ZK_VERSION 3.4.6
+
+ENV JAVA_HOME /home/admin/jdk1.8.0_141
+ENV MVN_HOME /home/admin/apache-maven-3.6.1
+ENV HADOOP_HOME /home/admin/hadoop-$HADOOP_VERSION
+ENV HIVE_HOME /home/admin/apache-hive-$HIVE_VERSION-bin
+ENV HADOOP_CONF $HADOOP_HOME/etc/hadoop
+ENV HBASE_HOME /home/admin/hbase-$HBASE_VERSION
+ENV SPARK_HOME /home/admin/$SPARK_VERSION
+ENV ZK_HOME /home/admin/zookeeper-$ZK_VERSION
+ENV PATH 
$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HIVE_HOME/bin:$HBASE_HOME/bin:$MVN_HOME/bin:$SPARK_HOME/bin
+
+USER root
+
+WORKDIR /home/admin
+
+# install tools
+RUN yum -y install lsof.x86_64 wget.x86_64 tar.x86_64 git.x86_64 
mysql-server.x86_64 mysql.x86_64
+
+# install mvn
+RUN wget 
http://mirrors.ocf.berkeley.edu/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz
+RUN tar -zxvf apache-maven-3.6.1-bin.tar.gz
+COPY conf/maven/settings.xml $MVN_HOME/conf/settings.xml
+RUN rm -f apache-maven-3.6.1-bin.tar.gz
+
+# install npm
+RUN curl -sL https://rpm.nodesource.com/setup_8.x | bash -
+RUN yum install -y nodejs
+
+# setup jdk
+RUN wget --no-cookies --no-check-certificate --header "Cookie: 
gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; 
oraclelicense=accept-securebackup-cookie" 
"http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.tar.gz;
+RUN tar -zxvf /home/admin/jdk-8u141-linux-x64.tar.gz
+RUN rm -f /home/admin/jdk-8u141-linux-x64.tar.gz
+
+# setup hadoop
+RUN wget 
https://archive.apache.org/dist/hadoop/core/hadoop-2.7.0/hadoop-2.7.0.tar.gz
+RUN tar -zxvf /home/admin/hadoop-$HADOOP_VERSION.tar.gz
+RUN rm -f /home/admin/hadoop-$HADOOP_VERSION.tar.gz
+
+COPY conf/hadoop/* $HADOOP_CONF/
+RUN mkdir -p /data/hadoop
+
+# setup hbase
+RUN wget https://archive.apache.org/dist/hbase/1.1.2/hbase-1.1.2-bin.tar.gz
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] weibin0516 commented on a change in pull request #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
weibin0516 commented on a change in pull request #797: KYLIN-4114 Provided a 
self-contained docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#discussion_r314225993
 
 

 ##
 File path: docker/use-guide.md
 ##
 @@ -0,0 +1,61 @@
+Docker for Kylin is convenient for users to quickly try Kylin, as well as for 
developers to debug and verify code modifications.
+
+In the image, including:
+* Jdk 1.8
+* Hadoop 2.7.0
+* Hive 1.2.1
+* Hbase 1.1.2
+* Spark 2.3.1
+* Zookeeper 3.4.6
+* Mysql
+* Maven 3.6.1
+
+## Build image
+Execute the following command to build image
+```
+sh build_image.sh
+```
+This operation may last for several tens of minutes due to the download of the 
installation package.
+
+## Launch container
+After image built, execute
+```
+sh run_container.sh
+```
+to launch container which will automatically start the following services:
+* Mysql
+* NameNode
+* DataNode
+* ResourceManager
+* NodeManager
+* HMaster
+
+Then execute command [docker 
exec](https://docs.docker.com/engine/reference/commandline/exec/) to run 
command in then running Kylin container.
+
+## For user to try Kylin
+All Kylin service dependences are ready, just need to download Kylin release 
binary from http://kylin.apache.org/download/ and start Kylin service.
+
+For example:
+```
+cd /home/admin
+wget 
https://archive.apache.org/dist/kylin/apache-kylin-3.0.0-alpha2/apache-kylin-3.0.0-alpha2-bin-hbase1x.tar.gz
+tar -zxvf apache-kylin-3.0.0-alpha2-bin-hbase1x.tar.gz
+cd apache-kylin-3.0.0-alpha2-bin-hbase1x
+bin/kylin.sh start
+```
+
+## For developer to debug and verify
+When building image, the latest kylin source code will be copied to 
`/home/admin/kylin_sourcecode`.
+
+Execute the following commands to do package:
+```
+cd /home/admin/kylin_sourcecode
+build/script/package.sh
+```
+After the package is complete, use the package in 
`/home/admin/kylin_sourcecode/dist` to start the Kylin service.
+
+You can also run CI inside the container:
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] coveralls edited a comment on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
coveralls edited a comment on issue #797: KYLIN-4114 Provided a self-contained 
docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-520164477
 
 
   ## Pull Request Test Coverage Report for [Build 
4910](https://coveralls.io/builds/25149884)
   
   * **0** of **0**   changed or added relevant lines in **0** files are 
covered.
   * **2** unchanged lines in **1** file lost coverage.
   * Overall coverage increased (+**0.006%**) to **28.095%**
   
   ---
   
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-|--|--: |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/25149884/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124)
 | 2 | 68.46% |
   
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/25149884/badge)](https://coveralls.io/builds/25149884)
 |
   | :-- | --: |
   | Change from base [Build 4909](https://coveralls.io/builds/25147213): |  
0.006% |
   | Covered Lines: | 23434 |
   | Relevant Lines: | 83411 |
   
   ---
   #   - [Coveralls](https://coveralls.io)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (KYLIN-4131) Broadcaster memory leak

2019-08-15 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907939#comment-16907939
 ] 

ASF subversion and git services commented on KYLIN-4131:


Commit 194272986e6dec96a78f1228d27e0208a68a268b in kylin's branch 
refs/heads/master from chao long
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1942729 ]

KYLIN-4131 Fix Broadcaster memory leak


> Broadcaster memory leak
> ---
>
> Key: KYLIN-4131
> URL: https://issues.apache.org/jira/browse/KYLIN-4131
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Critical
> Fix For: v3.0.0
>
> Attachments: image-2019-08-12-16-39-46-262.png, 
> image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png, 
> image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png
>
>
> h3. *Test evidence*
> Use MAT(Memory Analyzer) to analyze this issue.
>  # start  Kylin, dump a jvm heap info snapshot and import into MAT. We can 
> see there is only one Broadcaster and ProjectManager object.
>  !image-2019-08-12-17-22-15-783.png|width=721,height=520!
>  # Call 'list cube' api and 'reload metadata' api many times. We can see 
> there are many BroadCaster and Manager objects in the memory, which should be 
> recycled by GC, but they are not.
>  !image-2019-08-12-17-22-23-477.png|width=724,height=498!
>  
> *Root cause*
> Broadcaster hold the ref of Listener object defined in Manager, and the 
> Listener object hold the ref of Manager, because the impl class of Listener 
> is an inner class of Manager.
> Broadcaster.java
> {code:java}
> private Map> listenerMap = Maps.newConcurrentMap();
> {code}
>  ProjectManager.java -> ProjectSyncListener
> {code:java}
> private class ProjectSyncListener extends Broadcaster.Listener {
> @Override
> public void onEntityChange(Broadcaster broadcaster, String entity, Event 
> event, String cacheKey)
> throws IOException {
> String project = cacheKey;
> if (event == Event.DROP) {
> removeProjectLocal(project);
> return;
> }
> reloadProjectQuietly(project);
> broadcaster.notifyProjectSchemaUpdate(project);
> broadcaster.notifyProjectDataUpdate(project);
> }
> }
> {code}
> And Broadcaster can't be recycled by GC because it holds a running thread, 
> which cause the Manager object also not being recycled.
> *After Fix*
>  The Broadcaster objects and Manager objects are recycled by GC.
> !image-2019-08-12-17-16-05-532.png!
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (KYLIN-4131) Broadcaster memory leak

2019-08-15 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907938#comment-16907938
 ] 

ASF GitHub Bot commented on KYLIN-4131:
---

nichunen commented on pull request #799: KYLIN-4131 Fix Broadcaster memory leak
URL: https://github.com/apache/kylin/pull/799
 
 
   
 

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Broadcaster memory leak
> ---
>
> Key: KYLIN-4131
> URL: https://issues.apache.org/jira/browse/KYLIN-4131
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Critical
> Fix For: v3.0.0
>
> Attachments: image-2019-08-12-16-39-46-262.png, 
> image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png, 
> image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png
>
>
> h3. *Test evidence*
> Use MAT(Memory Analyzer) to analyze this issue.
>  # start  Kylin, dump a jvm heap info snapshot and import into MAT. We can 
> see there is only one Broadcaster and ProjectManager object.
>  !image-2019-08-12-17-22-15-783.png|width=721,height=520!
>  # Call 'list cube' api and 'reload metadata' api many times. We can see 
> there are many BroadCaster and Manager objects in the memory, which should be 
> recycled by GC, but they are not.
>  !image-2019-08-12-17-22-23-477.png|width=724,height=498!
>  
> *Root cause*
> Broadcaster hold the ref of Listener object defined in Manager, and the 
> Listener object hold the ref of Manager, because the impl class of Listener 
> is an inner class of Manager.
> Broadcaster.java
> {code:java}
> private Map> listenerMap = Maps.newConcurrentMap();
> {code}
>  ProjectManager.java -> ProjectSyncListener
> {code:java}
> private class ProjectSyncListener extends Broadcaster.Listener {
> @Override
> public void onEntityChange(Broadcaster broadcaster, String entity, Event 
> event, String cacheKey)
> throws IOException {
> String project = cacheKey;
> if (event == Event.DROP) {
> removeProjectLocal(project);
> return;
> }
> reloadProjectQuietly(project);
> broadcaster.notifyProjectSchemaUpdate(project);
> broadcaster.notifyProjectDataUpdate(project);
> }
> }
> {code}
> And Broadcaster can't be recycled by GC because it holds a running thread, 
> which cause the Manager object also not being recycled.
> *After Fix*
>  The Broadcaster objects and Manager objects are recycled by GC.
> !image-2019-08-12-17-16-05-532.png!
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[GitHub] [kylin] nichunen merged pull request #799: KYLIN-4131 Fix Broadcaster memory leak

2019-08-15 Thread GitBox
nichunen merged pull request #799: KYLIN-4131 Fix Broadcaster memory leak
URL: https://github.com/apache/kylin/pull/799
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [kylin] codecov-io edited a comment on issue #797: KYLIN-4114 Provided a self-contained docker image for Kylin

2019-08-15 Thread GitBox
codecov-io edited a comment on issue #797: KYLIN-4114 Provided a self-contained 
docker image for Kylin
URL: https://github.com/apache/kylin/pull/797#issuecomment-520163608
 
 
   # [Codecov](https://codecov.io/gh/apache/kylin/pull/797?src=pr=h1) Report
   > Merging [#797](https://codecov.io/gh/apache/kylin/pull/797?src=pr=desc) 
into 
[master](https://codecov.io/gh/apache/kylin/commit/57fb3c9f97994c5c55c7edfc9555fa0eb0c90b94?src=pr=desc)
 will **increase** coverage by `<.01%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/kylin/pull/797/graphs/tree.svg?width=650=JawVgbgsVo=150=pr)](https://codecov.io/gh/apache/kylin/pull/797?src=pr=tree)
   
   ```diff
   @@ Coverage Diff  @@
   ## master #797  +/-   ##
   
   + Coverage 25.62%   25.63%   +<.01% 
   + Complexity 6048 6047   -1 
   
 Files  1398 1398  
 Lines 8341183411  
 Branches  1168011680  
   
   + Hits  2137121379   +8 
   + Misses5998559980   -5 
   + Partials   2055 2052   -3
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/kylin/pull/797?src=pr=tree) | Coverage Δ 
| Complexity Δ | |
   |---|---|---|---|
   | 
[.../apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://codecov.io/gh/apache/kylin/pull/797/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2N1Ym9pZC9UcmVlQ3Vib2lkU2NoZWR1bGVyLmphdmE=)
 | `63.84% <0%> (-2.31%)` | `0% <0%> (ø)` | |
   | 
[...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/797/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh)
 | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | |
   | 
[...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/797/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh)
 | `70.21% <0%> (+3.64%)` | `7% <0%> (ø)` | :arrow_down: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/kylin/pull/797?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/kylin/pull/797?src=pr=footer). Last 
update 
[57fb3c9...636c11e](https://codecov.io/gh/apache/kylin/pull/797?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (KYLIN-4137) Accelerate metadata reloading

2019-08-15 Thread nichunen (JIRA)


[ 
https://issues.apache.org/jira/browse/KYLIN-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16907894#comment-16907894
 ] 

nichunen commented on KYLIN-4137:
-

+1

> Accelerate metadata reloading
> -
>
> Key: KYLIN-4137
> URL: https://issues.apache.org/jira/browse/KYLIN-4137
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Affects Versions: Future
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
>
> Now, org.apache.kylin.metadata.cachesync.CachedCrudAssist#reloadAll is using 
> an inappropriate method to deal with MySQLJdbcMetadata.
> As the every call of reloadAt(path) will access the database, which is almost 
> fine to HBase but really unfriendly to the MySQL(RDBMS).
> I think we should get all of the resource with single request instead of the 
> separate request to get every resource.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)