[jira] [Created] (KYLIN-3955) Real-time streaming tech blog
Ma Gang created KYLIN-3955: -- Summary: Real-time streaming tech blog Key: KYLIN-3955 URL: https://issues.apache.org/jira/browse/KYLIN-3955 Project: Kylin Issue Type: Improvement Components: Documentation Reporter: Ma Gang Assignee: Ma Gang Real-time streaming tech blog -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3821) Expose real-time streaming data consuming lag info
[ https://issues.apache.org/jira/browse/KYLIN-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16774770#comment-16774770 ] Ma Gang commented on KYLIN-3821: It can be done by adding a new method in IStreamingConnector to fetch the latest streaming source position. > Expose real-time streaming data consuming lag info > -- > > Key: KYLIN-3821 > URL: https://issues.apache.org/jira/browse/KYLIN-3821 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: Ma Gang >Assignee: XiaoXiang Yu >Priority: Major > > Expose real-time streaming data consuming lag info, so that user can easily > know the lag information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3821) Expose real-time streaming data consuming lag info
[ https://issues.apache.org/jira/browse/KYLIN-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang reassigned KYLIN-3821: -- Assignee: XiaoXiang Yu (was: Ma Gang) > Expose real-time streaming data consuming lag info > -- > > Key: KYLIN-3821 > URL: https://issues.apache.org/jira/browse/KYLIN-3821 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: Ma Gang >Assignee: XiaoXiang Yu >Priority: Major > > Expose real-time streaming data consuming lag info, so that user can easily > know the lag information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3821) Expose real-time streaming data consuming lag info
Ma Gang created KYLIN-3821: -- Summary: Expose real-time streaming data consuming lag info Key: KYLIN-3821 URL: https://issues.apache.org/jira/browse/KYLIN-3821 Project: Kylin Issue Type: Improvement Components: Real-time Streaming Reporter: Ma Gang Assignee: Ma Gang Expose real-time streaming data consuming lag info, so that user can easily know the lag information. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3797) Too many or filters may break Kylin server when flatting filter
Ma Gang created KYLIN-3797: -- Summary: Too many or filters may break Kylin server when flatting filter Key: KYLIN-3797 URL: https://issues.apache.org/jira/browse/KYLIN-3797 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang Kylin needs to convert filter into a flat format, like: OR(AND(f1,f2), AND(f3,f4,f5)…) , so that Kylin can build HBase scan/filter accordingly, but when there are too many or filters in query, like: AND(OR(f1,f2,...,f1000), OR(g1,g2,...,g1000), OR(h1,h2,...,h1000)), then the generated flat filter size will be 1000*1000*1000=1 billion, the flatting process will cause Kylin server OOM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3789) Stream receiver FQDN is too long to show on GUI
Ma Gang created KYLIN-3789: -- Summary: Stream receiver FQDN is too long to show on GUI Key: KYLIN-3789 URL: https://issues.apache.org/jira/browse/KYLIN-3789 Project: Kylin Issue Type: Sub-task Components: Real-time Streaming Reporter: Ma Gang Assignee: Pan, Julian There are two places to show streaming receiver info(system level and cube level), sometimes the receiver's FQDN is too long to show, maybe we can just show a limit length name, and give the full name when hover. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3787) NPE throws when dimension value has null when query real-time data
[ https://issues.apache.org/jira/browse/KYLIN-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang resolved KYLIN-3787. Resolution: Resolved > NPE throws when dimension value has null when query real-time data > -- > > Key: KYLIN-3787 > URL: https://issues.apache.org/jira/browse/KYLIN-3787 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > NPE throws when dimension value has null when query real-time data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3787) NPE throws when dimension value has null when query real-time data
Ma Gang created KYLIN-3787: -- Summary: NPE throws when dimension value has null when query real-time data Key: KYLIN-3787 URL: https://issues.apache.org/jira/browse/KYLIN-3787 Project: Kylin Issue Type: Sub-task Reporter: Ma Gang Assignee: Ma Gang NPE throws when dimension value has null when query real-time data -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3768) Save streaming metadata a standard kylin path in zookeeper
[
https://issues.apache.org/jira/browse/KYLIN-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ma Gang resolved KYLIN-3768.
Resolution: Won't Fix
Don't do the change, since if change the zk path to
"/kylin/\{META_URL_PREFIX}/stream" will break the mvn test(many unit test
extends LocalFileMetadataTestCase class and metadata url is set to
"../examples/test_metadata/" which is not valid path in zookeeper)
> Save streaming metadata a standard kylin path in zookeeper
> --
>
> Key: KYLIN-3768
> URL: https://issues.apache.org/jira/browse/KYLIN-3768
> Project: Kylin
> Issue Type: Sub-task
> Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> Currently we save streaming metadata in the zk path:
> /kylin/stream/${DEPLOY_ENV}, it should be changed to use a standard path like
> the distribute lock path: /kylin/\{METADATA_URL_PREFIX}/stream
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3768) Save streaming metadata a standard kylin path in zookeeper
Ma Gang created KYLIN-3768:
--
Summary: Save streaming metadata a standard kylin path in zookeeper
Key: KYLIN-3768
URL: https://issues.apache.org/jira/browse/KYLIN-3768
Project: Kylin
Issue Type: Sub-task
Components: Real-time Streaming
Reporter: Ma Gang
Assignee: Ma Gang
Currently we save streaming metadata in the zk path:
/kylin/stream/${DEPLOY_ENV}, it should be changed to use a standard path like
the distribute lock path: /kylin/\{METADATA_URL_PREFIX}/stream
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3747) Use FQDN to register a streaming receiver instead of ip
[ https://issues.apache.org/jira/browse/KYLIN-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang resolved KYLIN-3747. Resolution: Resolved > Use FQDN to register a streaming receiver instead of ip > --- > > Key: KYLIN-3747 > URL: https://issues.apache.org/jira/browse/KYLIN-3747 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > When streaming receiver is started, use FQDN to register it instead of ip, > since the FQDN is more stable especially in cloud env. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3747) Use FQDN to register a streaming receiver instead of ip
Ma Gang created KYLIN-3747: -- Summary: Use FQDN to register a streaming receiver instead of ip Key: KYLIN-3747 URL: https://issues.apache.org/jira/browse/KYLIN-3747 Project: Kylin Issue Type: Sub-task Reporter: Ma Gang Assignee: Ma Gang When streaming receiver is started, use FQDN to register it instead of ip, since the FQDN is more stable especially in cloud env. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3745) real-time segment state changed from active to immutable is not sequently
[ https://issues.apache.org/jira/browse/KYLIN-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang resolved KYLIN-3745. Resolution: Resolved > real-time segment state changed from active to immutable is not sequently > - > > Key: KYLIN-3745 > URL: https://issues.apache.org/jira/browse/KYLIN-3745 > Project: Kylin > Issue Type: Sub-task > Components: Streaming >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > real-time segment state changed from active to immutable is not sequently, > for example, segment 201812271000-201812271100 is active, but the segment > 201812271100-201812271200 is immutable -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3745) real-time segment state changed from active to immutable is not sequently
Ma Gang created KYLIN-3745: -- Summary: real-time segment state changed from active to immutable is not sequently Key: KYLIN-3745 URL: https://issues.apache.org/jira/browse/KYLIN-3745 Project: Kylin Issue Type: Sub-task Components: Streaming Reporter: Ma Gang Assignee: Ma Gang real-time segment state changed from active to immutable is not sequently, for example, segment 201812271000-201812271100 is active, but the segment 201812271100-201812271200 is immutable -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3654) New Kylin Streaming
[ https://issues.apache.org/jira/browse/KYLIN-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3654: --- Attachment: How to Use New Kylin Streaming.pdf > New Kylin Streaming > --- > > Key: KYLIN-3654 > URL: https://issues.apache.org/jira/browse/KYLIN-3654 > Project: Kylin > Issue Type: New Feature > Components: Job Engine, Metadata, Query Engine, Streaming >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > Attachments: How to Use New Kylin Streaming.pdf, New Kylin Streaming > Design.pdf > > > eBay Kylin team has developed a new Kylin streaming solution, the basic idea > is to build a streaming cluster to ingest data from streaming source(Kafka), > and provide query for real-time data, the data preparation latency is > milliseconds, which means the data is queryable almost when it is ingested, > attach is the architecture design doc. > We would like to contribute the feature to community, please let us know if > you have any concern. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3690) New streaming backend implementation
[ https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716018#comment-16716018 ] Ma Gang commented on KYLIN-3690: [PR 382|https://github.com/apache/kylin/pull/382] is created, and later will provide a simple usage doc. > New streaming backend implementation > > > Key: KYLIN-3690 > URL: https://issues.apache.org/jira/browse/KYLIN-3690 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > New streaming backend implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3690) New streaming backend implementation
[ https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709553#comment-16709553 ] Ma Gang commented on KYLIN-3690: Hi Hubert, The PR should be ready next week, because I did some refactor work after merge the existing code to the master, and I need further test after the refactoring. Sorry it comes a little bit late. > New streaming backend implementation > > > Key: KYLIN-3690 > URL: https://issues.apache.org/jira/browse/KYLIN-3690 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > New streaming backend implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3690) New streaming backend implementation
[ https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692947#comment-16692947 ] Ma Gang commented on KYLIN-3690: Thanks Hubert, the PR should be ready around Nov 30th, and the following process will be: code review, test, and release, when it can be release depends on the readiness of the feature. > New streaming backend implementation > > > Key: KYLIN-3690 > URL: https://issues.apache.org/jira/browse/KYLIN-3690 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > New streaming backend implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3692) New streaming ui implementation
[ https://issues.apache.org/jira/browse/KYLIN-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang closed KYLIN-3692. -- Resolution: Duplicate > New streaming ui implementation > --- > > Key: KYLIN-3692 > URL: https://issues.apache.org/jira/browse/KYLIN-3692 > Project: Kylin > Issue Type: Sub-task >Reporter: Ma Gang >Assignee: Pan, Julian >Priority: Major > > New streaming ui implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3691) New streaming ui implementation
Ma Gang created KYLIN-3691: -- Summary: New streaming ui implementation Key: KYLIN-3691 URL: https://issues.apache.org/jira/browse/KYLIN-3691 Project: Kylin Issue Type: Sub-task Reporter: Ma Gang Assignee: Pan, Julian New streaming ui implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3692) New streaming ui implementation
Ma Gang created KYLIN-3692: -- Summary: New streaming ui implementation Key: KYLIN-3692 URL: https://issues.apache.org/jira/browse/KYLIN-3692 Project: Kylin Issue Type: Sub-task Reporter: Ma Gang Assignee: Pan, Julian New streaming ui implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3690) New streaming backend implementation
Ma Gang created KYLIN-3690: -- Summary: New streaming backend implementation Key: KYLIN-3690 URL: https://issues.apache.org/jira/browse/KYLIN-3690 Project: Kylin Issue Type: Sub-task Reporter: Ma Gang Assignee: Ma Gang New streaming backend implementation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3654) New Kylin Streaming
Ma Gang created KYLIN-3654: -- Summary: New Kylin Streaming Key: KYLIN-3654 URL: https://issues.apache.org/jira/browse/KYLIN-3654 Project: Kylin Issue Type: New Feature Components: Job Engine, Metadata, Query Engine, Streaming Reporter: Ma Gang Assignee: Ma Gang Attachments: New Kylin Streaming Design.pdf eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming source(Kafka), and provide query for real-time data, the data preparation latency is milliseconds, which means the data is queryable almost when it is ingested, attach is the architecture design doc. We would like to contribute the feature to community, please let us know if you have any concern. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652999#comment-16652999
]
Ma Gang edited comment on KYLIN-3601 at 10/17/18 6:00 AM:
--
I suggest to test using 1,5,10,20,50 concurrency level and see what's the
result, you may find the qps/avg_latency will be downgrade since which
concurrency level, and then use jstack to dump some stacktrace when query is
running, and copy the result here, I may help to find what's the bottleneck.
The qps is highly dependent on the query latency for one query, and Kylin
cannot support too large concurrency, if you find the Kylin server is the
bottleneck to support your concurrent client, you need to add more Kylin
servers.
Mostly Kylin server will be treated as a database(mysql, etc), which is hided
behind your application server, so there should not be too many concurrent
clients.
was (Author: magang):
I suggest to test using 1,5,10,20,50 concurrency level and see what's the
result, you may find the qps/avg_latency will be downgrade since which
concurrency level, and then use jstack to dump some stacktrace when query is
running, and copy the result here, I may help to find what's the bottleneck.
The qps is highly dependent on the query latency for one query, and Kylin
cannot support too large concurrency, if you find the Kylin server is the
bottleneck to support your concurrent client, you need to add more Kylin
servers.
Mostly Kylin server will be treated as a database(mysql, etc), which is hided
behind your application server, so there should not too may concurrent clients.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652999#comment-16652999
]
Ma Gang commented on KYLIN-3601:
I suggest to test using 1,5,10,20,50 concurrency level and see what's the
result, you may find the qps/avg_latency will be downgrade since which
concurrency level, and then use jstack to dump some stacktrace when query is
running, and copy the result here, I may help to find what's the bottleneck.
The qps is highly dependent on the query latency for one query, and Kylin
cannot support too large concurrency, if you find the Kylin server is the
bottleneck to support your concurrent client, you need to add more Kylin
servers.
Mostly Kylin server will be treated as a database(mysql, etc), which is hided
behind your application server, so there should not too may concurrent clients.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651174#comment-16651174
]
Ma Gang commented on KYLIN-3601:
What's your expected query performance(load and latency)? Could you provide the
details performance data(including concurrency level, avg latency, etc)? Do
you have any proof that the preparedStatement creation is the bottleneck of the
system?
Below is my test result on my env(cube info and env info not provided):
not use preparedstatement cache:
===
concurrency qps avg_latency(ms) 50%(ms) 90%(ms)
5 19.79 252.672 249 277
10 22.74 439.764 432 499
20 23.74 842.411 829 1008
50 24.07 2077.124 1964 2485
use preparedstatement cache:
===
concurrency qps avg_latency(ms) 50%(ms) 90%(ms)
5 77.63 64.411 62 76
10 76.75 130.285 124 163
20 74.20 269.540 251 335
50 70.96 704.581 638 1005
You can see that the load performance is from around 20 qps to around 70 qps,
more than 3 times increment. And the concurrency is limited, more than 50
concurrent client will greatly downgrade the system performance, the bottleneck
should be the rpc call to Hbase(not very sure, I didn't dig into it).
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649978#comment-16649978
]
Ma Gang commented on KYLIN-3601:
I use apache ab to do some test: ab -n 10 -c 2 -T
'application/json;charset=UTF-8' -p query_request.json -A
[http://localhost/kylin/api/query]
the result is:
Document Path: /kylin/api/query
Document Length: 951 bytes
Concurrency Level: 2
Time taken for tests: 739.392 seconds
Complete requests: 1
Failed requests: 3746
(Connect: 0, Receive: 0, Length: 3746, Exceptions: 0)
Write errors: 0
Non-2xx responses: 30
Total transferred: 13928764 bytes
Total POSTed: 4610461
HTML transferred: 9728194 bytes
Requests per second: 13.52 [#/sec] (mean)
Time per request: 147.878 [ms] (mean)
Time per request: 73.939 [ms] (mean, across all concurrent requests)
the log in kylin server side is:
2018-10-15 09:07:40,448 DEBUG [Query 2f60cebe-3b9f-432a-b971-7d2285e2e25c-615]
service.QueryService:664 :
BorrowedCount:20737,DestroyedCount:62,CreatedCount:70,ReturnedCount:20737,IdleCount:8,ActiveCount:
0
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649767#comment-16649767
]
Ma Gang commented on KYLIN-3601:
So you mean that only 1 concurrent client can also reproduce it? I will try to
reproduce it in my side.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
Ma Gang created KYLIN-3632: -- Summary: Add configuration that can switch on/off preparedStatement cache in Kylin server Key: KYLIN-3632 URL: https://issues.apache.org/jira/browse/KYLIN-3632 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang Fix For: v2.5.1 From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn on/off for each request by adding a new field "enableStatementCache" in the query request, by default it is on. We need to add a switch in server level, the preparedStatement cache can take effective only when the switch is on, by default, it will be set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649722#comment-16649722
]
Ma Gang commented on KYLIN-3601:
Looks weird, and should not happen theoretically. [~kofiori] what performance
test tool you use to do the test? Is one concurrent client only send the next
request after it get response for the previous request?
And you should increase the concurrency time by time to do the test, for
example: 1, 5, 10, 20, 50, 100 concurrent clients, and check the issue happens
in which concurrency level.
For borrow fail case, you may add some log after throw NoSuchElementException
when borrow prepareContext from pool.
Also the bottleneck may happen in Hbase side or even in tomcat, could you check
the query log for the timeout query, what makes the query slow? you may dump
some stacktrace randomly when lots of query failed.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3629) NullPointException throws when use preparedStatement cache in some case
[
https://issues.apache.org/jira/browse/KYLIN-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ma Gang updated KYLIN-3629:
---
Attachment: KYLIN-3629.patch
> NullPointException throws when use preparedStatement cache in some case
> ---
>
> Key: KYLIN-3629
> URL: https://issues.apache.org/jira/browse/KYLIN-3629
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Attachments: KYLIN-3629.patch
>
>
> NullPointException throws when use preparedStatement cache in some case that
> there is no realization exists in the OLAPContext, for example the following
> sql request will cause issue:
> {color:#205081}{"sql":"select t1.lstg_site_id,t1.seller_id, count(1) from
> (select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
> lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t1 left join
> (select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
> lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t2 on
> t1.lstg_site_id = t2.lstg_site_id group by
> t1.lstg_site_id,t1.seller_id","offset":0,"limit":5,"params":[\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1731"},\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1732"}],"acceptPartial":true,"project":"default","backdoorToggles":
> {}}{color}
> the exception stacktrace is like:
> {color:#205081}org.apache.kylin.rest.exception.InternalErrorException:
> Unknown error.\n\tat
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:424)\n\tat
>
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:351)\n\tat
>
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)\n\tat
> sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:498)\n\tat
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)\n\tat
>
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:661)\n\tat
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>
>
[jira] [Created] (KYLIN-3629) NullPointException throws when use preparedStatement cache in some case
Ma Gang created KYLIN-3629:
--
Summary: NullPointException throws when use preparedStatement
cache in some case
Key: KYLIN-3629
URL: https://issues.apache.org/jira/browse/KYLIN-3629
Project: Kylin
Issue Type: Bug
Components: Query Engine
Reporter: Ma Gang
Assignee: Ma Gang
NullPointException throws when use preparedStatement cache in some case that
there is no realization exists in the OLAPContext, for example the following
sql request will cause issue:
{color:#205081}{"sql":"select t1.lstg_site_id,t1.seller_id, count(1) from
(select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t1 left join
(select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t2 on
t1.lstg_site_id = t2.lstg_site_id group by
t1.lstg_site_id,t1.seller_id","offset":0,"limit":5,"params":[\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1731"},\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1732"}],"acceptPartial":true,"project":"default","backdoorToggles":
{}}{color}
the exception stacktrace is like:
{color:#205081}org.apache.kylin.rest.exception.InternalErrorException: Unknown
error.\n\tat
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:424)\n\tat
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:351)\n\tat
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)\n\tat
sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)\n\tat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
java.lang.reflect.Method.invoke(Method.java:498)\n\tat
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)\n\tat
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)\n\tat
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:661)\n\tat
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n\tat
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n\tat
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631561#comment-16631561
]
Ma Gang commented on KYLIN-3601:
How many concurrent clients you use to do the performance test? You should use
less than kylin.query.statement-cache-max-num-per-key concurrent clients to do
the test. I didn't use blocked strategy for the pool, which means when borrow
fail, it will throw Exception and create a PreparedContext out of the pool. You
can use use apis: getNumIdle(Key) and getNumActive(Key) to check how many
preparedcontext are pooled.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3569) Server with query mode still can submit/build job
[ https://issues.apache.org/jira/browse/KYLIN-3569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16619993#comment-16619993 ] Ma Gang commented on KYLIN-3569: I think that is by design, query server can accept any restful request(including submit job request), and job server is responsible to schedule jobs. In a typical Kylin cluster setup, Kylin servers that behind LB(nginx, F5, etc) should have query server permission, so that it can accept any restful request, and the servers that are configured only as job server should not be configured in LB. For the permission issue, you should configure the ACL properly, to ensure that the BI tools use the user that only have read permission for your project. > Server with query mode still can submit/build job > - > > Key: KYLIN-3569 > URL: https://issues.apache.org/jira/browse/KYLIN-3569 > Project: Kylin > Issue Type: Bug > Components: Job Engine, REST Service, Security >Affects Versions: v2.4.1 > Environment: CentOS 6.7, HBase 1.2.0+cdh5.14.2+456 >Reporter: Zongwei Li >Priority: Major > Labels: build, documentation, security > Attachments: kylinCode.png > > > From the Docs at Kylin site, > [http://kylin.apache.org/docs24/install/kylin_cluster.html] > * *query* : run query engine only; Kylin query engine accepts and answers > your SQL queries > It seems that if server set with 'kylin.server.mode=query', it should not can > support submit/build job. But as we tested, server with query mode still can > submit/build job from UI or RESTFul API. > We analyzed the source code, found that there didn't exist any protect logic > to check whether server is at 'job' or 'build' mode in service layer for > submit/build job. Already attach the source code in this issue. > This issue really confused us, because we considered query server cannot > build job in Kylin Docs and many Kylin books. And query server will exposed > to 3rd BI tool to query the data, if we forget to configure the suitable ACL > for Cubes, then the 3rd BI tool can trigger build job in any time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604139#comment-16604139 ] Ma Gang commented on KYLIN-3521: Add another patch KYLIN-3521_V3.patch to fix. Since Shaofeng is busy now, [~yaho] could you help to apply the patch. > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521_V2.patch, KYLIN-3521_V3.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang reopened KYLIN-3521: > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521_V2.patch, KYLIN-3521_V3.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3521: --- Attachment: KYLIN-3521_V3.patch > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521_V2.patch, KYLIN-3521_V3.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604115#comment-16604115 ] Ma Gang commented on KYLIN-3521: Thanks [~yichen.zhou], my bad. Why we need to check the configuration value in UT? I don't think it make any sense. Thought this patch is just make a simple configuration change..., anyway I will fix it and provide another patch. > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521_V2.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3536) PrepareStatement cache issue when there are new segments built
[ https://issues.apache.org/jira/browse/KYLIN-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602832#comment-16602832 ] Ma Gang commented on KYLIN-3536: [~Shaofengshi], please help to review and apply to master and 2.5.0 branch if it is ok, thanks! > PrepareStatement cache issue when there are new segments built > -- > > Key: KYLIN-3536 > URL: https://issues.apache.org/jira/browse/KYLIN-3536 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Ma Gang >Assignee: Shaofeng SHI >Priority: Minor > Attachments: KYLIN-3536.patch > > > PrepareStatement cache will have issue when there are new segments built, > because the OLAPContext is cached with preparedStatement, and the choosed > Realization is cached in the OLAPContext, so if there are new segments built > after cache, the new segments will not be used for the query. The fix is > always use the latest Realization Instance in the OLAPContext. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3536) PrepareStatement cache issue when there are new segments built
[ https://issues.apache.org/jira/browse/KYLIN-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang reassigned KYLIN-3536: -- Assignee: Shaofeng SHI (was: Ma Gang) > PrepareStatement cache issue when there are new segments built > -- > > Key: KYLIN-3536 > URL: https://issues.apache.org/jira/browse/KYLIN-3536 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Ma Gang >Assignee: Shaofeng SHI >Priority: Minor > Attachments: KYLIN-3536.patch > > > PrepareStatement cache will have issue when there are new segments built, > because the OLAPContext is cached with preparedStatement, and the choosed > Realization is cached in the OLAPContext, so if there are new segments built > after cache, the new segments will not be used for the query. The fix is > always use the latest Realization Instance in the OLAPContext. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3536) PrepareStatement cache issue when there are new segments built
[ https://issues.apache.org/jira/browse/KYLIN-3536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3536: --- Attachment: KYLIN-3536.patch > PrepareStatement cache issue when there are new segments built > -- > > Key: KYLIN-3536 > URL: https://issues.apache.org/jira/browse/KYLIN-3536 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Attachments: KYLIN-3536.patch > > > PrepareStatement cache will have issue when there are new segments built, > because the OLAPContext is cached with preparedStatement, and the choosed > Realization is cached in the OLAPContext, so if there are new segments built > after cache, the new segments will not be used for the query. The fix is > always use the latest Realization Instance in the OLAPContext. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3536) PrepareStatement cache issue when there are new segments built
Ma Gang created KYLIN-3536: -- Summary: PrepareStatement cache issue when there are new segments built Key: KYLIN-3536 URL: https://issues.apache.org/jira/browse/KYLIN-3536 Project: Kylin Issue Type: Bug Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang PrepareStatement cache will have issue when there are new segments built, because the OLAPContext is cached with preparedStatement, and the choosed Realization is cached in the OLAPContext, so if there are new segments built after cache, the new segments will not be used for the query. The fix is always use the latest Realization Instance in the OLAPContext. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16602041#comment-16602041 ] Ma Gang commented on KYLIN-3521: Thanks [~Shaofengshi], upload the new patch, please review, also set the "kylin.cube.cubeplanner.enabled-for-existing-cube" to true. > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521.patch, KYLIN-3521_V2.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3521: --- Attachment: KYLIN-3521_V2.patch > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521.patch, KYLIN-3521_V2.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16601754#comment-16601754 ] Ma Gang commented on KYLIN-3521: Patch upload, please help to review [~Shaofengshi] > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3521: --- Attachment: KYLIN-3521.patch > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.0 > > Attachments: KYLIN-3521.patch > > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3522) PrepareStatement cache issue
[ https://issues.apache.org/jira/browse/KYLIN-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598543#comment-16598543 ] Ma Gang commented on KYLIN-3522: Patch uploaded, please help to review, the fix is to clear the filter's previous variable values before bind it. > PrepareStatement cache issue > > > Key: KYLIN-3522 > URL: https://issues.apache.org/jira/browse/KYLIN-3522 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Attachments: KYLIN-3522.patch > > > When enable server side prepareStatement cache, OLAPContext is cached with > prepareStatement object, because filter is also in the OLAPContext, so the > condition values in the comparedFilter is also cached, that may cause problem > in some cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3522) PrepareStatement cache issue
[ https://issues.apache.org/jira/browse/KYLIN-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3522: --- Attachment: KYLIN-3522.patch > PrepareStatement cache issue > > > Key: KYLIN-3522 > URL: https://issues.apache.org/jira/browse/KYLIN-3522 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Attachments: KYLIN-3522.patch > > > When enable server side prepareStatement cache, OLAPContext is cached with > prepareStatement object, because filter is also in the OLAPContext, so the > condition values in the comparedFilter is also cached, that may cause problem > in some cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3522) PrepareStatement cache issue
Ma Gang created KYLIN-3522: -- Summary: PrepareStatement cache issue Key: KYLIN-3522 URL: https://issues.apache.org/jira/browse/KYLIN-3522 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang When enable server side prepareStatement cache, OLAPContext is cached with prepareStatement object, because filter is also in the OLAPContext, so the condition values in the comparedFilter is also cached, that may cause problem in some cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598387#comment-16598387 ] Ma Gang commented on KYLIN-3521: +1 > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Priority: Minor > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. As > Cube planner only works when build the first segment, it only affect the > cuboid scheduler of a new Cube. Old cubes will not be affected. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3434) Support prepare statement in Kylin server side
Ma Gang created KYLIN-3434: -- Summary: Support prepare statement in Kylin server side Key: KYLIN-3434 URL: https://issues.apache.org/jira/browse/KYLIN-3434 Project: Kylin Issue Type: Improvement Reporter: Ma Gang Assignee: Ma Gang Kylin use calcite as sql engine, when a sql comes to Kylin server, it requires to be parsed, optimized, code gen, and then query Kylin's cube storage, the previous 3 steps often take 50-150 ms to complete(depends on the complexity of the sql). If we support to cache the parsed result in Kylin server, the 3 steps will be saved. The idea is to cache calcite's PreparedStatement object and related OLAPContexts in the server side, when the prepare request comes with the same sql, reuse the PreparedStatement to do the execution. Since the PreparedStatement is not thread safe, so I planned to use ObjectPool to cache the PreparedStatement.(use apache commons-pool lib) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3396) NPE throws when materialize lookup table to HBase
[ https://issues.apache.org/jira/browse/KYLIN-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3396: --- Attachment: KYLIN-3396.patch > NPE throws when materialize lookup table to HBase > - > > Key: KYLIN-3396 > URL: https://issues.apache.org/jira/browse/KYLIN-3396 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Attachments: KYLIN-3396.patch > > > NPE throws when materialize lookup table to HBase: > java.util.concurrent.ExecutionException: > java.lang.reflect.InvocationTargetException > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.kylin.provision.BuildCubeWithEngine.runTestAndAssertSucceed(BuildCubeWithEngine.java:237) > at > org.apache.kylin.provision.BuildCubeWithEngine.testCase(BuildCubeWithEngine.java:223) > at > org.apache.kylin.provision.BuildCubeWithEngine.build(BuildCubeWithEngine.java:201) > at > org.apache.kylin.provision.BuildCubeWithEngine.main(BuildCubeWithEngine.java:93) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.kylin.provision.BuildCubeWithEngine$TestCallable.call(BuildCubeWithEngine.java:263) > at > org.apache.kylin.provision.BuildCubeWithEngine$TestCallable.call(BuildCubeWithEngine.java:248) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > *Caused by: java.lang.NullPointerException at > org.apache.kylin.job.execution.ExecutableManager.getInstance(ExecutableManager.java:54) > at > org.apache.kylin.job.execution.AbstractExecutable.getManager(AbstractExecutable.java:81) > at > org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403) > at > org.apache.kylin.storage.hbase.lookup.HBaseLookupMRSteps.addMaterializeLookupTableSteps(HBaseLookupMRSteps.java:89) > at > org.apache.kylin.storage.hbase.lookup.HBaseLookupMRSteps.addMaterializeLookupTablesSteps(HBaseLoo* > kupMRSteps.java:75) > at > org.apache.kylin.storage.hbase.lookup.HBaseLookupMaterializer.materializeLookupTablesForCube(HBaseLookupMaterializer.java:38) > at > org.apache.kylin.engine.mr.BatchCubingJobBuilder2.addMaterializeLookupTableSteps(BatchCubingJobBuilder2.java:113) > at > org.apache.kylin.engine.mr.BatchCubingJobBuilder2.build(BatchCubingJobBuilder2.java:75) > at > org.apache.kylin.engine.mr.MRBatchCubingEngine2.createBatchCubingJob(MRBatchCubingEngine2.java:42) > at > org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:56) > at > org.apache.kylin.provision.BuildCubeWithEngine.buildSegment(BuildCubeWithEngine.java:380) > at > org.apache.kylin.provision.BuildCubeWithEngine.doBuildAndMergeOnCube(BuildCubeWHiithEngine.java:307) > at > org.apache.kylin.provision.BuildCubeWithEngine.testLeftJoinCube(BuildCubeWithEngine.java:289) > ... 10 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3396) NPE throws when materialize lookup table to HBase
Ma Gang created KYLIN-3396: -- Summary: NPE throws when materialize lookup table to HBase Key: KYLIN-3396 URL: https://issues.apache.org/jira/browse/KYLIN-3396 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Ma Gang Assignee: Ma Gang NPE throws when materialize lookup table to HBase: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.kylin.provision.BuildCubeWithEngine.runTestAndAssertSucceed(BuildCubeWithEngine.java:237) at org.apache.kylin.provision.BuildCubeWithEngine.testCase(BuildCubeWithEngine.java:223) at org.apache.kylin.provision.BuildCubeWithEngine.build(BuildCubeWithEngine.java:201) at org.apache.kylin.provision.BuildCubeWithEngine.main(BuildCubeWithEngine.java:93) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.kylin.provision.BuildCubeWithEngine$TestCallable.call(BuildCubeWithEngine.java:263) at org.apache.kylin.provision.BuildCubeWithEngine$TestCallable.call(BuildCubeWithEngine.java:248) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) *Caused by: java.lang.NullPointerException at org.apache.kylin.job.execution.ExecutableManager.getInstance(ExecutableManager.java:54) at org.apache.kylin.job.execution.AbstractExecutable.getManager(AbstractExecutable.java:81) at org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:403) at org.apache.kylin.storage.hbase.lookup.HBaseLookupMRSteps.addMaterializeLookupTableSteps(HBaseLookupMRSteps.java:89) at org.apache.kylin.storage.hbase.lookup.HBaseLookupMRSteps.addMaterializeLookupTablesSteps(HBaseLoo* kupMRSteps.java:75) at org.apache.kylin.storage.hbase.lookup.HBaseLookupMaterializer.materializeLookupTablesForCube(HBaseLookupMaterializer.java:38) at org.apache.kylin.engine.mr.BatchCubingJobBuilder2.addMaterializeLookupTableSteps(BatchCubingJobBuilder2.java:113) at org.apache.kylin.engine.mr.BatchCubingJobBuilder2.build(BatchCubingJobBuilder2.java:75) at org.apache.kylin.engine.mr.MRBatchCubingEngine2.createBatchCubingJob(MRBatchCubingEngine2.java:42) at org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:56) at org.apache.kylin.provision.BuildCubeWithEngine.buildSegment(BuildCubeWithEngine.java:380) at org.apache.kylin.provision.BuildCubeWithEngine.doBuildAndMergeOnCube(BuildCubeWHiithEngine.java:307) at org.apache.kylin.provision.BuildCubeWithEngine.testLeftJoinCube(BuildCubeWithEngine.java:289) ... 10 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469954#comment-16469954
]
Ma Gang commented on KYLIN-3221:
Already send to pull request for the change:
[https://github.com/apache/kylin/pull/139]
> Some improvements for lookup table
> ---
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> There are two limitations for current look table design:
> # lookup table size is limited, because table snapshot need to be cached in
> Kylin server, too large snapshot table will break the server.
> # lookup table snapshot references are stored in all segments of the cube,
> cannot support global snapshot table, the global snapshot table means when
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the
> existing lookup table design, below is the initial document, any comments and
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
> {{private}} {{List snapshotTableDescList =
> Collections.emptyList();}}|
> SnapshotTableDesc defines how table is stored and whether it is global or
> not, currently we can support two types of store:
> # "metaStore", table snapshot is stored in the metadata store, it is the
> same as current design, and this is the default option.
> # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
> {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>
> @JsonProperty("local_cache_enable")
> private boolean enableLocalCache = true;
>
> {{@JsonProperty}}{{(}}{{"global"}}{{)}}
> {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>
> Add 'snapshots' property in CubeInstance, to store snapshots resource path
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
> {{private}} {{Map snapshots; }}{{// tableName ->
> tableResoucePath mapping}}|
>
> Add new meta model ExtTableSnapshot to describe the extended table snapshot
> information, the information is stored in a new metastore path:
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
> {{private}} {{TableSignature signature;}}
>
> {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
> {{private}} {{String storageLocationIdentifier;}}
>
> @JsonProperty("key_columns")
> private String[] keyColumns; // the key columns of the table
>
> @JsonProperty("storage_type")
> private String storageType;
>
> {{@JsonProperty}}{{(}}{{"size"}}{{)}}
> {{private}} {{long}} {{size;}}
>
> {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
> {{private}} {{long}} {{rowCnt;}}|
>
> Add new section in 'Advance Setting' tab when do cube design, user can set
> table snapshot properties for each table, and by default, it is segment level
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use
> MapReduce job convert the hive source table to hfiles, and then bulk load
> hfiles to HTable. So it will add two job steps to do the lookup table
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
> the first 2 bytes is shard number, HBase table can be pre-split, the shard
> size is configurable through Kylin's properties:
> "kylin.snapshot.ext.shard-mb", default size is 500MB.
> 1 column family c, multiple columns which column name is the index of the
> column in the table definition
> |c|
> |1|2|...|
>
> h2. Query
> For key lookup query, directly call hbase get api to get entire row according
> to key (call local cache if there is local cache enable)
> For queries that need fetch keys according to the derived columns, iterate
> all rows to get related keys. (call local cache if there is local cache
> enable)
> For queries that only hit the lookup table, iterate all rows and let calcite
> to do aggregation
[jira] [Created] (KYLIN-3377) Some improvements for lookup table - snapshot management
Ma Gang created KYLIN-3377: -- Summary: Some improvements for lookup table - snapshot management Key: KYLIN-3377 URL: https://issues.apache.org/jira/browse/KYLIN-3377 Project: Kylin Issue Type: Sub-task Components: Job Engine, REST Service Reporter: Ma Gang Assignee: Ma Gang including build snapshot independently, view snapshot information, etc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3376) Some improvements for lookup table - query change
Ma Gang created KYLIN-3376: -- Summary: Some improvements for lookup table - query change Key: KYLIN-3376 URL: https://issues.apache.org/jira/browse/KYLIN-3376 Project: Kylin Issue Type: Sub-task Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang query part change -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3375) Some improvements for lookup table - build change
Ma Gang created KYLIN-3375: -- Summary: Some improvements for lookup table - build change Key: KYLIN-3375 URL: https://issues.apache.org/jira/browse/KYLIN-3375 Project: Kylin Issue Type: Sub-task Components: Job Engine Reporter: Ma Gang Assignee: Ma Gang build change for new lookup table -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3374) Some improvements for lookup table - metadata change
Ma Gang created KYLIN-3374: -- Summary: Some improvements for lookup table - metadata change Key: KYLIN-3374 URL: https://issues.apache.org/jira/browse/KYLIN-3374 Project: Kylin Issue Type: Sub-task Components: Metadata Reporter: Ma Gang Assignee: Ma Gang Some improvements for lookup table - metadata change -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3373) Some improvements for lookup table - UI part change
Ma Gang created KYLIN-3373: -- Summary: Some improvements for lookup table - UI part change Key: KYLIN-3373 URL: https://issues.apache.org/jira/browse/KYLIN-3373 Project: Kylin Issue Type: Sub-task Reporter: Ma Gang Assignee: Pan, Julian UI part change -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453408#comment-16453408
]
Ma Gang commented on KYLIN-3221:
Also add a new action button 'Lookup Refresh' for each cube, when click the
button, a dialog will popup, let user choose which lookup table need to be
refreshed, and if the table is not set to global, user can choose some or all
segments that the related snapshot need to be refreshed, then user can click
'submit' to submit a new job to build the table snapshot independently.
> Some improvements for lookup table
> ---
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> There are two limitations for current look table design:
> # lookup table size is limited, because table snapshot need to be cached in
> Kylin server, too large snapshot table will break the server.
> # lookup table snapshot references are stored in all segments of the cube,
> cannot support global snapshot table, the global snapshot table means when
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the
> existing lookup table design, below is the initial document, any comments and
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
> {{private}} {{List snapshotTableDescList =
> Collections.emptyList();}}|
> SnapshotTableDesc defines how table is stored and whether it is global or
> not, currently we can support two types of store:
> # "metaStore", table snapshot is stored in the metadata store, it is the
> same as current design, and this is the default option.
> # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
> {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>
> @JsonProperty("local_cache_enable")
> private boolean enableLocalCache = true;
>
> {{@JsonProperty}}{{(}}{{"global"}}{{)}}
> {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>
> Add 'snapshots' property in CubeInstance, to store snapshots resource path
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
> {{private}} {{Map snapshots; }}{{// tableName ->
> tableResoucePath mapping}}|
>
> Add new meta model ExtTableSnapshot to describe the extended table snapshot
> information, the information is stored in a new metastore path:
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
> {{private}} {{TableSignature signature;}}
>
> {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
> {{private}} {{String storageLocationIdentifier;}}
>
> @JsonProperty("key_columns")
> private String[] keyColumns; // the key columns of the table
>
> @JsonProperty("storage_type")
> private String storageType;
>
> {{@JsonProperty}}{{(}}{{"size"}}{{)}}
> {{private}} {{long}} {{size;}}
>
> {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
> {{private}} {{long}} {{rowCnt;}}|
>
> Add new section in 'Advance Setting' tab when do cube design, user can set
> table snapshot properties for each table, and by default, it is segment level
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use
> MapReduce job convert the hive source table to hfiles, and then bulk load
> hfiles to HTable. So it will add two job steps to do the lookup table
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
> the first 2 bytes is shard number, HBase table can be pre-split, the shard
> size is configurable through Kylin's properties:
> "kylin.snapshot.ext.shard-mb", default size is 500MB.
> 1 column family c, multiple columns which column name is the index of the
> column in the table definition
> |c|
> |1|2|...|
>
> h2. Query
> For key lookup query, directly call hbase get api to get entire row according
> to key (call local cache if
[jira] [Updated] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ma Gang updated KYLIN-3221:
---
Description:
There are two limitations for current look table design:
# lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break the server.
# lookup table snapshot references are stored in all segments of the cube,
cannot support global snapshot table, the global snapshot table means when the
lookup table is updated, it will take effective for all segments.
To resolve the above limitations, we decide to do some improvements for the
existing lookup table design, below is the initial document, any comments and
suggestions are welcome.
h2. Metadata
Will add a new property in CubeDesc to describe how lookup tables will be
snapshot, it can be defined during the cube design
|{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
{{private}} {{List snapshotTableDescList =
Collections.emptyList();}}|
SnapshotTableDesc defines how table is stored and whether it is global or not,
currently we can support two types of store:
# "metaStore", table snapshot is stored in the metadata store, it is the same
as current design, and this is the default option.
# "hbaseStore', table snapshot is stored in an additional hbase table.
|{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
{{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
@JsonProperty("local_cache_enable")
private boolean enableLocalCache = true;
{{@JsonProperty}}{{(}}{{"global"}}{{)}}
{{private}} {{boolean}} {{global = }}{{false}}{{;}}|
Add 'snapshots' property in CubeInstance, to store snapshots resource path for
each table, when the table snapshot is set to global in cube design:
|{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
{{private}} {{Map snapshots; }}{{// tableName ->
tableResoucePath mapping}}|
Add new meta model ExtTableSnapshot to describe the extended table snapshot
information, the information is stored in a new metastore path:
/ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
following info:
|{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"signature"}}{{)}}
{{private}} {{TableSignature signature;}}
{{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
{{private}} {{String storageLocationIdentifier;}}
@JsonProperty("key_columns")
private String[] keyColumns; // the key columns of the table
@JsonProperty("storage_type")
private String storageType;
{{@JsonProperty}}{{(}}{{"size"}}{{)}}
{{private}} {{long}} {{size;}}
{{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
{{private}} {{long}} {{rowCnt;}}|
Add new section in 'Advance Setting' tab when do cube design, user can set
table snapshot properties for each table, and by default, it is segment level
and store to metadata store
h2. Build
If user specify 'hbaseStore' storageType for any lookup table, will use
MapReduce job convert the hive source table to hfiles, and then bulk load
hfiles to HTable. So it will add two job steps to do the lookup table
materialization.
h2. HBase Lookup Table Schema
all data are stored in raw value
suppose the lookup table has primary keys: key1,key2
rowkey will be:
||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
|shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
the first 2 bytes is shard number, HBase table can be pre-split, the shard size
is configurable through Kylin's properties: "kylin.snapshot.ext.shard-mb",
default size is 500MB.
1 column family c, multiple columns which column name is the index of the
column in the table definition
|c|
|1|2|...|
h2. Query
For key lookup query, directly call hbase get api to get entire row according
to key (call local cache if there is local cache enable)
For queries that need fetch keys according to the derived columns, iterate all
rows to get related keys. (call local cache if there is local cache enable)
For queries that only hit the lookup table, iterate all rows and let calcite to
do aggregation and filter. (call local cache if there is local cache enable)
h2. Management
For each lookup table, admin can view how many snapshots it has in Kylin, and
can view each snapshot type/size information and which cube/segments the
snapshot is referenced, the snapshot tables that have no reference can be
deleted.
Add a new action button 'Lookup Refresh' for each cube, when click the button,
a dialog will popup, let user choose which lookup table need to refresh, and if
the table is not set to global, user can choose some or all segments that the
related snapshot need to be refresh, then user can click 'submit' to submit a
new job to build the table snapshot
[jira] [Commented] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453381#comment-16453381
]
Ma Gang commented on KYLIN-3221:
According to performance test, when save big lookup table in hbase, and wants
to get lots of keys from htable, it will take lots of time, per test, get about
110k random keys from htable will take about 200s.
So I add a local LRU disk cache in query server to improve the lookup join
performance, currently I use RocksDB as the disk cache, it is configurable and
can be disable in the cube configuration. If the local cache is enable, will
add another step in build job to warm up the cache. Per test, the rocksdb cache
performance is good, in my vm with 4 core, 10G ram, HDD disk, it take about 4
seconds to randomly get 110k keys, and scan performance is also very good.
> Some improvements for lookup table
> ---
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> There are two limitations for current look table design:
> # lookup table size is limited, because table snapshot need to be cached in
> Kylin server, too large snapshot table will break the server.
> # lookup table snapshot references are stored in all segments of the cube,
> cannot support global snapshot table, the global snapshot table means when
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the
> existing lookup table design, below is the initial document, any comments and
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
> {{private}} {{List snapshotTableDescList =
> Collections.emptyList();}}|
> SnapshotTableDesc defines how table is stored and whether it is global or
> not, currently we can support two types of store:
> # "metaStore", table snapshot is stored in the metadata store, it is the
> same as current design, and this is the default option.
> # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
> {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>
> @JsonProperty("local_cache_enable")
> private boolean enableLocalCache = true;
>
> {{@JsonProperty}}{{(}}{{"global"}}{{)}}
> {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>
> Add 'snapshots' property in CubeInstance, to store snapshots resource path
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
> {{private}} {{Map snapshots; }}{{// tableName ->
> tableResoucePath mapping}}|
>
> Add new meta model ExtTableSnapshot to describe the extended table snapshot
> information, the information is stored in a new metastore path:
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
> {{private}} {{TableSignature signature;}}
>
> {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
> {{private}} {{String storageLocationIdentifier;}}
>
> @JsonProperty("key_columns")
> private String[] keyColumns; // the key columns of the table
>
> @JsonProperty("storage_type")
> private String storageType;
>
> {{@JsonProperty}}{{(}}{{"size"}}{{)}}
> {{private}} {{long}} {{size;}}
>
> {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
> {{private}} {{long}} {{rowCnt;}}|
>
> Add new section in 'Advance Setting' tab when do cube design, user can set
> table snapshot properties for each table, and by default, it is segment level
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use
> MapReduce job convert the hive source table to hfiles, and then bulk load
> hfiles to HTable. So it will add two job steps to do the lookup table
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
> the first 2 bytes is shard number, HBase table can be pre-split, the shard
> size is configurable through Kylin's properties:
>
[jira] [Updated] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ma Gang updated KYLIN-3221:
---
Description:
There are two limitations for current look table design:
# lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break the server.
# lookup table snapshot references are stored in all segments of the cube,
cannot support global snapshot table, the global snapshot table means when the
lookup table is updated, it will take effective for all segments.
To resolve the above limitations, we decide to do some improvements for the
existing lookup table design, below is the initial document, any comments and
suggestions are welcome.
h2. Metadata
Will add a new property in CubeDesc to describe how lookup tables will be
snapshot, it can be defined during the cube design
|{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
{{private}} {{List snapshotTableDescList =
Collections.emptyList();}}|
SnapshotTableDesc defines how table is stored and whether it is global or not,
currently we can support two types of store:
# "metaStore", table snapshot is stored in the metadata store, it is the same
as current design, and this is the default option.
# "hbaseStore', table snapshot is stored in an additional hbase table.
|{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
{{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
{{@JsonProperty}}{{(}}{{"global"}}{{)}}
{{private}} {{boolean}} {{global = }}{{false}}{{;}}|
Add 'snapshots' property in CubeInstance, to store snapshots resource path for
each table, when the table snapshot is set to global in cube design:
|{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
{{private}} {{Map snapshots; }}{{// tableName ->
tableResoucePath mapping}}|
Add new meta model ExtTableSnapshot to describe the extended table snapshot
information, the information is stored in a new metastore path:
/ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
following info:
|{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"signature"}}{{)}}
{{private}} {{TableSignature signature;}}
{{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
{{private}} {{String storageLocationIdentifier;}}
@JsonProperty("key_columns")
private String[] keyColumns; // the key columns of the table
@JsonProperty("storage_type")
private String storageType;
{{@JsonProperty}}{{(}}{{"size"}}{{)}}
{{private}} {{long}} {{size;}}
{{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
{{private}} {{long}} {{rowCnt;}}|
Add new section in 'Advance Setting' tab when do cube design, user can set
table snapshot properties for each table, and by default, it is segment level
and store to metadata store
h2. Build
If user specify 'hbaseStore' storageType for any lookup table, will use
MapReduce job convert the hive source table to hfiles, and then bulk load
hfiles to HTable. So it will add two job steps to do the lookup table
materialization.
h2. HBase Lookup Table Schema
all data are stored in raw value
suppose the lookup table has primary keys: key1,key2
rowkey will be:
||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
|shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
the first 2 bytes is shard number, HBase table can be pre-split, the shard size
is configurable through Kylin's properties: "kylin.snapshot.ext.shard-mb",
default size is 500MB.
1 column family c, multiple columns which column name is the index of the
column in the table definition
|c|
|1|2|...|
h2. Query
For key lookup query, directly call hbase get api to get entire row according
to key.
For queries that need fetch keys according to the derived columns, iterate all
rows to get related keys.
For queries that only hit the lookup table, iterate all rows and let calcite to
do aggregation and filter.
h2. Management
For each lookup table, admin can view how many snapshots it has in Kylin, and
can view each snapshot type/size information and which cube/segments the
snapshot is referenced, the snapshot tables that have no reference can be
deleted.
h2. Cleanup
When clean up metadata store, need to remove snapshot stored in HBase. And need
to clean up metadata store periodically by cronjob.
h2. Future
# Add coprocessor for lookup table, to improve the performance of lookup table
query, and queries that filter by derived columns.
# Add secondly index support for external snapshot table.
was:
There are two limitations for current look table design:
# lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break the server.
# lookup table snapshot references are
[jira] [Updated] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ma Gang updated KYLIN-3221:
---
Description:
There are two limitations for current look table design:
# lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break the server.
# lookup table snapshot references are stored in all segments of the cube,
cannot support global snapshot table, the global snapshot table means when the
lookup table is updated, it will take effective for all segments.
To resolve the above limitations, we decide to do some improvements for the
existing lookup table design, below is the initial document, any comments and
suggestions are welcome.
h2. Metadata
Will add a new property in CubeDesc to describe how lookup tables will be
snapshot, it can be defined during the cube design
|{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
{{private}} {{List snapshotTableDescList =
Collections.emptyList();}}|
SnapshotTableDesc defines how table is stored and whether it is global or not,
currently we can support two types of store:
# "metaStore", table snapshot is stored in the metadata store, it is the same
as current design, and this is the default option.
# "hbaseStore', table snapshot is stored in an additional hbase table.
|{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
{{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
{{@JsonProperty}}{{(}}{{"global"}}{{)}}
{{private}} {{boolean}} {{global = }}{{false}}{{;}}|
Add 'snapshots' property in CubeInstance, to store snapshots resource path for
each table, when the table snapshot is set to global in cube design:
|{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
{{private}} {{Map snapshots; }}{{// tableName ->
tableResoucePath mapping}}|
Add new meta model ExtTableSnapshot to describe the extended table snapshot
information, the information is stored in a new metastore path:
/ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
following info:
|{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"signature"}}{{)}}
{{private}} {{TableSignature signature;}}
{{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
{{private}} {{String storageLocationIdentifier;}}
@JsonProperty("key_columns")
private String[] keyColumns; // the key columns of the table
@JsonProperty("storage_type")
private String storageType;
{{@JsonProperty}}{{(}}{{"size"}}{{)}}
{{private}} {{long}} {{size;}}
{{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
{{private}} {{long}} {{rowCnt;}}|
Add new section in 'Advance Setting' tab when do cube design, user can set
table snapshot properties for each table, and by default, it is segment level
and store to metadata store
h2. Build
If user specify 'hbaseStore' storageType for any lookup table, will use
MapReduce job convert the hive source table to hfiles, and then bulk load
hfiles to HTable. So it will add two job steps to do the lookup table
materialization.
h2. HBase Lookup Table Schema
all data are stored in raw value
suppose the lookup table has primary keys: key1,key2
rowkey will be:
||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
|shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
the first 2 bytes is shard number, HBase table can be pre-split, the shard size
is configurable through Kylin's properties: "kylin.snapshot.ext.shard-mb",
default size is 500MB.
1 column family c, multiple columns which column name is the index of the
column in the table definition
|c|
|1|2|...|
h2. Query
For key lookup query, directly call hbase get api to get entire row according
to key.
For queries that need fetch keys according to the derived columns, iterate all
rows to get related keys.
For queries that only hit the lookup table, iterate all rows and let calcite to
do aggregation and filter.
h2. Management
For each lookup table, admin can view how many snapshots it has in Kylin, and
can view each snapshot type/size information and which cube/segments the
snapshot is referenced, the snapshot tables that have no reference can be
deleted.
h2. Cleanup
When clean up metadata store, need to remove snapshot stored in HBase. And need
to clean up metadata store periodically by cronjob.
h2. Future
# Add coprocessor for lookup table, to improve the performance of lookup table
query, and queries that filter by derived columns.
# Add secondly index support for external snapshot table.
was:
There are two limitations for current look table design:
# lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break the server.
# lookup table snapshot references are stored
[jira] [Commented] (KYLIN-2899) Enable segment level query cache
[
https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416822#comment-16416822
]
Ma Gang commented on KYLIN-2899:
Add some draft design and performance test here
h2. Motivation
Currently Kylin use sql as the cache key, when sql comes, if result exists in
the cache, it will directly returned the cached result and don't need to query
hbase, when there is new segment build or existing segment refresh, all related
cache result need to be evicted. For some frequently build cube such as
streaming cube, the cache miss will increase dramatically, that may decrease
the query performance.
Since for Kylin cube most historical segments are immutable, the same query
against historical segments should be always same, don't need to be evicted for
new segment building. So we decide to implement the segment level cache, it is
a complement of the existing front-end cache, the idea is similar as the
level1/level2 cache in operating system.
h2. Design
h3.
How to enable
By default, the segment-level closed, and open only all following conditions
satisfied:
1. "kylin.query.segment-cache-enabled" config is set to true, it can be set at
cube level.
2. there is memcached configured in Kylin, because segment query result can be
very large, may consume lots of memory if no external cache enabled.
h3. What is cached
cache key is \{cubeName} + "_" + \{segmentUUID} + "_" + \{serlized
GTScanRequest string}
cache value is SegmentQueryResult:
{code:java}
// result byte array for all regions of the the segment
private Collection regionResults;
// store segment query stats for cube planer usage
private byte[] cubeSegmentStatisticsBytes;
{code}
h3.
How it works
Before calling segment endpoint rpc, if the segment level cache is enabled, it
will try to get the SegmentQueryResult from cache, if the result exist,
directly return the result, else call the endpint rpc to get result, then save
the result to cache for future usage. If the query result is very big, it will
be chunked automatically.
The cache result will not be evicted explictly, it depends on the ttl
configuration and LRU mechanism of the memcached, by default the ttl is set to
7 days.
h2. Performance
Since memcached performance is very good, it often takes 1-10 ms to get data
from memcached, and don't need to do further aggregation/filter, so most of
time the performance is better than HBase coprocessor rpc. Especially for the
queries that need large aggregation/filter in the HBase region server, and no
fuzzy key can be used, sometimes the performance has more than 10 times
increase, below is some test result:
Query1:
select s1, s2, s3, s4, s5, s6, s7, s8, sum(pcount) c from
shop_exp_path_analytics_flat where site_id = 0 AND device = 'Mobile' AND s5 =
'Checkout: Success' group by s1, s2, s3, s4, s5, s6, s7, s8
Below is some number for the query
total scan count: 2348
hit cuboid row count: 10,063,375
not use segment level cache: 2.247s
using segment level cache 0.16s
Query2:
hit cuboid row count: 800,317,603
Total scan count: 62347166
not use segment level cache: 12.823
use segment level cache: 0.519
Query3:
Total scan count: 64
Result row count: 58
not use segment level cache: 0.173
use segment level cache: 0.153
> Enable segment level query cache
>
>
> Key: KYLIN-2899
> URL: https://issues.apache.org/jira/browse/KYLIN-2899
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Zhong Yanghong
>Assignee: Ma Gang
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3320) CubeStatsReader cannot print stats properly for some cube
[
https://issues.apache.org/jira/browse/KYLIN-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415063#comment-16415063
]
Ma Gang commented on KYLIN-3320:
attach the patch, this is caused by the code:
{code:java}
// CubeStatsReader.printCuboidInfoTree() method
List children = scheduler.getSpanningCuboid(cuboidID);
Collections.sort(children);{code}
the returned cuboid list is immutable for TreeCuboidScheduler, the
Collections.sort() will cause exception.
the fix is return a copy cuboid list from the TreeCuboidScheduler
> CubeStatsReader cannot print stats properly for some cube
> --
>
> Key: KYLIN-3320
> URL: https://issues.apache.org/jira/browse/KYLIN-3320
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Minor
> Attachments: fix_KYLIN-3320.patch
>
>
> For the cubes that have cuboid_bytes set in the CubeInstance, the cuboid
> stats cannot print properly using tool CubeStatsReader
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3320) CubeStatsReader cannot print stats properly for some cube
[ https://issues.apache.org/jira/browse/KYLIN-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3320: --- Attachment: fix_KYLIN-3320.patch > CubeStatsReader cannot print stats properly for some cube > -- > > Key: KYLIN-3320 > URL: https://issues.apache.org/jira/browse/KYLIN-3320 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Attachments: fix_KYLIN-3320.patch > > > For the cubes that have cuboid_bytes set in the CubeInstance, the cuboid > stats cannot print properly using tool CubeStatsReader -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3320) CubeStatsReader cannot print stats properly for some cube
Ma Gang created KYLIN-3320: -- Summary: CubeStatsReader cannot print stats properly for some cube Key: KYLIN-3320 URL: https://issues.apache.org/jira/browse/KYLIN-3320 Project: Kylin Issue Type: Improvement Components: Tools, Build and Test Reporter: Ma Gang Assignee: Ma Gang For the cubes that have cuboid_bytes set in the CubeInstance, the cuboid stats cannot print properly using tool CubeStatsReader -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3221) Some improvements for lookup table
Ma Gang created KYLIN-3221:
--
Summary: Some improvements for lookup table
Key: KYLIN-3221
URL: https://issues.apache.org/jira/browse/KYLIN-3221
Project: Kylin
Issue Type: Improvement
Components: General
Reporter: Ma Gang
Assignee: Ma Gang
There are two limitations for current look table design:
# lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break the server.
# lookup table snapshot references are stored in all segments of the cube,
cannot support global snapshot table, the global snapshot table means when the
lookup table is updated, it will take effective for all segments.
To resolve the above limitations, we decide to do some improvements for the
existing lookup table design, below is the initial document, any comments and
suggestions are welcome.
h2. Metadata
Will add a new property in CubeDesc to describe how lookup tables will be
snapshot, it can be defined during the cube design
|{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
{{private}} {{List snapshotTableDescList =
Collections.emptyList();}}|
SnapshotTableDesc defines how table is stored and whether it is global or not,
currently we can support two types of store:
# "metaStore", table snapshot is stored in the metadata store, it is the same
as current design, and this is the default option.
# "hbaseStore', table snapshot is stored in an additional hbase table.
|{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
{{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
{{@JsonProperty}}{{(}}{{"global"}}{{)}}
{{private}} {{boolean}} {{global = }}{{false}}{{;}}|
Add 'snapshots' property in CubeInstance, to store snapshots resource path for
each table, when the table snapshot is set to global in cube design:
|{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
{{private}} {{Map snapshots; }}{{// tableName ->
tableResoucePath mapping}}|
Add new meta model ExtTableSnapshot to describe the extended table snapshot
information, the information is stored in a new metastore path:
/ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
following info:
|{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
{{private}} {{String tableName;}}
{{@JsonProperty}}{{(}}{{"signature"}}{{)}}
{{private}} {{TableSignature signature;}}
{{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
{{private}} {{String storageLocationIdentifier;}}
{{@JsonProperty}}{{(}}{{"size"}}{{)}}
{{private}} {{long}} {{size;}}
{{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
{{private}} {{long}} {{rowCnt;}}|
Add new section in 'Advance Setting' tab when do cube design, user can set
table snapshot properties for each table, and by default, it is segment level
and store to metadata store
h2. Build
If user specify 'hbaseStore' storageType for any lookup table, will use
MapReduce job convert the hive source table to hfiles, and then bulk load
hfiles to HTable. So it will add two job steps to do the lookup table
materialization.
h2. HBase Lookup Table Schema
all data are stored in raw value
suppose the lookup table has primary keys: key1,key2
rowkey will be:
||2 bytes||len1 bytes||2 bytes||len2 bytes||
|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
1 column family c, multiple columns which column name is the index of the
column in the table definition
|c|
|1|2|...|
h2. Query
For key lookup query, directly call hbase get api to get entire row according
to key.
For queries that need fetch keys according to the derived columns, iterate all
rows to get related keys.
For queries that only hit the lookup table, iterate all rows and let calcite to
do aggregation and filter.
h2. Management
For each lookup table, admin can view how many snapshots it has in Kylin, and
can view each snapshot type/size information and which cube/segments the
snapshot is referenced, the snapshot tables that have no reference can be
deleted.
h2. Cleanup
When clean up metadata store, need to remove snapshot stored in HBase. And need
to clean up metadata store periodically by cronjob.
h2. Future
# Add coprocessor for lookup table, to improve the performance of lookup table
query, and queries that filter by derived columns.
# Add secondly index support for external snapshot table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2899) Enable segment level query cache
[ https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348084#comment-16348084 ] Ma Gang commented on KYLIN-2899: By default, it will be closed, and open only all following conditions satisfied: 1. "kylin.query.segment-cache-enabled" config is set to true, it can be set in cube level. 2. there is memcached configured in Kylin. Segment level cache can improve query performance for the cube that is built frequently, for example the streaming cube, the global cache will be invalidated frequently. > Enable segment level query cache > > > Key: KYLIN-2899 > URL: https://issues.apache.org/jira/browse/KYLIN-2899 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Ma Gang >Priority: Major > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3209) Optimize job partial statistics path is inconsistent with existing one
[
https://issues.apache.org/jira/browse/KYLIN-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343091#comment-16343091
]
Ma Gang commented on KYLIN-3209:
attach the patch
> Optimize job partial statistics path is inconsistent with existing one
> --
>
> Key: KYLIN-3209
> URL: https://issues.apache.org/jira/browse/KYLIN-3209
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Minor
> Attachments: fix_KYLIN-3209.patch
>
>
> batch build partial statistics file path is
> .../statistics/statistics_\{shardID}
> optimize job partial statistics file path is
> .../statistics/cuboid_statistics.seq_\{shardID}
> And in SaveStatistics step, it use "statistics" as prefix to filter files in
> hdfs, so to make it consistent, change the optimize job statistics file name
> to .../statistics/statistics_\{shardID}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3209) Optimize job partial statistics path is inconsistent with existing one
[
https://issues.apache.org/jira/browse/KYLIN-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ma Gang updated KYLIN-3209:
---
Attachment: fix_KYLIN-3209.patch
> Optimize job partial statistics path is inconsistent with existing one
> --
>
> Key: KYLIN-3209
> URL: https://issues.apache.org/jira/browse/KYLIN-3209
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Minor
> Attachments: fix_KYLIN-3209.patch
>
>
> batch build partial statistics file path is
> .../statistics/statistics_\{shardID}
> optimize job partial statistics file path is
> .../statistics/cuboid_statistics.seq_\{shardID}
> And in SaveStatistics step, it use "statistics" as prefix to filter files in
> hdfs, so to make it consistent, change the optimize job statistics file name
> to .../statistics/statistics_\{shardID}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3209) Optimize job partial statistics path is inconsistent with existing one
Ma Gang created KYLIN-3209:
--
Summary: Optimize job partial statistics path is inconsistent with
existing one
Key: KYLIN-3209
URL: https://issues.apache.org/jira/browse/KYLIN-3209
Project: Kylin
Issue Type: Improvement
Components: Job Engine
Reporter: Ma Gang
Assignee: Ma Gang
batch build partial statistics file path is .../statistics/statistics_\{shardID}
optimize job partial statistics file path is
.../statistics/cuboid_statistics.seq_\{shardID}
And in SaveStatistics step, it use "statistics" as prefix to filter files in
hdfs, so to make it consistent, change the optimize job statistics file name to
.../statistics/statistics_\{shardID}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3020) org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin HadoopShellExecutable
[ https://issues.apache.org/jira/browse/KYLIN-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3020: --- Attachment: KYLIN-3020.patch The root cause is GenericOptionsParser class is not thread safe, and when multiple jobs are running concurrently will cause some concurrency issue, attach the fix patch. > org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin > HadoopShellExecutable > -- > > Key: KYLIN-3020 > URL: https://issues.apache.org/jira/browse/KYLIN-3020 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Ma Gang > Attachments: KYLIN-3020.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3020) org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin HadoopShellExecutable
[ https://issues.apache.org/jira/browse/KYLIN-3020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16249192#comment-16249192 ] Ma Gang commented on KYLIN-3020: Sometimes NPE throws in create flat table steps or bulkload hfile steps, and often resume job make it work: java.lang.NullPointerException at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:283) at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:487) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:170) at org.apache.hadoop.util.GenericOptionsParser.(GenericOptionsParser.java:153) at org.apache.kylin.engine.mr.MRUtil.getParser(MRUtil.java:100) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:90) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:126) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:153) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:52) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:153) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:158) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745 > org.apache.hadoop.util.ToolRunner is not threadsafe and misused in kylin > HadoopShellExecutable > -- > > Key: KYLIN-3020 > URL: https://issues.apache.org/jira/browse/KYLIN-3020 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Ma Gang > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2965) RealizationChooser cost calculation logic different with CubeInstance
[ https://issues.apache.org/jira/browse/KYLIN-2965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-2965: --- Attachment: kylin-2965.patch attach the minor fix patch > RealizationChooser cost calculation logic different with CubeInstance > - > > Key: KYLIN-2965 > URL: https://issues.apache.org/jira/browse/KYLIN-2965 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Attachments: kylin-2965.patch > > > In RealizationChooser.RealizationCost, it uses cube's all > dimensions(including all derived columns) to calculate the cost, but in > CubeInstance, it use row key column count. Think it is a bug, should not > include the derived column number to calculate cost -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2965) RealizationChooser cost calculation logic different with CubeInstance
Ma Gang created KYLIN-2965: -- Summary: RealizationChooser cost calculation logic different with CubeInstance Key: KYLIN-2965 URL: https://issues.apache.org/jira/browse/KYLIN-2965 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang Priority: Minor In RealizationChooser.RealizationCost, it uses cube's all dimensions(including all derived columns) to calculate the cost, but in CubeInstance, it use row key column count. Think it is a bug, should not include the derived column number to calculate cost -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481358#comment-15481358 ] Ma Gang commented on KYLIN-1827: No, it is not a big issue, just happens in very occasional case. > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Fix For: v1.5.4 > > Attachments: fix_send_notification.patch, > fix_send_notification_2.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1932) Query did not filter unrelated cuboid shards when cuboid is shard on specific column
[ https://issues.apache.org/jira/browse/KYLIN-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463883#comment-15463883 ] Ma Gang commented on KYLIN-1932: Hi [~mahongbin], I have filter on the sharded column, currently I don't have test cases, just found that the performance doesn't improved when filter by sharded column. And I looked into related code, found some suspicious: 1. in CubeHBaseEndpointRPC.getEPKeyRanges() method, just use the cuboid's shard ranges, didn't use the column shard information that may be filtered. 2. in CubeHBaseRPC.preparedHBaseScan method, use LazyRowKeyEncoder as row key encoder, which override calculateShard() method, that doesn't do any real shard calculating. > Query did not filter unrelated cuboid shards when cuboid is shard on specific > column > > > Key: KYLIN-1932 > URL: https://issues.apache.org/jira/browse/KYLIN-1932 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Ma Gang >Assignee: hongbin ma > > This is related to KYLIN-1453, I just check the code, and looks like it > didn't work as expected, [~mahongbin] could you help to confirm it is a bug > or not? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1819) Exception swallowed when start DefaultScheduler fail
[ https://issues.apache.org/jira/browse/KYLIN-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418461#comment-15418461 ] Ma Gang commented on KYLIN-1819: Not merged yet, please go ahead thanks! > Exception swallowed when start DefaultScheduler fail > > > Key: KYLIN-1819 > URL: https://issues.apache.org/jira/browse/KYLIN-1819 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: fix_swallow_scheduler_start_exception.patch > > > Start job scheduler need to acquire job lock from zookeeper, when lock > acquire fail, it will throw an IllegalException, but because the scheduler is > started in a new thread, the exception thrown by the thread will be ignored, > and the server still started successfully, and no exceptions are logged. That > make it hard for trouble shooting, should change to make server started fail > when the scheduler started fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385185#comment-15385185 ] Ma Gang commented on KYLIN-1827: Hi Shaofeng, any concern on the patch? at least the bug in getProjectName/getDeployEnvName method in CubingJob should be fixed, those methods just return a constant string. > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Fix For: v1.6.0 > > Attachments: fix_send_notification.patch, > fix_send_notification_2.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1827: --- Attachment: fix_send_notification_2.patch send notification when build failure because of metadata store fail shortly > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Fix For: v1.6.0 > > Attachments: fix_send_notification.patch, > fix_send_notification_2.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378806#comment-15378806 ] Ma Gang edited comment on KYLIN-1827 at 7/15/16 3:36 AM: - send notification when build failure because of metadata store fail shortly, Shaofeng SHI please help to review. was (Author: magang): send notification when build failure because of metadata store fail shortly > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Fix For: v1.6.0 > > Attachments: fix_send_notification.patch, > fix_send_notification_2.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15378736#comment-15378736 ] Ma Gang edited comment on KYLIN-1827 at 7/15/16 1:53 AM: - Reopen the issue, because the previous patch doesn't handle the metadata persist exception which may be caused by hbase metadata store unavailable shortly, that may happen in the onExecuteStart()/onExecuteFinish() methods. Will provide another patch. was (Author: magang): Reopen the issue, because the previous patch doesn't handle the metadata persist exception which may be caused by hbase unavailable shortly, that may happen in the onExecuteStart()/onExecuteFinish() methods. Will provide another patch. > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Fix For: v1.6.0 > > Attachments: fix_send_notification.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang reopened KYLIN-1827: Reopen the issue, because the previous patch doesn't handle the metadata persist exception which may be caused by hbase unavailable shortly, that may happen in the onExecuteStart()/onExecuteFinish() methods. Will provide another patch. > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Fix For: v1.6.0 > > Attachments: fix_send_notification.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1827: --- Attachment: fix_send_notification.patch Did not find that it is just a easy fix..., please help to review. > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: fix_send_notification.patch > > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
[ https://issues.apache.org/jira/browse/KYLIN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1872: --- Attachment: query_visible_interruptable-master.patch patch for master > Make query visible and interruptible, improve server's stablility > - > > Key: KYLIN-1872 > URL: https://issues.apache.org/jira/browse/KYLIN-1872 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: query_visible_interruptable-1.4rc.patch, > query_visible_interruptable-master.patch > > > Problem: > 1. Large query result will break kylin server, for example: select * from > fact_table. Even when properties "kylin.query.scan.threshold" and > "kylin.query.mem.budget" are set properly, OOM still happens, because the > hbase rpc thread is not interrupted, the result will continually go to kylin > server. And server will run OOM quickly when there are multiple such queries. > 2. Tow many slow queries will occupy all tomcat threads, and make server > unresponsed. > 3. There's no corelation id for a specified query, so it is hard to find the > rpc log for a specified query, if there are too many queries running > concurrently. > Solution: > 1. Interrupt the rpc thread and main query thread when return result size > larger than the config limit size. > 2. Make query visible. Admin can view all running queries, and detail of each > query. >Split the query into following steps: >1) sql parse >2) cube plan >3) query cache >4) multiple cube segment query > a. for each segment request, have muliple endpoint range request. > b. for each endpoint range request, have multiple coprocessor request. > c. for each coprocessor request, have multiple region server rpc. >Admin can view the startTime/endTime of each step, and the thread stack > trace if the step is running. > 3. Add query id as corelation id in the rpc log. > 4. Admin can interrupt a running query, to release the thread, memory, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
[ https://issues.apache.org/jira/browse/KYLIN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374191#comment-15374191 ] Ma Gang commented on KYLIN-1872: Yes, the interrupting only happens on client side, the only purpose is to protect Kylin server from running OOM because of large query, rather than protect the region server. Totally agree that the final solution is to interrupt the coprocessor, that's a long way to go. Do you have any idea to accomplish this? The only solution comes to my mind is: use another coprocessor interface to set specify query is stopped, that state is stored in a static map, and the visitCube interface check the query state in period, and if found it is stopped, quit the procedure. Or store the state in an external storage. For the scan count limit, we can just put the limit in the CubeVisitRequest, just like timeout property, so that the coprocessor don't return too large result. > Make query visible and interruptible, improve server's stablility > - > > Key: KYLIN-1872 > URL: https://issues.apache.org/jira/browse/KYLIN-1872 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: query_visible_interruptable-1.4rc.patch > > > Problem: > 1. Large query result will break kylin server, for example: select * from > fact_table. Even when properties "kylin.query.scan.threshold" and > "kylin.query.mem.budget" are set properly, OOM still happens, because the > hbase rpc thread is not interrupted, the result will continually go to kylin > server. And server will run OOM quickly when there are multiple such queries. > 2. Tow many slow queries will occupy all tomcat threads, and make server > unresponsed. > 3. There's no corelation id for a specified query, so it is hard to find the > rpc log for a specified query, if there are too many queries running > concurrently. > Solution: > 1. Interrupt the rpc thread and main query thread when return result size > larger than the config limit size. > 2. Make query visible. Admin can view all running queries, and detail of each > query. >Split the query into following steps: >1) sql parse >2) cube plan >3) query cache >4) multiple cube segment query > a. for each segment request, have muliple endpoint range request. > b. for each endpoint range request, have multiple coprocessor request. > c. for each coprocessor request, have multiple region server rpc. >Admin can view the startTime/endTime of each step, and the thread stack > trace if the step is running. > 3. Add query id as corelation id in the rpc log. > 4. Admin can interrupt a running query, to release the thread, memory, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
[ https://issues.apache.org/jira/browse/KYLIN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372668#comment-15372668 ] Ma Gang commented on KYLIN-1872: Read the hbase 0.98 coprocessor code, looks like interrupting rpc thread is ok, but the rpc response from region server still goes to client side, and dropped immediately. The only benefit to do this is it can release the hbase rpc thread earlier. Another option is do not interrupt the rpc thread, just mark the query is stopped, and before adding the response to queue, if the query is stopped, throw exception and the response is then dropped. > Make query visible and interruptible, improve server's stablility > - > > Key: KYLIN-1872 > URL: https://issues.apache.org/jira/browse/KYLIN-1872 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: query_visible_interruptable-1.4rc.patch > > > Problem: > 1. Large query result will break kylin server, for example: select * from > fact_table. Even when properties "kylin.query.scan.threshold" and > "kylin.query.mem.budget" are set properly, OOM still happens, because the > hbase rpc thread is not interrupted, the result will continually go to kylin > server. And server will run OOM quickly when there are multiple such queries. > 2. Tow many slow queries will occupy all tomcat threads, and make server > unresponsed. > 3. There's no corelation id for a specified query, so it is hard to find the > rpc log for a specified query, if there are too many queries running > concurrently. > Solution: > 1. Interrupt the rpc thread and main query thread when return result size > larger than the config limit size. > 2. Make query visible. Admin can view all running queries, and detail of each > query. >Split the query into following steps: >1) sql parse >2) cube plan >3) query cache >4) multiple cube segment query > a. for each segment request, have muliple endpoint range request. > b. for each endpoint range request, have multiple coprocessor request. > c. for each coprocessor request, have multiple region server rpc. >Admin can view the startTime/endTime of each step, and the thread stack > trace if the step is running. > 3. Add query id as corelation id in the rpc log. > 4. Admin can interrupt a running query, to release the thread, memory, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15370301#comment-15370301 ] Ma Gang commented on KYLIN-1827: Hi Shaofeng, I will contribute the patch, but it may take some time, because I'm working on others. > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
[ https://issues.apache.org/jira/browse/KYLIN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1872: --- Description: Problem: 1. Large query result will break kylin server, for example: select * from fact_table. Even when properties "kylin.query.scan.threshold" and "kylin.query.mem.budget" are set properly, OOM still happens, because the hbase rpc thread is not interrupted, the result will continually go to kylin server. And server will run OOM quickly when there are multiple such queries. 2. Tow many slow queries will occupy all tomcat threads, and make server unresponsed. 3. There's no corelation id for a specified query, so it is hard to find the rpc log for a specified query, if there are too many queries running concurrently. Solution: 1. Interrupt the rpc thread and main query thread when return result size larger than the config limit size. 2. Make query visible. Admin can view all running queries, and detail of each query. Split the query into following steps: 1) sql parse 2) cube plan 3) query cache 4) multiple cube segment query a. for each segment request, have muliple endpoint range request. b. for each endpoint range request, have multiple coprocessor request. c. for each coprocessor request, have multiple region server rpc. Admin can view the startTime/endTime of each step, and the thread stack trace if the step is running. 3. Add query id as corelation id in the rpc log. 4. Admin can interrupt a running query, to release the thread, memory, etc. was: Problem: 1. Large query result will break kylin server, for example: select * from fact_table. Even when properties "kylin.query.scan.threshold" and "kylin.query.mem.budget" are set properly, OOM still happens, because the hbase rpc thread is not interrupted, the result will continually go to kylin server. And server will run OOM quickly when there are multiple such queries. 2. Tow many slow queries will occupy all tomcat threads, and make server unresponsed. 3. There's no corelation id for a specified query, so it is hard to find the rpc log for a specified query, if there are too many queries running concurrently. Solution: 1. Interrupt the rpc thread and main query thread when return result size larger than the config limit size. 2. Make query visible. Admin can view all running queries, and detail of each query. Split the query into following steps: 1) sql parse 2) cube plan 3) query cache 4) multiple cube segment query a. for each segment request, have muliple endpoint range request. b. for each endpoint range request, have multiple coprocessor request. c. for eache coprocessor request, have multiple region server rpc. Admin can view the startTime/endTime of each step, and the thread stack trace if the step is running. 3. Add query id as corelation id in the rpc log. 4. Admin can interrupt a running query, to release the thread, memory, etc. > Make query visible and interruptible, improve server's stablility > - > > Key: KYLIN-1872 > URL: https://issues.apache.org/jira/browse/KYLIN-1872 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: query_visible_interruptable-1.4rc.patch > > > Problem: > 1. Large query result will break kylin server, for example: select * from > fact_table. Even when properties "kylin.query.scan.threshold" and > "kylin.query.mem.budget" are set properly, OOM still happens, because the > hbase rpc thread is not interrupted, the result will continually go to kylin > server. And server will run OOM quickly when there are multiple such queries. > 2. Tow many slow queries will occupy all tomcat threads, and make server > unresponsed. > 3. There's no corelation id for a specified query, so it is hard to find the > rpc log for a specified query, if there are too many queries running > concurrently. > Solution: > 1. Interrupt the rpc thread and main query thread when return result size > larger than the config limit size. > 2. Make query visible. Admin can view all running queries, and detail of each > query. >Split the query into following steps: >1) sql parse >2) cube plan >3) query cache >4) multiple cube segment query > a. for each segment request, have muliple endpoint range request. > b. for each endpoint range request, have multiple coprocessor request. > c. for each coprocessor request, have multiple region server rpc. >Admin can view the startTime/endTime of each step, and the thread stack > trace if the step is running. > 3. Add query id as corelation id in the rpc log. > 4. Admin can interrupt a running query, to release the thread, memory, etc.
[jira] [Updated] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
[ https://issues.apache.org/jira/browse/KYLIN-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1872: --- Attachment: query_visible_interruptable-1.4rc.patch attached is the patch for 1.4-rc branch > Make query visible and interruptible, improve server's stablility > - > > Key: KYLIN-1872 > URL: https://issues.apache.org/jira/browse/KYLIN-1872 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: query_visible_interruptable-1.4rc.patch > > > Problem: > 1. Large query result will break kylin server, for example: select * from > fact_table. Even when properties "kylin.query.scan.threshold" and > "kylin.query.mem.budget" are set properly, OOM still happens, because the > hbase rpc thread is not interrupted, the result will continually go to kylin > server. And server will run OOM quickly when there are multiple such queries. > 2. Tow many slow queries will occupy all tomcat threads, and make server > unresponsed. > 3. There's no corelation id for a specified query, so it is hard to find the > rpc log for a specified query, if there are too many queries running > concurrently. > Solution: > 1. Interrupt the rpc thread and main query thread when return result size > larger than the config limit size. > 2. Make query visible. Admin can view all running queries, and detail of each > query. >Split the query into following steps: >1) sql parse >2) cube plan >3) query cache >4) multiple cube segment query > a. for each segment request, have muliple endpoint range request. > b. for each endpoint range request, have multiple coprocessor request. > c. for eache coprocessor request, have multiple region server rpc. >Admin can view the startTime/endTime of each step, and the thread stack > trace if the step is running. > 3. Add query id as corelation id in the rpc log. > 4. Admin can interrupt a running query, to release the thread, memory, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1872) Make query visible and interruptible, improve server's stablility
Ma Gang created KYLIN-1872: -- Summary: Make query visible and interruptible, improve server's stablility Key: KYLIN-1872 URL: https://issues.apache.org/jira/browse/KYLIN-1872 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang Problem: 1. Large query result will break kylin server, for example: select * from fact_table. Even when properties "kylin.query.scan.threshold" and "kylin.query.mem.budget" are set properly, OOM still happens, because the hbase rpc thread is not interrupted, the result will continually go to kylin server. And server will run OOM quickly when there are multiple such queries. 2. Tow many slow queries will occupy all tomcat threads, and make server unresponsed. 3. There's no corelation id for a specified query, so it is hard to find the rpc log for a specified query, if there are too many queries running concurrently. Solution: 1. Interrupt the rpc thread and main query thread when return result size larger than the config limit size. 2. Make query visible. Admin can view all running queries, and detail of each query. Split the query into following steps: 1) sql parse 2) cube plan 3) query cache 4) multiple cube segment query a. for each segment request, have muliple endpoint range request. b. for each endpoint range request, have multiple coprocessor request. c. for eache coprocessor request, have multiple region server rpc. Admin can view the startTime/endTime of each step, and the thread stack trace if the step is running. 3. Add query id as corelation id in the rpc log. 4. Admin can interrupt a running query, to release the thread, memory, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
[ https://issues.apache.org/jira/browse/KYLIN-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1827: --- Description: Currently mail notification is only sent in the onExecuteFinished() method, but no notification will be sent when RuntimeException throws, that may cause user miss some important job build failure, especially for some automation merge jobs. Sometimes job state update fails(the hbase metastore is unavailable in a short time), it will make the job always look like in a running state, but actually it is failed, should send mail to notify user. (was: Currently mail notification only is sent in the onExecuteFinished() method, but no notification will be sent when RuntimeException throws, that may cause user miss some important job build failure, especially for some automation merge jobs. Sometimes job state update fails(the hbase metastore is unavailable in a short time), it will make the job always look like in a running state, but actually it is failed, should send mail to notify user.) > Send mail notification when runtime exception throws during build/merge cube > > > Key: KYLIN-1827 > URL: https://issues.apache.org/jira/browse/KYLIN-1827 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > > Currently mail notification is only sent in the onExecuteFinished() method, > but no notification will be sent when RuntimeException throws, that may cause > user miss some important job build failure, especially for some automation > merge jobs. Sometimes job state update fails(the hbase metastore is > unavailable in a short time), it will make the job always look like in a > running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube
Ma Gang created KYLIN-1827: -- Summary: Send mail notification when runtime exception throws during build/merge cube Key: KYLIN-1827 URL: https://issues.apache.org/jira/browse/KYLIN-1827 Project: Kylin Issue Type: Improvement Components: Job Engine Affects Versions: v1.5.2, v1.5.1 Reporter: Ma Gang Assignee: Ma Gang Currently mail notification only is sent in the onExecuteFinished() method, but no notification will be sent when RuntimeException throws, that may cause user miss some important job build failure, especially for some automation merge jobs. Sometimes job state update fails(the hbase metastore is unavailable in a short time), it will make the job always look like in a running state, but actually it is failed, should send mail to notify user. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1819) Exception swallowed when start DefaultScheduler fail
[ https://issues.apache.org/jira/browse/KYLIN-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1819: --- Attachment: fix_swallow_scheduler_start_exception 1. don't use new thread to init the default scheduler, since the scheduler is a must for the "JOB"/"ALL" mode kylin server, scheduler start in a new thread make user not aware that the scheduler start fail. 2. remove the ConnectionStateListener interface from DefaultScheduler, since it is complicated to implement HA for job engine. > Exception swallowed when start DefaultScheduler fail > > > Key: KYLIN-1819 > URL: https://issues.apache.org/jira/browse/KYLIN-1819 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v1.5.1, v1.5.2 >Reporter: Ma Gang >Assignee: Ma Gang > Attachments: fix_swallow_scheduler_start_exception > > > Start job scheduler need to acquire job lock from zookeeper, when lock > acquire fail, it will throw an IllegalException, but because the scheduler is > started in a new thread, the exception thrown by the thread will be ignored, > and the server still started successfully, and no exceptions are logged. That > make it hard for trouble shooting, should change to make server started fail > when the scheduler started fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1819) Exception swallowed when start DefaultScheduler fail
Ma Gang created KYLIN-1819: -- Summary: Exception swallowed when start DefaultScheduler fail Key: KYLIN-1819 URL: https://issues.apache.org/jira/browse/KYLIN-1819 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v1.5.2, v1.5.1 Reporter: Ma Gang Assignee: Ma Gang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KYLIN-1811) Error step may be skipped sometimes when resume a cube job
Ma Gang created KYLIN-1811: -- Summary: Error step may be skipped sometimes when resume a cube job Key: KYLIN-1811 URL: https://issues.apache.org/jira/browse/KYLIN-1811 Project: Kylin Issue Type: Bug Affects Versions: v1.5.2, v1.5.1 Reporter: Ma Gang Assignee: Ma Gang Fix For: v1.5.3 Error step may be skipped sometimes when resume a cube job, this should be a concurrency issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KYLIN-1740) HDFS storage leak in HBaseResourceStore.deleteResourceImpl() in case of large cell of dictionary or table snapshot
[ https://issues.apache.org/jira/browse/KYLIN-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-1740: --- Attachment: fix_storage_leak_1.4rc.patch > HDFS storage leak in HBaseResourceStore.deleteResourceImpl() in case of large > cell of dictionary or table snapshot > -- > > Key: KYLIN-1740 > URL: https://issues.apache.org/jira/browse/KYLIN-1740 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Ma Gang > Attachments: fix_storage_leak.patch, fix_storage_leak_1.4rc.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KYLIN-1783) Can't add override property at cube design 'Configuration Overwrites' step.
[ https://issues.apache.org/jira/browse/KYLIN-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang closed KYLIN-1783. -- Resolution: Fixed Already been fixed in the latest version > Can't add override property at cube design 'Configuration Overwrites' step. > --- > > Key: KYLIN-1783 > URL: https://issues.apache.org/jira/browse/KYLIN-1783 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v1.5.3 >Reporter: Ma Gang >Assignee: Zhong,Jason > Fix For: v1.5.3 > > > When click the 'Property' button at cube design 'Configuration Overwrites' > step, nothing happens. And there is some js error in the browser's developer > console: > angular.js:10147 TypeError: Cannot read property 'hasOwnProperty' of undefined > at Scope.$scope.addNewProperty (cubeOverwriteProp.js:37) > at angular.js:10943 > at callback (angular.js:19274) > at Scope.$eval (angular.js:12851) > at Scope.$apply (angular.js:12949) > at HTMLButtonElement. (angular.js:19279) > at HTMLButtonElement.jQuery.event.dispatch (jquery.js:5226) > at HTMLButtonElement.elemData.handle (jquery.js:4878) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
