[
https://issues.apache.org/jira/browse/KYLIN-5571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17731966#comment-17731966
]
ASF subversion and git services commented on KYLIN-5571:
--------------------------------------------------------
Commit b43fa19ae5ee0af889c8c263150ccffadc49e5a6 in kylin's branch
refs/heads/kylin5 from fanfanAlice
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b43fa19ae5 ]
KYLIN-5571 Optimize the procedure of pushing down the query
> It takes too much time to calculate the data size during pushing down
> queries, which will lead to the queries un-stoppable.
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: KYLIN-5571
> URL: https://issues.apache.org/jira/browse/KYLIN-5571
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
> Affects Versions: 5.0-alpha
> Reporter: Guangyuan Feng
> Assignee: Guangyuan Feng
> Priority: Major
> Fix For: 5.0-beta
>
>
> During pushing down the query, KE will try to calculate the included data
> size to set Spark partitions, but if there were too many files on HDFS, it
> will take a lot of time to complete.
> So in order to improve this situation, the following things will be done:
> # Using a limited thread pool to calculate the data size
> # Add timeout for the calculation, so as to stop the query as soon as
> possible
> # Add new properties:
> {color:#4c9aff}_kylin.query.pushdown.auto-set-shuffle-partitions-multiple=3_{color}，the
> default Spark partition num
> _{color:#4c9aff}kylin.query.pushdown.auto-set-shuffle-partitions-timeout=30{color},_
> the maximum timeout, 30 seconds by default, to calculate the data size in
> order to adjust the Spark partition num
> After these changes, we can expected the query complete in a fixed duration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
