[
https://issues.apache.org/jira/browse/KYLIN-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148547#comment-16148547
]
Zhong Yanghong edited comment on KYLIN-2735 at 8/31/17 7:13 AM:
----------------------------------------------------------------
Related commit for branch yaho-cube-planner is as follows:
https://github.com/apache/kylin/commit/d216eba3e10af99cb58b9ca45582d2bf5c8a0a37
was (Author: yaho):
https://github.com/apache/kylin/commit/d216eba3e10af99cb58b9ca45582d2bf5c8a0a37
> Introduce an option to make job scheduler consider job priority
> ---------------------------------------------------------------
>
> Key: KYLIN-2735
> URL: https://issues.apache.org/jira/browse/KYLIN-2735
> Project: Kylin
> Issue Type: Sub-task
> Components: Job Engine
> Affects Versions: v2.2.0
> Reporter: Zhong Yanghong
> Assignee: Zhong Yanghong
>
> To optimize a cube with *N* ready segments, *N* optimize jobs + *1*
> checkpoint job are needed, which may occupy too much resources and make other
> jobs waiting too much time to be scheduled. Thus, job priority and its
> dynamic changing will be introduced. The idea is as follows:
> * Job Fetcher fetches job metadata from database every *M* seconds.
> * A Running Job List is maintained globally
> * {color:#f79232}A Job Priority Queue is maintained globally{color}
> * Before fetching job metadata,
> ** {color:#f79232}first check the job priority queue and schedule jobs
> meeting condition{color} {color:#f691b2}*C*{color}
> ** then if the size of Running Job List exceeds the max concurrent job limit
> *L*, this fetch round will be skipped
> * During fetch round,
> ** {color:#f79232} the priority of each job in the queue will increase.{color}
> ** {color:#f79232} each new job fetched will be assigned with a default
> priority.{color}
> ** {color:#f79232} the highest *L*{color} jobs will be added to Running Job
> List and be submitted to Job Execute Thread Pool
> * Once a job finishes, it will be removed from Running Job List and notify
> the Job Fetcher to fetch job metadata
> The condition {color:#f691b2}*C*{color} is as follows:
> * job priority should be above a threshold, to filter low priority jobs
> * job should wait at least once, to give chance for low priority jobs to be
> scheduled
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
