[
https://issues.apache.org/jira/browse/KYLIN-2735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-2735:
----------------------------------
Description:
To optimize a cube with *N* ready segments, *N* optimize jobs + *1* checkpoint
job are needed, which may occupy too much resources and make other jobs waiting
too much time to be scheduled. Thus, job priority and its dynamic changing will
be introduced. The idea is as follows:
* Job Fetcher fetches job metadata from database every *M* seconds.
* A Running Job List is maintained globally
* {color:#f79232}A Job Priority Queue is maintained globally{color}
* Before fetching job metadata,
** first check the job priority queue and schedule jobs meeting condition *C*
** then if the size of Running Job List exceeds the max concurrent job limit
*L*, this fetch round will be skipped
* During fetch round,
** {color:#f79232} the priority of each job in the queue will increase.{color}
** {color:#f79232} each new job fetched will be assigned with a default
priority.{color}
** {color:#f79232} the highest *L*{color} jobs will be added to Running Job
List and be submitted to Job Execute Thread Pool
* Once a job finishes, it will be removed from Running Job List and notify the
Job Fetcher to fetch job metadata
The condition *C* is as follows:
* job priority should be above a threshold, to filter low priority jobs
* job should wait at least once, to give chance for low priority jobs to be
scheduled
> Introduce an option to make job scheduler consider job priority
> ---------------------------------------------------------------
>
> Key: KYLIN-2735
> URL: https://issues.apache.org/jira/browse/KYLIN-2735
> Project: Kylin
> Issue Type: Sub-task
> Components: Job Engine
> Affects Versions: v2.2.0
> Reporter: Zhong Yanghong
> Assignee: Zhong Yanghong
>
> To optimize a cube with *N* ready segments, *N* optimize jobs + *1*
> checkpoint job are needed, which may occupy too much resources and make other
> jobs waiting too much time to be scheduled. Thus, job priority and its
> dynamic changing will be introduced. The idea is as follows:
> * Job Fetcher fetches job metadata from database every *M* seconds.
> * A Running Job List is maintained globally
> * {color:#f79232}A Job Priority Queue is maintained globally{color}
> * Before fetching job metadata,
> ** first check the job priority queue and schedule jobs meeting condition *C*
> ** then if the size of Running Job List exceeds the max concurrent job limit
> *L*, this fetch round will be skipped
> * During fetch round,
> ** {color:#f79232} the priority of each job in the queue will increase.{color}
> ** {color:#f79232} each new job fetched will be assigned with a default
> priority.{color}
> ** {color:#f79232} the highest *L*{color} jobs will be added to Running Job
> List and be submitted to Job Execute Thread Pool
> * Once a job finishes, it will be removed from Running Job List and notify
> the Job Fetcher to fetch job metadata
> The condition *C* is as follows:
> * job priority should be above a threshold, to filter low priority jobs
> * job should wait at least once, to give chance for low priority jobs to be
> scheduled
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
