[
https://issues.apache.org/jira/browse/KYLIN-5874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865900#comment-17865900
]
pengfei.zhan commented on KYLIN-5874:
-------------------------------------
h1. Dev design
Parameter explanation
kylin.job.max-concurrent-jobs (default value 20) Supports both system-level and
project-level configuration; when configured at the system level, the meaning
is the default value of the maximum number of parallel tasks for each project;
when you need to modify the maximum number of parallel tasks for an individual
project, you can configure the subparameter at the project level.
Behavioral difference from the version before 4.6.18: If the parameter value is
modified, it does not take effect on the tasks that were already in PENDING
state before the modification.
New parameter added:
kylin.jobs.node-max-concurrent-jobs (default value 30), only supported to be
configured at system level, meaning the maximum number of parallel tasks per
node, independent of the project. Different nodes can set different values.
Setting this parameter is used for underpinning, which can avoid OOM of the
node in some extreme cases.
h1. Implementation Principle
The master node adds a memory cache for the READY state task, and the logic of
the master node's timed produce task is changed to the following steps:
Clear invalid lock records
Read all tasks with statuses READY, PENDING, and RUNNIG from the task metadata
table (job_info).
Cache the tasks with READY status by project, the cached data structure is
PriorityQueue; PriorityQueue sorts the tasks according to their priority and
createTime.
Iterate through the PENDING and RUNNIG tasks and calculate the number of
running tasks for each project; read the kylin.job.max-concurrent-jobs value at
the project level, and subtract the number of running tasks to calculate how
many more tasks can be produced for each project.
Consume each project's READY task cache and produce the corresponding number of
tasks.
> Change 'max concurrent limit' to project level config
> -----------------------------------------------------
>
> Key: KYLIN-5874
> URL: https://issues.apache.org/jira/browse/KYLIN-5874
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: 5.0-alpha
> Reporter: pengfei.zhan
> Assignee: pengfei.zhan
> Priority: Major
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
