Wang, Gang commented on KYLIN-3140:

Yes. The administrator do need to take care of each failed job. And my point is 
the failed auto merge job should not block the user incremental building job.

Currently, the problem is that if the max-building-segments number is set to 1, 
before the failed auto merge job is handled properly and resumed successfully, 
user can not submit a new building job.

And if we differentiate the auto merge job and user build/refresh job, we can 
also set a concurrent threshold for each of them, which may protect Kylin 
server from OOM or other performance issue.

> Auto merge jobs should not block user build jobs
> ------------------------------------------------
>                 Key: KYLIN-3140
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3140
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Job Engine
>            Reporter: Wang, Gang
>            Assignee: Shaofeng SHI
>            Priority: Major
> Although in the latest version, Kylin support concurrent jobs. If the 
> concurrency is set to 1, there is some possibility that cube build job will 
> have dead lock. Say, when there is some issue which causes merge job failed, 
> even when you discard the job, another job will be launched and failed again 
> due to auto merge policy. And this failed merge job blocks user to build 
> incremental segment.
> Even if the concurrency is set to larger than 1, the auto merge jobs occupy 
> some concurrency quota. 
> While, from user perspective, they don't care much about the auto merge jobs, 
> and the auto merge jobs should not block the building/refresh jobs they 
> submit manually.
> A better way may be separating the auto merge jobs from the job queue, 
> parameter max-building-segments only limit jobs submitted by users.

This message was sent by Atlassian JIRA

Reply via email to