yindarui123 opened a new pull request #6415: URL: https://github.com/apache/dolphinscheduler/pull/6415
# 1. the necessity of task group queue Task group queue (TGQ) can achieve cross-project and cross-process concurrent control of tasks, reducing resource pressure on scheduling systems or other big data clusters. TGQ also supports priority-based control, which ensures that important tasks can be executed first. Users can also execute a task compulsively, ignoring the TGQ. # 2. the details about TGQ TGQ is essentially a flow limiter. By managing resources, the TGQ allows the tasks to obtain resources from the TGQ. In this way, the resources obtained by multiple tasks is limited and the worker node's pressure is avoided. The database optimistic lock is used to solve the thread-safety problem in the distributed concurrent scenario. Note that some tasks are not TGQ bound: - The tasks that do not need to be performed by workers; - The tasks which do not belongs to any TGQ; - The task is forcibly started by the user. ## 2.1 init a TGQ The user manually creates a TGQ. The size of the TGQ is specified by the user. ## 2.2 how do a TGQ works Each task configured with TGQ will apply for resources from the TGQ before being issued to the worker. If the TGQ has no available resources, the task will not be delivered to the worker, and the task will wait for the resource release and resend a request to TGQ. ## 2.3 recycle resources After receiving the response from the worker, TGQ will release the resources corresponding to the task. ## 2.4 fault tolerance In the distributed architecture, the fault tolerance mechanism is considerable. When the worker node is offline, the tasks with a fault tolerance mechanism running on the worker node will be re-executed by the master. In order to prevent the same task from repeatedly applying for resources, when a task succeeds in applying for resources, it should check whether the task is already in the TGQ, and if so, it will resend the task to the worker. If not, allocate resources. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
