yuanlihan opened a new pull request #9272: Add a dedicated thread to execute callbacks attached by TaskQueue URL: https://github.com/apache/druid/pull/9272 This PR adds a dedicated thread to execute callbacks of task status updates attached by `TaskQueue ` and then the specific thread which notifies the change of task status can return as soon as possible. ### Description Found that tasks sometimes are failed to assign within the default timeout(5 minutes) but the tasks were actually already running on middleManager according to the logs. Also found that the thread of pendingTasksExec, which supposed to be notified with the task status changing within the timeout, fails to be waked up. After diving into this issue, I think I have found the root cause. The Guava concurrent framework will schedule the callback to be run by this syncer thread and the task syncer thread possibly get stuck if it waits in the giant lock. Both the `remote` and `httpRemote` task runner have this issue. Since the callback handler attached by TaskQueue seem a bit heavyweight, it will be better to provide a dedicated thread to execute these callbacks. This issue can be fixed by adding a single thread executor to handle callbacks attached by `TaskQueue`. <hr> This PR has: - [X] been self-reviewed. - [X] been tested in a test Druid cluster. <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist above are strictly necessary, but it would be very helpful if you at least self-review the PR. --> <hr> ##### Key changed classes in this PR * `TaskQueue`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
