morningman opened a new issue #7367: URL: https://github.com/apache/incubator-doris/issues/7367
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description ## What is wrong? The current Broker Load execution is divided into two stages, `pending` and `loading`. The pending phase is responsible for obtaining file information, and the loading phase is responsible for executing the load. These two stages are run by two thread pools. The current submission logic may cause the actual execution time of the task to be inconsistent with the user's perception. For example, the user submits a broker load and check its status as `LOADING`. But in fact, the job may not start execution, but wait for scheduling. ## Solution The new design scheme is as follows: The `desired_max_waiting_jobs` parameter is used to control the number of broker load jobs running in the cluster. When the number of jobs exceeds this threshold, reject the new job directly. The newly submitted job is in the `PENDING` state. Only after the pending task is successfully submitted to the pending task thread pool will the job actually start (for example, to do "begin txn"). If the submission is not successful, continue to wait. The queue of the loading task thread pool is infinite, that is, the loading task thread pool will unconditionally accept loading tasks generated after the pending task ends. Because the number of jobs running in the cluster has been controlled by the pending task thread pool. ### Solution _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
