chengshiwen edited a comment on pull request #5113: URL: https://github.com/apache/incubator-dolphinscheduler/pull/5113#issuecomment-803823872
> Our goal is the scheduling capability of 10,000 tasks per second. Just imagine, if we have 10w+ tasks to schedule, this line of code adds 10w database requests. if we have `10w+` tasks to schedule, and there are `10` workers, we will see `100w+` queries to zk. In fact, the writing and reading performance level of zk is `2w`.  If we want to achieve `10w` task to schedule, first of all we must build performance monitoring. And secondly, we must sort out the priority of performance optimization tasks. The single database can no longer support, we need more slave database, add cache, optimize slow query SQL, and split tasks to different masters, etc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
