somandal opened a new pull request, #16981: URL: https://github.com/apache/pinot/pull/16981
Today we only limit the total tasks allowed to run on a minion instance based on the task type. This does not protect against scenarios where too many tasks across different tasks types get scheduled on a single minion. Scheduling too many tasks can lead to resource utilization issues like OOMs or high disk usage. This PR adds support to set an upper limit cap on the max concurrent tasks that can be run on on a given minion instance. This utilizes the Helix side Instance level config defined here: https://helix.apache.org/1.3.2-docs/tutorial_task_throttling.html Testing done: Did exhaustive testing using the HybridQuickStart. Some scenarios tested: - Set an instance level cap of 3, and a higher cap of 5 for the per task type level - saw that only 3 tasks were scheduled at a time - Set an instance level cap of 3, and a cap of 2 for the per task type level - if a single task type ran, only 2 were scheduled at a time, but if two different tasks types were run, then a maximum of 3 task were scheduled on the minion - Setting the default of -1 keeps the behavior the same as today, and the default Helix side limit is used - Tested config update via setting the config in the Pinot cluster configs, and that overrides the default cc @krishan1390 @swaminathanmanish @Jackie-Jiang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
