attilapiros commented on pull request #31790: URL: https://github.com/apache/spark/pull/31790#issuecomment-826039250
cc @dongjoon-hyun With this change pending PODs are not counted as outstanding PODs so their number can be quite high in k8s cluster. But still I would keep the allocation batch size to limit the max number of POD requests made at once. I am thinking about introducing a new limit for the max number of pending PODs (if k8s tends to struggle to handle high number of pending PODs). This new limit must significantly higher then the POD allocation size (we could even derive it from the batchsize with using constant multiplier like * 10 or make the factor configurable). WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
