ivoson commented on PR #37268: URL: https://github.com/apache/spark/pull/37268#issuecomment-1201323830
> Can we please add more of a high level design/overview here? > > Also I assume the intent here is to reuse executors since no dynamic allocation - please see #33941 Hi @tgravescs , thanks for your feedback. You are right, the idea here is to reuse executors with `default resource profile` when dynamic allocation is disabled so that we can specify task resource profiles when dynamic allocation is off. This PR introduces a new special `ResourceProfile` (`TaskResourceProfile`) which describe different task resources, but share the same set of executors with default resource profile. Pretty similar to the concept of PR #33941, `TaskResourceProfile` is kind of special `ResourceProfile` which is compatible to default resource profile, so that we can support stage level task resource profiles when dynamic allocation is off. > Why is this only being added for standalone mode? It should be the same for Yarn and K8s cluster also. But I didn't test it in Yarn and K8s cluster right now, so I just enabled for standalone cluster in this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
