tgravescs commented on PR #37268: URL: https://github.com/apache/spark/pull/37268#issuecomment-1208148350
> Yes, exactly. I put the ResourceProfileManager here because it knows about all the ResourceProfiles. > Adding this API just want to make sure that we have one interface to get compatible RP Ids for scheduler to assign tasks. And the implementation can be further enriched in future maybe for re-use executors with dynamic allocation on > and adding more reuse policy as https://github.com/apache/spark/pull/33941 does. > And we'll not let user to specify compatible ResourceProfiles. In this case, we will only have TaskResourceProfile compatible with Default ResourceProfile. Yeah so to me I think it makes more sense to put into the ResourceProfileManager API to say find me compatible ones but at the same time now that I said it, we may want the user to be able to specify something via the ResourceProfile. That is where like the reuse policy could be specified for instance, or if user really wanted to limit it to exactly one executor resource profile that might be nice. need to think about it a bit more. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
