Ngone51 commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-854768806
> if user can't specify it themselves with stage level api, you are saying Spark would internally do it for the user? Yes. And we can add a new conf for users to control the behavior. > so essentially this is extending the locality feature and then the only thing you would need in stage level scheduling api is ability to say use this new locality algorithm for this stage? Yes. I'm thinking a bit more: we probably even don't need the stage level scheduling API ability. After knowing the "mapping", we can use it directly in `resourcesMeetTaskRequirements`. The "mapping" is actually a hard-coded task requirement, and use stage level scheduling API ability to specify that requirement looks redundant and unnecessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
