viirya commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-819342631
> So, how would the plugin help when there's no previous location info? For example, in the plugin implementation, it can distribute tasks to executors more evenly. > I think this is the point that matches my point of case b above. > > But looking at the code, it seems the plugin is still applied after locality scheduling? Locality still works. Generally, this plugin API doesn't break locality if it is set. > > 1. Forcibly assigning state stores to executors can possibly lead to unreasonable scheduling decision. For example, we don't know if the executor satisfy resource requirement. > > I don't get this. Do you mean some executors may not be suitable for having a state store? For example, an executor is not capable for running the task? If we blindly assign stateful task to executor, we don't actually know if the executor is capable for the task. Only the scheduler knows the info. I think this is the major point discussed in my previous PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
