viirya commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-847209424
> It's required for the classic use case (when you really need to change executor resources dynamically) but not this case. > > In this case, we only leverage the task scheduling characteristic of the stage level scheduling. Thus, adding the task request for the state store resource only should be enough. Good to know that. So, does it mean I can remove the dynamic allocation check for our case without affecting classic stage-level scheduling? > This means that if an executor goes down it has to wait for something else on executor to start up the task specific state store - what is going to do that in this scenario? Or you wait a certain period and schedule it anywhere. I think this could be configurable. Users can configure it to schedule on anywhere and load from checkpointed data, or in other case just fail the streaming app, after a certain period. > I think this would mean scheduler would have some specific logic to be able to match task id to state store id, right? Otherwise stage level scheduling would schedule a task on anything in that list., which seems like at that point makes a list not relavent if Spark knows how to do some sort of mapping. Hmm, @Ngone51, is it true? For other resources like GPUs, it makes sense but in this case we need specific (task id)-(resource id, e.g. state store id, PVC claim name, etc.) bound. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
