viirya commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-846699726
> That said, personally, as I commented on above comment thread, I'd like to see the interface to let end users do the hack on Spark and do whatever they want, with taking their own risks. In other words, task level scheduling itself sounds OK to me, though I promise I'll respect the decision of committers on CORE area. > > Btw, IMHO, at least for now, initial state distribution and leveraging PVC are something which should be proved in various use cases / workloads in production, before taking them on the plate of discussions. Before then, I'd like to see Spark be customizable on their needs, so that they are no longer blocked on Spark side. This API is proposed to let other developers to change the way in Spark. And yes, taking with their own risks. Actually this might be most less intrusive way to support the improvements on SS side. We don't need to touch other core pieces but just let developers to work with the API. Maybe we even don't need to add the PVC stuff into Spark upstream. As it is pretty clean API implementation, we can just implement it at our side if Spark upstream doesn't want to take the complexity. Mentioning PVC is to show the use-case in our mind when we were asked about how we are going to use the API. Unfortunately so far this is unable to push forward. And I'm not sure right now how much change is needed for stage-level scheduling for similar function. But I also want to respect the decision of committers on core area, so I'd like to take a look on stage-level scheduling as it is suggested above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
