hvanhovell commented on PR #40649: URL: https://github.com/apache/spark/pull/40649#issuecomment-1496739368
@Hisoka-X thanks for the write up. We should be able to support most of this at the moment. GRPC supports this type of execution out of the box. The reason we did not really go for this, is because of API compatibility. The `SparkResult` does support incremental collect and can collect results in the background though. The thing that Martin was getting at in the ticket is more about what to do when disconnect happen. You probably want to reconnect in these cases, this does require some architectural rework. We are discussing how we should do this, there are quite a few trade offs here. Do you mind shelving this until we can provide a bit more clarity? Please let me know if you want in on these conversations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
