AngersZhuuuu commented on pull request #33780: URL: https://github.com/apache/spark/pull/33780#issuecomment-906121166
> > The problem is that not everyone calls sc.stop... in interactive environments you also have people killing it weirdly, there were a bunch of shut down cases in client mode that made determining the real success/failure an issue. > > It was intentionally decided to just always return success in those cases. If this properly handles those cases I'm fine with it but I need to go try to remember what they all were and make sure they are handled. We should not report FAIL unless its truly a failure. > > Got your point..I miss the case such as killing spark-shell weirdly...then how about: > > * add a configuration such as spark.yarn.clientModeAMDisconnect as Failed, when true, make it as failed, when false let it behavior as before? then let user to choose? Two company's scheduler platform meet such case, so we can have a choose. Since we always non't need to rerun pysaprk/spark-shell/spark-sql? > * For pyspark/spark-shell/spark-sql such interactive mode, make it behavior as before and for normal batch job, make it failed when disconnected? I prefer the first way, since always scheduler platform will care about this thing, if scheduler platform call spark-sql/spark-shell/pyspark etc, it will stop the application when finish computation. Only the interactive command line initiated by the user will be killed at will -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
