sujith71955 commented on issue #24036: [SPARK-27036][SPARK-SQL] Cancel the running jobs in the background if broadcast future timeout error occurs URL: https://github.com/apache/spark/pull/24036#issuecomment-471207820 > Is it necessary to kill all these jobs? I get that they will probably fail without the broadcast, but, it's also possible they won't. You are right, the query will be failed, but few points i want to highlight here is, a) Broadcast job which is running in the background may unnecessarily occupy the resources , even-though the query will fail in the end , so it will be better if we can cancel these useless jobs/tasks which we are not sure how much time it can run. b) Another point is regarding web ui, In UI broadcast job will be shown as completed which may sipley mislead the user.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
