cloud-fan commented on code in PR #42199:
URL: https://github.com/apache/spark/pull/42199#discussion_r1332544711
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -77,6 +79,11 @@ object SQLExecution {
}
val rootExecutionId = sc.getLocalProperty(EXECUTION_ROOT_ID_KEY).toLong
executionIdToQueryExecution.put(executionId, queryExecution)
+ val originalInterruptOnCancel =
sc.getLocalProperty(SPARK_JOB_INTERRUPT_ON_CANCEL)
+ if (originalInterruptOnCancel == null) {
+ val interruptOnCancel =
sparkSession.conf.get(SQLConf.INTERRUPT_ON_CANCEL)
+ sc.setInterruptOnCancel(interruptOnCancel)
Review Comment:
I think it's OK to interrupt tasks when we explicitly cancel jobs, but
`sc.setInterruptOnCancel` does more than this. It interrupts tasks if the stage
is finished (completed -> interrupt speculative tasks, failed -> interrupt
remaining running tasks). This has side effects and we'd better avoid it.
Is it possible we can also call `cancelJobGroup` explicitly in the sql-shell?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
