andygrove opened a new issue, #7372: URL: https://github.com/apache/datafusion-comet/issues/7372
### What is the problem the feature request solves? When a Spark job running on a Comet Native Executor cluster (#7289) is cancelled, for example from the Spark UI, by `SparkContext.cancelJob` or by a Spark Connect interrupt, the query keeps running on the cluster. The driver waits in `NativeExecutorsClient.submit` until the whole query ends, and nothing tells the scheduler to stop it. ### Describe the potential solution Submit the query without waiting for it, keep its job id, and cancel the Ballista job when Spark cancels the task or job that's waiting on it, through a Flight action on the scheduler that calls Ballista's job cancellation. ### Additional context The scheduler's Flight service is `native/native-executors/src/frontend.rs`, which waits for each job in `run`. Part of #7288. ### Willingness to contribute Yes. I can contribute this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
