andygrove commented on issue #6295: URL: https://github.com/apache/datafusion-comet/issues/6295#issuecomment-6223479588
Measured again on `main` at c230bf4e98 (release build, Spark 4.1.3, `local[8]`). Each query is cancelled with `cancelJobGroup` 2 s after it starts, with `interruptOnCancel = true`. The table is the time from the cancel to the last task of the job ending, from two runs each. | Query (203-column, 4M-row Parquet table, 8 tasks of about 400 MB) | Comet | Spark | |---|---:|---:| | native scan feeding a native hash shuffle write (`REPARTITION(32, id)`) | 2.2 to 2.7 s | 51 to 260 ms | | native scan feeding a native sort (`ORDER BY`) | 4.2 to 4.4 s | 2 to 5 ms | The job failed within 70 ms of the cancel in both engines, so Comet's slots stay busy after the job is reported cancelled. The Comet times are each task's remaining work: every task ran to completion. That is the map-stage and sort counterpart of the direct-read kill check #6805 added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
