andygrove opened a new issue, #7557: URL: https://github.com/apache/datafusion-comet/issues/7557
### What is the problem the feature request solves? With `spark.comet.nativeExecutors.enabled` on (#7289), a query run with `spark.comet.enabled=false` is still sent to the Comet Native Executor cluster, and the driver refuses it. `CometNativeExecutors.applies` checks `spark.comet.nativeExecutors.enabled` but not `spark.comet.enabled`. So `CometRule` offloads a plan that `CometScanRule` and `CometExecRule` left unconverted, because both return early when Comet is off (`isCometLoaded`). The stage-graph builder then fails on the plan's top operator with "Comet Native Executors cannot run this query, because of ...: it does not run natively". Turning Comet off for a single query is the usual way to keep that query in Spark, for example a step that generates data with a Scala UDF before the queries Comet should run, or a test's `withSQLConf`. With the native executors on, that query fails instead. ### Describe the potential solution Have `CometNativeExecutors.applies` require `spark.comet.enabled` as well as `spark.comet.nativeExecutors.enabled`, so a query with Comet off runs in Spark, as it would without the cluster. Add a test in `CometNativeExecutorsSuite` that runs an aggregate with Comet off and checks that its plan has no `CometNativeExecutorsExec`. ### Additional context Part of #7288. ### Willingness to contribute Yes. I can contribute this fix independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
