comphead opened a new pull request, #7681:
URL: https://github.com/apache/datafusion-comet/pull/7681
## Which issue does this PR close?
No issue was filed for this small addition.
## Rationale for this change
Checking whether Comet ran a query natively currently takes
`new
ExtendedExplainInfo().getFallbackReasons(df.queryExecution.executedPlan)` plus
a separate check
that Comet is enabled at all. About 20 suites repeat the first part, and
applications that embed
Comet need the same check. A plan-only check is also misleading on its own:
when Comet disables
itself (for example without off-heap memory or without the Comet shuffle
manager), the plan has no
fallback reasons because Comet never tagged it, so "no fallback reasons"
does not mean "accelerated".
## What changes are included in this PR?
Two methods on `ExtendedExplainInfo`, the class users already reach through
`spark.sql.extendedExplainProviders`:
- `getFallbackReasons(df: DataFrame): Seq[String]` returns the fallback
reasons of the DataFrame's
executed plan. It delegates to the existing `getFallbackReasons(plan:
SparkPlan)`.
- `planAccelerated(df: DataFrame): Boolean` is true when Comet is enabled
for the DataFrame's session
and the plan has no fallback reasons. "Enabled" uses
`CometSparkSessionExtensions.isCometLoaded`,
the same check the Comet planner rules make, rather than
`spark.comet.enabled` alone.
Neither method runs the query. For an adaptive query that has not run yet,
the executed plan is the
initial plan. The scaladoc and the user guide snippet in
`understanding-comet-plans.md` say to run
the query first to get the final plan.
The session conf is read through `df.queryExecution.sparkSession`, because
on Spark 4.x `DataFrame`
is the API `Dataset`, which has `queryExecution` but not `sessionState`.
## How are these changes tested?
A new test in `CometExecSuite`, "getFallbackReasons and planAccelerated for
a DataFrame", covers:
- A query with a known fallback (sampling with replacement):
`getFallbackReasons(df)` contains the
reason, matches the plan overload, and `planAccelerated` is false.
- A fully native filter and project: no reasons, and `planAccelerated` is
true.
- The same query with `spark.comet.enabled=false`: `planAccelerated` is
false.
`spotless:check` passes locally. The suite has not been run locally yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]