andygrove opened a new pull request, #4892: URL: https://github.com/apache/datafusion-comet/pull/4892
## Which issue does this PR close? There is no dedicated tracking issue for this change. ## Rationale for this change `spark.comet.exec.pyarrowUdf.enabled` is `false` by default. When it is off, `mapInArrow` / `mapInPandas` operators silently fall back to vanilla Spark execution, and nothing in the plan tells the user that a faster native path (`CometMapInBatchExec`, which passes Arrow columnar data directly to Python UDFs) is available behind a config flag. Comet already surfaces this kind of guidance for expressions via `NativeOptIn` + `withInfo`, which writes a non-fallback `[COMET-INFO: ...]` message into extended-explain output pointing the user at the config that would opt into a native path. This PR applies the same idiom at the operator level for the PyArrow UDF feature. ## What changes are included in this PR? - In `EliminateRedundantTransitions`, the `EligibleMapInBatch` extractor no longer gates on `spark.comet.exec.pyarrowUdf.enabled`. It now matches whenever an operator *would* run natively (version-shimmed `mapInArrow` / `mapInPandas` matcher, strippable Comet columnar child, `useLargeVarTypes` off). - The conf check moves into the rewrite arm: when the feature is enabled the operator is rewritten to `CometMapInBatchExec` as before; when it is disabled the vanilla Spark operator is left in place but annotated with a `[COMET-INFO]` opt-in hint (reusing `NativeOptIn.message` so the wording stays consistent with the expression-level hints). Because the hint reuses the full eligibility check with only the conf inverted, it fires only on operators that would actually go native if the flag were flipped, and stays silent when `useLargeVarTypes` or a non-Comet child would prevent the rewrite anyway. This is Spark 4.x-only, matching where the feature exists (the 3.x matchers return `None`). ## How are these changes tested? Added two JVM-only cases to the existing `CometMapInBatchSuite` (which exercises the rule against a stubbed `MapInArrowExec` over a Comet leaf without spinning up Python): one asserts the `[COMET-INFO]` hint and config key appear in extended-explain output when the feature is disabled, and one asserts no hint is emitted when it is enabled. No plan-stability golden files are affected, as no TPCDS query uses these operators. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
