viirya opened a new issue, #6129: URL: https://github.com/apache/datafusion-comet/issues/6129
## What is the problem the feature request solves? Spark 4.1 scalar PySpark `@arrow_udf` expressions run through `ArrowEvalPythonExec`. Comet currently falls back to Spark at this operator, breaking a native execution pipeline and adding Python worker IPC and Arrow serialization costs. ## Describe the potential solution Add an opt-in native Comet operator for Spark 4.1+ scalar Arrow UDFs. The Spark shim should serialize supported `ArrowEvalPythonExec` functions and arguments into Comet's operator plan. Rust should load each Spark function command with PyO3, pass Arrow arrays through the Arrow C Data interface, invoke the Python callable, validate the result, and return it to the native pipeline. Guard planning with a native capability check and fall back to Spark for unsupported features. Initial scope: scalar Arrow UDFs with native child plans, positional or named arguments, and one or more independent functions. Keep the Cargo feature and Spark configuration opt-in. Regular Python UDFs, pandas UDFs, broadcast variables, Python includes, environment overrides, chained Python UDFs, and large Arrow variable types need separate support. ## Additional context A local Spark 4.1.3 benchmark using `pyarrow.compute.negate` on 50 million rows measured 1.276 s for Spark `ArrowEvalPythonExec` versus 0.468 s for the native path (5 measured iterations after warmup; 2.73x). With Comet aggregation and shuffle disabled, the same UDF query measured 1.341 s versus 0.407 s (3.30x). These are local microbenchmark results and need broader CI and workload validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
