andygrove opened a new issue, #5123: URL: https://github.com/apache/datafusion-comet/issues/5123
### What is the problem the feature request solves? Comet has an opt-in native path (`spark.comet.exec.pyarrowUdf.enabled`) that keeps data in Arrow format across `mapInArrow` / `mapInPandas`, avoiding Spark's Arrow to row to Arrow round trip. #4234 extends this to scalar pandas UDFs (`ArrowEvalPythonExec`) and `applyInPandas` (`FlatMapGroupsInPandasExec`). The remaining Arrow-based Python operators still fall back, even though they exchange Arrow batches with the Python worker and the same optimization applies in principle: - `AggregateInPandasExec` / `ArrowAggregatePythonExec` (grouped-aggregate pandas UDFs, renamed in Spark 4.1) - `WindowInPandasExec` / `ArrowWindowPythonExec` (window pandas UDFs, renamed in Spark 4.1) - `FlatMapGroupsInArrowExec` (`applyInArrow`, Spark 4.0+) - `FlatMapCoGroupsInPandasExec` / `FlatMapCoGroupsInArrowExec` (`cogroup(...).applyInPandas` / `applyInArrow`) - `ArrowEvalPythonUDTFExec` (Arrow-based Python UDTFs, Spark 3.5+) For Python-heavy analytics workloads these operators sit in the middle of otherwise native stages, so each one both pays the row round trip and splits the native stage. ### Describe the potential solution Extend the pyarrowUdf native path to these operators after #4234 lands, in roughly the order listed, since grouped-aggregate and window pandas UDFs are the most common in analytics code. The grouped variants additionally need the grouping/partitioning handled natively, which the `FlatMapGroupsInPandasExec` work in #4234 establishes the pattern for. Pickled (non-Arrow) UDF operators are out of scope since there is no columnar boundary to preserve. `AttachDistributedSequenceExec` is also out of scope here. ### Additional context Related: #4234, the PyArrow UDF user guide, #4384 (fuzz harness for the pyarrow UDF vector-copy path). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
