GitHub user carloea2 added a comment to the discussion: Support Arrow batches as an explicit Python UDF input and output mode
I ran the three-way comparison you suggested: current pandas helpers, direct Arrow input/output with the same pandas body, and equivalent pure Arrow kernels. It covers string, numeric and mixed ten-column inputs with nulls/timestamps, batch limits of 1,024/4,096/10,000, and identity/filter/per-batch grouped count/in-memory IPC sink cases. For one 10,000-row mixed-input batch, the same pandas filter measured **1,453.748 → 2.866 ms**, with added peak RSS **11.07 → 2.14 MiB**. The equivalent pure Arrow filter measured **0.579 ms / 0.60 MiB**. So the framework row detour is measurable even when the UDF keeps pandas; the pure Arrow case also changes the UDF implementation. I added all three input mixes to the opening post and the [full matrix, methods and raw-data package](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/benchmark/results.md). These are local conversion/UDF-kernel results, not end-to-end runs. Timing uses five samples after warmup; RSS is separate, two fresh processes per case. Values and schemas were checked across paths. MainLoop, real partition/hash dispatch, backpressure, Flight/JVM and backend storage are excluded, so workflow gains still need measurement. The original Table-only shape reproduced at about 69.5× under these settings; its baseline differs from these BatchOperator-helper cases, so I am not treating either ratio as a general speedup. The API remains opt-in for batch UDFs; existing row-at-a-time operators keep their current behavior. GitHub link: https://github.com/apache/texera/discussions/8477#discussioncomment-19404258 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
