GitHub user carloea2 added a comment to the discussion: Support Arrow batches 
as an explicit Python UDF input and output mode

I ran the three-way comparison you suggested: current pandas helpers, direct 
Arrow input/output with the same pandas body, and equivalent pure Arrow 
kernels. It covers string, numeric and mixed ten-column inputs with 
nulls/timestamps, batch limits of 1,024/4,096/10,000, and 
identity/filter/per-batch grouped count/in-memory IPC sink cases.

For one 10,000-row mixed-input batch, the same pandas filter measured 
**1,453.748 → 2.866 ms**, with added peak RSS **11.07 → 2.14 MiB**. The 
equivalent pure Arrow filter measured **0.579 ms / 0.60 MiB**. So the framework 
row detour is measurable even when the UDF keeps pandas; the pure Arrow case 
also changes the UDF implementation. I added all three input mixes to the 
opening post and the [full matrix, methods and raw-data 
package](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/benchmark/results.md).

These are local conversion/UDF-kernel results, not end-to-end runs. Timing uses 
five samples after warmup; RSS is separate, two fresh processes per case. 
Values and schemas were checked across paths. MainLoop, real partition/hash 
dispatch, backpressure, Flight/JVM and backend storage are excluded, so 
workflow gains still need measurement. The original Table-only shape reproduced 
at about 69.5× under these settings; its baseline differs from these 
BatchOperator-helper cases, so I am not treating either ratio as a general 
speedup.

The API remains opt-in for batch UDFs; existing row-at-a-time operators keep 
their current behavior.

GitHub link: 
https://github.com/apache/texera/discussions/8477#discussioncomment-19404258

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to