GitHub user carloea2 edited a discussion: Support Arrow batches as an explicit 
Python UDF input and output mode

### Feature Summary

Allow Python UDFs to explicitly receive and return Arrow batches, avoiding the 
framework's intermediate Python Tuple/Series conversions while keeping existing 
UDF APIs unchanged.

Arrow already reaches the Python worker. Today, InputManager exposes lazy 
Arrow-backed Tuples; BatchOperator builds pandas input from rows and iterates 
pandas output; DataProcessor converts and finalizes output Tuples; 
OutputManager rebuilds Arrow from row values. Those row-object steps add work 
around a UDF that already processes batches.

The proposal removes those detours. A UDF that still uses pandas internally 
pays Arrow → pandas → Arrow, but avoids the framework's Tuple/Series round 
trip. A pure Arrow UDF can also avoid pandas construction.

### Architecture and work removed

![Current Python row detours and proposed direct Arrow batch 
path](https://raw.githubusercontent.com/carloea2/texera/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/arrow-batch-combined.png)

The component names remain; the orange row-conversion steps disappear from the 
proposed path. Schema validation, partitioning, serialization and configured 
storage still require work. The lower row describes proposed batch handling in 
the existing managers, not an implemented engine path.

[Editable 
figure](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/arrow-batch-combined.drawio).

### Proposed Solution or Design

Introduce an explicit opt-in API, for example:

```python
import pyarrow as pa
import pyarrow.compute as pc
from pyamber import ArrowBatchOperator  # Proposed API


class FilterPrices(ArrowBatchOperator):
    BATCH_SIZE = 4096

    def process_batch(self, batch: pa.Table, port: int):
        yield batch.filter(pc.greater(batch["price"], 100))
```

The engine would deliver bounded Arrow batches directly and accept Arrow output 
without expanding every row into a Tuple. Syntax alone would not remove the 
current conversions.

- Keep existing TupleOperatorV2, BatchOperator, and TableOperator behavior 
unchanged.
- Respect configured batch sizes, final partial batches, input-port boundaries, 
and per-port ordering.
- Align output selection with 
[#8914](https://github.com/apache/texera/discussions/8914): `port` in 
`process_batch` identifies the input port; the proposed bare yield targets 
output port 0, while `PortOutput(value, port=n)` explicitly selects another 
output port.
- Validate each yielded Arrow batch against its selected output port's schema 
before routing or storage. Specify casting and null, timestamp, binary, and 
validation-error behavior.
- Match the existing Texera partition destination for the same key values and 
configuration. An arbitrary Arrow hash is insufficient. Flush each output 
link's buffered tail with that link's port schema before completion markers.
- Cooperate with control and backpressure between invocations and output 
yields. Arbitrary UDF code cannot be safely preempted. Do not automatically 
rerun a UDF as fallback, since it may have side effects; define what happens to 
valid output preceding a validation failure.

Start with ArrowBatchOperator. A separate ArrowTableOperator could later 
support whole-port input at completion.

### New local benchmarks — November 4, 2026

**These are conversion/UDF-kernel measurements. The direct Arrow paths are 
standalone prototypes, not integrated engine operators or measured workflow 
speedups.**

The run reused the existing audit worktree at 
`ec3a9dd3ca9c361cbd7f339d8bc78669024e84e8`, without changing engine source. It 
compared the current 
InputManager/BatchOperator/Tuple-finalization/Arrow-reconstruction helpers with 
(1) direct Arrow input/output around the same pandas UDF body and (2) an 
equivalent pure Arrow UDF.

The matrix covers ten-column string, numeric and mixed inputs with nulls and 
timestamps, identity, filter, per-batch grouped count, and an in-memory Arrow 
IPC sink. Batch limits were 1,024, 4,096 and 10,000 rows.

For **10,000 input rows in one batch, filtering about half the rows**, median 
elapsed time in milliseconds:

| Input | Current helpers | Direct Arrow + same pandas filter | Pure Arrow 
filter |
| --- | ---: | ---: | ---: |
| Strings | 1,844.101 | 6.588 | 1.756 |
| Numeric | 1,533.630 | 1.696 | 0.521 |
| Mixed | 1,453.748 | 2.866 | 0.579 |

Peak process RSS increase above the loaded-input baseline, in MiB, measured 
separately:

| Input | Current helpers | Direct Arrow + same pandas filter | Pure Arrow 
filter |
| --- | ---: | ---: | ---: |
| Strings | 45.39 | 5.32 | 3.00 |
| Numeric | 11.95 | 2.31 | 0.85 |
| Mixed | 11.07 | 2.14 | 0.60 |

Time is the median of five samples after one warmup; memory is the median of 
two fresh processes per path. The memory figures describe the first operation 
after imports and input construction, not steady-state worker RSS. All measured 
outputs were checked for values and schema equivalence, and 111 preflight 
checks covered empty, one-row and partial batches, nulls and equivalent 
operations. Allocation tracing and an attribution profile were run separately 
from the timing/RSS measurements.

Environment: Windows 11, Intel Core Ultra 9 275HX, Python 3.12.13, PyArrow 
23.0.1, pandas 2.2.3 and NumPy 2.1.0; one CPU affinity and one thread for the 
relevant compute pools. The machine also had other active work.

The driver excludes MainLoop switching, real partition/hash dispatch, 
backpressure, inspection/retry, Arrow Flight/JVM execution and backend storage. 
It is single-port, retains materialized output batches, and includes small 
manager setup that a long-lived worker would amortize. Per-port routing and 
completion-flush correctness still need engine integration tests. Full 
scan/filter/aggregation/sink workflow comparisons remain pending.

[Full results and batch-size 
matrix](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/benchmark/results.md)
 · [Reproduction package: scripts, raw samples, environment, source hashes and 
checks](https://raw.githubusercontent.com/carloea2/texera/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/arrow-batch-benchmarks.zip).
 The package's README documents the local Windows paths used by its entrypoint.

The original approximately 77× experiment remains a historical 
Table-conversion-only result on ten string columns. Re-running that original 
input/round-trip shape under the new single-thread settings measured 468.4 ms 
versus 6.74 ms, about 69.5×. It omitted BatchOperator's output-row/finalization 
work and is a different baseline from the current-helper cases above; neither 
result predicts a workflow speedup.

### Affected Area

Workflow Engine (Amber)

Originally raised in #8476. Continuing the proposal here for discussion.

GitHub link: https://github.com/apache/texera/discussions/8477

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to