GitHub user carloea2 edited a discussion: Support Arrow batches as an explicit Python UDF input and output mode
### Feature Summary Allow Python UDFs to explicitly receive and return Arrow batches, avoiding the framework's intermediate Python Tuple/Series conversions while keeping existing UDF APIs unchanged. Arrow already reaches the Python worker. Today, InputManager exposes lazy Arrow-backed Tuples; BatchOperator builds pandas input from rows and iterates pandas output; DataProcessor converts and finalizes output Tuples; OutputManager rebuilds Arrow from row values. Those row-object steps add work around a UDF that already processes batches. The proposal removes those detours. A UDF that still uses pandas internally pays Arrow → pandas → Arrow, but avoids the framework's Tuple/Series round trip. A pure Arrow UDF can also avoid pandas construction. ### Architecture and work removed  The component names remain; the orange row-conversion steps disappear from the proposed path. Schema validation, partitioning, serialization and configured storage still require work. The lower row describes proposed batch handling in the existing managers, not an implemented engine path. [Editable figure](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/arrow-batch-combined.drawio). ### Proposed Solution or Design Introduce an explicit opt-in API, for example: ```python import pyarrow as pa import pyarrow.compute as pc from pyamber import ArrowBatchOperator # Proposed API class FilterPrices(ArrowBatchOperator): BATCH_SIZE = 4096 def process_batch(self, batch: pa.Table, port: int): yield batch.filter(pc.greater(batch["price"], 100)) ``` The engine would deliver bounded Arrow batches directly and accept Arrow output without expanding every row into a Tuple. Syntax alone would not remove the current conversions. - Keep existing TupleOperatorV2, BatchOperator, and TableOperator behavior unchanged. - Respect configured batch sizes, final partial batches, input-port boundaries, and per-port ordering. - Align output selection with [#8914](https://github.com/apache/texera/discussions/8914): `port` in `process_batch` identifies the input port; the proposed bare yield targets output port 0, while `PortOutput(value, port=n)` explicitly selects another output port. - Validate each yielded Arrow batch against its selected output port's schema before routing or storage. Specify casting and null, timestamp, binary, and validation-error behavior. - Match the existing Texera partition destination for the same key values and configuration. An arbitrary Arrow hash is insufficient. Flush each output link's buffered tail with that link's port schema before completion markers. - Cooperate with control and backpressure between invocations and output yields. Arbitrary UDF code cannot be safely preempted. Do not automatically rerun a UDF as fallback, since it may have side effects; define what happens to valid output preceding a validation failure. Start with ArrowBatchOperator. A separate ArrowTableOperator could later support whole-port input at completion. ### New local benchmarks — November 4, 2026 **These are conversion/UDF-kernel measurements. The direct Arrow paths are standalone prototypes, not integrated engine operators or measured workflow speedups.** The run reused the existing audit worktree at `ec3a9dd3ca9c361cbd7f339d8bc78669024e84e8`, without changing engine source. It compared the current InputManager/BatchOperator/Tuple-finalization/Arrow-reconstruction helpers with (1) direct Arrow input/output around the same pandas UDF body and (2) an equivalent pure Arrow UDF. The matrix covers ten-column string, numeric and mixed inputs with nulls and timestamps, identity, filter, per-batch grouped count, and an in-memory Arrow IPC sink. Batch limits were 1,024, 4,096 and 10,000 rows. For **10,000 input rows in one batch, filtering about half the rows**, median elapsed time in milliseconds: | Input | Current helpers | Direct Arrow + same pandas filter | Pure Arrow filter | | --- | ---: | ---: | ---: | | Strings | 1,844.101 | 6.588 | 1.756 | | Numeric | 1,533.630 | 1.696 | 0.521 | | Mixed | 1,453.748 | 2.866 | 0.579 | Peak process RSS increase above the loaded-input baseline, in MiB, measured separately: | Input | Current helpers | Direct Arrow + same pandas filter | Pure Arrow filter | | --- | ---: | ---: | ---: | | Strings | 45.39 | 5.32 | 3.00 | | Numeric | 11.95 | 2.31 | 0.85 | | Mixed | 11.07 | 2.14 | 0.60 | Time is the median of five samples after one warmup; memory is the median of two fresh processes per path. The memory figures describe the first operation after imports and input construction, not steady-state worker RSS. All measured outputs were checked for values and schema equivalence, and 111 preflight checks covered empty, one-row and partial batches, nulls and equivalent operations. Allocation tracing and an attribution profile were run separately from the timing/RSS measurements. Environment: Windows 11, Intel Core Ultra 9 275HX, Python 3.12.13, PyArrow 23.0.1, pandas 2.2.3 and NumPy 2.1.0; one CPU affinity and one thread for the relevant compute pools. The machine also had other active work. The driver excludes MainLoop switching, real partition/hash dispatch, backpressure, inspection/retry, Arrow Flight/JVM execution and backend storage. It is single-port, retains materialized output batches, and includes small manager setup that a long-lived worker would amortize. Per-port routing and completion-flush correctness still need engine integration tests. Full scan/filter/aggregation/sink workflow comparisons remain pending. [Full results and batch-size matrix](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/benchmark/results.md) · [Reproduction package: scripts, raw samples, environment, source hashes and checks](https://raw.githubusercontent.com/carloea2/texera/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/arrow-batch-benchmarks.zip). The package's README documents the local Windows paths used by its entrypoint. The original approximately 77× experiment remains a historical Table-conversion-only result on ten string columns. Re-running that original input/round-trip shape under the new single-thread settings measured 468.4 ms versus 6.74 ms, about 69.5×. It omitted BatchOperator's output-row/finalization work and is a different baseline from the current-helper cases above; neither result predicts a workflow speedup. ### Affected Area Workflow Engine (Amber) Originally raised in #8476. Continuing the proposal here for discussion. GitHub link: https://github.com/apache/texera/discussions/8477 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
