GitHub user carloea2 added a comment to the discussion: Support Arrow batches 
as an explicit Python UDF input and output mode

Following up on my earlier reply: I updated the opening proposal to align the 
Arrow path with the output-port contract in #8914:

- `process_batch(batch, port)` identifies the input port. Bare output would 
target port 0; `PortOutput(batch, port=n)` would select the output port, 
following your proposed wrapper.
- Validate/cast each yielded batch against that selected output port's schema 
before routing or storage, including null/timestamp/binary behavior.
- Preserve the existing partition destination for each key and configuration. A 
different Arrow hash is not a compatible substitute.
- Flush final partial input batches and each output link's buffered tail before 
completion markers, using the corresponding output-port schema.

The new [benchmark 
report](https://github.com/carloea2/texera/blob/658529d733109d09371a01b3782d50c7eebafc3b/docs/benchmarks/arrow-batches-2026-11-04/benchmark/results.md)
 measures conversion and UDF kernels only. Its driver is single-port and does 
not exercise real hash routing, per-link completion flushing, or 
MainLoop/backpressure. Those remain integration and conformance checks; the 
timings do not establish those semantics.

The diagram keeps DataProcessor and OutputManager as the existing owners, with 
proposed batch handling inside that path. The next design step is to share 
#8914's port-carrying output representation and the routing/schema/flush tests, 
rather than introduce a second output-port mechanism.

GitHub link: 
https://github.com/apache/texera/discussions/8477#discussioncomment-19404260

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to