GitHub user carloea2 added a comment to the discussion: Support Arrow batches as an explicit Python UDF input and output mode
Thanks @sshiv012 and @Ma77Ball. For a prototype, I propose comparing the current pandas UDF path, a pure Arrow UDF, and an Arrow UDF that converts to pandas. I would use numeric-heavy, string-heavy, and mixed columns with nulls and timestamps, across several batch sizes. The end-to-end runs would use identical scan, filter, aggregation, and sink workflows, measuring total time, throughput, and peak memory, with output equivalence checks. Surya, I will coordinate with #8914 on explicit output ports, per-port schema validation, matching partition hashes, and completion flushing. The 77x remains a conversion-only result, not an expected workflow speedup. GitHub link: https://github.com/apache/texera/discussions/8477#discussioncomment-19189603 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
