GitHub user Ma77Ball added a comment to the discussion: Support Arrow batches as an explicit Python UDF input and output mode
Two things worth pinning down.�� 1. The 77x is close to a best case. The benchmark uses all string columns, and strings are the most expensive type to convert row by row. Numbers and timestamps convert much more cheaply, so a mixed workload should see a smaller speedup. Worth running the end-to-end benchmark across a few column-type mixes.�� 2. Also worth being clear on what the operator gets. Arrow already reaches the worker, but operators today only see pandas (batch/table) or row tuples, never Arrow. So the win is mainly for batch/table operators, and biggest only when they stay in Arrow the whole way. One that still wants pandas internally pays Arrow to pandas to Arrow, so that path is worth measuring too. For row-at-a-time operators, there's no saving, and keeping the new class opt-in looks right. GitHub link: https://github.com/apache/texera/discussions/8477#discussioncomment-19141369 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
