HyukjinKwon commented on PR #59313: URL: https://github.com/apache/spark/pull/59313#issuecomment-6225142853
@FelixYBW Thank you for the report and the reproduction. The bug is in the Arrow columnar input path of Python UDFs (SPARK-56350): it kept the pass-through columns of a batch until the result of the batch came back from Python, after the source had refilled or released their vectors. @sarutak reproduced it without native data sources too, with `ArrowCachedBatchSerializer`. It is fixed in SPARK-60294, #59551, and #59424, which replaces this PR, adds a test of Arrow UDFs over a native data source. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
