HyukjinKwon commented on PR #59551: URL: https://github.com/apache/spark/pull/59551#issuecomment-6225340126
@sarutak Thank you for running the benchmark, and for finding that it fails on master and v4.2.0 since SPARK-55715. I added your numbers to the description, with the cause. @viirya Thank you for the review. On sending Arrow-backed batches to the row queue path instead: I measured it on 2,000,000 rows in one partition, with 100-character strings in `data`, a UDF on `data` and the `noop` sink, in 5 runs after a warm-up: | Path | Mean | |---|---| | Arrow columnar path, with the copies | 1458 ms | | Row queue path, forced | 1862 ms | | Columnar input disabled | 1661 ms | The row queue path is slower than without columnar input, since it converts the joined rows back to columns, which the row-based parent then flattens again, while the Arrow columnar path with the copies stays 1.14x faster than without columnar input. So I kept the copy, and fixed the list views and the value counts as you and @sarutak suggested. For the backports, branch-4.2 has no pipelined mode, so the guard against the listener is not needed there. On memory, the comment of `evalArrowColumnar` now says that the copies are not counted by the task memory manager and cannot spill, and that their memory grows with how far the input runs ahead of the results, as for a source that allocates new vectors for every batch. The description has the peaks that @sarutak measured. Releasing the input batches once they are written, as you suggested, applies to all three paths, so I opened SPARK-60306 for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
