viirya commented on PR #57099: URL: https://github.com/apache/spark/pull/57099#issuecomment-4928697283
Added one more commit (2e405eb6ffb) after discussion: since Spark 4.2 ships with this code frozen, `_to_pylist` now checks the installed PyArrow version and short-circuits to the native `column.to_pylist()` when it contains the apache/arrow#50326 fix (planned for PyArrow 25.0.1 per the dev-list discussion). This way Spark 4.2 users automatically pick up the native fast path — and any future PyArrow-side improvements — just by upgrading PyArrow, while older PyArrow keeps the bulk conversion here. The version constant is documented and can be bumped in a patch release if the fix ships in a different version. Tests pin the gate off so the bulk paths stay covered regardless of the CI's PyArrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
