gaogaotiantian commented on PR #57137: URL: https://github.com/apache/spark/pull/57137#issuecomment-4930590267
First of all, I believe the input part is not removed. Then, I don't believe the output part is meaningful. > Spark's own per-row result converters (defensive list copies, dict→entry-list, Row→dict) that run before pa.array. Arrow can never do this for us — it doesn't know what a Row is — and this half uses no NumPy and doesn't depend on the PyArrow version. It's a permanent simplification of Spark's own hot loop, guarded so that anything unexpected falls back to the existing per-row path with identical error behavior. I don't believe this is correct because the row to arrow converter already deal with cases where the data can be directly converted. The converter would be `None` and no copy will be performed. Could you name a case where the new mechanism works but the old one doesn't? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
