viirya commented on PR #59551: URL: https://github.com/apache/spark/pull/59551#issuecomment-6225766264
Following up on my review: I wrote there that the copy is needed, and I'd like to take that back. I don't think path 1 should copy every pass-through column unconditionally. Path 1 was designed to keep the pass-through columns as they are, and copying them on every batch turns it into a different path, with an allocator outside memory accounting, the `ListViewVector` gap, and the lifecycle code that comes with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
