jorisvandenbossche commented on issue #38806: URL: https://github.com/apache/arrow/issues/38806#issuecomment-1836591138
@hendrikmakait thanks for raising the issue! I agree it would be good to have more explicit methods to copy pyarrow objects (Table, RecordBatch, Array), instead of people relying on the "concat trick" Some related issues about adding deep copy functionality for arrays: https://github.com/apache/arrow/issues/37878, https://github.com/apache/arrow/issues/30503 > Intuitively, I would have thought that copy.deepcopy(table) as well as pickle.loads(pickle.dumps(table)) would serve my purpose. Yes, but unfortunately pickle has the problem that it saves the full buffer instead of only the sliced part. That's a long standing issue with our implementation of pickling, see https://github.com/apache/arrow/issues/26685 (and `copy.deepcopy` relies on pickle) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
