pitrou commented on pull request #7648: URL: https://github.com/apache/arrow/pull/7648#issuecomment-654729844
The main source of potential inefficiency here is that the Schema is exported/imported once for each chunk. We may or may not case immediately about this. Also, note that you can transfer a Table as a sequence of RecordBatches, rather than a sequence of ChunkedArrays. If you have many many columns, it would probably be more efficient. (in C++, you can use `TableBatchReader` to export a Table as RecordBatches, and `Table::FromRecordBatches` to recreate a Table from the RecordBatches) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
