drin commented on issue #13827: URL: https://github.com/apache/arrow/issues/13827#issuecomment-1209654088
I think the absolute fastest way is to break up the columns into different files, in which case you'll have far fewer inefficiencies. I think structurally, IPC and parquet are similar, so performance should not drop when switching to IPC from parquet with regard to how much of a file must be read to select a subset of columns. I have ideas of experiments to run to make recommendations, but let me try and see if these happen to exist anywhere... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
