code1704 opened a new issue, #13827: URL: https://github.com/apache/arrow/issues/13827
We are building a high performance training system, and we do care about the performance a lot. We store the training data in arrow ipc format file, say, there are 100M rows and 1000 columns. We just need to read data of 10 columns each time for training. Seems with the arrow ipc format file, we have to read the whole file first to get the 10 columns. Is there's any suggestion if we can just read 10 columns only to get better performance? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
