kou commented on issue #38275: URL: https://github.com/apache/arrow/issues/38275#issuecomment-1769817895
Thanks for providing additional information. In your use case (random access), [File format](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) instead of [Stream format](https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format) may be better. Because we need to read all data from the beginning for Stream format but we don't need to do it for File format. If you don't need to use all data in your Arrow files, you don't need to load some (many) data into memory from disk by using File format. Could you try File format? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
