NightMachinary commented on issue #4802: URL: https://github.com/apache/arrow/issues/4802#issuecomment-1019985875
I have a `(10**6, 10**4)` numpy array. I want to store this in an efficient format and be able to load batches from this file to use in an out-of-core algorithm. Is Arrow suitable for this purpose? How do I do it? PS: I actually generate the big numpy array as 10 parts of `(10**5, 10**4)`; so I need to be able to append to the storage file. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
