comath opened a new issue, #5153: URL: https://github.com/apache/arrow-rs/issues/5153
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** I have machine learning dataset that exceed my system's ram which I access in a random pattern. I currently split the data into a flat f32 array written to the disk, and a set of parquet files. This is awkward and requires complex glue code. **Describe the solution you'd like** I'd like to use a memmap like this: https://huggingface.co/docs/datasets/about_arrow **Describe alternatives you've considered** Switching to another format. rkvy works, but is not nearly as interoperable. **Additional context** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
