adriangb opened a new issue, #9911: URL: https://github.com/apache/arrow-rs/issues/9911
There was an interesting discussion on the Parquet mailing list about writing large values into Parquet files. Summary doc: https://docs.google.com/document/d/1nntcYM98PFSkHT70RexSBPtCnWqg1uRJ5_7m--ZgbsA/edit?tab=t.0#heading=h.k4r8orckhbx0 This did make me think about the situation in arrow-rs. The idea of offloading large completed pages to a temporary file and then reading them back when writing out sounds very interesting to me as a pragmatic solution to the problem. Is this something we could / should support in arrow-rs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
