alamb commented on PR #11273: URL: https://github.com/apache/arrow-rs/pull/11273#issuecomment-6603254983
In terms of trying to help get this PR / set of PRs in / merged, I would like to suggest we start with a motivating example / benchmark that we can use to motivate the work more explicitly and thus attract reviewer attention -- especially as I think the use case is something that is widely interesting to people. Specifically I think your high level usecase is to implement compaction of multiple parquet files remaining both CPU and memory efficient, which I guaratnee you is **SUPER** relevant to many others and will be very exciting. However, as currently described "native REE support" making this connection requires quite a bit of intimate knowledge (and I am not sure i would have put it together if I hadn't spoken with you in person first) So, perhaps we can start with: 1. A test / harness that shows your usecase with an annotated example (e.g. generate 10 100MB parquet files, and then merge them together into a new single parquet file, reporting the time and peak memory usgae) 2. Some sort of high level writeup (blog post? Design doc?) that explains the usecase and then sketches the solution to it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
