anjakefala commented on issue #40301: URL: https://github.com/apache/arrow/issues/40301#issuecomment-2071134732
We ran benchmarks on @felipecrv and unfortunately the results weren't promising. With @pitrou's explanation on how to interpret the memory allocation results, and learning more about how the `ThreadPool` works, having a `ParallelFor` for one step of Arrow analysis doesn't significantly reduce memory usage. And the memory usage isn't as high as I was previously concerned it was. I'm going to go ahead and close this issue for now! Thanks everyone for helping me look into this. @jorisvandenbossche @felipecrv and @pitrou :heart: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
