pitrou commented on issue #40301: URL: https://github.com/apache/arrow/issues/40301#issuecomment-2046785908
For the concrete example here, it is still surprising that allocating 26 memo tables, all with only one (!) element, would result in 200+MB of memory allocations on jemalloc, and 1+GB on mimalloc. Our hash table is presized for 32 elements, which shouldn't probably result in so much memory being allocated. (caveat: I don't know what memray is accounting exactly here) So before trying to add heuristics for parallelization, perhaps we should investigate this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
