pitrou commented on issue #45236: URL: https://github.com/apache/arrow/issues/45236#issuecomment-2634370103
Ok, so it seems to be because you are using memray to account for memory. With PyArrow 19.0, by default memray tells me 1.8 GB have been allocated, however `ps` only shows 169 MB RSS. The explanation is that memray is not able to instrument the memory pool Arrow uses by default: https://github.com/bloomberg/memray/issues/577 To get reliable memray results with Arrow, you have to switch to the system memory pool, for example by setting the environment variable `ARROW_DEFAULT_MEMORY_POOL=system`. Then memray shows 178 MB of allocated memory for the same example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
