alamb commented on issue #6363: URL: https://github.com/apache/arrow-rs/issues/6363#issuecomment-2343438092
Hi @haohuaijin -- the symptoms you describe certainly sounds non ideal I wonder if the problem is with the reporting of memory used (`get_array_memory_size`) or if that much memory is really being allocated by the IPC decoder. For example, if something in the IPC decoder was using 1MB buffers by default, then 1000 columns * 1MB buffers would result in 1GB of memory used. So I agree this sounds like a bug, but my guess is that it is a real memory allocation issue rather than a reporting issue (but I haven't confirmed this) Perhaps you can use something like https://github.com/KDE/heaptrack to track the actual allocations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
