safrooze commented on issue #12116: Excessive memory allocation without static_alloc URL: https://github.com/apache/incubator-mxnet/issues/12116#issuecomment-413052714 The network does indeed run out of memory with larger loop count. ``` mxnet.base.MXNetError: [23:50:33] src/storage/./pooled_storage_manager.h:119: cudaMalloc failed: out of memory ``` I also tried setting `MXNET_GPU_MEM_POOL_RESERVE=100` and that has an interesting behavior: The peak memory usage doesn't change, but at the point where typically memory would stabilize, it resets back to ~4GB and climbs back up and again resets back and continues this pattern. I should mention that inference is composed of two hybridized networks. For each inference instance, the first network is called once and then the next network is called several times with fixed input shapes. The peak memory usage is a function of number of times the second network is called (i.e. the number of loop iterations). Without setting `MXNET_GPU_MEM_POOL_RESERVE`, if the network doesn't run out of memory for each inference instance, the memory utilization (i.e. buffer pool size) stabilizes and stays constant for subsequent inference runs.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
