andrei5055 commented on issue #17335: Excessive GPU memory usage with dynamic shape input using Gluon interface URL: https://github.com/apache/incubator-mxnet/issues/17335#issuecomment-606830034 I created a special MXNet tool for the collection of memory consumption statistics. Using this tool, I got the following memory allocation snapshot for this stress test.  As we could see here, the memory actually used during the stress test for each batch size looks OK (see red and very tiny blue parts of the graph). But memory, which reserved in the cache (shown in yellow), grows exponentially with the batch size and for some reason, it is not released. I am not sure if it is the only problem with this stress test. When the program crashes, I could see some error messages regarding some multithread issue in Python. I will continue to work on that problem.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
