Hi all,
I launched a Theano Python script with the lib.cnmem=0.9 flag. I then killed the script, and was expecting the GPU memory to get released, but it wasn't. - pkill -9 python and sudo nvidia-smi --gpu-reset -i 0 did not help (GPU 0 is the one where the GPU memory didn't get released). - sudo fuser -v /dev/nvidia* didn't show any unwanted process. How is that possible, and how can I get my memory back, without having to reboot the server? Here is the output of nvidia-smi Thu Aug 18 14:27:58 2016 +------------------------------------------------------+ | NVIDIA-SMI 352.63 Driver Version: 352.63 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce GTX TIT... Off | 0000:02:00.0 Off | N/A | | 29% 61C P2 71W / 250W | 11341MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 1 GeForce GTX TIT... Off | 0000:03:00.0 Off | N/A | | 22% 42C P0 71W / 250W | 23MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 2 GeForce GTX TIT... Off | 0000:82:00.0 Off | N/A | | 22% 35C P0 69W / 250W | 23MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ | 3 GeForce GTX TIT... Off | 0000:83:00.0 Off | N/A | | 0% 33C P0 60W / 250W | 23MiB / 12287MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ I use a GeForce GTX Titan Maxwell with Ubuntu 14.04.4 LTS x64, and Theano 0.8.2. I have the same issue on two different machines (that have approximately the same configuration). Thanks, Franck -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
