Thanks, I tried `sudo nvidia-smi --gpu-reset -i 0`, it did not help (GPU 0 is the one where the GPU memory didn't get released).
On 23 August 2016 at 09:26, Frédéric Bastien <[email protected]> wrote: > With nvidia-smi, I think you can reset a GPU. I never did it, but I know > some cluster admin did that with recent version. Maybe this will help you. > > Le 22 août 2016 22:13, "Franck Dernoncourt" <[email protected]> > a écrit : > >> I haven't had a chance to reboot yet as other processes are running, but >> that should indeed release the GPU memory. I was hoping for a solution that >> doesn't require rebooting. The issue happened to me on two different >> computers when running Theano programs, so I thought perhaps other Theano >> users faced the same issue and found a solution (I pinged >> <https://devtalk.nvidia.com/default/topic/958159/cuda-programming-and-performance/11-gb-of-gpu-ram-used-and-no-process-listed-by-nvidia-smi/post/4954186/#4954186> >> the Nvidia forum, no luck there yet). >> >> On 22 August 2016 at 22:06, Frédéric Bastien <[email protected]> >> wrote: >> >>> Did you reboot? This isn't a problem with Theano but with nvidia driver. >>> >>> Le 22 août 2016 18:04, "Franck Dernoncourt" < >>> [email protected]> a écrit : >>> >>>> After killing python, 11341MiB of GPU memory (out of 12287MiB) is still >>>> not released. >>>> >>>> On 22 August 2016 at 18:00, Frédéric Bastien < >>>> [email protected]> wrote: >>>> >>>>> If python is dead then Theano can't use GPU memory anymore. The small >>>>> memory you see used is used by nvidia driver, not Theano. >>>>> >>>>> Fred >>>>> >>>>> Le 20 août 2016 08:56, "Franck Dernoncourt" < >>>>> [email protected]> a écrit : >>>>> >>>>>> Thanks, I ran `pkill -9 python`, there is no Python process running >>>>>> (I checked with `htop`). >>>>>> >>>>>> On 20 August 2016 at 08:46, André Lopes <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> open htop and kill the python process that is still running. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "theano-users" group. >>>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>>>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "theano-users" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "theano-users" group. >>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "theano-users" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "theano-users" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "theano-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "theano-users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/theano-users/YTbjKw-KYUE/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
