With nvidia-smi, I think you can reset a GPU. I never did it, but I know some cluster admin did that with recent version. Maybe this will help you.
Le 22 août 2016 22:13, "Franck Dernoncourt" <[email protected]> a écrit : > I haven't had a chance to reboot yet as other processes are running, but > that should indeed release the GPU memory. I was hoping for a solution that > doesn't require rebooting. The issue happened to me on two different > computers when running Theano programs, so I thought perhaps other Theano > users faced the same issue and found a solution (I pinged > <https://devtalk.nvidia.com/default/topic/958159/cuda-programming-and-performance/11-gb-of-gpu-ram-used-and-no-process-listed-by-nvidia-smi/post/4954186/#4954186> > the Nvidia forum, no luck there yet). > > On 22 August 2016 at 22:06, Frédéric Bastien <[email protected]> > wrote: > >> Did you reboot? This isn't a problem with Theano but with nvidia driver. >> >> Le 22 août 2016 18:04, "Franck Dernoncourt" <[email protected]> >> a écrit : >> >>> After killing python, 11341MiB of GPU memory (out of 12287MiB) is still >>> not released. >>> >>> On 22 August 2016 at 18:00, Frédéric Bastien <[email protected] >>> > wrote: >>> >>>> If python is dead then Theano can't use GPU memory anymore. The small >>>> memory you see used is used by nvidia driver, not Theano. >>>> >>>> Fred >>>> >>>> Le 20 août 2016 08:56, "Franck Dernoncourt" < >>>> [email protected]> a écrit : >>>> >>>>> Thanks, I ran `pkill -9 python`, there is no Python process running >>>>> (I checked with `htop`). >>>>> >>>>> On 20 August 2016 at 08:46, André Lopes <[email protected]> >>>>> wrote: >>>>> >>>>>> open htop and kill the python process that is still running. >>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "theano-users" group. >>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "theano-users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "theano-users" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "theano-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "theano-users" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/theano-users/YTbjKw-KYUE/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
