My memory is fusy, but I think that setting the driver in permanent mode could fix this. It is mostly a disply problem.
nvidia-smi -pm 1 On Tue, Aug 23, 2016 at 9:47 AM, Franck Dernoncourt < [email protected]> wrote: > Thanks, I tried `sudo nvidia-smi --gpu-reset -i 0`, it did not help (GPU 0 > is the one where the GPU memory didn't get released). > > On 23 August 2016 at 09:26, Frédéric Bastien <[email protected]> > wrote: > >> With nvidia-smi, I think you can reset a GPU. I never did it, but I know >> some cluster admin did that with recent version. Maybe this will help you. >> >> Le 22 août 2016 22:13, "Franck Dernoncourt" <[email protected]> >> a écrit : >> >>> I haven't had a chance to reboot yet as other processes are running, but >>> that should indeed release the GPU memory. I was hoping for a solution that >>> doesn't require rebooting. The issue happened to me on two different >>> computers when running Theano programs, so I thought perhaps other Theano >>> users faced the same issue and found a solution (I pinged >>> <https://devtalk.nvidia.com/default/topic/958159/cuda-programming-and-performance/11-gb-of-gpu-ram-used-and-no-process-listed-by-nvidia-smi/post/4954186/#4954186> >>> the Nvidia forum, no luck there yet). >>> >>> On 22 August 2016 at 22:06, Frédéric Bastien <[email protected] >>> > wrote: >>> >>>> Did you reboot? This isn't a problem with Theano but with nvidia driver. >>>> >>>> Le 22 août 2016 18:04, "Franck Dernoncourt" < >>>> [email protected]> a écrit : >>>> >>>>> After killing python, 11341MiB of GPU memory (out of 12287MiB) is >>>>> still not released. >>>>> >>>>> On 22 August 2016 at 18:00, Frédéric Bastien < >>>>> [email protected]> wrote: >>>>> >>>>>> If python is dead then Theano can't use GPU memory anymore. The small >>>>>> memory you see used is used by nvidia driver, not Theano. >>>>>> >>>>>> Fred >>>>>> >>>>>> Le 20 août 2016 08:56, "Franck Dernoncourt" < >>>>>> [email protected]> a écrit : >>>>>> >>>>>>> Thanks, I ran `pkill -9 python`, there is no Python process running >>>>>>> (I checked with `htop`). >>>>>>> >>>>>>> On 20 August 2016 at 08:46, André Lopes <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> open htop and kill the python process that is still running. >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "theano-users" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/theano-users/YTbjKw-KYUE/u >>>>>>>> nsubscribe. >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "theano-users" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "theano-users" group. >>>>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>>>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "theano-users" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "theano-users" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>>> pic/theano-users/YTbjKw-KYUE/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "theano-users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "theano-users" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/to >> pic/theano-users/YTbjKw-KYUE/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
