Speeds up a little, but still not 100% utilization. On Tuesday, November 15, 2016 at 7:09:52 AM UTC-7, nouiz wrote: > > With device=gpu flag also add the flag lib.cnmem=1 > > This will speed up that code. The new backend so something similar by > default. > > Fred > > Le 14 nov. 2016 12:15, "Michael Klachko" <[email protected] > <javascript:>> a écrit : > >> I will try testing it on Pascal Titan X card when I have time tomorrow, >> and will report back. >> >> On Sat, Nov 12, 2016 at 9:36 PM, Ragav Venkatesan <[email protected] >> <javascript:>> wrote: >> >>> When I was debugging, I also discovered that if I use ignore_border = >>> False for pooling, it doesn't run on GPU in libgpuarray backend. >>> ignore_border = True does. Is there anything to this ? >>> >>> >>> On Saturday, November 12, 2016 at 7:45:22 PM UTC-7, Ragav Venkatesan >>> wrote: >>>> >>>> in htop I usually have one CPU running 100% for both cases. >>>> >>>> >>>> On Saturday, November 12, 2016 at 7:43:16 PM UTC-7, Michael Klachko >>>> wrote: >>>>> >>>>> I'm not sure, but just by looking at CPU usage (top command on Linux) >>>>> you should be able to see the difference. >>>>> >>>>> On Sat, Nov 12, 2016 at 6:19 PM, Ragav Venkatesan < >>>>> [email protected]> wrote: >>>>> >>>>>> Both are using CUdNNs.. I am wondering if some ops are running on the >>>>>> CPU, how do I find that out ? >>>>>> >>>>>> On Friday, November 11, 2016 at 10:00:39 PM UTC-7, Michael Klachko >>>>>> wrote: >>>>>>> >>>>>>> Do both versions use CuDNN? If gpu0 version didn't use it, that >>>>>>> would explain the difference. Also, look at CPU usage for gpu0 version >>>>>>> - it >>>>>>> could be that some ops are running on CPU instead of GPU. >>>>>>> >>>>>>> On Fri, Nov 11, 2016 at 2:20 PM, Ragav Venkatesan < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Running on GTX 1080, cuda0 for device runs for 1.69 minutes at 98% >>>>>>>> , gpu0 runs for 5.12 minutes at 34% . Both runs the same code >>>>>>>> cnn_tutorial >>>>>>>> from theano tutorials. The code is not modified or changed at all. >>>>>>>> floatX=float32, mode = FAST_RUN, nvcc.fastmath = True and nvcc.allowgc >>>>>>>> =True. >>>>>>>> >>>>>>>> On Thursday, November 10, 2016 at 4:47:38 PM UTC-7, Michael Klachko >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Yes. It depends on the size of your network/input - the smaller it >>>>>>>>> is, the harder it is to keep 3k cores busy all the time. >>>>>>>>> Regarding timing, you don't need to write much code: >>>>>>>>> >>>>>>>>> import time >>>>>>>>> start_time = time.time() >>>>>>>>> your code here >>>>>>>>> print "Code ran for {:.1f} minutes".format((time.time() - >>>>>>>>> start_time)/60) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Nov 10, 2016 at 3:26 PM, Ragav Venkatesan < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I'm writing a code to test this, but why do you ask this ? Is >>>>>>>>>> there a case where nvidia-smi might give me 35% util when the GPU is >>>>>>>>>> actually running the code as fast as it can ? >>>>>>>>>> >>>>>>>>>> On Wednesday, November 9, 2016 at 5:36:14 PM UTC-7, Michael >>>>>>>>>> Klachko wrote: >>>>>>>>>>> >>>>>>>>>>> Ragav, so when GPU is 98% utilized, is the training faster than >>>>>>>>>>> when it's 35% utilized? Have you timed it? >>>>>>>>>>> >>>>>>>>>>> On Wed, Nov 9, 2016 at 4:09 PM, Ragav Venkatesan < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> After investigating further I don't think this is a speed or >>>>>>>>>>>> slow issue. I think the newer version of CUDA/cuDNN using the cuda >>>>>>>>>>>> backend >>>>>>>>>>>> is not using the GPU fully. The older version (7.5/5103) of >>>>>>>>>>>> CUDA/cuDNN >>>>>>>>>>>> produce 98% GPU util but the same code on the latest versions >>>>>>>>>>>> (8.0/5105) >>>>>>>>>>>> don't. The code by the way is the lenet tutorial from theano, so >>>>>>>>>>>> its not >>>>>>>>>>>> some weird coding error also. Using the libgpuarray backend, I am >>>>>>>>>>>> able to >>>>>>>>>>>> produce 98% util even with CUDA/cuDNN (8/5105). >>>>>>>>>>>> >>>>>>>>>>>> On Wednesday, November 9, 2016 at 9:48:40 AM UTC-7, nouiz wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> It could be that the new back-end (libgpuarray) is faster and >>>>>>>>>>>>> more efficient in that cases. So just use that back-end :) >>>>>>>>>>>>> >>>>>>>>>>>>> The speed difference between both back-end isn't constant, but >>>>>>>>>>>>> should be a little bit faster with the new back-end in average. >>>>>>>>>>>>> >>>>>>>>>>>>> We have found a few speed regression in the new back-end, but >>>>>>>>>>>>> they where fixed. If you found one, just tell us and we'll fix >>>>>>>>>>>>> it. But the >>>>>>>>>>>>> probably is still low of having slowdown in the new back-end. >>>>>>>>>>>>> >>>>>>>>>>>>> We just merged one such fix with indexing. Make sure to update >>>>>>>>>>>>> libgpuarray and recompile it if you want to be sure to have the >>>>>>>>>>>>> fastest >>>>>>>>>>>>> version. >>>>>>>>>>>>> >>>>>>>>>>>>> Fred >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Nov 8, 2016 at 1:56 PM, Ragav Venkatesan < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, here is a problem I'm getting and I am not sure how to >>>>>>>>>>>>>> solve this. If I use the libgpuarray backend on the cnn_tutorial >>>>>>>>>>>>>> I am >>>>>>>>>>>>>> getting a 98% gpu tutilization with cudnn 5105. If I use cuda >>>>>>>>>>>>>> backend, I am >>>>>>>>>>>>>> only getting about 35% utilization. >>>>>>>>>>>>>> Anyidea why this might be so ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Monday, October 24, 2016 at 9:38:17 AM UTC-7, nouiz wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What errors do you have? Delete your Theano cache, just in >>>>>>>>>>>>>>> case and be sure to use Theano dev version. The last release >>>>>>>>>>>>>>> don't support >>>>>>>>>>>>>>> it I think. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Fred >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Oct 24, 2016 at 12:33 PM, Michael Klachko < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes, it's supported, I'm using it right now (CUDA 8.0 on >>>>>>>>>>>>>>>> Ubuntu 14.04): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>> import theano >>>>>>>>>>>>>>>> Using gpu device 0: TITAN X (Pascal) (CNMeM is enabled with >>>>>>>>>>>>>>>> initial size: 30.0% of memory, cuDNN 5105) >>>>>>>>>>>>>>>> >>> print theano.__version__ >>>>>>>>>>>>>>>> 0.9.0dev3.dev-20fd30a38d34687e9d944140042762ca9fca6276 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Saturday, October 22, 2016 at 2:54:00 PM UTC-7, Ragav >>>>>>>>>>>>>>>> Venkatesan wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I updated and I'm getting some weird errors. With Cuda >>>>>>>>>>>>>>>>> backend, convolutions only run on CPU and with libgpuarray >>>>>>>>>>>>>>>>> backend GPUs >>>>>>>>>>>>>>>>> only run at about 35% util. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> --- >>>>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>>>> Google Groups "theano-users" group. >>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails >>>>>>>>>>>>>>>> from it, send an email to [email protected]. >>>>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "theano-users" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> --- >>>>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>>>> in the Google Groups "theano-users" group. >>>>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>>>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe >>>>>>>>>>>> . >>>>>>>>>>>> To unsubscribe from this group and all its topics, send an >>>>>>>>>>>> email to [email protected]. >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "theano-users" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe >>>>>>>>>> . >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to [email protected]. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>> >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to a topic in >>>>>>>> the Google Groups "theano-users" group. >>>>>>>> To unsubscribe from this topic, visit >>>>>>>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe >>>>>>>> . >>>>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>>> [email protected]. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> -- >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "theano-users" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe >>>>>> . >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "theano-users" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected] <javascript:>. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "theano-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> >
-- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
