Both are using CUdNNs.. I am wondering if some ops are running on the CPU,
how do I find that out ?
On Friday, November 11, 2016 at 10:00:39 PM UTC-7, Michael Klachko wrote:
>
> Do both versions use CuDNN? If gpu0 version didn't use it, that would
> explain the difference. Also, look at CPU usage for gpu0 version - it could
> be that some ops are running on CPU instead of GPU.
>
> On Fri, Nov 11, 2016 at 2:20 PM, Ragav Venkatesan <ragav.ve...@gmail.com
> <javascript:>> wrote:
>
>> Running on GTX 1080, cuda0 for device runs for 1.69 minutes at 98% , gpu0
>> runs for 5.12 minutes at 34% . Both runs the same code cnn_tutorial from
>> theano tutorials. The code is not modified or changed at all.
>> floatX=float32, mode = FAST_RUN, nvcc.fastmath = True and nvcc.allowgc
>> =True.
>>
>> On Thursday, November 10, 2016 at 4:47:38 PM UTC-7, Michael Klachko wrote:
>>>
>>> Yes. It depends on the size of your network/input - the smaller it is,
>>> the harder it is to keep 3k cores busy all the time.
>>> Regarding timing, you don't need to write much code:
>>>
>>> import time
>>> start_time = time.time()
>>> your code here
>>> print "Code ran for {:.1f} minutes".format((time.time() -
>>> start_time)/60)
>>>
>>>
>>>
>>>
>>> On Thu, Nov 10, 2016 at 3:26 PM, Ragav Venkatesan <ragav.ve...@gmail.com
>>> > wrote:
>>>
>>>> I'm writing a code to test this, but why do you ask this ? Is there a
>>>> case where nvidia-smi might give me 35% util when the GPU is actually
>>>> running the code as fast as it can ?
>>>>
>>>> On Wednesday, November 9, 2016 at 5:36:14 PM UTC-7, Michael Klachko
>>>> wrote:
>>>>>
>>>>> Ragav, so when GPU is 98% utilized, is the training faster than when
>>>>> it's 35% utilized? Have you timed it?
>>>>>
>>>>> On Wed, Nov 9, 2016 at 4:09 PM, Ragav Venkatesan <
>>>>> ragav.ve...@gmail.com> wrote:
>>>>>
>>>>>> After investigating further I don't think this is a speed or slow
>>>>>> issue. I think the newer version of CUDA/cuDNN using the cuda backend is
>>>>>> not using the GPU fully. The older version (7.5/5103) of CUDA/cuDNN
>>>>>> produce
>>>>>> 98% GPU util but the same code on the latest versions (8.0/5105) don't.
>>>>>> The
>>>>>> code by the way is the lenet tutorial from theano, so its not some weird
>>>>>> coding error also. Using the libgpuarray backend, I am able to produce
>>>>>> 98%
>>>>>> util even with CUDA/cuDNN (8/5105).
>>>>>>
>>>>>> On Wednesday, November 9, 2016 at 9:48:40 AM UTC-7, nouiz wrote:
>>>>>>>
>>>>>>> It could be that the new back-end (libgpuarray) is faster and more
>>>>>>> efficient in that cases. So just use that back-end :)
>>>>>>>
>>>>>>> The speed difference between both back-end isn't constant, but
>>>>>>> should be a little bit faster with the new back-end in average.
>>>>>>>
>>>>>>> We have found a few speed regression in the new back-end, but they
>>>>>>> where fixed. If you found one, just tell us and we'll fix it. But the
>>>>>>> probably is still low of having slowdown in the new back-end.
>>>>>>>
>>>>>>> We just merged one such fix with indexing. Make sure to update
>>>>>>> libgpuarray and recompile it if you want to be sure to have the fastest
>>>>>>> version.
>>>>>>>
>>>>>>> Fred
>>>>>>>
>>>>>>> On Tue, Nov 8, 2016 at 1:56 PM, Ragav Venkatesan <
>>>>>>> ragav.ve...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Ok, here is a problem I'm getting and I am not sure how to solve
>>>>>>>> this. If I use the libgpuarray backend on the cnn_tutorial I am
>>>>>>>> getting a
>>>>>>>> 98% gpu tutilization with cudnn 5105. If I use cuda backend, I am only
>>>>>>>> getting about 35% utilization.
>>>>>>>> Anyidea why this might be so ?
>>>>>>>>
>>>>>>>> On Monday, October 24, 2016 at 9:38:17 AM UTC-7, nouiz wrote:
>>>>>>>>>
>>>>>>>>> What errors do you have? Delete your Theano cache, just in case
>>>>>>>>> and be sure to use Theano dev version. The last release don't support
>>>>>>>>> it I
>>>>>>>>> think.
>>>>>>>>>
>>>>>>>>> Fred
>>>>>>>>>
>>>>>>>>> On Mon, Oct 24, 2016 at 12:33 PM, Michael Klachko <
>>>>>>>>> michael...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, it's supported, I'm using it right now (CUDA 8.0 on Ubuntu
>>>>>>>>>> 14.04):
>>>>>>>>>>
>>>>>>>>>> >>> import theano
>>>>>>>>>> Using gpu device 0: TITAN X (Pascal) (CNMeM is enabled with
>>>>>>>>>> initial size: 30.0% of memory, cuDNN 5105)
>>>>>>>>>> >>> print theano.__version__
>>>>>>>>>> 0.9.0dev3.dev-20fd30a38d34687e9d944140042762ca9fca6276
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Saturday, October 22, 2016 at 2:54:00 PM UTC-7, Ragav
>>>>>>>>>> Venkatesan wrote:
>>>>>>>>>>>
>>>>>>>>>>> I updated and I'm getting some weird errors. With Cuda backend,
>>>>>>>>>>> convolutions only run on CPU and with libgpuarray backend GPUs only
>>>>>>>>>>> run at
>>>>>>>>>>> about 35% util.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "theano-users" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to theano-users...@googlegroups.com.
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "theano-users" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to theano-users...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>
>>>>>> ---
>>>>>> You received this message because you are subscribed to a topic in
>>>>>> the Google Groups "theano-users" group.
>>>>>> To unsubscribe from this topic, visit
>>>>>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe
>>>>>> .
>>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>>> theano-users...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>>
>>>> ---
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "theano-users" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> theano-users...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "theano-users" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/theano-users/bSTnP3yLorw/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> theano-users...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
