Yes. It depends on the size of your network/input - the smaller it is, the
harder it is to keep 3k cores busy all the time.
Regarding timing, you don't need to write much code:
import time
start_time = time.time()
your code here
print "Code ran for {:.1f} minutes".format((time.time() -
start_time)/60)
On Thu, Nov 10, 2016 at 3:26 PM, Ragav Venkatesan <
[email protected]> wrote:
> I'm writing a code to test this, but why do you ask this ? Is there a case
> where nvidia-smi might give me 35% util when the GPU is actually running
> the code as fast as it can ?
>
> On Wednesday, November 9, 2016 at 5:36:14 PM UTC-7, Michael Klachko wrote:
>>
>> Ragav, so when GPU is 98% utilized, is the training faster than when it's
>> 35% utilized? Have you timed it?
>>
>> On Wed, Nov 9, 2016 at 4:09 PM, Ragav Venkatesan <[email protected]>
>> wrote:
>>
>>> After investigating further I don't think this is a speed or slow issue.
>>> I think the newer version of CUDA/cuDNN using the cuda backend is not using
>>> the GPU fully. The older version (7.5/5103) of CUDA/cuDNN produce 98% GPU
>>> util but the same code on the latest versions (8.0/5105) don't. The code by
>>> the way is the lenet tutorial from theano, so its not some weird coding
>>> error also. Using the libgpuarray backend, I am able to produce 98% util
>>> even with CUDA/cuDNN (8/5105).
>>>
>>> On Wednesday, November 9, 2016 at 9:48:40 AM UTC-7, nouiz wrote:
>>>>
>>>> It could be that the new back-end (libgpuarray) is faster and more
>>>> efficient in that cases. So just use that back-end :)
>>>>
>>>> The speed difference between both back-end isn't constant, but should
>>>> be a little bit faster with the new back-end in average.
>>>>
>>>> We have found a few speed regression in the new back-end, but they
>>>> where fixed. If you found one, just tell us and we'll fix it. But the
>>>> probably is still low of having slowdown in the new back-end.
>>>>
>>>> We just merged one such fix with indexing. Make sure to update
>>>> libgpuarray and recompile it if you want to be sure to have the fastest
>>>> version.
>>>>
>>>> Fred
>>>>
>>>> On Tue, Nov 8, 2016 at 1:56 PM, Ragav Venkatesan <[email protected]
>>>> > wrote:
>>>>
>>>>> Ok, here is a problem I'm getting and I am not sure how to solve this.
>>>>> If I use the libgpuarray backend on the cnn_tutorial I am getting a 98%
>>>>> gpu
>>>>> tutilization with cudnn 5105. If I use cuda backend, I am only getting
>>>>> about 35% utilization.
>>>>> Anyidea why this might be so ?
>>>>>
>>>>> On Monday, October 24, 2016 at 9:38:17 AM UTC-7, nouiz wrote:
>>>>>>
>>>>>> What errors do you have? Delete your Theano cache, just in case and
>>>>>> be sure to use Theano dev version. The last release don't support it I
>>>>>> think.
>>>>>>
>>>>>> Fred
>>>>>>
>>>>>> On Mon, Oct 24, 2016 at 12:33 PM, Michael Klachko <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Yes, it's supported, I'm using it right now (CUDA 8.0 on Ubuntu
>>>>>>> 14.04):
>>>>>>>
>>>>>>> >>> import theano
>>>>>>> Using gpu device 0: TITAN X (Pascal) (CNMeM is enabled with initial
>>>>>>> size: 30.0% of memory, cuDNN 5105)
>>>>>>> >>> print theano.__version__
>>>>>>> 0.9.0dev3.dev-20fd30a38d34687e9d944140042762ca9fca6276
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Saturday, October 22, 2016 at 2:54:00 PM UTC-7, Ragav Venkatesan
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I updated and I'm getting some weird errors. With Cuda backend,
>>>>>>>> convolutions only run on CPU and with libgpuarray backend GPUs only
>>>>>>>> run at
>>>>>>>> about 35% util.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "theano-users" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "theano-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "theano-users" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>> pic/theano-users/bSTnP3yLorw/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> [email protected].
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "theano-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/to
> pic/theano-users/bSTnP3yLorw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
