Thank you Fred.

Yes I am using device=gpu0. I will switch to the new backend and test again.

On float64, do you mean int64? If yes, am puzzled by that too. In my code I
never explicit cast to int64. Instead I use tensor.ivector() to index
matrices and cast them explicitly into int32. For example:

x = T.ivector()

z = T.cast(y, dtype='int32')

Do you think these things cause the problem?

Thank you,
Haining

Haining Yu on Gmail

On Wed, Aug 9, 2017 at 2:36 PM, Frédéric Bastien <frederic.bast...@gmail.com
> wrote:

> My guess is that you use the old GPU backend. Can you confirm you use the
> Theano flag device=gpu? And that also you have float64 in the graph. The
> old backend don't support them. I suggest that you install the just
> released 0.10 beta and that you use the new backend with device=cuda.
>
> Also,you can use the flag warn_float64=pdb to find where you have them and
> make sore they are float32. This will be faster.
>
> Fred
>
> Le lun. 31 juil. 2017 14:42, Haining Yu <hainin...@gmail.com> a écrit :
>
>> Hi,
>>
>> I am running a RNN/GRU model for a fairly large dataset with the goal of
>> sequence prediction. When I profile my code, I found one GpuFromHost takes
>> ~30% of computation time. See part of profiling results below:
>>
>> <% time> <sum %> <apply time> <time per call> <#call> <id> <Mflops>
>> <Gflops/s> <Apply name>
>>   30.2%    73.0%     462.776s       3.71e-01s   1248   221
>>       GpuFromHost(Subtensor{:int64:}.0)
>>     input 0: dtype=float32, shape=(512, 1024, 2048), strides=(-4096, 4,
>> 2097152)
>>     output 0: dtype=float32, shape=(512, 1024, 2048), strides=(2097152,
>> 2048, 1)
>>
>> theano.printing.debugprint shows that the call is generated in gradient
>> calculation; see snippet below. There is also a HostFromGpu a couple of
>> layers below.
>>
>>  | | | | |GpuFromHost [id FN] ''   221
>>  | | | |   |Subtensor{:int64:} [id FO] ''   220
>>  | | | |     |Subtensor{::int64} [id FP] ''   219
>>  | | | |     | |InplaceDimShuffle{1,2,0} [id FQ] ''   218
>>  | | | |     | | |Reshape{3} [id FR] ''   217
>>  | | | |     | |   |CrossentropyCategorical1HotGrad [id FS] ''   216
>>  | | | |     | |   | |Elemwise{Second}[(0, 0)] [id FT] ''   215
>>  | | | |     | |   | | |CrossentropyCategorical1Hot [id FU] ''   209
>>  | | | |     | |   | | | |HostFromGpu [id FV] ''   206
>>
>> I have heard about the cost of using GpuFromHost (and its counterpart
>> HostFromGpu) and had moved almost all data to GPU (via shared
>> variables). So I don't understand why the call is needed. In particular I
>> don't understand:
>>
>> 1. If all my data are on GPU and theano is optimized for GPU, why is the
>> GpuFromHost even generated?
>> 2. Is the call generated because the memory is too large? The call tries
>> to move 512 x 1024 x 2048 x 4 = 4.2GB memory. But my Tesla K80 should have
>> 12GB memory thus the need to move seems remote on the surface. Overall
>> memory consumption seems OK under profiling.
>> 3. Does the call have anything to do with CrossentropyCategorical1Hot? I
>> assume CrossentropyCategorical1Hot  has been optimized for GPU. But the
>> code shows that a HostFromGPU is called before CrossentropyCategorical1Hot
>> is applied. I am not sure if CrossentropyCategorical1Hot has any memory
>> requirement (e.g., c-contiguous).
>> 4. Should I try any GPU assertion to debug the root cause of the problem?
>>
>> Any hint is appreciated.
>>
>> Thank you,
>> Haining
>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to theano-users+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "theano-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/theano-users/CjR0L_KroOU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to