Re: [theano-users] Re: CNMeM is disabled, CuDNN not available

[anishi gupta]

Sir I think GPUArray will itself control GPU memory allocation, there is no
need of cnmem now as it is depreciated whic is controlled via CUDA backend.

On Mon, Mar 6, 2017 at 12:20 PM, anishi gupta 
wrote:

> Hello Sir,
>
> When I used command "device=gpuN python file.py" then code is successfullt
> running on gpu.As you said theano FLAGS will override .rc file settings so
> I think there is no need of making .theanorc file. Kindly tell me how to
> make cuda memory faster if I dont use cnmem...I want to gain speedup in gpu
> as compare to cpu
>
>
>
> On Fri, Mar 3, 2017 at 7:08 PM, Ramana Subramanyam 
> wrote:
>
>> Hi,
>> You don't set the device parameter. Set your device to gpuN and try
>> running again. cnmem is from the old backend, which will be depreciated
>> soon. You should start using device=cudaN and config.gpuarray.preallocate 
>> instead
>> of lib.cnmem. Also, you seem to override the config file with THEANO_FLAGS.
>> Use either of them (remember FLAGS can override .rc file settings).
>>
>> Ramana
>>
>> On Friday, March 3, 2017 at 7:06:24 PM UTC+5:30, Ani wrote:
>>>
>>> No sir I have put
>>> [lib]
>>> cnmem=1
>>> but it shows error of lib command not found..what to do sir now.
>>> On 3 Mar 2017 5:07 p.m., "Kiuhnm Mnhuik"  wrote:
>>>
>>> You probably forgot [lib]:
>>>
>>> [lib]
>>> cnmem = 1
>>>
>>> Hello Sir...

 I have successfully installed CuDNN and it is also available...but
 problem is with cnmem..When I wrote cnmem=1 in my .theanorc file which is
 at home directory..it gives error on running any code with cpu or
 gpu..please have a look at error(screenshot is attached).

 Thanks in advance
 Regards
 Anishi Gupta

 On Saturday, 28 May 2016 11:05:29 UTC+5:30, Poornachandra Sandur wrote:
>
> Hi,
>CNMem is  fast CUDA memory allocator ... if you
> want to set it you can do it by writing the below lines in your .theanorc
> file  ...
>
> [lib]
> cnmem=1
>
>
> ---
>
> For CUDNN you can do it with the following steps :
>
> *First download the file : cudnn-7.0-linux-x64-v3.0-prod.tgz*
>
> *Extract it to home directory*
>
> *and set the LD_LIBRARY_PATH to the above extracted directory*
>
> *and then follow the below steps :*
>
>
>
> *sudo cp /home/poornachandra/cuda/include/cudnn.h
> /usr/local/cuda-7.5/include/*
>
> *sudo cp /home/poornachandra/cuda/lib64/libcudnn*
> /usr/local/cuda-7.5/lib64/*
>
>
> On Fri, May 27, 2016 at 1:22 PM, Ramana Subramanyam  > wrote:
>
>> Hello there,
>> If anyone is running theano on OSX, you need to make  a few changes
>> to Poornachandra's answer. Instead of setting the path of 
>> *LD_LIBRARY_PATH,
>> *set *DYLD_LIBRARY_PATH *to the extracted directory of CUDNN. I have
>> extracted them into `/home/packages/` and I am using CUDA 7.5. Copy those
>> files to the following destination,
>>
>>
>> 1) sudo cp ~/packages/cuda/lib/libcudnn* /Developer/NVIDIA/CUDA-7.5/
>> lib/
>>
>>
>>
>> 2) sudo cp ~/packages/cuda/include/cudnn.h /Developer/NVIDIA/CUDA-7.5
>> /include/
>>
>>
>>
>> Hope this helps!
>>
>> On Tuesday, March 15, 2016 at 7:52:50 AM UTC+5:30, Ashutosh Modi
>> wrote:
>>>
>>> Hi,
>>>
>>> I am running the latest version of Theano 0.8 on linux server using
>>> the GPUs. When I run my code I get the following message :
>>>
>>> Using gpu device 0: GeForce GTX TITAN X (CNMeM is disabled, CuDNN
>>> not available)
>>>
>>>
>>> Is it normal or something is lacking or is it some bug? I ran my
>>> code after clearing the .thano directory.
>>>
>>>
>>> Please help me in resolving this.
>>>
>>>
>>> Thanks,
>>>
>>> Ashutosh
>>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "theano-users" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to theano-users...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> warm regards,
> Poornachandra Sandur,
> Mob.No.+91-9901913049
>
 --
>>>
>>> ---
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "theano-users" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>> pic/theano-users/4qKbh5C_9e4/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> theano-users...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the

Re: [theano-users] Re: CNMeM is disabled, CuDNN not available

[anishi gupta]

Hello Sir,

When I used command "device=gpuN python file.py" then code is successfullt
running on gpu.As you said theano FLAGS will override .rc file settings so
I think there is no need of making .theanorc file. Kindly tell me how to
make cuda memory faster if I dont use cnmem...I want to gain speedup in gpu
as compare to cpu



On Fri, Mar 3, 2017 at 7:08 PM, Ramana Subramanyam 
wrote:

> Hi,
> You don't set the device parameter. Set your device to gpuN and try
> running again. cnmem is from the old backend, which will be depreciated
> soon. You should start using device=cudaN and config.gpuarray.preallocate 
> instead
> of lib.cnmem. Also, you seem to override the config file with THEANO_FLAGS.
> Use either of them (remember FLAGS can override .rc file settings).
>
> Ramana
>
> On Friday, March 3, 2017 at 7:06:24 PM UTC+5:30, Ani wrote:
>>
>> No sir I have put
>> [lib]
>> cnmem=1
>> but it shows error of lib command not found..what to do sir now.
>> On 3 Mar 2017 5:07 p.m., "Kiuhnm Mnhuik"  wrote:
>>
>> You probably forgot [lib]:
>>
>> [lib]
>> cnmem = 1
>>
>> Hello Sir...
>>>
>>> I have successfully installed CuDNN and it is also available...but
>>> problem is with cnmem..When I wrote cnmem=1 in my .theanorc file which is
>>> at home directory..it gives error on running any code with cpu or
>>> gpu..please have a look at error(screenshot is attached).
>>>
>>> Thanks in advance
>>> Regards
>>> Anishi Gupta
>>>
>>> On Saturday, 28 May 2016 11:05:29 UTC+5:30, Poornachandra Sandur wrote:

 Hi,
CNMem is  fast CUDA memory allocator ... if you want
 to set it you can do it by writing the below lines in your .theanorc file
  ...

 [lib]
 cnmem=1


 ---

 For CUDNN you can do it with the following steps :

 *First download the file : cudnn-7.0-linux-x64-v3.0-prod.tgz*

 *Extract it to home directory*

 *and set the LD_LIBRARY_PATH to the above extracted directory*

 *and then follow the below steps :*



 *sudo cp /home/poornachandra/cuda/include/cudnn.h
 /usr/local/cuda-7.5/include/*

 *sudo cp /home/poornachandra/cuda/lib64/libcudnn*
 /usr/local/cuda-7.5/lib64/*


 On Fri, May 27, 2016 at 1:22 PM, Ramana Subramanyam 
 wrote:

> Hello there,
> If anyone is running theano on OSX, you need to make  a few changes to
> Poornachandra's answer. Instead of setting the path of *LD_LIBRARY_PATH,
> *set *DYLD_LIBRARY_PATH *to the extracted directory of CUDNN. I have
> extracted them into `/home/packages/` and I am using CUDA 7.5. Copy those
> files to the following destination,
>
>
> 1) sudo cp ~/packages/cuda/lib/libcudnn* /Developer/NVIDIA/CUDA-7.5/
> lib/
>
>
>
> 2) sudo cp ~/packages/cuda/include/cudnn.h /Developer/NVIDIA/CUDA-7.5/
> include/
>
>
>
> Hope this helps!
>
> On Tuesday, March 15, 2016 at 7:52:50 AM UTC+5:30, Ashutosh Modi wrote:
>>
>> Hi,
>>
>> I am running the latest version of Theano 0.8 on linux server using
>> the GPUs. When I run my code I get the following message :
>>
>> Using gpu device 0: GeForce GTX TITAN X (CNMeM is disabled, CuDNN not
>> available)
>>
>>
>> Is it normal or something is lacking or is it some bug? I ran my code
>> after clearing the .thano directory.
>>
>>
>> Please help me in resolving this.
>>
>>
>> Thanks,
>>
>> Ashutosh
>>
> --
>
> ---
> You received this message because you are subscribed to the Google
> Groups "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to theano-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



 --
 warm regards,
 Poornachandra Sandur,
 Mob.No.+91-9901913049

>>> --
>>
>> ---
>> You received this message because you are subscribed to a topic in the
>> Google Groups "theano-users" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/theano-users/4qKbh5C_9e4/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> theano-users...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "theano-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/theano-users/4qKbh5C_9e4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 

With Best Wishes and Regards
Anishi Gupta

-- 

--- 
You received this 

[theano-users] Re: Gradients are always 0 for custom loss function

[taromakino]

Thanks Jesse, so are there operations that are "safe" to use and others 
that aren't? Where can I find this information? Also, I've used T.eq before 
in another custom loss function which works correctly and doesn't return 0 
gradients, but my use case there is in computing array indices, such as the 
way I'm using it in this line:

pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
preprocess.up_index])

Is T.eq ok to use in some contexts and not others?

On Sunday, March 5, 2017 at 9:14:20 PM UTC-6, Jesse Livezey wrote:
>
> The gradient of T.eq will be zero (almost) everywhere and you're using it 
> to compute num_win and num_lose.
>
> On Sunday, March 5, 2017 at 2:42:14 PM UTC-8, 
> tarom...@alum.northwestern.edu wrote:
>>
>> Also, the return values of this loss function are small compared to 
>> cross-entropy, some sample values after random initialization were around 
>> +/- 0.01. There is a LSTM layer and the input sequences are thousands of 
>> elements long, so I suspected vanishing gradients. However, I'm printing 
>> out the min, max, and mean of the gradients w.r.t each parameter, and they 
>> are all exactly equal to 0, which seems to indicate a different problem.
>>
>> On Sunday, March 5, 2017 at 3:59:42 PM UTC-6, 
>> tarom...@alum.northwestern.edu wrote:
>>>
>>> I have defined a custom loss function, and despite the loss function 
>>> returning correct values given the inputs, the gradients are all always 0 
>>> w.r.t each of my parameters. I am not suppressing any theano errors 
>>> including the disconnected input error, so I can't explain what is causing 
>>> this. I have copied the loss function below; in words, I first convert a 3 
>>> class softmax output into a one hot representation, then I compare a subset 
>>> of it to the response and compute a quantity of interest. More generally, I 
>>> was under the impression that if one could express a function using theano 
>>> ops, it could be used as a loss function. Is this not the case?
>>>
>>> def calc_one_hot_loss(pred, y, mask):
>>> mask_flat = T.flatten(mask)
>>> length = T.sum(mask_flat, dtype='int32')
>>> pred_unmasked = pred[mask_flat.nonzero()]
>>> max_indices = T.argmax(pred_unmasked, axis=1)
>>> pred_zero = T.set_subtensor(pred_unmasked[:], 0)
>>> pred_one_hot = T.set_subtensor(pred_zero[T.arange(length), 
>>> max_indices], 1)
>>> y_unmasked = y[mask_flat.nonzero()]
>>> unchanged_col = pred_one_hot[:, preprocess.unchanged_index]
>>> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>> preprocess.up_index])
>>> pred_down = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>>> preprocess.down_index])
>>> y_up = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>>> preprocess.up_index])
>>> y_down = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>>> preprocess.down_index])
>>> diff_up = T.abs_(pred_up - y_up)
>>> diff_down = T.abs_(pred_down - y_down)
>>> diff_sum = diff_up + diff_down
>>> num_win = T.sum(T.eq(diff_sum, 0))
>>> num_lose = T.sum(T.eq(diff_sum, 2))
>>> loss = -1 * (num_win - num_lose) / length
>>> return loss
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Gradients are always 0 for custom loss function

[Jesse Livezey]

The gradient of T.eq will be zero (almost) everywhere and you're using it 
to compute num_win and num_lose.

On Sunday, March 5, 2017 at 2:42:14 PM UTC-8, 
tarom...@alum.northwestern.edu wrote:
>
> Also, the return values of this loss function are small compared to 
> cross-entropy, some sample values after random initialization were around 
> +/- 0.01. There is a LSTM layer and the input sequences are thousands of 
> elements long, so I suspected vanishing gradients. However, I'm printing 
> out the min, max, and mean of the gradients w.r.t each parameter, and they 
> are all exactly equal to 0, which seems to indicate a different problem.
>
> On Sunday, March 5, 2017 at 3:59:42 PM UTC-6, 
> tarom...@alum.northwestern.edu wrote:
>>
>> I have defined a custom loss function, and despite the loss function 
>> returning correct values given the inputs, the gradients are all always 0 
>> w.r.t each of my parameters. I am not suppressing any theano errors 
>> including the disconnected input error, so I can't explain what is causing 
>> this. I have copied the loss function below; in words, I first convert a 3 
>> class softmax output into a one hot representation, then I compare a subset 
>> of it to the response and compute a quantity of interest. More generally, I 
>> was under the impression that if one could express a function using theano 
>> ops, it could be used as a loss function. Is this not the case?
>>
>> def calc_one_hot_loss(pred, y, mask):
>> mask_flat = T.flatten(mask)
>> length = T.sum(mask_flat, dtype='int32')
>> pred_unmasked = pred[mask_flat.nonzero()]
>> max_indices = T.argmax(pred_unmasked, axis=1)
>> pred_zero = T.set_subtensor(pred_unmasked[:], 0)
>> pred_one_hot = T.set_subtensor(pred_zero[T.arange(length), max_indices], 
>> 1)
>> y_unmasked = y[mask_flat.nonzero()]
>> unchanged_col = pred_one_hot[:, preprocess.unchanged_index]
>> pred_up = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.up_index])
>> pred_down = T.flatten(pred_one_hot[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.down_index])
>> y_up = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.up_index])
>> y_down = T.flatten(y_unmasked[T.eq(unchanged_col, 0).nonzero(), 
>> preprocess.down_index])
>> diff_up = T.abs_(pred_up - y_up)
>> diff_down = T.abs_(pred_down - y_down)
>> diff_sum = diff_up + diff_down
>> num_win = T.sum(T.eq(diff_sum, 0))
>> num_lose = T.sum(T.eq(diff_sum, 2))
>> loss = -1 * (num_win - num_lose) / length
>> return loss
>>
>>
>>
>>
>>
>>
>>
>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: how do I force a elemwise op included within tensor elemwise loop

[Adam Becker]

Nevermind, I figured it out. I had to disable constant folding and wrap 
with tensor.elemwise.

On Sunday, March 5, 2017 at 10:28:39 AM UTC+8, Adam Becker wrote:
>
> Hi,
>
> I'm writing a elemwise Op with special purpose, it's c_code should be 
> different when it's working with tensor objects. (Actually, it's for 
> gh:#5471 )
>
> Current I'm only working with the CPU version.
>
> Here's the approach I'm taking so far, I modified this line in 
> tensor/elemwise.py 
>  
> into:
>
> defines = "#define TENSOR_ELEMWISE"
> undefs = "#undef TENSOR_ELEMWISE"
>
> The elemwise Op looks like:
>
> class MyOp(scalar.ScalarOp):
> nin = 0
> # < ... >
> def c_code(self, node, name, inputs, outputs, sub):
> out, = outputs
> op_name = type(self).__name__
> code = '''
> #ifndef TENSOR_ELEMWISE
> %(out)s = 1227;
> #else
> %(out)s = 1337;
> #endif
> ''' % locals()
> return code
>
> However, even though I build a graph with matrix input, the elemwise Op 
> still takes the "1227" branch.
> Further investigation of generated code shows MyOp is compiled into a 
> separate mod.cpp than the main elemwise loop.
>
> I suppose this could be due to a few optimizations. However, as I'm still 
> unfamiliar with opt code base, I couldn't figure out where.
>
> Any clue how to force the c_code of MyOp into the elemwise loop, so that 
> the "1337" branch will be taken when dealing with tensor?
>
> Thanks.
>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.