Re: [theano-users] 2nd (and 3rd) highest value of a 2D tensor

[Frédéric Bastien]

Check this issue:

https://github.com/Theano/Theano/issues/5608

Using scan for that is a bad idea, it would be very slow. There isn't a
very good solution right now, but in this issue, it indicate some way to do
that.

Fred

On Thu, May 4, 2017 at 5:58 AM Andre Holzner 
wrote:

> Good morning,
>
> I want to get the 2nd (and possibly 3rd) highest value in a 2D tensor
> (output of a convolutional layer).
>
> What would be the simplest way of implementing this in Theano ?
>
> I saw that there is support for iterating (
> http://deeplearning.net/software/theano/library/scan.html )
> and IfElse operations (
> http://deeplearning.net/software/theano/tutorial/conditions.html), that
> in principle should allow me to implement this ? Will this also generate
> CUDA code when I run
> with the GPU backend or will the operation be performed on the CPU ?
>
> thanks a lot,
>
> Andre
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Graph optimization and memory usage

[Dustin Webb]

Hi all:

I'm having trouble with Theano graph optimizations and memory usage. I have
a very small network (~8 parameters) that I stack into a larger network
(yielding ~800 parameters). However the parameters are shared so it's not
really increasing the number of parameters that much. Compiling the network
seems to take an exorbitant amount of memory. I'm using Keras as a simple
way to construct it so it's conceivable that the problem isn't with Theano.
However a) the problem appears to be when theano.function is called and b)
when I use FAST_COMPILE it takes almost no time and almost no time and very
little memory to compile (~1 sec and 100MB vs. well over 32GB of memory and
it never finishes). As such I think the problem is with the optimizations
and I'm wondering if there is an easy way to identify which optimization(s)
might be causing the the problem.

Thanks,
-- 
Dustin Webb 


-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [theano-users] Re: Different processes on different gpus

[Frédéric Bastien]

The multi-gpu getting used by one process was fixed. I forgot if the fix is
in THeano 0.9 or the development version.

I would recommand to use the new gpu back-end:

https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

You can use theano.gpuarray.use('cuda0') similar to the old back-end.

You can also use this linux shell trick:

THEANO_FLAGS=device=cuda0 python ...

this will create that flag only for the new created process.

On Thu, May 4, 2017 at 5:05 AM Richard Hankins 
wrote:

> Hi,
>
> Sorry for the late reply. Haven't looked at this in a while. But just
> checked my set up and i've got device = cpu in .theanorc. But i'm not
> forcing the device.
> Checking nvidia-smi both my GPU's are in default compute mode. To select
> different devices I use the following:
>
> import theano.sandbox.cuda
> theano.sandbox.cuda.use("gpuX")
>
> Hope this helps,
>
> Richard
>
> On Thu, Apr 27, 2017 at 8:33 PM, anurag kumar  wrote:
>
>> Is there a final solution to this problem ? I am having similar problem.
>>
>> Best,
>> Anurag
>>
>> On Sunday, May 1, 2016 at 9:24:59 AM UTC-4, RHankins wrote:
>>>
>>> Sorry I meant "One on gpu0 and one on gpu1 (It begins by running a
>>> process of gpu1 then starts another on gpu0)".
>>>
>>> On Sunday, May 1, 2016 at 2:19:58 PM UTC+1, RHankins wrote:

 Right. I know why it was throwing an error when i had device=cpu
 because i also had force_device=True. Setting device=cpu and not setting
 force_device allows me to select different gpus using
 theano.sandbox.cuda.use().

 But i'm still having the problem with it running on mulitple gpus. If I
 select gpu0 (Titan X) it runs a single process on the correct gpu. If I
 select gpu1 (GTX 980) to run exactly the same code it runs 2 processes. One
 on gpu0 and one on gpu1 (It begins by running a process of gpu0 then starts
 another on gpu1). It doesn't matter if they are run simultaneously or not.
 Or if when they are run simultaneously, something was already running on
 gpu0 or gpu1. Should I use nvidia-smi to force the code to run on a single
 gpu using

 nvidia-smi  −−compute−mode=EXCLUSIVE_PROCESS?


 My only concern is if I want to run different programs at the same time
 I will end up having mutiple processes running on the same gpu. So will
 they interfer with each other if they are importing the same modules? Is it
 okay to run multiple processes on the same gpu? Will it effect the results?
 Or does it not matter?

 Cheers,

 R

 On Friday, April 29, 2016 at 9:55:33 PM UTC+1, nouiz wrote:
>
> You can ignore those 2 errrors. It is just that those test seem too
> sensitive.
>
> If you set a device in your theanorc file that isn't 'cpu' and call
> use() on another one, it is normal that Theano don't like this, as only 1
> GPU is supported in the current back-end. The new one support multiple 
> GPU.
>
> Does it work if you try to use gpu0? Does something was already
> running on gpu1?
>
> Fred
>
> On Fri, Apr 29, 2016 at 11:52 AM, RHankins 
> wrote:
>
>> Hi Fred,
>>
>> Thanks for your response. I'm using cuda back-end so I didn't install
>> libgpuarray. Or am I supposed to install libgpuarray as well? When I say
>> tests, I just mean testing some new code out not running theano tests (as
>> in nose tests).
>>
>> I'm using Lasagne as I saw that you suggested to someone else to use
>>
>> import theano.sandbox.cuda
>>
>> theano.sandbox.cuda.use("gpu1")
>>
>>
>> If in .theanorc device = gpu0 I get the following message
>>
>> WARNING (theano.sandbox.cuda): Ignoring call to use(1), GPU number 0
>> is already in use.
>>
>>
>> If in .theanorc device = cpu I get the following message
>>
>> WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu1 is
>> not available (error: cuda unavailable)
>>
>>
>> I updated Theano and Lasagne to the lastest versions - (0.9.0dev0)
>> and (0.2.dev1) respectively but I've still got the same problem. But in
>> addition to this now when I run theano.test() it won't pass. I get the
>> following errors
>>
>>
>> ==
>> ERROR: test_grad (theano.tensor.tests.test_basic.ArctanhInplaceTester)
>> --
>> Traceback (most recent call last):
>> File
>> "/usr/local/lib/python2.7/dist-packages/theano/tensor/tests/test_basic.py",
>> line 483, in test_grad
>> eps=_grad_eps)
>> File
>> "/usr/local/lib/python2.7/dist-packages/theano/tests/unittest_tools.py",
>> line 91, in verify_grad
>> T.verify_grad(op, pt, n_tests, rng, *args, 

[theano-users] Re: Is there a way to speed this operation in theano

[Jesse Livezey]

I think tensordot should do what you want
http://deeplearning.net/software/theano/library/tensor/basic.html#theano.tensor.tensordot
something like
result = T.tensordot(prob, cases, axes=1)



On Friday, May 5, 2017 at 3:17:14 AM UTC-7, Šarūnas S. wrote:
>
> I was shown that in *numpy* I could speed it up in the following way:
>
> result = np.einsum('ijk,ijk->ik', prob, cases)[:,None,:]
>
>
> result = np.matmul(prob.transpose(2,0,1), cases.T).T
>
>
> Bot give me the expected speedup in *numpy*, but neither is implemented 
> in *Theano*. Is there a way to do the same in *Theano* on the *GPU*?
>
>
>
> On Friday, 5 May 2017 11:15:26 UTC+3, Šarūnas S. wrote:
>>
>> In my current theano script the bottleneck is equivalent to the following 
>> numpy code:
>>
>> import numpy as np
>>
>> # 3D example
>> axis = 0
>> prob = np.random.random( ( 1, 1000, 50 ) )
>> cases = np.random.random( ( 1000, 1000, 50 ) )
>>
>> start = time.time(  )
>> for i in xrange( 1000 ):
>> result = ( cases * prob ).sum( axis=1-axis, keepdims=True )
>> print '3D naive method took {} seconds'.format( time.time() - start )
>> print result.shape
>> print
>>
>> I had seen in 2D case that replacing elementwise+sum with a dot product 
>> gave me 5x speedup. Are there any theano matrix operations that could help 
>> me out here? 
>>
>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Re: Variable number of arguments in scan

[Gilles Degottex]

Sorry, slight confusion, by "chose the order arbitrarily" I didn't mean on 
the fly. I meant be able to selection what ever order, but once for all 
once running.
I'm not actually changing anything once compiled.



On Friday, 5 May 2017 02:56:03 UTC+1, Adam Becker wrote:
>
> I don't think this works. The inner function of scan will be converted to 
> a graph, then getting compiled inside ScanOp. If you change nonlocal 
> variable "order" on the fly, the change won't be reflected on the compiled 
> function.
>
> If the inner loop itself can be written as scan, you can just make a 
> nested scan instead. Compile the innermost graph with scan first (by hand), 
> then pass it as fn to the outer scan.
>
> On Wednesday, May 3, 2017 at 8:04:46 PM UTC+8, Gilles Degottex wrote:
>>
>> Hi,
>>
>> I'm new on this list, so I hope I'm not duplicating a subject that has 
>> already been covered and answered.
>>
>> I'm trying to implement High-Order RNN (https://arxiv.org/abs/1605.00064) 
>> and I would like to let me chose the order arbitrarily, which changes the 
>> number parameters to pass to the inner scan function.
>> For this purpose, I'm doing something like:
>>
>> def step(in_t, *args):
>>
>> W_xi_ = args[-2]
>> b_i_ = args[-1]
>> args = args[:-2]
>>
>> h_t = T.dot(in_t, W_xi_)
>>
>> for p in xrange(order):
>> h_t += T.dot(args[p], args[order+p])
>>
>> h_t = nonlinearity(h_t + b_i_)
>>
>> return h_t
>>
>> h, _ = theano.scan(step, sequences=[invalues],
>> outputs_info=[dict(initial=hid_init, taps=range(-order,0))],
>> non_sequences=W_hip+[W_xi, b_i], strict=True)
>>
>> W_hip being a list of shared matrices, one for each tap.
>>
>> Basically, it compiles, I can train such a model, so it looks like it 
>> works.
>>
>> However, I've strictly no clue if the code is doing what I'm expecting it 
>> to do.
>> Is is okay to use a for loop in th inner scan function?
>> Does scan behave nicely with a variable number of argument?
>>
>> Any tips to help verifying/checking that a code is doing what it is 
>> supposed to do is also very welcome.
>>
>> Cheers,
>>
>>
>>

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[theano-users] Is there a way to speed this operation in theano

[Šarūnas S .]

In my current theano script the bottleneck is equivalent to the following 
numpy code:

import numpy as np

# 3D example
axis = 0
prob = np.random.random( ( 1, 1000, 50 ) )
cases = np.random.random( ( 1000, 1000, 50 ) )

start = time.time(  )
for i in xrange( 1000 ):
result = ( cases * prob ).sum( axis=1-axis, keepdims=True )
print '3D naive method took {} seconds'.format( time.time() - start )
print result.shape
print

I had seen in 2D case that replacing elementwise+sum with a dot product 
gave me 5x speedup. Are there any theano matrix operations that could help 
me out here? 

-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.