Hello,
I'm trying to parallelize an existing code (which runs fine with
THEANO_FLAGS=devices=cuda0,..., and without theano.shared(...,
target='dev0') or variable.transfer('dev0') like statements) to support
multi GPU, and I'm having some problems with code containing
THEANO_FLAGS=contexts=dev0->cuda0 and theano.scan.
I'm using Theano 0.9.0.dev4 and libgpuarry & pygpu from Github commit
9efa49341620c2d81857db5551e49a43806a00c9
.
Here is a very simplified example code of my problem:
import os
#os.environ['THEANO_FLAGS'] =
'contexts=dev0->cuda0,floatX=float32,exception_verbosity=high'
import numpy as np
import theano as th
import theano.tensor as tt
x_ti=tt.matrix()#.transfer('dev0')
y = th.shared(np.array([0., 0., 0.]).astype('float32'))#, target='dev0')
def step(x_i, prev_h_i):
return x_i + prev_h_i + y
ret, upd = th.scan(step, sequences=[x_ti],
outputs_info=[tt.zeros_like(x_ti[0])],
#.transfer('dev0')],
n_steps=x_ti.shape[0])
a=np.ones((5,3)).astype('float32')
a[:,1] *= 2
a[:,2] *= 3
inputs = {x_ti:a}
print 'a'
print a
print 'h'
print ret.eval(inputs)
When this code is run without the commented out parts and with
THEANO_FLAGS="device=cuda0", it produces expected results. When the
contexts flag and the target='dev0' etc. are used, however, I get the
following result:
Using cuDNN version 5105 on context dev0
Mapped name dev0 to device cuda0: GeForce GTX TITAN X (0000:02:00.0)
a
[[ 1. 2. 3.]
[ 1. 2. 3.]
[ 1. 2. 3.]
[ 1. 2. 3.]
[ 1. 2. 3.]]
h
ERROR (theano.gof.opt): SeqOptimizer apply <theano.gpuarray.opt.GraphToGPU
object at 0x7fb05da70c10>
ERROR (theano.gof.opt): Traceback:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/yuno/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "/home/yuno/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 87, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/opt.py",
line 388, in apply
outputs = new_ops(*[mapping[i] for i in node.inputs], return_list=True)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/tensor/basic.py",
line 2935, in __call__
ret = super(Alloc, self).__call__(val, *shapes, **kwargs)
File "/home/yuno/.local/lib/python2.7/site-packages/theano/gof/op.py",
line 604, in __call__
node = self.make_node(*inputs, **kwargs)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/basic_ops.py",
line 825, in make_node
value = as_gpuarray_variable(value, context_name=self.context_name)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/basic_ops.py",
line 74, in as_gpuarray_variable
return gpu_from_host(context_name)(x)
File "/home/yuno/.local/lib/python2.7/site-packages/theano/gof/op.py",
line 604, in __call__
node = self.make_node(*inputs, **kwargs)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/basic_ops.py",
line 651, in make_node
dtype=x.dtype)()])
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/type.py",
line 170, in __init__
get_context(self.context_name)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/type.py",
line 88, in get_context
raise ContextNotDefined("context name %s not defined" % (name,))
ContextNotDefined: context name None not defined
[[ 1. 2. 3.]
[ 2. 4. 6.]
[ 3. 6. 9.]
[ 4. 8. 12.]
[ 5. 10. 15.]]
That is, some exception is raised due to a node with context name None
(which I can't really identify in my code), but it still runs. Same thing
happens with my real project code, where several dozens of these errors pop
up but it still seems to work somehow and trains my RNN.
Because using device=cuda0 flag defines a mapping for None, I also tried
using THEANO_FLAGS=device=cuda0,contexts=dev0->cuda0 .
This produces the following:
Using cuDNN version 5105 on context None
Mapped name None to device cuda0: GeForce GTX TITAN X (0000:02:00.0)
Mapped name dev0 to device cuda0: GeForce GTX TITAN X (0000:02:00.0)
Traceback (most recent call last):
File "scantest.py", line 10, in <module>
y = th.shared(np.array([0., 0., 0.]).astype('float32'), target='dev0')
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/compile/sharedvalue.py",
line 247, in shared
allow_downcast=allow_downcast, **kwargs)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/type.py",
line 613, in gpuarray_shared_constructor
get_context(target)
File
"/home/yuno/.local/lib/python2.7/site-packages/theano/gpuarray/type.py",
line 88, in get_context
raise ContextNotDefined("context name %s not defined" % (name,))
theano.gpuarray.type.ContextNotDefined: context name dev0 not defined
which is even stranger since we surely have context name dev0 defined...
Anyone know what's going on here? Am I using theano.scan wrong within this
context?
Thanks,
Hosang
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
