Hi,
I am having some issues getting theano to run correctly on GPUs using
cuDNN. The same code runs and compiles without any errors when running it
on a CPU node or a GPU node without cuDNN enabled.
I am a non-root user working on a cluster machine. The cluster has a number
of GPU nodes (Tesla K20m, k40m...). The system is running Scientific Linux
release 6.6 (Carbon).
The system admin has installed CUDA 5.5 and CUDA 6.5, using cuDNN with
these versions does not work as theano expects a later version of cuDNN
that is not compatible with 6.5.
I have locally installed the bleeding edge version of Theano. I have also
tried using the current release version and some archived version with no
variation in the errors reported.
I have locally installed CUDA 7.5 as well as cudNN V5.
When running the theano code theano recognises that cuDNN 5005 is
installed, but I later get crashes in the code (that don't occur when
running the code on a CPU or GPU with cuDNN disabled).
I was wondering if there was any additional configuration that I need to do
in order to get theano to work correctly with cuDNN?
This is the crash log that occurs ONLY when cuDNN is enabled. Thanks for
your time.
Traceback (most recent call last):
File "runner.py", line 306, in <module>
main()
File "runner.py", line 241, in main
queries_exp = __import__(args.exp_model).queries_exp
File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-
convnet/exp_multi_conv_cosim.py", line 923, in <module>
queries_exp = EntityVectorLinkExp()
File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-
convnet/exp_multi_conv_cosim.py", line 51, in __init__
self._setup()
File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-
convnet/exp_multi_conv_cosim.py", line 543, in _setup
on_unused_input='ignore',
File
"/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/function.py",
line 326, in function
output_keys=output_keys)
File
"/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/pfunc.py",
line 484, in pfunc
output_keys=output_keys)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/
theano/compile/function_module.py", line 1788, in orig_function
output_keys=output_keys).create(
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/
theano/compile/function_module.py", line 1467, in __init__
optimizer_profile = optimizer(fgraph)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 102, in __call__
return self.optimize(fgraph)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 90, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 90, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 235, in apply
sub_prof = optimizer.optimize(fgraph)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 90, in optimize
ret = self.apply(fgraph, *args, **kwargs)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 2262, in apply
lopt_change = self.process_node(fgraph, node, lopt)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 1825, in process_node
lopt, node)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 1719, in warn_inplace
return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node)
File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
line 1705, in warn
raise exc
AssertionError
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
