Hi,
Can you try running "nosetests -v" or "theano-nose -v" in command line,
rather than from Python?
This should at least give the name of the test during which it crashes.
On 2017-11-21 07:49 PM, Ruben Dario Fonnegra Tarazona wrote:
Hi.
I'm having ploblems executing code in theano. I installed the dev
version and it runs perfectly in CPU. However, when I try to run
anything in the gpu (even LeNet with MNIST) the model doesn't even run
and it appears an only message saying "Segmentation fault. Core dumped".
Despite of that, I tried to verify theano installation using the command
THEANO_FLAGS='';python -c "import theano; theano.test()" and it did not
work (I attach the output in log file). I tried several things but I
couldn't solve the problem. I attach a file with the output after
executing the command, and my theanorc file. I work with a Kubuntu
16.04, CUDA 8 and cuDNN 6.2 in the Quadro M4000 series (8GB - the
problem is not memory allocation). I hope you could help me solve the
problem. Thanks in advice, and I will be very attentive for your answer.
What I've tried?
- Install stable, bleeding edge and dev theano version, using their
corresponding libgpuarray library (same issue)
- Manually compile OpenBLAS for installation
- Use environment variable CUDA_LAUNCH_BLOCKING set to 1
- Use several floatX types (float16, float32, float64)
- Use device=cpu and device=cuda0 flags
Note: It might be the CUDA drivers; however, I can use TensorFlow
running on GPU without any problem.
output of theano.test()
---------------------------------------------------------------------
HP-Z840-Workstation:~/Data$ THEANO_FLAGS=''; python -c "import theano;
theano.test()"
Theano version 1.0.0
theano is installed in /home/bluegum1/Theano/theano
NumPy version 1.13.3
NumPy relaxed strides checking option: True
NumPy is installed in
/home/bluegum1/.local/lib/python2.7/site-packages/numpy
Python version 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]
nose version 1.3.7
Using cuDNN version 6021 on context None
Mapped name None to device cuda: Quadro M4000 (0000:04:00.0)
..........................................ERROR (theano.gof.opt):
Optimization failure due to: insert_bad_dtype
ERROR (theano.gof.opt): node:
Elemwise{add,no_inplace}(<TensorType(float64, vector)>,
<TensorType(float64, vector)>)
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "/home/bluegum1/Theano/theano/gof/opt.py", line 2059, in
process_node
remove=remove)
File "/home/bluegum1/Theano/theano/gof/toolbox.py", line 569, in
replace_all_validate_remove
chk = fgraph.replace_all_validate(replacements, reason)
File "/home/bluegum1/Theano/theano/gof/toolbox.py", line 518, in
replace_all_validate
fgraph.replace(r, new_r, reason=reason, verbose=False)
File "/home/bluegum1/Theano/theano/gof/fg.py", line 486, in replace
". The type of the replacement must be the same.", old, new)
BadOptimization: BadOptimization Error
Variable: id 140331434464144 Elemwise{Cast{float32}}.0
Op Elemwise{Cast{float32}}(Elemwise{add,no_inplace}.0)
Value Type: <type 'NoneType'>
Old Value: None
New Value: None
Reason: insert_bad_dtype. The type of the replacement must be the same.
Old Graph:
Elemwise{add,no_inplace} [id A] <TensorType(float64, vector)> ''
|<TensorType(float64, vector)> [id B] <TensorType(float64, vector)>
|<TensorType(float64, vector)> [id C] <TensorType(float64, vector)>
New Graph:
Elemwise{Cast{float32}} [id D] <TensorType(float32, vector)> ''
|Elemwise{add,no_inplace} [id A] <TensorType(float64, vector)> ''
Hint: relax the tolerance by setting tensor.cmp_sloppy=1
or even tensor.cmp_sloppy=2 for less-strict comparison
......................................S............................./home/bluegum1/Theano/theano/compile/nanguardmode.py:150:
RuntimeWarning: All-NaN slice encountered
return np.isinf(np.nanmax(arr)) or np.isinf(np.nanmin(arr))
/home/bluegum1/Theano/theano/compile/nanguardmode.py:150:
RuntimeWarning: All-NaN axis encountered
return np.isinf(np.nanmax(arr)) or np.isinf(np.nanmin(arr))
............................................../home/bluegum1/Theano/theano/gof/vm.py:886:
UserWarning: CVM does not support memory profile, using Stack VM.
'CVM does not support memory profile, using Stack VM.')
............/home/bluegum1/Theano/theano/compile/profiling.py:283:
UserWarning: You are running the Theano profiler with CUDA enabled.
Theano GPU ops execution is asynchronous by default. So by default, the
profile is useless. You must set the environment variable
CUDA_LAUNCH_BLOCKING to 1 to tell the CUDA driver to synchronize the
execution to get a meaningful profile.
warnings.warn(msg)
....................0.0581646137166
0.0581646137166
0.0581646137166
0.0581646137166
.................................................................................................................................................................../home/bluegum1/Theano/theano/gof/vm.py:889:
UserWarning: LoopGC does not support partial evaluation, using Stack VM.
'LoopGC does not support partial evaluation, '
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................Violación
de segmento (`core' generado)
---------------------------------------------------------------------
theanorc file
---------------------------------------------------------------------
[global]
floatX = float32
#device = cuda0
#device = cpu
#optimizer=fast_run
#optimizer=fast_compile #Desabilita la GPU
#optimizer=None
[cuda]
root = /usr/local/cuda-8.0
[nvcc]
fastmath = True
[lib]
cnmem = 1.0
--
---
You received this message because you are subscribed to the Google
Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
Pascal Lamblin
--
---
You received this message because you are subscribed to the Google Groups "theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
