Hi,
the *RuntimeError: error selecting convolution algo: 
CUDNN_STATUS_INTERNAL_ERROR* is caused by the THEANO_FLAGS set in .theanorc

[dnn.conv]
algo_fwd =  time_once
algo_bwd_filter = time_once
algo_bwd_data = time_once 

If I delete these flags, the runtimeError disappears and the output is:

[luca@cuda core]$ THEANO_FLAGS=profile=True,profile_memory=True python 
run_multi_conv.py
Mapped name None to device cuda: Tesla K40c
/home/luca/data/Theano-master/theano/gof/vm.py:859: UserWarning: CVM does 
not support memory profile, using Stack VM.
  'CVM does not support memory profile, using Stack VM.')
Using cuDNN version 5103 on context None
Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16
Disabling C code for Elemwise{Cast{float32}} due to unsupported float16
Disabling C code for MaxAndArgmax due to unsupported float16

start time:
18/10/2016
11:50:25

Images for training: 574
Images for validation: 102


Image_dim_1: 90
Image_dim_2: 90
Image_dim_3: 90

... training neural network 57

training @ iter =  0
> /home/luca/data/Theano-master/theano/gof/link.py(123)raise_with_op()
-> if exc_info is None:
(Pdb) c
Traceback (most recent call last):
  File "run_multi_conv.py", line 41, in <module>
    run_experiments()
  File "run_multi_conv.py", line 33, in run_experiments
    Zoom = 0.0
  File 
"/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py",
 
line 334, in __init__
    training_cost_ij=train_model(a, b) 
  File "/home/luca/data/Theano-master/theano/compile/function_module.py", 
line 866, in __call__
    self.fn() if output_subset is None else\
  File "/home/luca/data/Theano-master/theano/gof/vm.py", line 514, in 
__call__
    storage_map=storage_map)
  File "/home/luca/data/Theano-master/theano/gof/link.py", line 123, in 
raise_with_op
    if exc_info is None:
  File "pygpu/gpuarray.pyx", line 1941, in pygpu.gpuarray.GpuArray.__repr__ 
(pygpu/gpuarray.c:24742)
  File 
"/home/luca/anaconda2/lib/python2.7/site-packages/numpy/core/numeric.py", 
line 482, in asarray
    return array(a, dtype, copy=False, order=order)
  File "pygpu/gpuarray.pyx", line 1572, in 
pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224)
  File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray 
(pygpu/gpuarray.c:17346)
  File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read 
(pygpu/gpuarray.c:6114)
pygpu.gpuarray.GpuArrayException: an illegal memory access was encountered
Function profiling
==================
  Message: 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py:281
  Time in 0 calls to Function.__call__: 0.000000e+00s
  Total compile time: 3.674333e+01s
    Number of Apply nodes: 189
    Theano Optimizer time: 1.165505e+00s
       Theano validate time: 5.227184e-02s
    Theano Linker time (includes C, CUDA code generation/compiling): 
3.555397e+01s
       Import time 1.860213e-02s
       Node make_thunk time 3.554579e+01s

Time in all call to theano.grad() 3.407788e-02s
Time since theano import 101.933s
Here are tips to potentially make your code run faster
                 (if you think of new ones, suggest them on the mailing 
list).
                 Test them first, as they are not guaranteed to always 
provide a speedup.
  Sorry, no tip for today.
Function profiling
==================
  Message: 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py:282
  Time in 0 calls to Function.__call__: 0.000000e+00s
  Total compile time: 5.410502e+00s
    Number of Apply nodes: 97
    Theano Optimizer time: 5.746739e-01s
       Theano validate time: 1.644993e-02s
    Theano Linker time (includes C, CUDA code generation/compiling): 
4.824444e+00s
       Import time 4.149914e-03s
       Node make_thunk time 4.820603e+00s

Time in all call to theano.grad() 3.407788e-02s
Time since theano import 101.934s
Here are tips to potentially make your code run faster
                 (if you think of new ones, suggest them on the mailing 
list).
                 Test them first, as they are not guaranteed to always 
provide a speedup.
  Sorry, no tip for today.
Function profiling
==================
  Message: Sum of all(2) printed profiles at exit excluding Scan op profile.
  Time in 0 calls to Function.__call__: 0.000000e+00s
  Total compile time: 4.215383e+01s
    Number of Apply nodes: 189
    Theano Optimizer time: 1.740179e+00s
       Theano validate time: 6.872177e-02s
    Theano Linker time (includes C, CUDA code generation/compiling): 
4.037841e+01s
       Import time 2.275205e-02s
       Node make_thunk time 4.036640e+01s

Time in all call to theano.grad() 3.407788e-02s
Time since theano import 101.934s
Here are tips to potentially make your code run faster
                 (if you think of new ones, suggest them on the mailing 
list).
                 Test them first, as they are not guaranteed to always 
provide a speedup.
  Sorry, no tip for today.




-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to