Hi, 
I made this test:

- floatX=float16, device=cuda;

- updated Theano to the dev version;

- used theano.tensor.nnet.conv3d

 out = theano.tensor.nnet.conv3d(
                        input=input,  
                        filters=self.W, 
                        input_shape = None,
                        filter_shape= None,
                        border_mode = 'valid',
                        subsample=(1,1,1),
                        filter_flip = True,
                        filter_dilation=(1,1,1)
                        ) 

- profile=True,profile_memory=True;

- edited theano/gof/link.py and added "import pdb; pdb.set_trace()" at the 
beginning of the definition of "raise_with_op" ; 

This is the output:

[luca@cuda core]$ THEANO_FLAGS=profile=True,profile_memory=True python 
run_multi_conv.py
Mapped name None to device cuda: Tesla K40c
/home/luca/data/Theano-master/theano/gof/vm.py:859: UserWarning: CVM does 
not support memory profile, using Stack VM.
  'CVM does not support memory profile, using Stack VM.')
Using cuDNN version 5103 on context None
Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16
Disabling C code for Elemwise{Cast{float32}} due to unsupported float16
Disabling C code for MaxAndArgmax due to unsupported float16


start time:
17/10/2016
14:23:03


Images for training: 574
Images for validation: 102
Epochs: 1000


Image_dim_1: 90
Image_dim_2: 90
Image_dim_3: 90


... training neural network 57


training @ iter =  0
> /home/luca/data/Theano-master/theano/gof/link.py(123)raise_with_op()
-> if exc_info is None:
(Pdb) u
> /home/luca/data/Theano-master/theano/gof/vm.py(514)__call__()
-> storage_map=storage_map)
(Pdb) u
> 
/home/luca/data/Theano-master/theano/compile/function_module.py(866)__call__()
-> self.fn() if output_subset is None else\
(Pdb) u
> 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py(333)__init__()
-> training_cost_ij=train_model(a, b)
(Pdb) u
> 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py(33)run_experiments()
-> Zoom = 0.0
(Pdb) d
> 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py(333)__init__()
-> training_cost_ij=train_model(a, b)
(Pdb) d
> 
/home/luca/data/Theano-master/theano/compile/function_module.py(866)__call__()
-> self.fn() if output_subset is None else\
(Pdb) d
> /home/luca/data/Theano-master/theano/gof/vm.py(514)__call__()
-> storage_map=storage_map)
(Pdb) d
> /home/luca/data/Theano-master/theano/gof/link.py(123)raise_with_op()
-> if exc_info is None:
(Pdb) d
*** Newest frame
(Pdb) c
Traceback (most recent call last):
  File "run_multi_conv.py", line 41, in <module>
    run_experiments()
  File "run_multi_conv.py", line 33, in run_experiments
    Zoom = 0.0
  File 
"/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py",
 
line 333, in __init__
    training_cost_ij=train_model(a, b) 
  File "/home/luca/data/Theano-master/theano/compile/function_module.py", 
line 866, in __call__
    self.fn() if output_subset is None else\
  File "/home/luca/data/Theano-master/theano/gof/vm.py", line 514, in 
__call__
    storage_map=storage_map)
  File "/home/luca/data/Theano-master/theano/gof/link.py", line 123, in 
raise_with_op
    if exc_info is None:
  File "/home/luca/data/Theano-master/theano/gof/vm.py", line 483, in 
__call__
    _, dt = self.run_thunk_of_node(current_apply)
  File "/home/luca/data/Theano-master/theano/gof/vm.py", line 390, in 
run_thunk_of_node
    rval = self.thunks[idx]()
  File "/home/luca/data/Theano-master/theano/gof/op.py", line 829, in rval
    fill_storage()
  File "/home/luca/data/Theano-master/theano/gof/cc.py", line 1701, in 
__call__
    reraise(exc_type, exc_value, exc_trace)
  File "<string>", line 2, in reraise
RuntimeError: error selecting convolution algo: CUDNN_STATUS_INTERNAL_ERROR
Apply node that caused the error: GpuDnnConvGradW{algo='time_once', 
inplace=True}(GpuContiguous.0, GpuContiguous.0, 
GpuAllocEmpty{dtype='float16', context_name=None}.0, 
GpuDnnConvDesc{border_mode='valid', subsample=(1, 1, 1), conv_mode='conv', 
precision='float32'}.0, Constant{1.0}, Constant{0.0})
Toposort index: 186
Inputs types: [GpuArrayType<None>(float16, (True, True, False, False, 
False)), GpuArrayType<None>(float16, (True, False, False, False, False)), 
GpuArrayType<None>(float16, (False, True, False, False, False)), 
<theano.gof.type.CDataType object at 0x7f63c9543410>, Scalar(float32), 
Scalar(float32)]
Inputs shapes: [(1, 1, 90, 90, 90), (1, 150, 86, 86, 86), (150, 1, 5, 5, 
5), 'No shapes', (), ()]
Inputs strides: [(1458000, 1458000, 16200, 180, 2), (190816800, 1272112, 
14792, 172, 2), (250, 250, 50, 10, 2), 'No strides', (), ()]
Inputs values: ['not shown', 'not shown', 'not shown', <capsule object NULL 
at 0x7f63bdce4f30>, 1.0, 0.0]
Outputs clients: [[Rebroadcast{?,0}(GpuDnnConvGradW{algo='time_once', 
inplace=True}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a 
back-trace of when this node was created. This can be done with by setting 
the Theano flag 'optimizer=fast_compile'. If that does not work, Theano 
optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and 
storage map footprint of this apply node.
Function profiling
==================
  Message: 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py:280
  Time in 0 calls to Function.__call__: 0.000000e+00s
  Total compile time: 3.621856e+01s
    Number of Apply nodes: 189
    Theano Optimizer time: 1.217813e+00s
       Theano validate time: 5.148840e-02s
    Theano Linker time (includes C, CUDA code generation/compiling): 
3.498101e+01s
       Import time 1.756597e-02s
       Node make_thunk time 3.497302e+01s

Time in all call to theano.grad() 3.205299e-02s
Time since theano import 119.636s
Here are tips to potentially make your code run faster
                 (if you think of new ones, suggest them on the mailing 
list).
                 Test them first, as they are not guaranteed to always 
provide a speedup.
  Sorry, no tip for today.
Function profiling
==================
  Message: 
/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py:281
  Time in 0 calls to Function.__call__: 0.000000e+00s
  Total compile time: 5.112607e+00s
    Number of Apply nodes: 97
    Theano Optimizer time: 5.799150e-01s
       Theano validate time: 1.637220e-02s
    Theano Linker time (includes C, CUDA code generation/compiling): 
4.521184e+00s
       Import time 2.564192e-03s
       Node make_thunk time 4.516305e+00s

Time in all call to theano.grad() 3.205299e-02s
Time since theano import 119.636s
Here are tips to potentially make your code run faster
                 (if you think of new ones, suggest them on the mailing 
list).
                 Test them first, as they are not guaranteed to always 
provide a speedup.
  Sorry, no tip for today.
Function profiling
==================
  Message: Sum of all(2) printed profiles at exit excluding Scan op profile.
  Time in 0 calls to Function.__call__: 0.000000e+00s
  Total compile time: 4.133116e+01s
    Number of Apply nodes: 189
    Theano Optimizer time: 1.797728e+00s
       Theano validate time: 6.786060e-02s
    Theano Linker time (includes C, CUDA code generation/compiling): 
3.950219e+01s
       Import time 2.013016e-02s
       Node make_thunk time 3.948932e+01s

Time in all call to theano.grad() 3.205299e-02s
Time since theano import 119.636s
Here are tips to potentially make your code run faster
                 (if you think of new ones, suggest them on the mailing 
list).
                 Test them first, as they are not guaranteed to always 
provide a speedup.
  Sorry, no tip for today.




-- 

--- 
You received this message because you are subscribed to the Google Groups 
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to