On Mon, Oct 17, 2016, luca.wagner.0...@gmail.com wrote:
> File "<string>", line 2, in reraise
> RuntimeError: error selecting convolution algo: CUDNN_STATUS_INTERNAL_ERROR
Here you go, that is the actual error that occurs.
It is not really descriptive... Can you try to make a small graph with
only that node and see if it runs?
Arnaud, do you know if "<capsule object NULL at 0x...>" is normal?
> Apply node that caused the error: GpuDnnConvGradW{algo='time_once',
> inplace=True}(GpuContiguous.0, GpuContiguous.0,
> GpuAllocEmpty{dtype='float16', context_name=None}.0,
> GpuDnnConvDesc{border_mode='valid', subsample=(1, 1, 1), conv_mode='conv',
> precision='float32'}.0, Constant{1.0}, Constant{0.0})
> Toposort index: 186
> Inputs types: [GpuArrayType<None>(float16, (True, True, False, False,
> False)), GpuArrayType<None>(float16, (True, False, False, False, False)),
> GpuArrayType<None>(float16, (False, True, False, False, False)),
> <theano.gof.type.CDataType object at 0x7f63c9543410>, Scalar(float32),
> Scalar(float32)]
> Inputs shapes: [(1, 1, 90, 90, 90), (1, 150, 86, 86, 86), (150, 1, 5, 5,
> 5), 'No shapes', (), ()]
> Inputs strides: [(1458000, 1458000, 16200, 180, 2), (190816800, 1272112,
> 14792, 172, 2), (250, 250, 50, 10, 2), 'No strides', (), ()]
> Inputs values: ['not shown', 'not shown', 'not shown', <capsule object NULL
> at 0x7f63bdce4f30>, 1.0, 0.0]
> Outputs clients: [[Rebroadcast{?,0}(GpuDnnConvGradW{algo='time_once',
> inplace=True}.0)]]
>
> HINT: Re-running with most Theano optimization disabled could give you a
> back-trace of when this node was created. This can be done with by setting
> the Theano flag 'optimizer=fast_compile'. If that does not work, Theano
> optimizations can be disabled with 'optimizer=None'.
> HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and
> storage map footprint of this apply node.
> Function profiling
> ==================
> Message:
> /home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py:280
> Time in 0 calls to Function.__call__: 0.000000e+00s
> Total compile time: 3.621856e+01s
> Number of Apply nodes: 189
> Theano Optimizer time: 1.217813e+00s
> Theano validate time: 5.148840e-02s
> Theano Linker time (includes C, CUDA code generation/compiling):
> 3.498101e+01s
> Import time 1.756597e-02s
> Node make_thunk time 3.497302e+01s
>
> Time in all call to theano.grad() 3.205299e-02s
> Time since theano import 119.636s
> Here are tips to potentially make your code run faster
> (if you think of new ones, suggest them on the mailing
> list).
> Test them first, as they are not guaranteed to always
> provide a speedup.
> Sorry, no tip for today.
> Function profiling
> ==================
> Message:
> /home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/mpr_convnet_class.py:281
> Time in 0 calls to Function.__call__: 0.000000e+00s
> Total compile time: 5.112607e+00s
> Number of Apply nodes: 97
> Theano Optimizer time: 5.799150e-01s
> Theano validate time: 1.637220e-02s
> Theano Linker time (includes C, CUDA code generation/compiling):
> 4.521184e+00s
> Import time 2.564192e-03s
> Node make_thunk time 4.516305e+00s
>
> Time in all call to theano.grad() 3.205299e-02s
> Time since theano import 119.636s
> Here are tips to potentially make your code run faster
> (if you think of new ones, suggest them on the mailing
> list).
> Test them first, as they are not guaranteed to always
> provide a speedup.
> Sorry, no tip for today.
> Function profiling
> ==================
> Message: Sum of all(2) printed profiles at exit excluding Scan op profile.
> Time in 0 calls to Function.__call__: 0.000000e+00s
> Total compile time: 4.133116e+01s
> Number of Apply nodes: 189
> Theano Optimizer time: 1.797728e+00s
> Theano validate time: 6.786060e-02s
> Theano Linker time (includes C, CUDA code generation/compiling):
> 3.950219e+01s
> Import time 2.013016e-02s
> Node make_thunk time 3.948932e+01s
>
> Time in all call to theano.grad() 3.205299e-02s
> Time since theano import 119.636s
> Here are tips to potentially make your code run faster
> (if you think of new ones, suggest them on the mailing
> list).
> Test them first, as they are not guaranteed to always
> provide a speedup.
> Sorry, no tip for today.
>
>
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to theano-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
Pascal
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to theano-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
