Hi, The issue is that cuDNN has no "sum" pooling, but Theano still tries to optimizer CPU sum-pooling by creating a cuDNN Op, which crashes at the creation.
The right solution would be to update the optimization doing that (https://github.com/Theano/Theano/issues/4929). A workaround for you now would be to explicitly use "average_inc_padding" and multiply by the window size. On Wed, Aug 31, 2016, Tom Ball wrote: > Hi, > > Sorry I missed part of the error message when I copied it over. > > Using gpu device 0: Tesla K40m (CNMeM is enabled with initial size: 95.0% > of memory, cuDNN 5005) > True > ++ which python2 > + PYTHON=/cm/shared/apps/python/2.7.5/bin/python2 > + WIKIPEDIA_DUMP=enwiki-latest-pages-articles.xml > + WIKIPEDIA_DOWNLOAD= > https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 > + WIKIPEDIA_W2V=enwiki-vecs > + python runner.py --queries /home/t/tj/tjb32/workspace/ > CNN_EL/data/data/ace-dev-set-queries.json --surface_count > /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-vecs-surface.json > --wordvecs /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-vecs.bin > --redirects > /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-vecs-redir.json > --wiki_dump > /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-latest-pages-articles.xml > --raw_output ./data/ace-dev-set.h5 --csv_output ./data/ace-dev-set.csv > Using gpu device 0: Tesla K40m (CNMeM is enabled with initial size: 95.0% > of memory, cuDNN 5005) > /home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/tensor/signal/downsample.py:6: > > UserWarning: downsample module has been moved to the > theano.tensor.signal.pool module. > "downsample module has been moved to the theano.tensor.signal.pool > module.") > /home/t/tj/tjb32/.local/lib/python2.7/site-packages/lasagne/layers/conv.py:489: > > UserWarning: The `image_shape` keyword argument to `tensor.nnet.conv2d` is > deprecated, it has been renamed to `input_shape`. > border_mode=border_mode) > ERROR (theano.gof.opt): Optimization failure due to: > local_avgpool_dnn_grad_stride > ERROR (theano.gof.opt): node: AveragePoolGrad{ds=(5, 1), > ignore_border=True, st=(5, 1), padding=(0, 0), mode='sum'}(HostFromGpu.0, > Rebroadcast{?,?,0,0}.0) > ERROR (theano.gof.opt): TRACEBACK: > ERROR (theano.gof.opt): Traceback (most recent call last): > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 1820, in process_node > replacements = lopt.transform(node) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/sandbox/cuda/dnn.py", > > line 3034, in local_avgpool_dnn_grad_stride > ret = GpuDnnPoolGrad(mode=mode)(gpu_contiguous(inp), > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/sandbox/cuda/dnn.py", > > line 1746, in __init__ > assert mode in ('max', 'average_inc_pad', 'average_exc_pad') > AssertionError > > Total queries possible: 0.984514648951, Testing queries possible: > 0.949673811743 > Number word vectors: 4402369 > Finding relevant pages from wikipedia > Loading model > Traceback (most recent call last): > File "runner.py", line 306, in <module> > main() > File "runner.py", line 241, in main > queries_exp = __import__(args.exp_model).queries_exp > File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity- > convnet/exp_multi_conv_cosim.py", line 923, in <module> > queries_exp = EntityVectorLinkExp() > File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity- > convnet/exp_multi_conv_cosim.py", line 51, in __init__ > self._setup() > File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity- > convnet/exp_multi_conv_cosim.py", line 543, in _setup > on_unused_input='ignore', > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/function.py", > > line 326, in function > output_keys=output_keys) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/pfunc.py", > > line 484, in pfunc > output_keys=output_keys) > File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/ > theano/compile/function_module.py", line 1788, in orig_function > output_keys=output_keys).create( > File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/ > theano/compile/function_module.py", line 1467, in __init__ > optimizer_profile = optimizer(fgraph) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 102, in __call__ > return self.optimize(fgraph) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 90, in optimize > ret = self.apply(fgraph, *args, **kwargs) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 235, in apply > sub_prof = optimizer.optimize(fgraph) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 90, in optimize > ret = self.apply(fgraph, *args, **kwargs) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 235, in apply > sub_prof = optimizer.optimize(fgraph) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 90, in optimize > ret = self.apply(fgraph, *args, **kwargs) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 2262, in apply > lopt_change = self.process_node(fgraph, node, lopt) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 1825, in process_node > lopt, node) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 1719, in warn_inplace > return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node) > File > "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py", > line 1705, in warn > raise exc > AssertionError > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- Pascal -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
