Hi,
Thanks for the response.
I have changed from mode='sum' to mode='average_inc_pad' for all layers.
This seems to have fixed the crash!
This is my first time using Theano and CNNs so i'm still very much a
novice. When you say multiply by the window size, what exactly do you mean
by that? Do you mean multiply the pool size by the window size?
For example I define a layer as such:
self.surface_pool1_l = lasagne.layers.Pool2DLayer(
self.surface_conv1_l,
name='surface_pool1',
pool_size=(self.sentence_length_short - self.num_words_to_use_conv, 1),
mode='average_inc_pad',
)
On Thursday, September 1, 2016 at 4:41:55 PM UTC+1, Pascal Lamblin wrote:
>
> Hi,
>
> The issue is that cuDNN has no "sum" pooling, but Theano still tries to
> optimizer CPU sum-pooling by creating a cuDNN Op, which crashes at the
> creation.
>
> The right solution would be to update the optimization doing that
> (https://github.com/Theano/Theano/issues/4929).
>
> A workaround for you now would be to explicitly use
> "average_inc_padding" and multiply by the window size.
>
> On Wed, Aug 31, 2016, Tom Ball wrote:
> > Hi,
> >
> > Sorry I missed part of the error message when I copied it over.
> >
> > Using gpu device 0: Tesla K40m (CNMeM is enabled with initial size:
> 95.0%
> > of memory, cuDNN 5005)
> > True
> > ++ which python2
> > + PYTHON=/cm/shared/apps/python/2.7.5/bin/python2
> > + WIKIPEDIA_DUMP=enwiki-latest-pages-articles.xml
> > + WIKIPEDIA_DOWNLOAD=
> >
> https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
>
> > + WIKIPEDIA_W2V=enwiki-vecs
> > + python runner.py --queries /home/t/tj/tjb32/workspace/
> > CNN_EL/data/data/ace-dev-set-queries.json --surface_count
> > /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-vecs-surface.json
> > --wordvecs /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-vecs.bin
> > --redirects
> /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-vecs-redir.json
> > --wiki_dump
> /home/t/tj/tjb32/workspace/CNN_EL/data/data/enwiki-latest-pages-articles.xml
>
> > --raw_output ./data/ace-dev-set.h5 --csv_output ./data/ace-dev-set.csv
> > Using gpu device 0: Tesla K40m (CNMeM is enabled with initial size:
> 95.0%
> > of memory, cuDNN 5005)
> >
> /home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/tensor/signal/downsample.py:6:
>
>
> > UserWarning: downsample module has been moved to the
> > theano.tensor.signal.pool module.
> > "downsample module has been moved to the theano.tensor.signal.pool
> > module.")
> >
> /home/t/tj/tjb32/.local/lib/python2.7/site-packages/lasagne/layers/conv.py:489:
>
>
> > UserWarning: The `image_shape` keyword argument to `tensor.nnet.conv2d`
> is
> > deprecated, it has been renamed to `input_shape`.
> > border_mode=border_mode)
> > ERROR (theano.gof.opt): Optimization failure due to:
> > local_avgpool_dnn_grad_stride
> > ERROR (theano.gof.opt): node: AveragePoolGrad{ds=(5, 1),
> > ignore_border=True, st=(5, 1), padding=(0, 0),
> mode='sum'}(HostFromGpu.0,
> > Rebroadcast{?,?,0,0}.0)
> > ERROR (theano.gof.opt): TRACEBACK:
> > ERROR (theano.gof.opt): Traceback (most recent call last):
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 1820, in process_node
> > replacements = lopt.transform(node)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/sandbox/cuda/dnn.py",
>
>
> > line 3034, in local_avgpool_dnn_grad_stride
> > ret = GpuDnnPoolGrad(mode=mode)(gpu_contiguous(inp),
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/sandbox/cuda/dnn.py",
>
>
> > line 1746, in __init__
> > assert mode in ('max', 'average_inc_pad', 'average_exc_pad')
> > AssertionError
> >
> > Total queries possible: 0.984514648951, Testing queries possible:
> > 0.949673811743
> > Number word vectors: 4402369
> > Finding relevant pages from wikipedia
> > Loading model
> > Traceback (most recent call last):
> > File "runner.py", line 306, in <module>
> > main()
> > File "runner.py", line 241, in main
> > queries_exp = __import__(args.exp_model).queries_exp
> > File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-
> > convnet/exp_multi_conv_cosim.py", line 923, in <module>
> > queries_exp = EntityVectorLinkExp()
> > File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-
> > convnet/exp_multi_conv_cosim.py", line 51, in __init__
> > self._setup()
> > File "/mnt/nfs2/inf/tjb32/workspace/CNN_EL/nlp-entity-
> > convnet/exp_multi_conv_cosim.py", line 543, in _setup
> > on_unused_input='ignore',
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/function.py",
>
>
> > line 326, in function
> > output_keys=output_keys)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/compile/pfunc.py",
>
>
> > line 484, in pfunc
> > output_keys=output_keys)
> > File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/
> > theano/compile/function_module.py", line 1788, in orig_function
> > output_keys=output_keys).create(
> > File "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/
> > theano/compile/function_module.py", line 1467, in __init__
> > optimizer_profile = optimizer(fgraph)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 102, in __call__
> > return self.optimize(fgraph)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 90, in optimize
> > ret = self.apply(fgraph, *args, **kwargs)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 235, in apply
> > sub_prof = optimizer.optimize(fgraph)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 90, in optimize
> > ret = self.apply(fgraph, *args, **kwargs)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 235, in apply
> > sub_prof = optimizer.optimize(fgraph)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 90, in optimize
> > ret = self.apply(fgraph, *args, **kwargs)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 2262, in apply
> > lopt_change = self.process_node(fgraph, node, lopt)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 1825, in process_node
> > lopt, node)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 1719, in warn_inplace
> > return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt,
> node)
> > File
> "/home/t/tj/tjb32/.local/lib/python2.7/site-packages/theano/gof/opt.py",
> > line 1705, in warn
> > raise exc
> > AssertionError
> >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "theano-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected] <javascript:>.
> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
> Pascal
>
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
