Do you have multiple cudnn version installed? I have the impression Theano is in an environment with multiple cudnn version available.
Can you delete your Theano cache? This could also help. theano-cache purge On Wed, Dec 13, 2017 at 9:20 AM Beatriz G. <[email protected]> wrote: > After trying a lot of thigns, I have decided to uninstall and install > theano, and a new version has installed, the new version requires cuda, so > my theanorc file is now like: > > [global] > device = cuda > > floatX = float32 > > > [blas] > ldflags = -lopenblas > > > [nvcc] > # flags=-D_FORCE_INLINES > optimizer_including=cudnn > > [cuda] > root=/usr/local/cuda-9.1 > > > And I get the following output after trying Lenet: > > > Using cuDNN version 7005 on context None > Mapped name None to device cuda: GeForce GTX 750 Ti (0000:06:00.0) > > ... loading data > ... building the model > LENET.py:108: UserWarning: DEPRECATION: the 'ds' parameter is not going to > exist anymore as it is going to be replaced by the parameter 'ws'. > ignore_border=True > > Traceback (most recent call last): > File "LENET.py", line 394, in <module> > evaluate_lenet5() > File "LENET.py", line 228, in evaluate_lenet5 > y: test_set_y[index * batch_size: (index + 1) * batch_size] > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function.py", > line 317, in function > output_keys=output_keys) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", > line 486, in pfunc > output_keys=output_keys) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", > line 1841, in orig_function > fn = m.create(defaults) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", > line 1715, in create > input_storage=input_storage_lists, storage_map=storage_map) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line > 699, in make_thunk > storage_map=storage_map)[:3] > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/vm.py", > line 1084, in make_all > impl=impl)) > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/op.py", > line 955, in make_thunk > no_recycling) > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/op.py", > line 858, in make_c_thunk > output_storage=node_output_storage) > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", > line 1217, in make_thunk > keep_lock=keep_lock) > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", > line 1157, in __compile__ > keep_lock=keep_lock) > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", > line 1620, in cthunk_factory > key=key, lnk=self, keep_lock=keep_lock) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py", > line 1174, in module_from_key > module = lnk.compile_cmodule(location) > File "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cc.py", > line 1523, in compile_cmodule > preargs=preargs) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py", > line 2368, in compile_str > return dlimport(lib_filename) > File > "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/cmodule.py", > line 302, in dlimport > rval = __import__(module_name, {}, {}, [module_name]) > ImportError: ('The following error happened while compiling the node', > GpuDnnConv{algo='small', inplace=True, num_groups=1}(GpuContiguous.0, > GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, > GpuDnnConvDesc{border_mode='valid', subsample=(1, 1), dilation=(1, 1), > conv_mode='conv', precision='float32', num_groups=1}.0, Constant{1.0}, > Constant{0.0}), '\n', > '/home/bea/.theano/compiledir_Linux-4.4--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.12-64/tmpPD9sEN/97ac95f817846a3cb0867215657bdc2150272dcddf165864039b936dd3b77309.so: > undefined symbol: cudnnGetConvolutionGroupCount', > "[GpuDnnConv{algo='small', inplace=True, > num_groups=1}(<GpuArrayType<None>(float32, (False, True, False, False))>, > <GpuArrayType<None>(float32, 4D)>, <GpuArrayType<None>(float32, 4D)>, > <CDataType{cudnnConvolutionDescriptor_t}>, Constant{1.0}, Constant{0.0})]") > > > Regards. > > El miércoles, 13 de diciembre de 2017, 13:50:44 (UTC+1), Beatriz G. > escribió: >> >> Hi everyone. >> >> I used to work with Theano and it works perfectly, but after installing >> tensorflow with conda, and some dependencies to work with it, my Theano has >> stopped to work. >> >> I obtain the following error: >> >> Using gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled, cuDNN not >> available) >> ... loading data >> ... building the model >> ... training >> training @ iter = 0 >> Traceback (most recent call last): >> File "LENET.py", line 394, in <module> >> evaluate_lenet5() >> File "LENET.py", line 301, in evaluate_lenet5 >> cost_ij = train_model(minibatch_index) >> File >> "/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 871, in __call__ >> storage_map=getattr(self.fn, 'storage_map', None)) >> File >> "/home/bea/anaconda2/lib/python2.7/site-packages/theano/gof/link.py", line >> 314, in raise_with_op >> reraise(exc_type, exc_value, exc_trace) >> File >> "/home/bea/anaconda2/lib/python2.7/site-packages/theano/compile/function_module.py", >> line 859, in __call__ >> outputs = self.fn() >> RuntimeError: GpuCorrMM encountered a CUBLAS error: the library was not >> initialized >> This could be a known bug in CUDA, please see the GpuCorrMM() >> documentation. >> >> Apply node that caused the error: GpuCorrMM_gradWeights{valid, (1, >> 1)}(GpuContiguous.0, GpuContiguous.0) >> Toposort index: 28 >> Inputs types: [CudaNdarrayType(float32, (True, False, False, False)), >> CudaNdarrayType(float32, 4D)] >> Inputs shapes: [(1, 500, 28, 28), (1, 20, 5, 5)] >> Inputs strides: [(0, 784, 28, 1), (0, 25, 5, 1)] >> Inputs values: ['not shown', 'not shown'] >> Outputs clients: [[GpuDimShuffle{1,0,2,3}(GpuCorrMM_gradWeights{valid, >> (1, 1)}.0)]] >> >> HINT: Re-running with most Theano optimization disabled could give you a >> back-trace of when this node was created. This can be done with by setting >> the Theano flag 'optimizer=fast_compile'. If that does not work, Theano >> optimizations can be disabled with 'optimizer=None'. >> HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and >> storage map footprint of this apply node. >> >> I have tried to install cuda and cudnn, but it does not work (apart from >> the toolkit that i had already installed) >> >> My theanorc file looks like: >> [global] >> device = gpu >> floatX = float32 >> >> >> [blas] >> ldflags = -lopenblas >> >> >> [nvcc] >> flags=-D_FORCE_INLINES >> >> >> I would appreciate any advice or help. >> >> Regards. >> >> Beatriz. >> > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
