After struggling with this error for a day, I decided to upgrade CUDA to 9.1 and CuDNN to 7.1. After that I got "your driver might be too old" error, which was resolved by updating the driver to 396.24. Also, in the process I found out I had older CuDNN files in /usr/lib/x86_64-linux-gnu/ directory. Not sure how they got there, perhaps because sometimes I installed CuDNN using .deb package, and sometimes by manually copying the files. So it's probably not a good idea to mix .deb and .run cuda installation methods.
Anyway, now theano works fine now, but unfortunately my Tensorflow is broken because it does not support cuda 9.1 yet... Will probably have to compile it from source. On Thursday, May 10, 2018 at 11:30:38 AM UTC-7, Arnaud Bergeron wrote: > > This is a new one. It is also very weird since gemm doesn't involve > cuLinkAddData. This may be an error message from something else. > > First things first, since you are on cuda 9.0, I would recommend that you > update your driver to 384.111 or 390.*. If that doesn't help, then I'll > need some help reproducing the problem since I don't get that in any of my > environments. > > Le 8 mai 2018 à 18:15, Michael Klachko <[email protected] <javascript:>> > a écrit : > > I have CUDA 9.0 and CuDNN 7.0.5 on my Ubuntu 16.04, and Tensorflow works > fine. In order to install theano, I first installed miniconda, then ran > "conda > install theano pygpu" and it seemed to have installed fine. > > > > However, here's what I get: > > > $ python > Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) > [GCC 7.2.0] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import theano > Using cuDNN version 7005 on context None > ERROR (theano.gpuarray): Could not initialize pygpu, support disabled > Traceback (most recent call last): > File > "/home/michael/miniconda2/envs/las/lib/python3.6/site-packages/theano/gpuarray/__init__.py", > line 227, in <module> > use(config.device) > File > "/home/michael/miniconda2/envs/las/lib/python3.6/site-packages/theano/gpuarray/__init__.py", > line 214, in use > init_dev(device, preallocate=preallocate) > File > "/home/michael/miniconda2/envs/las/lib/python3.6/site-packages/theano/gpuarray/__init__.py", > line 159, in init_dev > pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True) > File "pygpu/blas.pyx", line 149, in pygpu.blas.gemm > File "pygpu/blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm > pygpu.gpuarray.GpuArrayException: (b'cuLinkAddData: CUDA_ERROR_UNKNOWN: > unknown error', 3) > > > > Here's the packages I have installed in this environment: > > > $ conda list > # packages in environment at /home/michael/miniconda2/envs/las: > # > # Name Version Build Channel > binutils_impl_linux-64 2.28.1 had2808c_3 > binutils_linux-64 7.2.0 26 > ca-certificates 2018.03.07 0 > certifi 2018.4.16 py36_0 > gcc_impl_linux-64 7.2.0 habb00fd_3 > gcc_linux-64 7.2.0 26 > gxx_impl_linux-64 7.2.0 hdf63c60_3 > gxx_linux-64 7.2.0 26 > intel-openmp 2018.0.0 8 > libedit 3.1 heed3624_0 > libffi 3.2.1 hd88cf55_4 > libgcc-ng 7.2.0 hdf63c60_3 > libgfortran-ng 7.2.0 hdf63c60_3 > libgpuarray 0.7.5 h14c3975_0 > libstdcxx-ng 7.2.0 hdf63c60_3 > mako 1.0.7 py36h0727276_0 > markupsafe 1.0 py36hd9260cd_1 > mkl 2018.0.2 1 > mkl-service 1.1.2 py36h17a0993_4 > mkl_fft 1.0.1 py36h3010b51_0 > mkl_random 1.0.1 py36h629b387_0 > ncurses 6.0 h9df7e31_2 > nose 1.3.7 py36hcdf7029_2 > numpy 1.14.2 py36hdbf6ddf_1 > openssl 1.0.2o h20670df_0 > pip 10.0.1 py36_0 > pygpu 0.7.5 py36h14c3975_0 > python 3.6.5 hc3d631a_2 > readline 7.0 ha6073c6_4 > scipy 1.0.1 py36hfc37229_0 > setuptools 39.1.0 py36_0 > six 1.11.0 py36h372c433_1 > sqlite 3.23.1 he433501_0 > theano 1.0.1 py36h6bb024c_0 > tk 8.6.7 hc745277_3 > wheel 0.31.0 py36_0 > xz 5.2.3 h5e939de_4 > zlib 1.2.11 ha838bed_2 > > > > Here's my .theanorc file: > > > [global] > device = cuda0 > optimizer_including = cudnn > floatX = float32 > > [dnn] > include_path = /usr/local/cuda/include > library_path = /usr/local/cuda/lib64 > > [lib] > cnmem = 0.7 > > [nvcc] > fastmath = True > > [blas] > # Only used for device = cpu > ldflags = -lopenblas > > [cuda] > root = /usr/local/cuda/bin > > > > Nvidia driver: Driver Version: 384.81 > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > For more options, visit https://groups.google.com/d/optout. > > > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
