Thanks for the investigation. Can you also try running the Python script from inside cuda-memcheck? For instance: CUDA_LAUNCH_BLOCKING=1 THEANO_FLAGS=... cuda-memcheck python /home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py
Thanks, On Wed, Nov 02, 2016, [email protected] wrote: > Hi, I did this test as Fred suggested: > > > - floatX=float16, device=cuda; > > - CUDA_LAUNCH_BLOCKING=1 > > - used theano.tensor.nnet.conv3d > > out = theano.tensor.nnet.conv3d( > input=input, > filters=self.W, > input_shape = None, > filter_shape= None, > border_mode = 'valid', > subsample=(1,1,1), > filter_flip = True, > filter_dilation=(1,1,1) > ) > > > this is the output (same error: pygpu.gpuarray. > GpuArrayException: an illegal memory access was encounteredt): > > Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) > [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > Anaconda is brought to you by Continuum Analytics. > Please check out: http://continuum.io/thanks and https://anaconda.org > >>> > runfile('/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py', > > wdir='/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core') > > Mapped name None to device cuda: Tesla K40c > Using cuDNN version 5103 on context None > Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16 > Disabling C code for Elemwise{Cast{float32}} due to unsupported float16 > Disabling C code for MaxAndArgmax due to unsupported float16 > > > start time: > 02/11/2016 > 10:39:10 > > > > ... training neural network 63 > > > > training @ iter = 0 > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File > "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", > > line 866, in runfile > execfile(filename, namespace) > File > "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", > > line 94, in execfile > builtins.execfile(filename, *where) > File > "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", > > line 43, in <module> > run_experiments() > File > "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", > > line 34, in run_experiments > Synthetic_data = False, > File "mpr_convnet_class.py", line 424, in __init__ > training_cost_ij=train_model(a, b) > File "/home/luca/data/Theano-master/theano/compile/function_module.py", > line 879, in __call__ > storage_map=getattr(self.fn, 'storage_map', None)) > File "/home/luca/data/Theano-master/theano/gof/link.py", line 172, in > raise_with_op > > "\nInputs values: %s" % scalar_values) > File "pygpu/gpuarray.pyx", line 1941, in pygpu.gpuarray.GpuArray.__repr__ > (pygpu/gpuarray.c:24742) > File > "/home/luca/anaconda2/lib/python2.7/site-packages/numpy/core/numeric.py", > line 482, in asarray > return array(a, dtype, copy=False, order=order) > File "pygpu/gpuarray.pyx", line 1572, in > pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224) > File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray > (pygpu/gpuarray.c:17346) > File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read > (pygpu/gpuarray.c:6114) > pygpu.gpuarray.GpuArrayException: an illegal memory access was encountered > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- Pascal -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
