Hi, I'm doing tests on Tesla K40 and Theano==0.9.0.dev3: whether I use float 16 or float32, theano.gpuarray.dnn.dnn_conv or theano.tensor.nnet.conv3d2d.conv3d it works only for small images but if I increase the images size there are problems with memory. I had not this issue when I was using float32 with the previous Theano version and same parameters and images size.
I try: floatX = float16 device = cuda #theano.gpuarray.dnn.dnn_conv out = dnn_conv(img= input, kerns= self.W, border_mode='valid', subsample=(1,1,1), conv_mode='conv', direction_hint=None, workmem=None, algo=None, precision=None) This is the output for small 3d images, the convnet is working: Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> runfile('/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py', wdir='/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core') Mapped name None to device cuda: Tesla K40c Using cuDNN version 5103 on context None Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16 Disabling C code for Elemwise{Cast{float32}} due to unsupported float16 Disabling C code for MaxAndArgmax due to unsupported float16 start time: 13/10/2016 11:43:00 Image_dim_1: 30 Image_dim_2: 30 Image_dim_3: 30 training @ iter = 0 training @ iter = 400 training cost 0.701 epoch 1, training batch 574/574, validation error 48.04 % ----------- if I increase the size of the images: pygpu.gpuarray.GpuArrayException: an illegal memory access was encountered Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> runfile('/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py', wdir='/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core') Mapped name None to device cuda: Tesla K40c Using cuDNN version 5103 on context None Disabling C code for Elemwise{mul,no_inplace} due to unsupported float16 Disabling C code for Elemwise{Cast{float32}} due to unsupported float16 Disabling C code for MaxAndArgmax due to unsupported float16 start time: 13/10/2016 11:52:45 Image_dim_1: 90 Image_dim_2: 90 Image_dim_3: 90 training @ iter = 0 Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile execfile(filename, namespace) File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile builtins.execfile(filename, *where) File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 42, in <module> run_experiments() File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 33, in run_experiments Zoom = 0.0 File "mpr_convnet_class.py", line 322, in __init__ training_cost_ij=train_model(a, b) File "/home/luca/data/Theano-master/theano/compile/function_module.py", line 879, in __call__ storage_map=getattr(self.fn, 'storage_map', None)) File "/home/luca/data/Theano-master/theano/gof/link.py", line 167, in raise_with_op "\nInputs values: %s" % scalar_values) File "pygpu/gpuarray.pyx", line 1941, in pygpu.gpuarray.GpuArray.__repr__ (pygpu/gpuarray.c:24742) File "/home/luca/anaconda2/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray return array(a, dtype, copy=False, order=order) File "pygpu/gpuarray.pyx", line 1572, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224) File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray (pygpu/gpuarray.c:17346) File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6114) pygpu.gpuarray.GpuArrayException: an illegal memory access was encountered ----------------- If I try float 32 and theano.gpuarray.dnn.dnn_conv I also have memory problems: pygpu.gpuarray.GpuArrayException: out of memory Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> runfile('/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py', wdir='/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core') Mapped name None to device cuda: Tesla K40c Using cuDNN version 5103 on context None Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile execfile(filename, namespace) File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile builtins.execfile(filename, *where) File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 42, in <module> run_experiments() File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 33, in run_experiments Zoom = 0.0 File "mpr_convnet_class.py", line 242, in __init__ b) File "mlp.py", line 199, in __init__ borrow=True, File "mlp.py", line 138, in __init__ self.W = shared(value=W_val, borrow=borrow, name=layer_name+'_W') File "/home/luca/data/Theano-master/theano/compile/sharedvalue.py", line 247, in shared allow_downcast=allow_downcast, **kwargs) File "/home/luca/data/Theano-master/theano/gpuarray/type.py", line 624, in gpuarray_shared_constructor context=type.context) File "pygpu/gpuarray.pyx", line 935, in pygpu.gpuarray.array (pygpu/gpuarray.c:12296) File "pygpu/gpuarray.pyx", line 633, in pygpu.gpuarray.pygpu_fromhostdata (pygpu/gpuarray.c:9119) File "pygpu/gpuarray.pyx", line 264, in pygpu.gpuarray.array_copy_from_host (pygpu/gpuarray.c:5008) pygpu.gpuarray.GpuArrayException: out of memory ------ If I try theano.tensor.nnet.conv3d2d.conv3d, floatX = float32, device = gpu I also have memory problems: MemoryError: ('Error allocating 14224896000 bytes of device memory (out of memory).', "you might consider using 'theano.shared(..., borrow=True)'") Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> runfile('/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py', wdir='/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core') Using gpu device 0: Tesla K40c (CNMeM is disabled, cuDNN 5103) Error allocating 14224896000 bytes of device memory (out of memory). Driver report 11961581568 bytes free and 12079136768 bytes total Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile execfile(filename, namespace) File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile builtins.execfile(filename, *where) File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 42, in <module> run_experiments() File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 33, in run_experiments Zoom = 0.0 File "mpr_convnet_class.py", line 242, in __init__ b) File "mlp.py", line 199, in __init__ borrow=True, File "mlp.py", line 138, in __init__ self.W = shared(value=W_val, borrow=borrow, name=layer_name+'_W') File "/home/luca/data/Theano-master/theano/compile/sharedvalue.py", line 247, in shared allow_downcast=allow_downcast, **kwargs) File "/home/luca/data/Theano-master/theano/sandbox/cuda/var.py", line 242, in float32_shared_constructor deviceval = type_support_filter(value, type.broadcastable, False, None) MemoryError: ('Error allocating 14224896000 bytes of device memory (out of memory).', "you might consider using 'theano.shared(..., borrow=True)'") ---------------- If I try theano.tensor.nnet.conv3d2d.conv3d, floatX = float16, device = gpu I have TypeError Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org >>> runfile('/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py', wdir='/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core') Using gpu device 0: Tesla K40c (CNMeM is disabled, cuDNN 5103) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 866, in runfile execfile(filename, namespace) File "/home/luca/anaconda2/lib/python2.7/site-packages/spyder/utils/site/sitecustomize.py", line 94, in execfile builtins.execfile(filename, *where) File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 42, in <module> run_experiments() File "/home/luca/data/DeepLearningTutorials/Theano-3D-Convnet-master/convnet3d/core/run_multi_conv.py", line 33, in run_experiments Zoom = 0.0 File "mpr_convnet_class.py", line 269, in __init__ train_model = theano.function([x,y],cost, updates=updates) File "/home/luca/data/Theano-master/theano/compile/function.py", line 326, in function output_keys=output_keys) File "/home/luca/data/Theano-master/theano/compile/pfunc.py", line 449, in pfunc no_default_updates=no_default_updates) File "/home/luca/data/Theano-master/theano/compile/pfunc.py", line 208, in rebuild_collect_shared raise TypeError(err_msg, err_sug) TypeError: ('An update must have the same type as the original shared variable (shared_var=DropoutLogisticRegression_W, shared_var.type=TensorType(float16, matrix), update_val=Elemwise{sub,no_inplace}.0, update_val.type=TensorType(float32, matrix)).', 'If the difference is related to the broadcast pattern, you can call the tensor.unbroadcast(var, axis_to_unbroadcast[, ...]) function to remove broadcastable dimensions.') Many thanks Luca -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to theano-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.