To get better error message from Theano, disable the GPU and use this flag. optimizer=fast_compile
In this way, Theano will probably give you a stack trace where you created the computation that cause problem. On lun. 21 août 2017 19:15 ephi5757 via theano-users < [email protected]> wrote: > Hi Frederic, > I am pre-processing the image data again to regenerate the training > and validation .hkl image files. I found in my code (alexnet\train.py) that > the program crashes before it completes the first iteration, i.e., as it > looks at the first of 5003 minibatches. In order to make room on my > external solid state hard drive, I deleted the training and validation file > folders named train_(or val_)hkl_b256_b_128, which I don't think are used > but take up 237GB of space... and kept the folders named train_(or > val_)hkl_b256_b_256. Perhaps in another day or two when the 1.2 M images > are reshaped into 5003 files each containing 256 images that are size (256 > x 256)... then I can try to run the train.py again and see if the errors > correct themselves. > This may have been my mistake for wanting to save space for my neural > net model output (weights and biases). > Best, > Arnold > > On Wednesday, August 16, 2017 at 10:00:43 PM UTC-4, nouiz wrote: > >> I think the problem are the values in the index vector. Double check that. >> >> Frédéric >> >> On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users < >> [email protected]> wrote: >> > I'm retraining my implementation of the neural network model AlexNet in >>> Theano and not long after it initializes the program crashes with the error >>> "ValueError: dimension mismatch in x,y_idx arguments." see traceback below. >>> Any comments or suggestions that you may offer would be helpful. Note >>> that the only discernible difference in this training in comparison to the >>> previous one is that I am using 5003 .hkl training image data files instead >>> of 5004. Nevertheless, I don't think this value needs to be fixed. >>> Looking forward to your reply. >>> Arnold >>> _______________________________________________________________. >>> >>> >>> C:\SciSoft\Git\theano_alexnet>python train.py >>> THEANO_FLAGS=mode=FAST_RUN, floatX=float32 >>> Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007) >>> Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007) >>> ... building the model >>> conv (cudnn) layer with shape_in: (3, 227, 227, 1) >>> conv (cudnn) layer with shape_in: (96, 27, 27, 1) >>> conv (cudnn) layer with shape_in: (256, 13, 13, 1) >>> conv (cudnn) layer with shape_in: (384, 13, 13, 1) >>> conv (cudnn) layer with shape_in: (384, 13, 13, 1) >>> fc layer with num_in: 9216 num_out: 4096 >>> dropout layer with P_drop: 0.5 >>> fc layer with num_in: 4096 num_out: 4096 >>> dropout layer with P_drop: 0.5 >>> softmax layer with num_in: 4096 num_out: 1000 >>> ... training >>> >>> >>> ______________________________________________________________________________. >>> Traceback (most recent call last): >>> File >>> "C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py", >>> line 266, in _bootstrap >>> self.run() >>> File >>> "C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py", >>> line 120, in run >>> self._target(*self._args, **self._kwargs) >>> File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net >>> recv_queue=load_recv_queue) >>> File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in >>> train_model_wrap >>> cost_ij = train_model() >>> File "c:\scisoft\git\theano\theano\compile\function_module.py", line >>> 871, in __call__ >>> storage_map=getattr(self.fn, 'storage_map', None)) >>> File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in >>> raise_with_op >>> reraise(exc_type, exc_value, exc_trace) >>> File "c:\scisoft\git\theano\theano\compile\function_module.py", line >>> 859, in __call__ >>> outputs = self.fn() >>> >>> ValueError: dimension mismatch in x,y_idx arguments >>> Apply node that caused the error: >>> GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0, >>> <CudaNdarrayType(float32, vector)>, GpuFromHost.0) >>> Toposort index: 298 >>> Inputs types: [CudaNdarrayType(float32, matrix), >>> CudaNdarrayType(float32, vector), CudaNdarrayType(float32, vector)] >>> Inputs shapes: [(256, 1000), (1000,), (1,)] >>> Inputs strides: [(1000, 1), (1,), (0,)] >>> Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])] >>> Outputs clients: >>> [[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)], >>> [GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0, >>> GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []] >>> . >>> _____________________________________________________________________. >>> >>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "theano-users" group. >>> >> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >> >> >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "theano-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
