Problem solved. In one of my configuration files (spec_1gpu.yaml) I forgot to reset my batch_size parameter back to 256. I had assigned batch_size = 1 when I was doing some analyses on single image files. Fortunately, several debugging print statements before and after crash points identified this mislabeled parameter. I am still grateful to the theano users group. It is a generous resource for the community. Best, Arnold
On Tuesday, August 22, 2017 at 9:16:11 PM UTC-4, nouiz wrote: > > To get better error message from Theano, disable the GPU and use this > flag. optimizer=fast_compile > > In this way, Theano will probably give you a stack trace where you created > the computation that cause problem. > > On lun. 21 août 2017 19:15 ephi5757 via theano-users < > [email protected] <javascript:>> wrote: > >> Hi Frederic, >> I am pre-processing the image data again to regenerate the training >> and validation .hkl image files. I found in my code (alexnet\train.py) that >> the program crashes before it completes the first iteration, i.e., as it >> looks at the first of 5003 minibatches. In order to make room on my >> external solid state hard drive, I deleted the training and validation file >> folders named train_(or val_)hkl_b256_b_128, which I don't think are used >> but take up 237GB of space... and kept the folders named train_(or >> val_)hkl_b256_b_256. Perhaps in another day or two when the 1.2 M images >> are reshaped into 5003 files each containing 256 images that are size (256 >> x 256)... then I can try to run the train.py again and see if the errors >> correct themselves. >> This may have been my mistake for wanting to save space for my >> neural net model output (weights and biases). >> Best, >> Arnold >> >> On Wednesday, August 16, 2017 at 10:00:43 PM UTC-4, nouiz wrote: >> >>> I think the problem are the values in the index vector. Double check >>> that. >>> >>> Frédéric >>> >>> On Wed, Aug 16, 2017 at 5:49 PM ephi5757 via theano-users < >>> [email protected]> wrote: >>> >> I'm retraining my implementation of the neural network model AlexNet in >>>> Theano and not long after it initializes the program crashes with the >>>> error >>>> "ValueError: dimension mismatch in x,y_idx arguments." see traceback below. >>>> Any comments or suggestions that you may offer would be helpful. Note >>>> that the only discernible difference in this training in comparison to the >>>> previous one is that I am using 5003 .hkl training image data files >>>> instead >>>> of 5004. Nevertheless, I don't think this value needs to be fixed. >>>> Looking forward to your reply. >>>> Arnold >>>> _______________________________________________________________. >>>> >>>> >>>> C:\SciSoft\Git\theano_alexnet>python train.py >>>> THEANO_FLAGS=mode=FAST_RUN, floatX=float32 >>>> Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007) >>>> Using gpu device 0: Quadro K4000M (CNMeM is disabled, CuDNN 3007) >>>> ... building the model >>>> conv (cudnn) layer with shape_in: (3, 227, 227, 1) >>>> conv (cudnn) layer with shape_in: (96, 27, 27, 1) >>>> conv (cudnn) layer with shape_in: (256, 13, 13, 1) >>>> conv (cudnn) layer with shape_in: (384, 13, 13, 1) >>>> conv (cudnn) layer with shape_in: (384, 13, 13, 1) >>>> fc layer with num_in: 9216 num_out: 4096 >>>> dropout layer with P_drop: 0.5 >>>> fc layer with num_in: 4096 num_out: 4096 >>>> dropout layer with P_drop: 0.5 >>>> softmax layer with num_in: 4096 num_out: 1000 >>>> ... training >>>> >>>> >>>> ______________________________________________________________________________. >>>> Traceback (most recent call last): >>>> File >>>> "C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py", >>>> >>>> line 266, in _bootstrap >>>> self.run() >>>> File >>>> "C:\SciSoft\WinPython-64bit-2.7.9.4\python-2.7.9.amd64\lib\multiprocessing\process.py", >>>> >>>> line 120, in run >>>> self._target(*self._args, **self._kwargs) >>>> File "C:\SciSoft\Git\theano_alexnet\train.py", line 128, in train_net >>>> recv_queue=load_recv_queue) >>>> File "C:\SciSoft\Git\theano_alexnet\train_funcs.py", line 171, in >>>> train_model_wrap >>>> cost_ij = train_model() >>>> File "c:\scisoft\git\theano\theano\compile\function_module.py", line >>>> 871, in __call__ >>>> storage_map=getattr(self.fn, 'storage_map', None)) >>>> File "c:\scisoft\git\theano\theano\gof\link.py", line 314, in >>>> raise_with_op >>>> reraise(exc_type, exc_value, exc_trace) >>>> File "c:\scisoft\git\theano\theano\compile\function_module.py", line >>>> 859, in __call__ >>>> outputs = self.fn() >>>> >>>> ValueError: dimension mismatch in x,y_idx arguments >>>> Apply node that caused the error: >>>> GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0, >>>> <CudaNdarrayType(float32, vector)>, GpuFromHost.0) >>>> Toposort index: 298 >>>> Inputs types: [CudaNdarrayType(float32, matrix), >>>> CudaNdarrayType(float32, vector), CudaNdarrayType(float32, vector)] >>>> Inputs shapes: [(256, 1000), (1000,), (1,)] >>>> Inputs strides: [(1000, 1), (1,), (0,)] >>>> Inputs values: ['not shown', 'not shown', CudaNdarray([ 275.])] >>>> Outputs clients: >>>> [[GpuCAReduce{add}{1}(GpuCrossentropySoftmaxArgmax1HotWithBias.0)], >>>> [GpuCrossentropySoftmax1HotWithBiasDx(GpuElemwise{Inv}[(0, 0)].0, >>>> GpuCrossentropySoftmaxArgmax1HotWithBias.1, GpuFromHost.0)], []] >>>> . >>>> _____________________________________________________________________. >>>> >>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "theano-users" group. >>>> >>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>> >>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "theano-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- --- You received this message because you are subscribed to the Google Groups "theano-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
