Here is the beginning of the error message:
> An unexpected GpuArrayException exception occurred. Traceback will be
> written to debug log. The error message was: b'out of memory'
The output of your dot product would be of size (482112, 4000) in float32,
for a total of 7.7 GB.
It may very well exceed the amount of free memory on the GPU.
On Wed, May 10, 2017, Siva Reddy Gangireddy wrote:
> Hi,
>
> I am getting the GpuArrayException while validating and saving the model.
> Not during the training.
>
> I couldn't understand the error. Could you look and tell me what might be
> the problem.
>
> Thank you.
>
> Log:............
>
> Using cuDNN version 5103 on context None
> Mapped name None to device cuda0: Tesla K80 (0000:05:00.0)
> /scratch/elec/puhe/Modules/opt/Theano/Theano-0c53fb5/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/sandbox/rng_mrg.py:1521:
>
> UserWarning: MRG_RandomStreams.multinomial_wo_replacement() is deprecated
> and will be removed in the next release of Theano. Please use
> MRG_RandomStreams.choice() instead.
> warnings.warn('MRG_RandomStreams.multinomial_wo_replacement() is '
> An unexpected GpuArrayException exception occurred. Traceback will be
> written to debug log. The error message was: b'out of memory'
> Apply node that caused the error: GpuDot22(GpuReshape{2}.0,
> layers/hidden_layer_1/layer_input/W)
> Toposort index: 62
> Inputs types: [GpuArrayType<None>(float32, (False, False)),
> GpuArrayType<None>(float32, (False, False))]
> Inputs shapes: [(482112, 200), (200, 4000)]
> Inputs strides: [(800, 4), (16000, 4)]
> Inputs values: ['not shown', 'not shown']
> Inputs type_num: [11, 11]
> Outputs clients: [[GpuReshape{3}(GpuDot22.0, MakeVector{dtype='int64'}.0)]]
>
> Debugprint of the apply node:
> GpuDot22 [id A] <GpuArrayType<None>(float32, (False, False))> ''
> |GpuReshape{2} [id B] <GpuArrayType<None>(float32, (False, False))> ''
> | |GpuAdvancedSubtensor1 [id C] <GpuArrayType<None>(float32, (False,
> False))> ''
> | | |layers/projection_layer/W [id D] <GpuArrayType<None>(float32, (False,
> False))>
> | | |GpuContiguous [id E] <GpuArrayType<None>(int64, (False,))> ''
> | | |GpuReshape{1} [id F] <GpuArrayType<None>(int64, (False,))> ''
> | | |GpuSubtensor{:int64:} [id G] <GpuArrayType<None>(int64, (False,
> False))> ''
> | | | |GpuFromHost<None> [id H] <GpuArrayType<None>(int64, (False,
> False))> ''
> | | | | |textscorer/batch_word_ids [id I] <TensorType(int64, matrix)>
> | | | |Constant{-1} [id J] <int64>
> | | |TensorConstant{(1,) of -1} [id K] <TensorType(int64, (True,))>
> | |MakeVector{dtype='int64'} [id L] <TensorType(int64, vector)> ''
> | |Elemwise{mul,no_inplace} [id M] <TensorType(int64, scalar)> ''
> | | |Elemwise{Switch}[(0, 2)] [id N] <TensorType(int64, scalar)> ''
> | | | |Elemwise{eq,no_inplace} [id O] <TensorType(bool, scalar)> ''
> | | | | |Elemwise{Switch}[(0, 2)] [id P] <TensorType(int64, scalar)> ''
>
> | | | | | |Elemwise{lt,no_inplace} [id Q] <TensorType(bool, scalar)> ''
>
> | | | | | | |Elemwise{add,no_inplace} [id R] <TensorType(int64, scalar)>
> ''
> | | | | | | | |TensorConstant{-1} [id S] <TensorType(int64, scalar)>
> | | | | | | | |Shape_i{0} [id T] <TensorType(int64, scalar)> ''
> | | | | | | | |textscorer/batch_word_ids [id I] <TensorType(int64,
> matrix)>
> | | | | | | |TensorConstant{0} [id U] <TensorType(int8, scalar)>
> | | | | | |TensorConstant{0} [id U] <TensorType(int8, scalar)>
> | | | | | |Elemwise{add,no_inplace} [id R] <TensorType(int64, scalar)>
> ''
> | | | | |TensorConstant{-1} [id V] <TensorType(int8, scalar)>
> | | | |Elemwise{switch,no_inplace} [id W] <TensorType(int64, scalar)> ''
>
> | | | | |Elemwise{lt,no_inplace} [id Q] <TensorType(bool, scalar)> ''
> | | | | |TensorConstant{0} [id U] <TensorType(int8, scalar)>
> | | | | |Elemwise{Composite{((i0 * (i1 * i2)) // (i1 * i2))}}[(0, 0)]
> [id X] <TensorType(int64, scalar)> ''
> | | | | |Shape_i{1} [id Y] <TensorType(int64, scalar)> ''
> | | | | | |layers/projection_layer/W [id D]
> <GpuArrayType<None>(float32, (False, False))>
> | | | | |Shape_i{1} [id Z] <TensorType(int64, scalar)> ''
> | | | | | |textscorer/batch_word_ids [id I] <TensorType(int64, matrix)>
> | | | | |Elemwise{add,no_inplace} [id R] <TensorType(int64, scalar)>
> ''
> | | | |Elemwise{Switch}[(0, 2)] [id P] <TensorType(int64, scalar)> ''
> | | |Shape_i{1} [id Z] <TensorType(int64, scalar)> ''
> | |Elemwise{switch,no_inplace} [id W] <TensorType(int64, scalar)> ''
> |layers/hidden_layer_1/layer_input/W [id BA] <GpuArrayType<None>(float32,
> (False, False))>
>
> Storage map footprint:
> - GpuReshape{2}.0, Shape: (482112, 200), ElemSize: 4 Byte(s), TotalSize:
> 385689600 Byte(s)
> - layers/hidden_layer_1/step_input/W, Shared Input, Shape: (1000, 4000),
> ElemSize: 4 Byte(s), TotalSize: 16000000 Byte(s)
> - layers/highway_layer_1/input/W, Shared Input, Shape: (1000, 2000),
> ElemSize: 4 Byte(s), TotalSize: 8000000 Byte(s)
> - textscorer/batch_word_ids, Input, Shape: (7534, 64), ElemSize: 8
> Byte(s), TotalSize: 3857408 Byte(s)
> - textscorer/batch_class_ids, Input, Shape: (7534, 64), ElemSize: 8
> Byte(s), TotalSize: 3857408 Byte(s)
> - GpuFromHost<None>.0, Shape: (7534, 64), ElemSize: 8 Byte(s), TotalSize:
> 3857408 Byte(s)
> - layers/hidden_layer_1/layer_input/W, Shared Input, Shape: (200, 4000),
> ElemSize: 4 Byte(s), TotalSize: 3200000 Byte(s)
> - textscorer/membership_probs, Input, Shape: (7533, 64), ElemSize: 4
> Byte(s), TotalSize: 1928448 Byte(s)
> - layers/output_layer/input/W, Shared Input, Shape: (1000, 404), ElemSize:
> 4 Byte(s), TotalSize: 1616000 Byte(s)
> - network/mask, Input, Shape: (7533, 64), ElemSize: 1 Byte(s), TotalSize:
> 482112 Byte(s)
> - GpuFromHost<None>.0, Shape: (7533, 64), ElemSize: 1 Byte(s), TotalSize:
> 482112 Byte(s)
> - InplaceGpuDimShuffle{0,1,x}.0, Shape: (7533, 64, 1), ElemSize: 1
> Byte(s), TotalSize: 482112 Byte(s)
> - <GpuArrayType<None>(int32, (False, False))>, Shared Input, Shape:
> (15360, 6), ElemSize: 4 Byte(s), TotalSize: 368640 Byte(s)
> - layers/projection_layer/W, Shared Input, Shape: (404, 200), ElemSize: 4
> Byte(s), TotalSize: 323200 Byte(s)
> - layers/hidden_layer_1/layer_input/b, Shared Input, Shape: (4000,),
> ElemSize: 4 Byte(s), TotalSize: 16000 Byte(s)
> - layers/highway_layer_1/input/b, Shared Input, Shape: (2000,), ElemSize:
> 4 Byte(s), TotalSize: 8000 Byte(s)
> - layers/output_layer/input/b, Shared Input, Shape: (404,), ElemSize: 4
> Byte(s), TotalSize: 1616 Byte(s)
> - Shape_i{1}.0, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - Shape_i{0}.0, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - Constant{1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - TensorConstant{1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - TensorConstant{1000}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
> Byte(s)
> - TensorConstant{-1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
> Byte(s)
> - Constant{-1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - Elemwise{Composite{((i0 * (i1 * i2)) // (i1 * i2))}}[(0, 0)].0, Shape:
> (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - TensorConstant{(1,) of -1}, Shape: (1,), ElemSize: 8 Byte(s), TotalSize:
> 8 Byte(s)
> - Elemwise{Switch}[(0, 2)].0, Shape: (), ElemSize: 8 Byte(s), TotalSize:
> 8.0 Byte(s)
> - Constant{0}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - TensorConstant{2}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - TensorConstant{-2}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
> Byte(s)
> - TensorConstant{0}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - TensorConstant{404}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
> Byte(s)
> - Constant{1000}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - Constant{2000}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
> - GpuArrayConstant{0.0}, Shape: (), ElemSize: 4 Byte(s), TotalSize: 4.0
> Byte(s)
> - GpuArrayConstant{[[[ 1.25]]]}, Shape: (1, 1, 1), ElemSize: 4 Byte(s),
> TotalSize: 4 Byte(s)
> - GpuArrayConstant{[[[ 0.80000001]]]}, Shape: (1, 1, 1), ElemSize: 4
> Byte(s), TotalSize: 4 Byte(s)
> - GpuArrayConstant{[[[ 1.]]]}, Shape: (1, 1, 1), ElemSize: 4 Byte(s),
> TotalSize: 4 Byte(s)
> - TensorConstant{-5.0}, Shape: (), ElemSize: 4 Byte(s), TotalSize: 4.0
> Byte(s)
> - TensorConstant{1000}, Shape: (), ElemSize: 2 Byte(s), TotalSize: 2.0
> Byte(s)
> - GpuArrayConstant{[[403]]}, Shape: (1, 1), ElemSize: 2 Byte(s),
> TotalSize: 2 Byte(s)
>
> - <TensorType(int8, scalar)>, Shared Input, Shape: (), ElemSize: 1
> Byte(s), TotalSize: 1.0 Byte(s)
> - TensorConstant{1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
> - TensorConstant{0}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
> - Elemwise{lt,no_inplace}.0, Shape: (), ElemSize: 1 Byte(s), TotalSize:
> 1.0 Byte(s)
> - TensorConstant{-1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0
> Byte(s)
> - Elemwise{eq,no_inplace}.0, Shape: (), ElemSize: 1 Byte(s), TotalSize:
> 1.0 Byte(s)
> - Elemwise{lt,no_inplace}.0, Shape: (), ElemSize: 1 Byte(s), TotalSize:
> 1.0 Byte(s)
> - TensorConstant{2}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
> - Constant{1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
> TotalSize: 430170233.0 Byte(s) 0.401 GB
> TotalSize inputs: 39658966.0 Byte(s) 0.037 GB
>
> ---
> Siva
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "theano-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
--
Pascal
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
