Hi,
I am getting the GpuArrayException while validating and saving the model.
Not during the training.
I couldn't understand the error. Could you look and tell me what might be
the problem.
Thank you.
Log:............
Using cuDNN version 5103 on context None
Mapped name None to device cuda0: Tesla K80 (0000:05:00.0)
/scratch/elec/puhe/Modules/opt/Theano/Theano-0c53fb5/lib/python3.6/site-packages/Theano-0.9.0-py3.6.egg/theano/sandbox/rng_mrg.py:1521:
UserWarning: MRG_RandomStreams.multinomial_wo_replacement() is deprecated
and will be removed in the next release of Theano. Please use
MRG_RandomStreams.choice() instead.
warnings.warn('MRG_RandomStreams.multinomial_wo_replacement() is '
An unexpected GpuArrayException exception occurred. Traceback will be
written to debug log. The error message was: b'out of memory'
Apply node that caused the error: GpuDot22(GpuReshape{2}.0,
layers/hidden_layer_1/layer_input/W)
Toposort index: 62
Inputs types: [GpuArrayType<None>(float32, (False, False)),
GpuArrayType<None>(float32, (False, False))]
Inputs shapes: [(482112, 200), (200, 4000)]
Inputs strides: [(800, 4), (16000, 4)]
Inputs values: ['not shown', 'not shown']
Inputs type_num: [11, 11]
Outputs clients: [[GpuReshape{3}(GpuDot22.0, MakeVector{dtype='int64'}.0)]]
Debugprint of the apply node:
GpuDot22 [id A] <GpuArrayType<None>(float32, (False, False))> ''
|GpuReshape{2} [id B] <GpuArrayType<None>(float32, (False, False))> ''
| |GpuAdvancedSubtensor1 [id C] <GpuArrayType<None>(float32, (False,
False))> ''
| | |layers/projection_layer/W [id D] <GpuArrayType<None>(float32, (False,
False))>
| | |GpuContiguous [id E] <GpuArrayType<None>(int64, (False,))> ''
| | |GpuReshape{1} [id F] <GpuArrayType<None>(int64, (False,))> ''
| | |GpuSubtensor{:int64:} [id G] <GpuArrayType<None>(int64, (False,
False))> ''
| | | |GpuFromHost<None> [id H] <GpuArrayType<None>(int64, (False,
False))> ''
| | | | |textscorer/batch_word_ids [id I] <TensorType(int64, matrix)>
| | | |Constant{-1} [id J] <int64>
| | |TensorConstant{(1,) of -1} [id K] <TensorType(int64, (True,))>
| |MakeVector{dtype='int64'} [id L] <TensorType(int64, vector)> ''
| |Elemwise{mul,no_inplace} [id M] <TensorType(int64, scalar)> ''
| | |Elemwise{Switch}[(0, 2)] [id N] <TensorType(int64, scalar)> ''
| | | |Elemwise{eq,no_inplace} [id O] <TensorType(bool, scalar)> ''
| | | | |Elemwise{Switch}[(0, 2)] [id P] <TensorType(int64, scalar)> ''
| | | | | |Elemwise{lt,no_inplace} [id Q] <TensorType(bool, scalar)> ''
| | | | | | |Elemwise{add,no_inplace} [id R] <TensorType(int64, scalar)>
''
| | | | | | | |TensorConstant{-1} [id S] <TensorType(int64, scalar)>
| | | | | | | |Shape_i{0} [id T] <TensorType(int64, scalar)> ''
| | | | | | | |textscorer/batch_word_ids [id I] <TensorType(int64,
matrix)>
| | | | | | |TensorConstant{0} [id U] <TensorType(int8, scalar)>
| | | | | |TensorConstant{0} [id U] <TensorType(int8, scalar)>
| | | | | |Elemwise{add,no_inplace} [id R] <TensorType(int64, scalar)>
''
| | | | |TensorConstant{-1} [id V] <TensorType(int8, scalar)>
| | | |Elemwise{switch,no_inplace} [id W] <TensorType(int64, scalar)> ''
| | | | |Elemwise{lt,no_inplace} [id Q] <TensorType(bool, scalar)> ''
| | | | |TensorConstant{0} [id U] <TensorType(int8, scalar)>
| | | | |Elemwise{Composite{((i0 * (i1 * i2)) // (i1 * i2))}}[(0, 0)]
[id X] <TensorType(int64, scalar)> ''
| | | | |Shape_i{1} [id Y] <TensorType(int64, scalar)> ''
| | | | | |layers/projection_layer/W [id D]
<GpuArrayType<None>(float32, (False, False))>
| | | | |Shape_i{1} [id Z] <TensorType(int64, scalar)> ''
| | | | | |textscorer/batch_word_ids [id I] <TensorType(int64, matrix)>
| | | | |Elemwise{add,no_inplace} [id R] <TensorType(int64, scalar)>
''
| | | |Elemwise{Switch}[(0, 2)] [id P] <TensorType(int64, scalar)> ''
| | |Shape_i{1} [id Z] <TensorType(int64, scalar)> ''
| |Elemwise{switch,no_inplace} [id W] <TensorType(int64, scalar)> ''
|layers/hidden_layer_1/layer_input/W [id BA] <GpuArrayType<None>(float32,
(False, False))>
Storage map footprint:
- GpuReshape{2}.0, Shape: (482112, 200), ElemSize: 4 Byte(s), TotalSize:
385689600 Byte(s)
- layers/hidden_layer_1/step_input/W, Shared Input, Shape: (1000, 4000),
ElemSize: 4 Byte(s), TotalSize: 16000000 Byte(s)
- layers/highway_layer_1/input/W, Shared Input, Shape: (1000, 2000),
ElemSize: 4 Byte(s), TotalSize: 8000000 Byte(s)
- textscorer/batch_word_ids, Input, Shape: (7534, 64), ElemSize: 8
Byte(s), TotalSize: 3857408 Byte(s)
- textscorer/batch_class_ids, Input, Shape: (7534, 64), ElemSize: 8
Byte(s), TotalSize: 3857408 Byte(s)
- GpuFromHost<None>.0, Shape: (7534, 64), ElemSize: 8 Byte(s), TotalSize:
3857408 Byte(s)
- layers/hidden_layer_1/layer_input/W, Shared Input, Shape: (200, 4000),
ElemSize: 4 Byte(s), TotalSize: 3200000 Byte(s)
- textscorer/membership_probs, Input, Shape: (7533, 64), ElemSize: 4
Byte(s), TotalSize: 1928448 Byte(s)
- layers/output_layer/input/W, Shared Input, Shape: (1000, 404), ElemSize:
4 Byte(s), TotalSize: 1616000 Byte(s)
- network/mask, Input, Shape: (7533, 64), ElemSize: 1 Byte(s), TotalSize:
482112 Byte(s)
- GpuFromHost<None>.0, Shape: (7533, 64), ElemSize: 1 Byte(s), TotalSize:
482112 Byte(s)
- InplaceGpuDimShuffle{0,1,x}.0, Shape: (7533, 64, 1), ElemSize: 1
Byte(s), TotalSize: 482112 Byte(s)
- <GpuArrayType<None>(int32, (False, False))>, Shared Input, Shape:
(15360, 6), ElemSize: 4 Byte(s), TotalSize: 368640 Byte(s)
- layers/projection_layer/W, Shared Input, Shape: (404, 200), ElemSize: 4
Byte(s), TotalSize: 323200 Byte(s)
- layers/hidden_layer_1/layer_input/b, Shared Input, Shape: (4000,),
ElemSize: 4 Byte(s), TotalSize: 16000 Byte(s)
- layers/highway_layer_1/input/b, Shared Input, Shape: (2000,), ElemSize:
4 Byte(s), TotalSize: 8000 Byte(s)
- layers/output_layer/input/b, Shared Input, Shape: (404,), ElemSize: 4
Byte(s), TotalSize: 1616 Byte(s)
- Shape_i{1}.0, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- Shape_i{0}.0, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- Constant{1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- TensorConstant{1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- TensorConstant{1000}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
Byte(s)
- TensorConstant{-1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
Byte(s)
- Constant{-1}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- Elemwise{Composite{((i0 * (i1 * i2)) // (i1 * i2))}}[(0, 0)].0, Shape:
(), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- TensorConstant{(1,) of -1}, Shape: (1,), ElemSize: 8 Byte(s), TotalSize:
8 Byte(s)
- Elemwise{Switch}[(0, 2)].0, Shape: (), ElemSize: 8 Byte(s), TotalSize:
8.0 Byte(s)
- Constant{0}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- TensorConstant{2}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- TensorConstant{-2}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
Byte(s)
- TensorConstant{0}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- TensorConstant{404}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0
Byte(s)
- Constant{1000}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- Constant{2000}, Shape: (), ElemSize: 8 Byte(s), TotalSize: 8.0 Byte(s)
- GpuArrayConstant{0.0}, Shape: (), ElemSize: 4 Byte(s), TotalSize: 4.0
Byte(s)
- GpuArrayConstant{[[[ 1.25]]]}, Shape: (1, 1, 1), ElemSize: 4 Byte(s),
TotalSize: 4 Byte(s)
- GpuArrayConstant{[[[ 0.80000001]]]}, Shape: (1, 1, 1), ElemSize: 4
Byte(s), TotalSize: 4 Byte(s)
- GpuArrayConstant{[[[ 1.]]]}, Shape: (1, 1, 1), ElemSize: 4 Byte(s),
TotalSize: 4 Byte(s)
- TensorConstant{-5.0}, Shape: (), ElemSize: 4 Byte(s), TotalSize: 4.0
Byte(s)
- TensorConstant{1000}, Shape: (), ElemSize: 2 Byte(s), TotalSize: 2.0
Byte(s)
- GpuArrayConstant{[[403]]}, Shape: (1, 1), ElemSize: 2 Byte(s),
TotalSize: 2 Byte(s)
- <TensorType(int8, scalar)>, Shared Input, Shape: (), ElemSize: 1
Byte(s), TotalSize: 1.0 Byte(s)
- TensorConstant{1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
- TensorConstant{0}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
- Elemwise{lt,no_inplace}.0, Shape: (), ElemSize: 1 Byte(s), TotalSize:
1.0 Byte(s)
- TensorConstant{-1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0
Byte(s)
- Elemwise{eq,no_inplace}.0, Shape: (), ElemSize: 1 Byte(s), TotalSize:
1.0 Byte(s)
- Elemwise{lt,no_inplace}.0, Shape: (), ElemSize: 1 Byte(s), TotalSize:
1.0 Byte(s)
- TensorConstant{2}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
- Constant{1}, Shape: (), ElemSize: 1 Byte(s), TotalSize: 1.0 Byte(s)
TotalSize: 430170233.0 Byte(s) 0.401 GB
TotalSize inputs: 39658966.0 Byte(s) 0.037 GB
---
Siva
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
