Hi There,
My scan function is outputting nans after a few iterations.
I tried the NaNGaurdMode but I can't make much sense of the output.
It's compiling fine and even running but I can't figure out whats wrong.
See output of monitor mode below. Any help would be really appreciated. I
think its something to do with what the previous value that is being used
in scan calculates - for some reason it shows an extra row (), (8, 3), *(9,*
5), (3, 5), (5, 5), (5,)] - but I checked all my dimensions and they seem
fine and tried initialising it with other values too.
Traceback (most recent call last):
File "/home/jim/PyCharmProjects/Experiments/Experiments/gaa/gaa_rdbn.py",
line 92, in <module>
main()
File "/home/jim/PyCharmProjects/Experiments/Experiments/gaa/gaa_rdbn.py",
line 89, in main
shuffle=False, y_col=3)
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/hyper_parameter/search.py",
line 83, in search_optimise
interactive_plot=interactive_plot, bootstrap=bootstrap, y_col=y_col,
shuffle=shuffle)
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/optimise/hyper_parameters.py",
line 56, in hp_search
interactive_plot=interactive_plot, print_gradients=print_gradients,
output_results=output_results)
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/model/rnn.py",
line 147, in train
data_id, d_format, checkpoint_every, interactive_plot, print_gradients,
output_results)
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/optimise/model_parameters.py",
line 78, in msgd
print(learner.train_func(batch_j))
File
"/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py",
line 859, in __call__
outputs = self.fn()
File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 297,
in __call__
link.raise_with_op(node, thunk)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line
314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 293,
in __call__
thunk()
File
"/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_op.py",
line 951, in rval
r = p(n, [x[0] for x in i], o)
File
"/usr/local/lib/python2.7/dist-packages/theano/scan_module/scan_op.py",
line 940, in <lambda>
self, node)
File "theano/scan_module/scan_perform.pyx", line 415, in
theano.scan_module.scan_perform.perform
(/home/jim/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/scan_perform/mod.cpp:4409)
File "theano/scan_module/scan_perform.pyx", line 397, in
theano.scan_module.scan_perform.perform
(/home/jim/.theano/compiledir_Linux-3.19--generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/scan_perform/mod.cpp:4193)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line
1014, in f
raise_with_op(node, *thunks)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line
314, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line
1012, in f
wrapper(i, node, *thunks)
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line
1028, in wrapper
f(*args)
File
"/usr/local/lib/python2.7/dist-packages/theano/compile/monitormode.py",
line 75, in eval
self.post_func(i, node, fn)
TypeError: 'str' object is not callable
Apply node that caused the error: Print{message='prev_a',
attrs=('__str__',), global_fn=<function _print_fn at
0x7f4876427140>}(initial_state_h[t-1])
Toposort index: 0
Inputs types: [TensorType(float64, vector)]
Inputs shapes: [(5,)]
Inputs strides: [(8,)]
Inputs values: [array([ 1., 1., 1., 1., 1.])]
Outputs clients: [[CGemv{no_inplace}(CGemv{no_inplace}.0,
TensorConstant{1.0}, W_1_HH_copy.T, Print{message='prev_a',
attrs=('__str__',), global_fn=<function _print_fn at 0x7f4876427140>}.0,
TensorConstant{1.0})]]
Debugprint of the apply node:
Print{message='prev_a', attrs=('__str__',), global_fn=<function _print_fn
at 0x7f4876427140>} [id A] <TensorType(float64, vector)> ''
|initial_state_h[t-1] [id B] <TensorType(float64, vector)>
HINT: Re-running with most Theano optimization disabled could give you a
back-trace of when this node was created. This can be done with by setting
the Theano flag 'optimizer=fast_compile'. If that does not work, Theano
optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and
storage map footprint of this apply node.
Apply node that caused the error: for{cpu,scan_fn}(Subtensor{int64}.0,
Subtensor{:int64:}.0, IncSubtensor{Set;:int64:}.0, W_1, W_1_HH, bias_1_to_h)
Toposort index: 92
Inputs types: [TensorType(int64, scalar), TensorType(float64, matrix),
TensorType(float64, matrix), TensorType(float64, matrix),
TensorType(float64, matrix), TensorType(float64, vector)]
Inputs shapes: [(), (8, 3), (9, 5), (3, 5), (5, 5), (5,)]
Inputs strides: [(), (24, 8), (40, 8), (40, 8), (40, 8), (8,)]
Inputs values: [array(8), 'not shown', 'not shown', 'not shown', 'not
shown', array([ 1., 1., 1., 1., 1.])]
Outputs clients: [[Subtensor{int64::}(for{cpu,scan_fn}.0, Constant{1}),
Elemwise{second,no_inplace}(for{cpu,scan_fn}.0, TensorConstant{(1, 1) of
0.0}), Shape(for{cpu,scan_fn}.0), Subtensor{::int64}(for{cpu,scan_fn}.0,
Constant{-1}), Subtensor{:int64:}(for{cpu,scan_fn}.0, Constant{-1})]]
Backtrace when the node is created(use Theano flag traceback.limit=N to
make it longer):
File "/home/jim/PyCharmProjects/Experiments/Experiments/gaa/gaa_rdbn.py",
line 92, in <module>
main()
File "/home/jim/PyCharmProjects/Experiments/Experiments/gaa/gaa_rdbn.py",
line 89, in main
shuffle=False, y_col=3)
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/hyper_parameter/search.py",
line 83, in search_optimise
interactive_plot=interactive_plot, bootstrap=bootstrap, y_col=y_col,
shuffle=shuffle)
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/optimise/hyper_parameters.py",
line 43, in hp_search
learner.build_hypothesis()
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/model/rnn.py",
line 82, in build_hypothesis
ff_layer.propagate()
File
"/home/jim/PyCharmProjects/SaotharlannDomhain/SaotharlannDomhain/model/hidden_layers.py",
line 208, in propagate
mode=theano.compile.MonitorMode(post_func='detect_nan').excluding('local_elemwise_fusion',
'inplace')
Debugprint of the apply node:
for{cpu,scan_fn} [id A] <TensorType(float64, matrix)> ''
|Subtensor{int64} [id B] <TensorType(int64, scalar)> ''
| |Shape [id C] <TensorType(int64, vector)> ''
| | |Subtensor{int64::} [id D] <TensorType(float64, matrix)>
'layer_0_x[0:]'
| |Constant{0} [id E] <int64>
|Subtensor{:int64:} [id F] <TensorType(float64, matrix)> ''
| |Subtensor{int64::} [id D] <TensorType(float64, matrix)>
'layer_0_x[0:]'
| |ScalarFromTensor [id G] <int64> ''
| |Subtensor{int64} [id B] <TensorType(int64, scalar)> ''
|IncSubtensor{Set;:int64:} [id H] <TensorType(float64, matrix)> ''
| |AllocEmpty{dtype='float64'} [id I] <TensorType(float64, matrix)> ''
| | |Elemwise{add,no_inplace} [id J] <TensorType(int64, scalar)> ''
| | | |Subtensor{int64} [id B] <TensorType(int64, scalar)> ''
| | | |Subtensor{int64} [id K] <TensorType(int64, scalar)> ''
| | | |Shape [id L] <TensorType(int64, vector)> ''
| | | | |Rebroadcast{0} [id M] <TensorType(float64, matrix)> ''
| | | | |DimShuffle{x,0} [id N] <TensorType(float64, row)> ''
| | | | |initial_state_h [id O] <TensorType(float64, vector)>
| | | |Constant{0} [id E] <int64>
| | |Subtensor{int64} [id P] <TensorType(int64, scalar)> ''
| | |Shape [id L] <TensorType(int64, vector)> ''
| | |Constant{1} [id Q] <int64>
| |Rebroadcast{0} [id M] <TensorType(float64, matrix)> ''
| |ScalarFromTensor [id R] <int64> ''
| |Subtensor{int64} [id K] <TensorType(int64, scalar)> ''
|W_1 [id S] <TensorType(float64, matrix)>
|W_1_HH [id T] <TensorType(float64, matrix)>
|bias_1_to_h [id U] <TensorType(float64, vector)>
Inner graphs of the scan ops:
for{cpu,scan_fn} [id A] <TensorType(float64, matrix)> ''
>Elemwise{add,no_inplace} [id V] <TensorType(float64, vector)> ''
> |bias_1_to_h_copy [id W] <TensorType(float64, vector)> -> [id U]
> |CGemv{no_inplace} [id X] <TensorType(float64, vector)> ''
> |CGemv{no_inplace} [id Y] <TensorType(float64, vector)> ''
> | |AllocEmpty{dtype='float64'} [id Z] <TensorType(float64, vector)>
''
> | | |Shape_i{1} [id BA] <TensorType(int64, scalar)> ''
> | | |W_1_copy [id BB] <TensorType(float64, matrix)> -> [id S]
> | |TensorConstant{1.0} [id BC] <TensorType(float64, scalar)>
> | |DimShuffle{1,0} [id BD] <TensorType(float64, matrix)>
'W_1_copy.T'
> | |layer_0_x[t] [id BE] <TensorType(float64, vector)> -> [id F]
> | |TensorConstant{0.0} [id BF] <TensorType(float64, scalar)>
> |TensorConstant{1.0} [id BC] <TensorType(float64, scalar)>
> |DimShuffle{1,0} [id BG] <TensorType(float64, matrix)>
'W_1_HH_copy.T'
> |Print{message='prev_a', attrs=('__str__',), global_fn=<function
_print_fn at 0x7f4876427140>} [id BH] <TensorType(float64, vector)> ''
> | |initial_state_h[t-1] [id BI] <TensorType(float64, vector)> -> [id H]
> |TensorConstant{1.0} [id BC] <TensorType(float64, scalar)>
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and
storage map footprint of this apply node.
--
---
You received this message because you are subscribed to the Google Groups
"theano-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
