mycpuorg commented on issue #15725: AMP issue in _gen_atomic_symbol URL: https://github.com/apache/incubator-mxnet/issues/15725#issuecomment-517510741 Here's the generic part of the Python stack trace that I can share. Thanks. ``` ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:548: in __call__ out = self.forward(*args) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:915: in forward return self._call_cached_op(x, *args) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:805: in _call_cached_op self._build_cache(*args) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:757: in _build_cache data, out = self._get_graph(*args) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:749: in _get_graph out = self.hybrid_forward(symbol, *grouped_inputs, **params) # pylint: disable=no-value-for-parameter src/aws_concierge_hrnn_model/model.py:298: in hybrid_forward embed_output, hidden = self.core(embed_input, hidden) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/rnn/rnn_layer.py:237: in __call__ return super(_RNNLayer, self).__call__(inputs, states, **kwargs) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:548: in __call__ out = self.forward(*args) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:932: in forward return self.hybrid_forward(symbol, x, *args, **params) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/rnn/rnn_layer.py:250: in hybrid_forward out = self._forward_kernel(F, inputs, states, sequence_length, **kwargs) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/rnn/rnn_layer.py:286: in _forward_kernel lstm_state_clip_nan=self._lstm_state_clip_nan) ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/contrib/amp/amp.py:98: in _new_fun atomic_sym = sym._gen_atomic_symbol() ../../../anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py:1384: in _gen_atomic_symbol check_call(_LIB.MXGenAtomicSymbolFromSymbol(self.handle, ctypes.byref(handle))) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ret = -1 def check_call(ret): """Check the return value of C API call. This function will raise an exception when an error occurs. Wrap every API call with this function. Parameters ---------- ret : int return value from API calls. """ if ret != 0: > raise MXNetError(py_str(_LIB.MXGetLastError())) E mxnet.base.MXNetError: [01:11:15] src/c_api/c_api_symbolic.cc:857: Check failed: source->outputs.size() == 1U (2 vs. 1) : Generating atomic symbol from other symbol only works for nongrouped symbol. E Stack trace: E [bt] (0) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x4a357b) [0x7f92a573657b] E [bt] (1) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXGenAtomicSymbolFromSymbol+0x19d) [0x7f92a78a5fbd] E [bt] (2) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f932a897ec0] E [bt] (3) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f932a89787d] E [bt] (4) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f932aaace2e] ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
