atopion opened a new issue #14751: Model build error: Check failed: i >= 0 && i < ndim(): index = -2 must be in range [0, -1) URL: https://github.com/apache/incubator-mxnet/issues/14751 ## Description On Windows: Building of keras models fails on adding Dense layers with this error message. This happens regardless if package is build from source or downloaded via pip, with or without cuda. I first encountered this error as I tried to start [https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/GAN/dcgan_mnist.py](https://github.com/roatienza/Deep-Learning-Experiments/blob/master/Experiments/Tensorflow/GAN/dcgan_mnist.py) with mxnet instead of tensorflow backend. Afterwards I tried a couple examples on [https://github.com/awslabs/keras-apache-mxnet/tree/master/examples](https://github.com/awslabs/keras-apache-mxnet/tree/master/examples) which all failed with this error. I did not find this error on any other issue or in any forum, so I opened a new issue. ## Environment info (Required) ```----------Python Info---------- Version : 3.7.2 Compiler : MSC v.1916 64 bit (AMD64) Build : ('tags/v3.7.2:9a3ffc0492', 'Dec 23 2018 23:09:28') Arch : ('64bit', 'WindowsPE') ------------Pip Info----------- Version : 19.0.3 Directory : E:\data\program\python\lib\site-packages\pip ----------MXNet Info----------- Version : 1.4.0 Directory : E:\data\program\python\lib\site-packages\mxnet Hashtag not found. Not installed from pre-built package. ----------System Info---------- Platform : Windows-10-10.0.17763-SP0 system : Windows node : W7 release : 10 version : 10.0.17763 ----------Hardware Info---------- machine : AMD64 processor : Intel64 Family 6 Model 94 Stepping 3, GenuineIntel Name Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0312 sec, LOAD: 0.7029 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0625 sec, LOAD: 1.0311 sec. Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0625 sec, LOAD: 0.9529 sec. Error open FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>, DNS finished in 0.04686379432678223 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0312 sec, LOAD: 4.7601 sec. Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0469 sec, LOAD: 0.2656 sec. ``` Package used (Python/R/Scala/Julia): Python 3.7.2 ## Build info Compiler (gcc/clang/mingw/visual studio): visual studio MXNet commit hash: dc48cd2a5a6460171bf9b842453866e731e6ff7d Build config: Build command: ```cmake -G "Visual Studio 15 2017 Win64" -T cuda=9.2,host=x64 -DUSE_CUDA=1 -DUSE_CUDNN=1 -DUSE_NVRTC=1 -DUSE_OPENCV=1 -DUSE_OPENMP=1 -DUSE_BLAS=open -DUSE_LAPACK=1 -DUSE_DIST_KVSTORE=0 -DCUDA_ARCH_LIST=Common -DCUDA_TOOLSET='9.2' -D CUDNN_INCLUDE='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\include' -D CUDNN_LIBRARY='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.2\lib\x64\cudnn.lib' -D OpenCV_DIR="E:\data\program\opencv\build" -Wno-dev E:\data\program\mxnet\incubator-mxnet msbuild mxnet.sln /p:Configuration=Release;Platform=x64 /maxcpucount ``` ## Error Message: ```PS python .\lstm_text_generation.py Using MXNet backend corpus length: 600893 total chars: 57 nb sequences: 200285 Vectorization... Build model... Traceback (most recent call last): File ".\lstm_text_generation.py", line 60, in <module> model.add(Dense(len(chars), activation='softmax')) File "E:\data\program\python\lib\site-packages\keras\engine\sequential.py", line 181, in add output_tensor = layer(self.outputs[0]) File "E:\data\program\python\lib\site-packages\keras\engine\base_layer.py", line 470, in __call__ output = self.call(inputs, **kwargs) File "E:\data\program\python\lib\site-packages\keras\layers\core.py", line 893, in call output = K.bias_add(output, self.bias, data_format='channels_last') File "E:\data\program\python\lib\site-packages\keras\backend\mxnet_backend.py", line 96, in func_wrapper train_symbol = func(*args, **kwargs) File "E:\data\program\python\lib\site-packages\keras\backend\mxnet_backend.py", line 3986, in bias_add x_dim = ndim(x) File "E:\data\program\python\lib\site-packages\keras\backend\mxnet_backend.py", line 537, in ndim shape = x.shape File "E:\data\program\python\lib\site-packages\keras\backend\mxnet_backend.py", line 4399, in shape return self._get_shape() File "E:\data\program\python\lib\site-packages\keras\backend\mxnet_backend.py", line 4408, in _get_shape _, out_shape, _ = self.symbol.infer_shape_partial() File "E:\data\program\python\lib\site-packages\mxnet\symbol\symbol.py", line 1068, in infer_shape_partial return self._infer_shape_impl(True, *args, **kwargs) File "E:\data\program\python\lib\site-packages\mxnet\symbol\symbol.py", line 1126, in _infer_shape_impl ctypes.byref(complete))) File "E:\data\program\python\lib\site-packages\mxnet\base.py", line 252, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: Error in operator _foreach0: Error in operator dot7: [12:53:29] e:\data\program\mxnet\incubator-mxnet\include\mxnet\tuple.h:202: Check failed: i >= 0 && i < ndim(): index = -2 must be in range [0, -1) ``` ## Steps to reproduce git clone https://github.com/awslabs/keras-apache-mxnet.git cd examples python ./lstm_text_generation.py
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
