QueensGambit opened a new issue #15337: Current MXNET-Dev master breaks code for loading certain models URL: https://github.com/apache/incubator-mxnet/issues/15337 ## Description The current MXNET master dev branch, pypi version 1.5.0b20190623 breaks the loading of certain MXNET-models (both in mxnet-mkl & mxnet-cu100), which previously were loaded successfully with mxnet==1.4.1. The model uses grouped depthwise (a.ka. depthwise seperable) convolutions. Other models (e.g. CrazyAraFish_0.5.0_RiseV1.zip) still work correctly as usual. ## Environment info ``` ----------Python Info---------- Version : 3.6.5 Compiler : GCC 7.2.0 Build : ('default', 'Apr 29 2018 16:14:56') Arch : ('64bit', '') ------------Pip Info----------- Version : 10.0.1 Directory : /home/queensgambit/anaconda3/lib/python3.6/site-packages/pip ----------MXNet Info----------- /home/queensgambit/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`. from ._conv import register_converters as _register_converters Version : 1.5.0 Directory : /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet Commit Hash : f44f6cfbe752fd8b8036307cecf6a30a30ad8557 ----------System Info---------- Platform : Linux-4.15.0-52-generic-x86_64-with-debian-buster-sid system : Linux node : Latitude-5590 release : 4.15.0-52-generic version : #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 ----------Hardware Info---------- machine : x86_64 processor : x86_64 Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 142 Model name: Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz Stepping: 10 CPU MHz: 3104.749 CPU max MHz: 3400,0000 CPU min MHz: 400,0000 BogoMIPS: 3600.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 6144K NUMA node0 CPU(s): 0-7 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d ----------Network Test---------- Setting timeout: 10 Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0263 sec, LOAD: 0.5751 sec. Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0536 sec, LOAD: 0.9013 sec. Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0508 sec, LOAD: 0.8830 sec. Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0420 sec, LOAD: 0.4008 sec. Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0243 sec, LOAD: 0.6710 sec. Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0284 sec, LOAD: 0.2558 sec. ``` I'm using python, but the same problem also occurs when building the MXNET-CPP package from source. ## Error Message: ```python isready self.symbol_path: /home/queensgambit/Programming/Deep_Learning/models/risev2/symbol/model-1.19246-0.603-symbol.json self.params_path: /home/queensgambit/Programming/Deep_Learning/models/risev2/params/model-1.19246-0.603-0223.params [00:35:51] src/nnvm/legacy_json_util.cc:209: Loading symbol saved by previous version v1.4.1. Attempting to upgrade... [00:35:51] src/nnvm/legacy_json_util.cc:217: Symbol successfully upgraded! Traceback (most recent call last): File "/home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1623, in simple_bind ctypes.byref(exe_handle))) File "/home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: Error in operator value: [00:35:51] include/mxnet/./tuple.h:202: Check failed: i >= 0 && i < ndim(): index = 0 must be in range [0, -1) Stack trace: [bt] (0) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25b3ab) [0x7f186bc433ab] [bt] (1) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2c5343) [0x7f186bcad343] [bt] (2) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x298bf82) [0x7f186e373f82] [bt] (3) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2471ee2) [0x7f186de59ee2] [bt] (4) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2474794) [0x7f186de5c794] [bt] (5) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::exec::GraphExecutor::Init(nnvm::Symbol, mxnet::Context const&, std::map<std::string, mxnet::Context, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::Context> > > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::unordered_map<std::string, mxnet::TShape, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::TShape> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::unordered_set<std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::string> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::unordered_map<std::string, mxnet::NDArray, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::NDArray> > >*, mxnet::Executor*, std::unordered_map<nnvm::NodeEntry, mxnet::NDArray, nnvm::NodeEntryHash, nnvm::NodeEntryEqual, std::allocator<std::pair<nnvm::NodeEntry const, mxnet::NDArray> > > const&)+0x355) [0x7f186de48455] [bt] (6) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Executor::SimpleBind(nnvm::Symbol, mxnet::Context const&, std::map<std::string, mxnet::Context, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::Context> > > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::unordered_map<std::string, mxnet::TShape, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::TShape> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::unordered_set<std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::string> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::unordered_map<std::string, mxnet::NDArray, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::NDArray> > >*, mxnet::Executor*)+0x8a8) [0x7f186de49688] [bt] (7) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXExecutorSimpleBindEx+0x221b) [0x7f186dd9884b] [bt] (8) /home/queensgambit/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f1872e3eec0] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "crazyara.py", line 668, in main self.setup_network() File "crazyara.py", line 166, in setup_network model_weights_dir=self.settings["model_weights_dir"])) File "/home/queensgambit/Programming/Deep_Learning/CrazyAra/DeepCrazyhouse/src/domain/agent/neural_net_api.py", line 95, in __init__ force_rebind=True, File "/home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1629, in simple_bind raise RuntimeError(error_msg) RuntimeError: simple_bind error. Arguments: data: (1, 34, 8, 8) force_rebind: True Error in operator value: [00:35:51] include/mxnet/./tuple.h:202: Check failed: i >= 0 && i < ndim(): index = 0 must be in range [0, -1) Stack trace: [bt] (0) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25b3ab) [0x7f186bc433ab] [bt] (1) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2c5343) [0x7f186bcad343] [bt] (2) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x298bf82) [0x7f186e373f82] [bt] (3) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2471ee2) [0x7f186de59ee2] [bt] (4) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2474794) [0x7f186de5c794] [bt] (5) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::exec::GraphExecutor::Init(nnvm::Symbol, mxnet::Context const&, std::map<std::string, mxnet::Context, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::Context> > > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::unordered_map<std::string, mxnet::TShape, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::TShape> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::unordered_set<std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::string> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::unordered_map<std::string, mxnet::NDArray, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::NDArray> > >*, mxnet::Executor*, std::unordered_map<nnvm::NodeEntry, mxnet::NDArray, nnvm::NodeEntryHash, nnvm::NodeEntryEqual, std::allocator<std::pair<nnvm::NodeEntry const, mxnet::NDArray> > > const&)+0x355) [0x7f186de48455] [bt] (6) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(mxnet::Executor::SimpleBind(nnvm::Symbol, mxnet::Context const&, std::map<std::string, mxnet::Context, std::less<std::string>, std::allocator<std::pair<std::string const, mxnet::Context> > > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::vector<mxnet::Context, std::allocator<mxnet::Context> > const&, std::unordered_map<std::string, mxnet::TShape, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::TShape> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::unordered_map<std::string, int, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, int> > > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::unordered_set<std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::string> > const&, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::unordered_map<std::string, mxnet::NDArray, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, mxnet::NDArray> > >*, mxnet::Executor*)+0x8a8) [0x7f186de49688] [bt] (7) /home/queensgambit/anaconda3/lib/python3.6/site-packages/mxnet/libmxnet.so(MXExecutorSimpleBindEx+0x221b) [0x7f186dd9884b] [bt] (8) /home/queensgambit/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f1872e3eec0] ``` ## Minimum reproducible example ## Steps to reproduce Download release `CrazyAra_0.5.0_RiseV2_mobile.zip` at: * https://github.com/QueensGambit/CrazyAra/releases Install [python-chess](https://github.com/niklasf/python-chess). ``` pip install chess ``` Extract `CrazyAra_0.5.0_RiseV2_mobile.zip` and run ``` $ python crazyara.py $ uci $ isready ``` from the commandline. More details for install instructions can be found here: * [Install Guide](https://github.com/QueensGambit/CrazyAra/wiki/Installation-Guide) Alternatively you can load the mxnet model from the `model/` directory manually in python. Does someones have an idea what recent change causes this? Can you include more automated unit tests for MXNET to ensure that the loading of different model types is preserved for version updates?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
