johnbroughton2017 opened a new issue #16239: train_imagenet.py fails with float16 + alexnet URL: https://github.com/apache/incubator-mxnet/issues/16239 ## Description I found the following error when training an alexnet using fp16. It is very easy reproduce using the command below. I have seen a few people reporting this bug yet I haven't been able to find a solution. If you replace alexnet with reset, it works fine. ## Error Message: ``` mxnet.base.MXNetError: [14:57:10] include/mxnet/././tensor_blob.h:236: Check failed: mshadow::DataType<DType>::kFlag == type_flag_: TBlob.get_with_shape: data type do not match specified type.Expected: 2 v.s. given 0 Stack trace: [bt] (0) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x4a3b8b) [0x7f377f32ab8b] [bt] (1) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x304366d) [0x7f3781eca66d] [bt] (2) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x3553e98) [0x7f37823dae98] [bt] (3) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2675018) [0x7f37814fc018] [bt] (4) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x267abf5) [0x7f3781501bf5] [bt] (5) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x265a8a1) [0x7f37814e18a1] [bt] (6) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x265ddb0) [0x7f37814e4db0] [bt] (7) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x265e046) [0x7f37814e5046] [bt] (8) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2659004) [0x7f37814e0004] ``` ## Minimum reproducible example ``` python train_imagenet.py --network alexnet --benchmark 1 --gpus 0 --batch-size 64 --dtype float16 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
