zhreshold opened a new issue #7368: Caffe converter test fails, causing CI to halt for all PRs URL: https://github.com/apache/incubator-mxnet/issues/7368 ## Steps to reproduce or if you are running standard examples, please provide the commands you have run that lead to the error. 1. python tools/caffe_converter/test_converter.py 2. 3. ## What have you tried to solve it? Tried to debug the metric_update process, figured out for vgg-16 and resnet, the network prediction output is wrong, we have the last conv output mixed to the final prediction. Thus causing the mismatched shape with label. ``` ('!!!labels', 1, (32L,)) ('!!!preds', 2, (32L, 1000L), (32L, 512L, 7L, 7L)) ``` For googlenet, the output is correct: ``` ('!!!labels', 1, (32L,)) ('!!!preds', 2, (32L, 1000L)) ``` @joey2014 https://builds.apache.org/blue/organizations/jenkins/incubator-mxnet/detail/master/152/pipeline/ ``` INFO:root:Saved checkpoint to "./model/vgg-16-0000.params" data/val-5k-256.rec [18:21:16] src/io/iter_image_recordio_2.cc:135: ImageRecordIOParser2: data/val-5k-256.rec, use 4 threads for decoding.. ('!!!labels', 1, (32L,)) ('!!!preds', 2, (32L, 1000L), (32L, 512L, 7L, 7L)) Traceback (most recent call last): File "test_converter.py", line 89, in <module> main() File "test_converter.py", line 86, in main test_imagenet_model_performance(m, val, gpus, batch_size) File "test_converter.py", line 38, in test_imagenet_model_performance **mean_args) File "/home/ubuntu/debug/incubator-mxnet/tools/caffe_converter/../../example/image-classification/score.py", line 60, in score mod.update_metric(m, batch.label) File "/home/ubuntu/verify/mxnet/python/mxnet/module/module.py", line 718, in update_metric self._exec_group.update_metric(eval_metric, labels) File "/home/ubuntu/verify/mxnet/python/mxnet/module/executor_group.py", line 565, in update_metric eval_metric.update_dict(labels_, preds) File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 91, in update_dict self.update(label, pred) File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 373, in update check_label_shapes(labels, preds) File "/home/ubuntu/verify/mxnet/python/mxnet/metric.py", line 24, in check_label_shapes "predictions {}".format(label_shape, pred_shape)) ValueError: Shape of labels 1 does not match shape of predictions 2 ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
With regards, Apache Git Services