ThomasDelteil commented on issue #9571: ACCURACY IS USING NUMPY, URGENT FIX URL: https://github.com/apache/incubator-mxnet/issues/9571#issuecomment-374411651 I see how that could be a problem, but I think that philosophically MXNet should let the users choose when to block. Typically I usually block after 100 batches or on every epoch. In a typical GPU with GBs of RAMs, the memory footprints of a few batches is negligible compared to the size of the network. Also blocking or not blocking, I think we can get good gains by storing the accuracy on the right context. When running on GPU, I got a 100% improvement by storing the accuracy on the GPU. Here is an example where I benchmarked the network trained in this tutorial: https://github.com/zackchase/mxnet-the-straight-dope/blob/master/chapter04_convolutional-neural-networks/cnn-gluon.ipynb That is the base implementation of the accuracy evaluation loop using mx.metric.accuracy ``` %%time test_accuracy = evaluate_accuracy_blocking(train_data, net) ``` ``` CPU times: user 1min 3s, sys: 2 s, total: 1min 5s Wall time: 24.9 s ``` That is my implementation using only non-blocking operations ``` %%time test_accuracy = evaluate_accuracy_non_blocking(train_data, net) ``` ``` CPU times: user 50.5 s, sys: 1.24 s, total: 51.7 s Wall time: 12 s ``` The same implementation but blocking on preds.wait_to_read() in the `.update()` function ``` %%time test_accuracy = evaluate_accuracy_wait_to_read(train_data, net) ``` ``` CPU times: user 56.6 s, sys: 1.18 s, total: 57.7 s Wall time: 13.5 s ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
