safrooze opened a new pull request #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705 Output of the benchmark is sent to stderr ## Description ## Benchmark loops through two batch-sizes (100,000 and 1,000,000) and two output dimensions (100 and 500) and generates random data on CPU and GPU and calls metric.update() on a list of metrics with the generated date. ## Checklist ## ### Essentials ### - [x] Passed code style checking (`make lint`) - [x] Changes are complete (i.e. I finished coding on this PR) - [x] All changes have test coverage: - [x] Code is well-documented: - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [ ] Added unit-test for benchmarking metric performance. ## Comments ## - Unit-test passes without GPU, but fails if GPU memory allocation fails - The output looks like this on a p2.x instance ``` mx.metric benchmarks Metric Ctx Batch Size Output Dim Elapsed Time ---------------------------------------------------------------------- acc cpu(0) 100000 100 0.069804 acc gpu(0) 100000 100 0.0055592 ---------------------------------------------------------------------- acc cpu(0) 100000 500 0.29323 acc gpu(0) 100000 500 0.034261 ---------------------------------------------------------------------- acc cpu(0) 1000000 100 0.66856 acc gpu(0) 1000000 100 0.057442 ---------------------------------------------------------------------- acc cpu(0) 1000000 500 2.9239 acc gpu(0) 1000000 500 0.27827 ---------------------------------------------------------------------- top_k_acc cpu(0) 100000 100 0.39707 top_k_acc gpu(0) 100000 100 0.39684 ---------------------------------------------------------------------- top_k_acc cpu(0) 100000 500 2.6537 top_k_acc gpu(0) 100000 500 2.6574 ---------------------------------------------------------------------- top_k_acc cpu(0) 1000000 100 4.0662 top_k_acc gpu(0) 1000000 100 4.0537 ---------------------------------------------------------------------- top_k_acc cpu(0) 1000000 500 26.581 top_k_acc gpu(0) 1000000 500 26.594 ---------------------------------------------------------------------- F1 cpu(0) 100000 2 0.2515 F1 gpu(0) 100000 2 0.25105 ---------------------------------------------------------------------- F1 cpu(0) 100000 2 0.25086 F1 gpu(0) 100000 2 0.24956 ---------------------------------------------------------------------- F1 cpu(0) 1000000 2 2.509 F1 gpu(0) 1000000 2 2.5127 ---------------------------------------------------------------------- F1 cpu(0) 1000000 2 2.5107 F1 gpu(0) 1000000 2 2.5094 ---------------------------------------------------------------------- Perplexity cpu(0) 100000 100 0.0058115 Perplexity gpu(0) 100000 100 0.0030518 ---------------------------------------------------------------------- Perplexity cpu(0) 100000 500 0.0054376 Perplexity gpu(0) 100000 500 0.0070541 ---------------------------------------------------------------------- Perplexity cpu(0) 1000000 100 0.042403 Perplexity gpu(0) 1000000 100 0.003443 ---------------------------------------------------------------------- Perplexity cpu(0) 1000000 500 0.041232 Perplexity gpu(0) 1000000 500 0.051778 ---------------------------------------------------------------------- MAE cpu(0) 100000 100 0.058175 MAE gpu(0) 100000 100 0.056117 ---------------------------------------------------------------------- MAE cpu(0) 100000 500 0.26928 MAE gpu(0) 100000 500 0.26553 ---------------------------------------------------------------------- MAE cpu(0) 1000000 100 0.53227 MAE gpu(0) 1000000 100 0.52565 ---------------------------------------------------------------------- MAE cpu(0) 1000000 500 2.6206 MAE gpu(0) 1000000 500 2.607 ---------------------------------------------------------------------- MSE cpu(0) 100000 100 0.041658 MSE gpu(0) 100000 100 0.041626 ---------------------------------------------------------------------- MSE cpu(0) 100000 500 0.215 MSE gpu(0) 100000 500 0.21492 ---------------------------------------------------------------------- MSE cpu(0) 1000000 100 0.43541 MSE gpu(0) 1000000 100 0.42094 ---------------------------------------------------------------------- MSE cpu(0) 1000000 500 2.1183 MSE gpu(0) 1000000 500 2.1229 ---------------------------------------------------------------------- RMSE cpu(0) 100000 100 0.042453 RMSE gpu(0) 100000 100 0.041688 ---------------------------------------------------------------------- RMSE cpu(0) 100000 500 0.21422 RMSE gpu(0) 100000 500 0.21395 ---------------------------------------------------------------------- RMSE cpu(0) 1000000 100 0.43216 RMSE gpu(0) 1000000 100 0.42024 ---------------------------------------------------------------------- RMSE cpu(0) 1000000 500 2.1158 RMSE gpu(0) 1000000 500 2.1298 ---------------------------------------------------------------------- ce cpu(0) 100000 100 0.017465 ce gpu(0) 100000 100 0.016886 ---------------------------------------------------------------------- ce cpu(0) 100000 500 0.084103 ce gpu(0) 100000 500 0.080693 ---------------------------------------------------------------------- ce cpu(0) 1000000 100 0.19837 ce gpu(0) 1000000 100 0.1848 ---------------------------------------------------------------------- ce cpu(0) 1000000 500 0.81667 ce gpu(0) 1000000 500 0.8098 ---------------------------------------------------------------------- nll_loss cpu(0) 100000 100 0.018017 nll_loss gpu(0) 100000 100 0.016982 ---------------------------------------------------------------------- nll_loss cpu(0) 100000 500 0.083593 nll_loss gpu(0) 100000 500 0.080484 ---------------------------------------------------------------------- nll_loss cpu(0) 1000000 100 0.19791 nll_loss gpu(0) 1000000 100 0.1856 ---------------------------------------------------------------------- nll_loss cpu(0) 1000000 500 0.81095 nll_loss gpu(0) 1000000 500 0.81938 ---------------------------------------------------------------------- pearsonr cpu(0) 100000 100 0.57283 pearsonr gpu(0) 100000 100 0.22794 ---------------------------------------------------------------------- pearsonr cpu(0) 100000 500 2.2202 pearsonr gpu(0) 100000 500 1.1238 ---------------------------------------------------------------------- pearsonr cpu(0) 1000000 100 4.4207 pearsonr gpu(0) 1000000 100 2.2353 ---------------------------------------------------------------------- pearsonr cpu(0) 1000000 500 21.999 pearsonr gpu(0) 1000000 500 11.147 ---------------------------------------------------------------------- ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
