[GitHub] safrooze commented on issue #9705: Added unittest for benchmarking metric performance
safrooze commented on issue #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705#issuecomment-369331546 I'll move this to nightly tests then. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] safrooze commented on issue #9705: Added unittest for benchmarking metric performance
safrooze commented on issue #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705#issuecomment-364517332 @eric-haibin-lin You're correct that nightly would be a more suitable place. One concern with nightly was that community wouldn't be able to see the results of the benchmark. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] safrooze commented on issue #9705: Added unittest for benchmarking metric performance
safrooze commented on issue #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705#issuecomment-363977854 OK I think I addressed all the feedback: * random is seeded * nd.wait_all() used before starting timing and before ending timing * Added batch-size values of 16, 64, 256, and 1024 * Datasize varies by number of output channels to keep total runtime down to a few minutes The modified code output looks like this: ``` Metric Data-Ctx Label-Ctx Data Size Batch Size Output Dim Elapsed Time -- acccpu(0)cpu(0) 131072 16 128 1.0015 acccpu(0)gpu(0) 131072 16 128 1.682 accgpu(0)cpu(0) 131072 16 128 2.6263 accgpu(0)gpu(0) 131072 16 128 3.3028 -- acccpu(0)cpu(0) 131072 64 128 0.42843 acccpu(0)gpu(0) 131072 64 128 0.568 accgpu(0)cpu(0) 131072 64 128 0.78586 accgpu(0)gpu(0) 131072 64 128 0.94317 -- acccpu(0)cpu(0) 131072 256128 0.19074 acccpu(0)gpu(0) 131072 256128 0.24228 accgpu(0)cpu(0) 131072 256128 0.21548 accgpu(0)gpu(0) 131072 256128 0.25075 -- acccpu(0)cpu(0) 131072 1024 128 0.1303 acccpu(0)gpu(0) 131072 1024 128 0.14127 accgpu(0)cpu(0) 131072 1024 128 0.055079 accgpu(0)gpu(0) 131072 1024 128 0.065515 -- ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] safrooze commented on issue #9705: Added unittest for benchmarking metric performance
safrooze commented on issue #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705#issuecomment-363977854 OK I think I addressed all the feedback: * random is seeded * nd.wait_all() used before starting timing and before ending timing * Added batch-size values of 16, 64, 256, and 1024 * Datasize varies by number of output channels to keep total runtime down to a few minutes The modified code output looks like this: ``` Metric Data-Ctx Label-Ctx Data Size Batch Size Output Dim Elapsed Time -- acccpu(0)cpu(0) 131072 16 128 1.0015 acccpu(0)gpu(0) 131072 16 128 1.682 accgpu(0)cpu(0) 131072 16 128 2.6263 accgpu(0)gpu(0) 131072 16 128 3.3028 -- acccpu(0)cpu(0) 131072 64 128 0.42843 acccpu(0)gpu(0) 131072 64 128 0.568 accgpu(0)cpu(0) 131072 64 128 0.78586 accgpu(0)gpu(0) 131072 64 128 0.94317 -- ``` This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] safrooze commented on issue #9705: Added unittest for benchmarking metric performance
safrooze commented on issue #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705#issuecomment-363328924 The intention is to observe a measurable elapsed time (hence large data size) and amplify the difference between CPU and GPU processing (hence processing all the data in one batch). A valid alternative is to use small batch size and iterate over multiple batches. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] safrooze commented on issue #9705: Added unittest for benchmarking metric performance
safrooze commented on issue #9705: Added unittest for benchmarking metric performance URL: https://github.com/apache/incubator-mxnet/pull/9705#issuecomment-363302529 @szha Please review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services