Lizonghang opened a new issue #12640: Time bottleneck in forward_backward and next(train_iter) URL: https://github.com/apache/incubator-mxnet/issues/12640 ## Description In BaseModule.fit function, <code>forward_backward</code> only needs 0.00055s, <code>update</code> only needs 0.079953s, <code>next(data_iter)</code> only needs 0.000265s. However, we run <code>forward_backward</code> and <code>next(data_iter)</code> together and spend 2.919867s. Could you please tell me what is the reason of this time bottleneck? ## Environment info (Required) Platform: MacOS Language: Python 2.7 Platform Version: MXNet 1.2.0 Device: 2.9 GHz Intel Core i5 (The test runs using single CPU) Memory: 8 GB 1867 MHz DDR3 Install Tool: pip ## Steps to reproduce (In BaseModule.fit function) * Run <code>forward_backward</code> only: ``` while True: data_batch = next_data_batch calc_start = time.time() self.forward_backward(data_batch) calc_end = time.time() self.logger.info("[CALC TIME] %f", calc_end - calc_start) ``` result: [CALC TIME] 0.000551 * Run <code>next(data_iter)</code> only: ``` while True: data_iter_start = time.time() next(data_iter) data_iter_end = time.time() self.logger.info("[DATAITER TIME] %f", data_iter_end - data_iter_start) ``` result: [DATAITER TIME] 0.000265 * Run <code>forward_backward</code> and <code>next(data_iter)</code> together: ``` while True: data_batch = next_data_batch calc_iter_start = time.time() self.forward_backward(data_batch) next_data_batch = next(data_iter) calc_iter_end = time.time() self.logger.info("[CALC AND DATAITER TIME] %f", calc_iter_end - calc_iter_start) ``` result: [CALC AND DATAITER TIME] 2.919867 ## What have you tried to solve it? 1. I read the source code of <code>forward_backward</code> and <code>ImageRecordIter</code> but have no idea. 2. I found little modify in <code>data_batch</code> variable would also cause the time bottleneck in the loop of <code>forward_backward</code>. Thanks a lot for your kind !
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
