chrishkchris opened a new pull request #560: SINGA-487 Accumulate gradients to reduce network latency URL: https://github.com/apache/singa/pull/560 This PR reduces the network latency by accumulate gradients in a memory buffer before sending out with NCCL. This can reduce much of the TCP/IP latency by reducing the number of NCCL API call. Together with the result of PR #555, here is a simple test to make sure the training is correct: ``` ubuntu@ip-172-31-26-214:~/singa/examples/autograd$ python3 mnist_multiprocess.py Starting Epoch 0: Training loss = 831.072205, training accuracy = 0.700454 Evaluation accuracy = 0.927015, Elapsed Time = 0.676089s Starting Epoch 1: Training loss = 248.684601, training accuracy = 0.916183 Evaluation accuracy = 0.958265, Elapsed Time = 0.545179s Starting Epoch 2: Training loss = 172.330597, training accuracy = 0.943042 Evaluation accuracy = 0.967928, Elapsed Time = 0.543617s Starting Epoch 3: Training loss = 139.254807, training accuracy = 0.953425 Evaluation accuracy = 0.973067, Elapsed Time = 0.530805s Starting Epoch 4: Training loss = 115.329491, training accuracy = 0.960737 Evaluation accuracy = 0.976049, Elapsed Time = 0.530590s Starting Epoch 5: Training loss = 101.911728, training accuracy = 0.966179 Evaluation accuracy = 0.974095, Elapsed Time = 0.529574s Starting Epoch 6: Training loss = 90.820244, training accuracy = 0.969969 Evaluation accuracy = 0.980983, Elapsed Time = 0.530502s Starting Epoch 7: Training loss = 86.718071, training accuracy = 0.971037 Evaluation accuracy = 0.977590, Elapsed Time = 0.531085s Starting Epoch 8: Training loss = 79.507553, training accuracy = 0.973675 Evaluation accuracy = 0.976562, Elapsed Time = 0.529935s Starting Epoch 9: Training loss = 78.784409, training accuracy = 0.974025 Evaluation accuracy = 0.980469, Elapsed Time = 0.530919s ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
