[GitHub] piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ
piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ URL: https://github.com/apache/incubator-mxnet/pull/8766#discussion_r152675141 ## File path: docs/faq/gradient_compression.md ## @@ -0,0 +1,107 @@ +# Gradient Compression + +Gradient Compression reduces communication bandwidth to make distributed training with GPUs more scalable and efficient without significant loss in convergence rate or accuracy. + + +## Benefits + +**Increased Speed** + +For architectures with fully connected components, the gradient compression capability is observed to speedup training by about 2x, depending on the size of the model and the network bandwidth of the instance. Bigger models see larger speedup with gradient compression. + +**Minimal Accuracy Loss** + +Gradient compression uses the approach of delaying the synchronization of weight updates which are small. Although small weight updates might not be sent for that batch, this information is not discarded. Once the weight updates for this location accumulate to become a larger value, they will be propagated. Since there is no information loss, but only delayed updates, it does not lead to a significant loss in accuracy or convergence rate. In distributed training experiments[1], the accuracy loss observed due to gradient compression was as low as 1% + + +## When to Use Gradient Compression + +When training models whose architectures include large fully connected components, it can be helpful to use gradient compression. For larger models, the communication cost becomes a major factor. Such models stand to benefit greatly with gradient compression. Review comment: Common in recurrent neural networks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ
piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ URL: https://github.com/apache/incubator-mxnet/pull/8766#discussion_r152674943 ## File path: docs/faq/gradient_compression.md ## @@ -0,0 +1,107 @@ +# Gradient Compression + +Gradient Compression reduces communication bandwidth to make distributed training with GPUs more scalable and efficient without significant loss in convergence rate or accuracy. + + +## Benefits + +**Increased Speed** + +For architectures with fully connected components, the gradient compression capability is observed to speedup training by about 2x, depending on the size of the model and the network bandwidth of the instance. Bigger models see larger speedup with gradient compression. Review comment: fully connected components -> fully connected layers This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ
piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ URL: https://github.com/apache/incubator-mxnet/pull/8766#discussion_r152674798 ## File path: docs/tutorials/basic/ndarray_indexing.md ## @@ -0,0 +1,377 @@ + +# NDArray Indexing - Array indexing features Review comment: why not merge this into the ndarray tutorial? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ
piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ URL: https://github.com/apache/incubator-mxnet/pull/8766#discussion_r152674342 ## File path: docs/faq/gradient_compression.md ## @@ -0,0 +1,107 @@ +# Gradient Compression + +Gradient Compression reduces communication bandwidth to make distributed training with GPUs more scalable and efficient without significant loss in convergence rate or accuracy. Review comment: Also there is no guarantee that you won't loss convergence or accuracy significantly. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ
piiswrong commented on a change in pull request #8766: NDArray Indexing tutorial and Gradient Compression FAQ URL: https://github.com/apache/incubator-mxnet/pull/8766#discussion_r152674195 ## File path: docs/faq/gradient_compression.md ## @@ -0,0 +1,107 @@ +# Gradient Compression + +Gradient Compression reduces communication bandwidth to make distributed training with GPUs more scalable and efficient without significant loss in convergence rate or accuracy. Review comment: Its not restricted to distributed or GPUs This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
