## Description ## SVRG stands for Stochastic Variance Reduced Gradient, which is an optimization technique that complements SGD. It is introduced in the paper _[Accelerating Stochastic Gradient Descent using Predicative Variance Reduction](https://papers.nips.cc/paper/4937-accelerating-stochastic-gradient-descent-using-predictive-variance-reduction.pdf)_ in 2013. Key Characteristics of SVRG: - Explicit Variance Reduction - Converge much faster compared to SGD on strongly convex smooth functions
SVRG Module should: - seamlessly supports both dense and sparse data, run on CPU and GPU instances on single machine and in distributed setting. - minimize the API differences with Python Module API. ## Checklist ## ### Essentials ### Please feel free to remove inapplicable items for your PR. - [ ] The PR title starts with [MXNET-854], where MXNET-854 refers to the relevant [JIRA issue](https://issues.apache.org/jira/browse/MXNET-854) created (except PRs with tiny changes) - [ ] All changes have test coverage: - [ ] Code is well-documented: - For user-facing API changes, API doc string has been updated. - For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable - [ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - SVRGModule inherits from Python Module and minimizes the API changes of Module API. Both intermediate level and high level APIs are supported - Functional tests and unittests have been tested for SVRGModule and SVRG Optimizer. ## Comments ## - Design decisions of why using Python Module instead of Optimizer can be found in the design doc [here](https://cwiki.apache.org/confluence/display/MXNET/SVRG+Optimization+in+MXNet+Python+Module) - This version of SVRGModule supports single machine run with single cpu, single gpu and multiple-gpus. Distributed SVRGModule is soon to be updated. - Benchmarking has been performed on this single machine version SVRGModule on single cpu on linear regression model. Distributed SVRGModule to be updated soon. Benchmarking results can be found in contrib/svrg_optimization_python/benchmarks. [ Full content available at: https://github.com/apache/incubator-mxnet/pull/12376 ] This message was relayed via gitbox.apache.org for [email protected]
