Hi, The practical way to run parallel SGD is to use a constant stepsize per epoch, and then set the step size \eta <- \eta * \rho for some \rho \in (0,1) (typical value is like 0.9). I would not recommend changing the step size at every time. See the discussion in Section 5 (Robust 1/k rates).
I don't understand your second question since there shouldn't be locks? On Tue, Mar 7, 2017 at 5:57 AM, 孟憲妤 <[email protected]> wrote: > Hello, > > I'm HsienYu Meng from Taiwan. I'm currently in my first year of graduate > studies at Tsinghua Univ. As you mentioned, the SGD project should > implement HOGWILD! algorithm and consider to set stepsize at each > iteration. However, the paper indicates that they assume the stepsize to be > a constant value so that the formula 7 make sense. I wonder if we change > the stepsize at each time, will the properties of HOGWILD! still holds? > Another question is, if the dataset is large and we partition it in memory, > which kind of losk-release should we apply? Sequential consistency, Release > consistency or lazy release consistency? > > Appreciate for your patience .Thanks! > > -- > *孟憲妤* > > *MengHsienyu**Department of Computer Science ,Tsinghua university * > *Beijing ,China* > > *100084* > > *Mobile:(+86)18201162149 <+86%20182%200116%202149>* > * (+886)0952424693 <+886%20952%20424%20693>* > > > _______________________________________________ > mlpack mailing list > [email protected] > http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack >
_______________________________________________ mlpack mailing list [email protected] http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
