Hi Tobias.
The neural net uses a different implementation of SGD than the
SGDClassifier (and the derived ones).
The objectives in this SGD are always convex, so there are no local
minima. I have not seen any paper
showing actual benefits in using minibatches for this kind of optimization.
This might be "not much code" but it is code nonetheless. If we added
all the features that are "not much code"
we will very quickly have A LOT more code. And having another parameter
to optimize over doesn't sound like
an advantage to me at all.
Feel free to prove me wrong by providing an implementation that is
faster than the current one using mini-batches ;)
Cheers,
Andy
On 05/04/2014 10:45 AM, Tobias Günther wrote:
Hi sklearners!
I think mini-batch training and averaging are both desirable features
for the SGD class.
As far as I know using mini-batches can prevent getting stuck in local
minima and might
converge faster than plain SGD. Also, there is the possibility to
parallelize the gradient
computation inside one mini-batch or use a matrix multiplication, so
it might speed up
computation as well.
I don't know what the coming NN class in sklearn uses, but I think
mini-batch SGD is often
used in NN training, so if the implementation uses the SGD class, it
could be valuable for
the performance of the NN in sklearn as well.
Anyway, as it is not too much code to add and batch-size just another
parameter for the
class, that can be turned off (or be 1) by default, optimized like any
other hyperparameter
and easily described to new users I see only advantages in adding it.
Best,
Tobias
On Sun, May 4, 2014 at 9:48 AM, Alexandre Gramfort
<alexandre.gramf...@telecom-paristech.fr
<mailto:alexandre.gramf...@telecom-paristech.fr>> wrote:
hi sklearners,
FYI Danny Sullivan https://github.com/dsullivan7
will work at Telecom ParisTech with me as a scikit-learn
engineer starting this summer. These topics
(SGD improvements, averaging, SAG etc.) are part
of the roadmap.
I think he will start by setting up a benchmark for online
supervised estimators on sparse and dense data and
then bench the different stochastic variants.
However I am sure that any work done before he starts
this summer will be useful :)
Before Danny actually starts we'll send an email to the list
to detail the plan and get feedback.
Best,
Alex
On Sat, May 3, 2014 at 7:08 PM, Andy <t3k...@gmail.com
<mailto:t3k...@gmail.com>> wrote:
> On 05/03/2014 05:37 PM, Mathieu Blondel wrote:
>
> Same feeling as Andy. I'd favor implementing averaging instead.
>
> I totally forgot that we don't have averaging yet ^^
> I'd be in favor of geometric averaging:
http://arxiv.org/abs/1212.2002
>
>
>
>
> Mathieu
>
>
> On Sun, May 4, 2014 at 12:23 AM, Andy <t3k...@gmail.com
<mailto:t3k...@gmail.com>> wrote:
>>
>> Hi Sean.
>> For linear classifiers I'm not really aware of benefits in doing
>> mini-batch training, and I don't think it is widely used
(someone correct me
>> if I'm wrong).
>> Usually we only like to add features that have a clear benefit
for the
>> users, to prevent scikit-learn from becoming bloated.
>>
>> Do you have a particular use-case where it is important?
>>
>> Cheers,
>> Andy
>>
>>
>> On 05/01/2014 02:04 AM, Sean Violante wrote:
>>
>> Hi
>>
>> I was wondering if there is any interest in implementing
minibatch/batch
>> for the SGD algorithm. As I understand it, this is not implemented
>>
>> "There is a compromise between the two forms, which is often called
>> "mini-batches", where the true gradient is approximated by a
sum over a
>> small number of training examples."
>>
>> http://en.wikipedia.org/wiki/Stochastic_gradient_descent
>>
>>
>>
>> This would be doing a partial_fit (on small number of training
examples)
>> but updating the weights only after each epoch rather than
after each
>> training sample
>>
>>
>> as far as I can see it would only require a flag in the
sgd_fast.pyx code.
>>
>> thanks
>>
>> Sean
>>
>>
>>
>>
------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing -
For FREE
>> Instantly run your Selenium tests across 300+ browser/OS
combos. Get
>> unparalleled scalability from the best Selenium testing platform
>> available.
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>>
>>
>>
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>>
>>
>>
>>
------------------------------------------------------------------------------
>> "Accelerate Dev Cycles with Automated Cross-Browser Testing -
For FREE
>> Instantly run your Selenium tests across 300+ browser/OS
combos. Get
>> unparalleled scalability from the best Selenium testing platform
>> available.
>> Simple to use. Nothing to install. Get started now for free."
>> http://p.sf.net/sfu/SauceLabs
>> _______________________________________________
>> Scikit-learn-general mailing list
>> Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>>
>
>
>
>
------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing -
For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
Get
> unparalleled scalability from the best Selenium testing platform
available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
>
>
>
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
>
>
>
------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing -
For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos.
Get
> unparalleled scalability from the best Selenium testing platform
available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Scikit-learn-general mailing list
> Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
>
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform
available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
<mailto:Scikit-learn-general@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
------------------------------------------------------------------------------
"Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
Instantly run your Selenium tests across 300+ browser/OS combos. Get
unparalleled scalability from the best Selenium testing platform available.
Simple to use. Nothing to install. Get started now for free."
http://p.sf.net/sfu/SauceLabs
_______________________________________________
Scikit-learn-general mailing list
Scikit-learn-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
