ConjugateGradient comes under 1 or 2may be we need 4 options...
>> >> >
>> >> > Thanks.
>> >> > Deb
>> >> >
>> >> >
>> >> > On Mon, Apr 7, 2014 at 11:23 PM, Debasish Das
>> >> >
>> >> > wrote:
>&
>> >> with LBFGS right ? Are you adding it to regression package as well ?
> >> >>
> >> >> Thanks.
> >> >> Deb
> >> >>
> >> >>
> >> >> On Mon, Apr 7, 2014 at 7:00 PM, DB Tsai wrote:
> >> >&g
PM, DB Tsai wrote:
>> >>>
>> >>> Hi guys,
>> >>>
>> >>> The latest PR uses Breeze's L-BFGS implement which is introduced by
>> >>> Xiangrui's sparse input format work in SPARK-1212.
>> >>>
>&
l/353
> >>>
> >>> Now, it works with the new sparse framework!
> >>>
> >>> Any feedback would be greatly appreciated.
> >>>
> >>> Thanks.
> >>>
> >>> Sincerely,
> >>>
> >>> DB Tsai
>
.
>>>
>>> Sincerely,
>>>
>>> DB Tsai
>>> -------------------
>>> My Blog: https://www.dbtsai.com
>>> LinkedIn: https://www.linkedin.com/in/dbtsai
>>>
>>>
>>> On Thu, Apr 3,
; Any feedback would be greatly appreciated.
>>>
>>> Thanks.
>>>
>>> Sincerely,
>>>
>>> DB Tsai
>>> -------------------
>>> My Blog: https://www.dbtsai.com
>>> LinkedIn: https:
btsai.com
>> LinkedIn: https://www.linkedin.com/in/dbtsai
>>
>>
>> On Thu, Apr 3, 2014 at 5:02 PM, DB Tsai wrote:
>> > -- Forwarded message --
>> > From: David Hall
>> > Date: Sat, Mar 15, 2014 at 10:02 AM
>> > Subject:
gt;>
>>
>> On Thu, Apr 3, 2014 at 5:02 PM, DB Tsai wrote:
>> > -- Forwarded message --
>> > From: David Hall
>> > Date: Sat, Mar 15, 2014 at 10:02 AM
>> > Subject: Re: MLLib - Thoughts about refactoring Updater for LBFGS?
>> > To:
d message --
> > From: David Hall
> > Date: Sat, Mar 15, 2014 at 10:02 AM
> > Subject: Re: MLLib - Thoughts about refactoring Updater for LBFGS?
> > To: DB Tsai
> >
> >
> > On Fri, Mar 7, 2014 at 10:56 PM, DB Tsai wrote:
> >>
> >&g
gt; Subject: Re: MLLib - Thoughts about refactoring Updater for LBFGS?
> To: DB Tsai
>
>
> On Fri, Mar 7, 2014 at 10:56 PM, DB Tsai wrote:
>>
>> Hi David,
>>
>> Please let me know the version of Breeze that LBFGS can be serialized,
>> and CachedDiffFunctio
Thanks for the pointers.
That's the plan...I am going to have default ALS, variable projection at
each ALS step, and then using the constrained first order solver that you
have...
I am hoping ECOS QP will beat all of them...I don't know...some of these
obvious assumptions does not work when the d
On Mon, Mar 31, 2014 at 10:15 AM, Debasish Das wrote:
> I added eclipse support in my qp branch:
>
> https://github.com/debasish83/breeze/tree/qp
Ok, great. Totally fine for this to come in with the QP.
>
>
> For the QP solver I will look into this solver http://www.joptimizer.com/
>
> Right n
I added eclipse support in my qp branch:
https://github.com/debasish83/breeze/tree/qp
For the QP solver I will look into this solver http://www.joptimizer.com/
Right now my plan is to use Professor Boyd's ECOS solver which is also
designed in the very similar lines but has been tested to solve e
On Sun, Mar 30, 2014 at 2:01 PM, Debasish Das wrote:
> Hi David,
>
> I have started to experiment with BFGS solvers for Spark GLM over large
> scale data...
>
> I am also looking to add a good QP solver in breeze that can be used in
> Spark ALS for constraint solves...More details on that soon...
Hi David,
I have started to experiment with BFGS solvers for Spark GLM over large
scale data...
I am also looking to add a good QP solver in breeze that can be used in
Spark ALS for constraint solves...More details on that soon...
I could not load up breeze 0.7 code onto eclipse...There is a fol
Hi Xiangrui,
I think it doesn't matter whether we use Fortran/Breeze/RISO for
optimizers since optimization only takes << 1% of time. Most of the
time is in gradientSum and lossSum parallel computation.
Sincerely,
DB Tsai
Machine Learning Engineer
Alpine Data Labs
---
Hi DB,
Thanks for doing the comparison! What were the running times for
fortran/breeze/riso?
Best,
Xiangrui
On Thu, Mar 6, 2014 at 4:21 PM, DB Tsai wrote:
> Hi David,
>
> I can converge to the same result with your breeze LBFGS and Fortran
> implementations now. Probably, I made some mistakes w
On Thu, Mar 6, 2014 at 4:26 PM, David Hall wrote:
> I'm not sure why Spark should be serializing LBFGS? Shouldn't it live on
> the controller node? Or is this a per-node thing?
>
> But no problem to make it serializable.
It will live in the controller node. Only RDD operations are per-node
thing.
On Thu, Mar 6, 2014 at 4:21 PM, DB Tsai wrote:
> Hi David,
>
> I can converge to the same result with your breeze LBFGS and Fortran
> implementations now. Probably, I made some mistakes when I tried
> breeze before. I apologize that I claimed it's not stable.
>
> See the test case in BreezeLBFGSS
Hi David,
I can converge to the same result with your breeze LBFGS and Fortran
implementations now. Probably, I made some mistakes when I tried
breeze before. I apologize that I claimed it's not stable.
See the test case in BreezeLBFGSSuite.scala
https://github.com/AlpineNow/spark/tree/dbtsai-bre
I did not. They would be nice to have.
On Wed, Mar 5, 2014 at 5:21 PM, Debasish Das wrote:
> David,
>
> There used to be standard BFGS testcases in Professor Nocedal's
> package...did you stress test the solver with them ?
>
> If not I will shoot him an email for them.
>
> Thanks.
> Deb
>
>
>
>
David,
There used to be standard BFGS testcases in Professor Nocedal's
package...did you stress test the solver with them ?
If not I will shoot him an email for them.
Thanks.
Deb
On Wed, Mar 5, 2014 at 2:00 PM, David Hall wrote:
> On Wed, Mar 5, 2014 at 1:57 PM, DB Tsai wrote:
>
> > Hi Dav
On Wed, Mar 5, 2014 at 1:57 PM, DB Tsai wrote:
> Hi David,
>
> On Tue, Mar 4, 2014 at 8:13 PM, dlwh wrote:
> > I'm happy to help fix any problems. I've verified at points that the
> > implementation gives the exact same sequence of iterates for a few
> different
> > functions (with a particular
Hi David,
On Tue, Mar 4, 2014 at 8:13 PM, dlwh wrote:
> I'm happy to help fix any problems. I've verified at points that the
> implementation gives the exact same sequence of iterates for a few different
> functions (with a particular line search) as the c port of lbfgs. So I'm a
> little surpris
t; > >
> > >> c. Few variants of line searches : I will discuss on it.
> > >> For the dbtsai-lbfgs branch seems like it already got merged by
> Jenkins.
> > > I don't think it's merged into master. Still have couple things needed
> > > to be cleaned up. Just open the PR to have public feedback.
> > >
> > >> Is this getting merged to the master or there will be revisions on it
> ?
> > >>
> > >> https://github.com/apache/spark/pull/53
> > >>
> > >> Thanks.
> > >> Deb
> > >
> > > Sincerely,
> > >
> > > DB Tsai
> > > Machine Learning Engineer
> > > Alpine Data Labs
> > > --
> > > Web: http://alpinenow.com/
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-spark-developers-list.1001551.n3.nabble.com/MLLib-Thoughts-about-refactoring-Updater-for-LBFGS-tp2493p3935.html
> > Sent from the Apache Spark Developers List mailing list archive at
> > Nabble.com.
> >
>
>
> >> Is this getting merged to the master or there will be revisions on it ?
> >>
> >> https://github.com/apache/spark/pull/53
> >>
> >> Thanks.
> >> Deb
> >
> > Sincerely,
> >
> > DB Tsai
> > Machine Learning Engineer
> > Alpine Data Labs
> > --
> > Web: http://alpinenow.com/
>
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/MLLib-Thoughts-about-refactoring-Updater-for-LBFGS-tp2493p3935.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
erged to the master or there will be revisions on it ?
>>
>> https://github.com/apache/spark/pull/53
>>
>> Thanks.
>> Deb
>
> Sincerely,
>
> DB Tsai
> Machine Learning Engineer
> Alpine Data Labs
> --
> Web: http://alpinenow.com/
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/MLLib-Thoughts-about-refactoring-Updater-for-LBFGS-tp2493p3935.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Hi Xiangrui,
It seems that Robert is busy recently. I setup the org.riso in
maven central for him, and I was waiting for his response for
a while without any news. So, I decided to maintain myself.
I'm more favor of using breeze for both sparse and optimization
core math library. When I tried L-B
Hi DB,
I saw you released the L-BFGS code under com.dbtsai.lbfgs on maven
central, so I assume that Robert (the author of RISO) is not going to
maintain it. Is it correct?
For the breeze implementation, do you mind sharing more details about
the issues you have?
I saw the hack you did to get reg
Hi Deb,
I've been working with David to add or enhance some features to breeze
to make its performance comparable to bare-bone implementations. I'm
going to update that PR this week with sparse support to KMeans. You
are certainly welcome to update the GLM part. Make sure you are using
the master
Yeah we should move f2j L-BFGS and L-BFGS-B to breeze..they already have 2
line searches..also the OWL-QN outline...
Hi Xiangrui,
What's the plan on the PR ?
https://github.com/apache/incubator-spark/pull/575
Will you add breeze as a dependency for the sparse support ?
I looked at your branch
h
Hi Deb,
I had tried breeze L-BFGS algorithm, and when I tried it couple weeks
ago, it's not as stable as the fortran implementation. I guessed the
problem is in the line search related thing. Since we may bring breeze
dependency for the sparse format support as you pointed out, we can
just try to
Hi Deb,
> a. OWL-QN for solving L1 natively in BFGS
Based on what I saw from
https://github.com/tjhunter/scalanlp-core/blob/master/learn/src/main/scala/breeze/optimize/OWLQN.scala
, it seems that it's not difficult to implement OWL-QN once LBFGS is
done.
>
> b. Bound constraints in BFGS : I saw
Hi DB,
I have forked the LBFGS repo. The features that I want in it are the
following:
a. OWL-QN for solving L1 natively in BFGS
b. Bound constraints in BFGS : I saw you have converted the fortran code.
Is there a license issue ? I can help in getting that up to speed as well.
c. Few variants
Hi Deb,
1) The LBFGS code I published to maven central is here,
https://github.com/dbtsai/lbfgs You can fork it, and publish it
locally so that mllib can use it. I also want to add orthant wise for
L1, so we can work together to make it happen.
I think ideally, we can rewrite it in scala, and mo
Hi DB,
1. Could you point to the BFGS repositories used to publish artifacts to
maven central ? What's the best way to add changes to it ? I fork the repo
at my github ? Basically as I mentioned before I need to add lbfgs-b,
orthant wise for L1 handling and few variants of line search to lbfgs...
Hi Deb,
The PR is here
https://github.com/apache/spark/pull/53
Hi Evan,
I think we need to refactor the optimization methods and also the way
we write algorithms. For example, if I want to use the new
optimization method in LogisticRegression.scala, I need to implement
LogisticRegressionWithLBF
Hi everyone,
Sorry I'm late to the thread here, but I want to point out a few things.
This is, of course, a most welcome contribution and it will be immediately
useful to everything currently using the stochastic gradient optimizers!
1) I'm all for refactoring the optimization methods to make the
Hi DB,
Could you please point me to your spark PR ?
Thanks.
Deb
On Tue, Feb 25, 2014 at 5:03 PM, DB Tsai wrote:
> Hi Deb, Xiangrui
>
> I just moved the LBFGS code to maven central, and cleaned up the code
> a little bit.
>
> https://github.com/AlpineNow/incubator-spark/commits/dbtsai-LBFGS
>
Hi Deb, Xiangrui
I just moved the LBFGS code to maven central, and cleaned up the code
a little bit.
https://github.com/AlpineNow/incubator-spark/commits/dbtsai-LBFGS
After looking at Mallet, the api is pretty simple, and it's probably
can be easily tested
based on my PR.
It will be tricky to j
Hi DB,
I am considering building on your PR and add Mallet as the dependency so
that we can run some basic comparisons test on large scale sparse datasets
that I have.
In the meantime, let's discuss if there are other optimization packages
that we should try.
My wishlist has bounded bfgs as well
I find some comparison between Mallet vs Fortran version. The result
is closed but not the same.
http://t3827.ai-mallet-development.aitalk.info/help-with-l-bfgs-t3827.html
Here is LBFGS-B
Cost: 0.6902411220175793
Gradient: -5.453609E-007, -2.858372E-008, -1.369706E-007
Theta: -0.01418621010217140
Hi Deb,
On Tue, Feb 25, 2014 at 7:07 AM, Debasish Das wrote:
> Continuation on last email sent by mistake:
>
> Is cpl license is compatible with apache ?
>
> http://opensource.org/licenses/cpl1.0.php
Based on what I read here, there is no problem to include CPL code in
apache project
as long as
Hi Deb,
CPL 1.0 is compatible if the inclusion is appropriately labeled
(https://www.apache.org/legal/3party.html). I think it is great to
have an L-BFGS optimizer in mllib, but we need to investigate some
time to figure out which one to use. I'm not sure whether jblas or
netlib-java will make a b
Continuation on last email sent by mistake:
Is cpl license is compatible with apache ?
http://opensource.org/licenses/cpl1.0.php
Mallet jars are available on maven. They have hessian based solvers which
looked interesting along with bfgs and cg.
Definitely the lbfgs f2j looks promising as the b
Hi DB, Xiangrui,
Mallet from cmu also has bfgs cg and a good optimization package. Do you
know if cpl license si
On Feb 22, 2014 11:50 AM, "Xiangrui Meng" wrote:
> Hi DB,
>
> It is great to have the L-BFGS optimizer in MLlib and thank you for taking
> care of the license issue. I looked through
Hi Yinxusen,
I agree. I found a hack to use the current Updater.scala (it contains
not only the logic of regularization but also the adaptive learning
rate in SGD which will not be used in general for other optimizers),
but I think we still need to have a better design to deal with it.
Sincerely,
Hi DB and Xiangrui,
L-BFGS is really useful. I think it will be cool if we can refactor the
interface of runMiniBatchSGD, so as to plug in new optimizer more easily
and compatible. What do you think of that ?
2014-02-23 3:49 GMT+08:00 Xiangrui Meng :
> Hi DB,
>
> It is great to have the L-BFGS
48 matches
Mail list logo
