Re: what is the best way to implement mini batches?

2014-12-15 Thread Earthson Lu
eate ShuffleRDD based on the index(eg. using index/10 as key) 3. using mapPartition to calculate each bach I also have a question: Can mini batches run in parallel? I think parallel all batches just like a full batch GD in some case. -- View this message in context: http://apache-spark-user-

Re: what is the best way to implement mini batches?

2014-12-15 Thread Imran Rashid
batches run in parallel? > I think parallel all batches just like a full batch GD in some case. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to

Re: what is the best way to implement mini batches?

2014-12-14 Thread Earthson
60.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264p20677.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional com

Re: what is the best way to implement mini batches?

2014-12-11 Thread Ilya Ganelin
wrote: >> > >> > any advice/comment on this would be much appreciated. >> > >> > >> > >> > -- >> > View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini >>

Re: what is the best way to implement mini batches?

2014-12-11 Thread Imran Rashid
ould be in). > > Matei > > > On Dec 11, 2014, at 10:16 AM, ll wrote: > > > > any advice/comment on this would be much appreciated. > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/wha

Re: what is the best way to implement mini batches?

2014-12-11 Thread Duy Huynh
epartition that you said (where you > map each record to the partition it should be in). > > Matei > > > On Dec 11, 2014, at 10:16 AM, ll wrote: > > > > any advice/comment on this would be much appreciated. > > > > > > > > -- > > View this me

Re: what is the best way to implement mini batches?

2014-12-11 Thread Matei Zaharia
ssage in context: > http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264p20635.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > ---

Re: what is the best way to implement mini batches?

2014-12-11 Thread ll
any advice/comment on this would be much appreciated. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264p20635.html Sent from the Apache Spark User List mailing list archive at Nabble.com

Re: what is the best way to implement mini batches?

2014-12-03 Thread Alex Minnaar
I am trying to do the same thing and also wondering what the best strategy is. Thanks From: ll Sent: Wednesday, December 3, 2014 10:28 AM To: u...@spark.incubator.apache.org Subject: what is the best way to implement mini batches? hi. what is the best

what is the best way to implement mini batches?

2014-12-03 Thread ll
).mapPartitions() work? thanks! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264.html Sent from the Apache Spark User List mailing list archive at Nabble.com