eate ShuffleRDD based on the index(eg. using index/10 as key)
3. using mapPartition to calculate each bach
I also have a question:
Can mini batches run in parallel?
I think parallel all batches just like a full batch GD in some case.
--
View this message in context:
http://apache-spark-user-
batches run in parallel?
> I think parallel all batches just like a full batch GD in some case.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to
60.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264p20677.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional com
wrote:
>> >
>> > any advice/comment on this would be much appreciated.
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-spark-user-list.
>> 1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini
>>
ould be in).
>
> Matei
>
> > On Dec 11, 2014, at 10:16 AM, ll wrote:
> >
> > any advice/comment on this would be much appreciated.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/wha
epartition that you said (where you
> map each record to the partition it should be in).
>
> Matei
>
> > On Dec 11, 2014, at 10:16 AM, ll wrote:
> >
> > any advice/comment on this would be much appreciated.
> >
> >
> >
> > --
> > View this me
ssage in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264p20635.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---
any advice/comment on this would be much appreciated.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264p20635.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
I am trying to do the same thing and also wondering what the best strategy is.
Thanks
From: ll
Sent: Wednesday, December 3, 2014 10:28 AM
To: u...@spark.incubator.apache.org
Subject: what is the best way to implement mini batches?
hi. what is the best
).mapPartitions() work?
thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/what-is-the-best-way-to-implement-mini-batches-tp20264.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
10 matches
Mail list logo