Hi All, I need an Akka like framework to implement model parallelism in neural networks, an architecture similar to that given in the link http://alexminnaar.com/implementing-the-distbelief-deep-neural-network-training-framework-with-akka.html. I need to divide a big neural network ( which can't fit into the memory of one machine) layer by layer and do message passing across actors which are distributed across different worker machines. I found Akka to be most suitable for the job.
Please suggest if it can be done by any other suitable frameworks. Regards, Disha On Sun, Dec 27, 2015 at 1:04 PM, Reynold Xin <r...@databricks.com> wrote: > We are just removing Spark's dependency on Akka. It has nothing to do with > whether user applications can use Akka or not. As a matter of fact, by > removing the Akka dependency from Spark, it becomes easier for user > applications to use Akka, because there is no more dependency conflict. > > For more information, see https://issues.apache.org/jira/browse/SPARK-5293 > > On Sat, Dec 26, 2015 at 9:31 PM, Soumya Simanta <soumya.sima...@gmail.com> > wrote: > >> >> >> Any rationale for removing Akka from Spark ? Also, what is the >> replacement ? >> >> Thanks >> >> On Dec 27, 2015, at 8:31 AM, Dean Wampler <deanwamp...@gmail.com> wrote: >> >> Note that Akka is being removed from Spark. Even if it weren't, I would >> consider keeping Akka processes separate from Spark processes, so you can >> monitor, debug, and scale them independently. So consider streaming data >> from Akka to Spark Streaming or go the other way, from Spark to Akka >> Streams. >> >> dean >> >> Dean Wampler, Ph.D. >> Author: Programming Scala, 2nd Edition >> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >> Typesafe <http://typesafe.com> >> @deanwampler <http://twitter.com/deanwampler> >> http://polyglotprogramming.com >> >> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Do you mind sharing your use case ? >>> >>> It may be possible to use a different approach than Akka. >>> >>> Cheers >>> >>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava <dishu....@gmail.com >>> > wrote: >>> >>>> Hi, >>>> >>>> I wanted to know how to use Akka framework with Spark starting from >>>> basics. I saw online that Spark uses Akka framework but I am not really >>>> sure if I can define Actors and use it in Spark. >>>> >>>> Also, how to integrate Akka with Spark as in how will I know how many >>>> Akka actors are running on each of my worker machines? Can I control that? >>>> >>>> Please help. The only useful resource which I could find online was >>>> Akka with Spark Streaming which was also not very clear. >>>> >>>> Thanks, >>>> >>>> Disha >>>> >>> >>> >> >
