As Reynold said, you can still use Akka with Spark, but now it's more like using any third-party library that isn't already a Spark dependency (at least once the current Akka dependency is fully removed).
Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Sun, Dec 27, 2015 at 4:06 AM, Disha Shrivastava <dishu....@gmail.com> wrote: > Hi All, > > I need an Akka like framework to implement model parallelism in neural > networks, an architecture similar to that given in the link > http://alexminnaar.com/implementing-the-distbelief-deep-neural-network-training-framework-with-akka.html. > I need to divide a big neural network ( which can't fit into the memory of > one machine) layer by layer and do message passing across actors which are > distributed across different worker machines. I found Akka to be most > suitable for the job. > > Please suggest if it can be done by any other suitable frameworks. > > Regards, > Disha > > On Sun, Dec 27, 2015 at 1:04 PM, Reynold Xin <r...@databricks.com> wrote: > >> We are just removing Spark's dependency on Akka. It has nothing to do >> with whether user applications can use Akka or not. As a matter of fact, by >> removing the Akka dependency from Spark, it becomes easier for user >> applications to use Akka, because there is no more dependency conflict. >> >> For more information, see >> https://issues.apache.org/jira/browse/SPARK-5293 >> >> On Sat, Dec 26, 2015 at 9:31 PM, Soumya Simanta <soumya.sima...@gmail.com >> > wrote: >> >>> >>> >>> Any rationale for removing Akka from Spark ? Also, what is the >>> replacement ? >>> >>> Thanks >>> >>> On Dec 27, 2015, at 8:31 AM, Dean Wampler <deanwamp...@gmail.com> wrote: >>> >>> Note that Akka is being removed from Spark. Even if it weren't, I would >>> consider keeping Akka processes separate from Spark processes, so you can >>> monitor, debug, and scale them independently. So consider streaming data >>> from Akka to Spark Streaming or go the other way, from Spark to Akka >>> Streams. >>> >>> dean >>> >>> Dean Wampler, Ph.D. >>> Author: Programming Scala, 2nd Edition >>> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) >>> Typesafe <http://typesafe.com> >>> @deanwampler <http://twitter.com/deanwampler> >>> http://polyglotprogramming.com >>> >>> On Sat, Dec 26, 2015 at 12:54 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Do you mind sharing your use case ? >>>> >>>> It may be possible to use a different approach than Akka. >>>> >>>> Cheers >>>> >>>> On Sat, Dec 26, 2015 at 10:08 AM, Disha Shrivastava < >>>> dishu....@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I wanted to know how to use Akka framework with Spark starting from >>>>> basics. I saw online that Spark uses Akka framework but I am not really >>>>> sure if I can define Actors and use it in Spark. >>>>> >>>>> Also, how to integrate Akka with Spark as in how will I know how many >>>>> Akka actors are running on each of my worker machines? Can I control that? >>>>> >>>>> Please help. The only useful resource which I could find online was >>>>> Akka with Spark Streaming which was also not very clear. >>>>> >>>>> Thanks, >>>>> >>>>> Disha >>>>> >>>> >>>> >>> >> >
