Thanks Bjorn for the response. I kind of gleaned from further reading here and elsewhere that akka streams does not run on a cluster.
By "not ready for Spark", simply means that we do not have a practice (hardware, ops, engineering) to support it. We _do_ however have practices supporting Storm and Hadoop. The project I am working on is getting data through Kafka. I am getting essentially measures and dimensions coming through Kafka. The process I am building looks like: Filter --> Partition by Dimension 1 --> Partition by Dimension 2 --> Create Aggregates If I want to be able to create aggregates / rollups across some dimension, I need to be able to have all of those messages land in the same "spot". Calculating perc50 or perc99 and other metrics. I suppose in theory I could use some combination of Akka Streams to do the simple Filter and Partitions, but then crunch the numbers in a cluster via Akka Clustering. On Tuesday, November 4, 2014 4:11:11 PM UTC-5, Björn Antonsson wrote: > > Hi Paul, > > Akka streams is still under development and marked as an experimental > feature. Currently they can _not_ take advantage of the nodes in an akka > cluster without writing all the code youserlf and connecting them over TCP. > Doing this automatically is of course something that we have been thinking > about, but it is still only on the drawing board. > > Why do you feel that you are not ready for Spark? It would be very good > feedback to know what to improve with akka streams. > > B/ > > On 4 November 2014 at 13:48:12, Paul Cleary ([email protected] > <javascript:>) wrote: > > Working on a project where we need to do processing of high-volume data > arriving via Kafka. > > We did a POC in Spark Streaming, but we are not yet ready for Spark. > > I was looking at using Storm, but the abstractions make me sad, at least > compared to how you can pipeline in Spark. > > I am investigating using Akka Streams as an alternative to Storm, since we > can support Akka + Play in production today. It feels like Akka Streams + > Akka Clustering makes for a way to setup distributed pipelines to process > large data flows. > > I was wondering if anyone can give me any pointers on how to setup Akka > Streams so that it runs on a cluster (or if it is even possible). > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > > -- > Björn Antonsson > Typesafe <http://typesafe.com/> – Reactive Apps on the JVM > twitter: @bantonsson <http://twitter.com/#!/bantonsson> > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
