Thank you for your response. I have to admit the use-case I was trying for is unusual and it was dependent on being able to serialize parts of the graph to other nodes. I did see the nice cookbook sample of balancing out processing to a number of workers, but in my case there would be no merge phase back to one output.
The main goal I was trying to achieve was to distribute a finite stream of data out to a collection of Spark partitions without having to first store the entire stream data set in memory on the master. On Tuesday, March 3, 2015 at 11:45:20 AM UTC-5, Björn Antonsson wrote: > > Hi, > > First of, there is no support for serializing parts of a graph and sending > them to other nodes for execution in akka-streams. That would be awesome to > have, but it's not there yet. > > Second, your shape in the graph is no longer a UniformFanOutShape, since > you have connected a Source to the Balance, and thus removed the inlet on > the graph you are building. Then you return the balance which hasn't got > the same shape as the thing you are building. > > Have you looked at the cookbook > <http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M4/scala/stream-cookbook.html>samples > > of how to balance out processing to a number f workers? > > B/ > > On 2 March 2015 at 23:06:52, rmarsch ([email protected] <javascript:>) > wrote: > > Hi, > > I have a use case that I believe Akka Streams may be well suited for, but > I'm not clearly seeing how to implement part of it from reading the > documentation. > > I have a single, finite input source from an iterator stream. I want to > use Balance to split this stream into N partitions and then be able to feed > these stream partitions each into a RunnableFlow that would be executed on > its own node in a distributed system. > > in ~> balance.out(0) ~> sink0 > balance.out(1) ~> sink1 > ... > balance.out(n) ~> sinkN > > I have some rough code sketched together, but I think I'm missing some > concepts to tie everything together. Runtime complains about the inlet for > the UniformInOutShape which I'm assuming is what I have in the inputGraph. > Any nudge in the right direction would be appreciated: > > > val iteratorBuilder: () => Iterator[T] = ??? > > > val inputGraph: Graph[UniformFanOutShape[T,T], Unit] = FlowGraph.partial() { > implicit builder : FlowGraph.Builder => > import FlowGraph.Implicits._ > > val stream: SourceShape[T] = builder.add(Source(iteratorBuilder)) > val balance = builder.add(Balance[T](numPartitions)) > > stream ~> balance > > balance > } > > partitions = (0 until numPartitions).map(i => { > val sink : Sink[T, Future[Stream[T]]] = Sink.fold(Stream.empty[T]) { (v, e) > => v.+:(e) } > > val outFlow: RunnableFlow[Future[Stream[T]]] = FlowGraph.closed(sink) { > implicit builder : FlowGraph.Builder => sink => > import FlowGraph.Implicits._ > val balance = builder.add(inputGraph) > > balance.out(i) ~> sink > } > > new Partition(i, outFlow) > }).toArray > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > To post to this group, send email to [email protected] > <javascript:>. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > -- > Björn Antonsson > Typesafe Inc. <http://typesafe.com> – Reactive Apps on the JVM > twitter: bantonsson <http://twitter.com/bantonsson> > > JOIN US. REGISTER TODAY! > <http://event.scaladays.org/scaladays-sanfran-2015> > Scala <http://event.scaladays.org/scaladays-sanfran-2015> > Days <http://event.scaladays.org/scaladays-sanfran-2015> > March 16th-18th, <http://event.scaladays.org/scaladays-sanfran-2015> > San Francisco <http://event.scaladays.org/scaladays-sanfran-2015> > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
