Hi Sam,
> Bjorn asked if I felt any examples were missing, and sadly my > original request (that I've been going on about for years, > sorry!) is indeed missing. It is the case of a fast producer and > a slow consumer that is ideal for parallelisation. > There are many examples of slow consumers, for example here is a global rate limiter pattern: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-cookbook.html#Globally_limiting_the_rate_of_a_set_of_streams Obviously that pattern is all about making the consumer slow (and therefore limiting the producer in turn). Also, any fast producer - slow consumer pair is automatically handled by the internal backpressure mechanism, there is nothing special to show about it in examples -- it always looks like source.via(flow).to(sink) independently of the rate of the components. > > I believe it may be possible to use the current 1.0-M2 to address > my bugbear by using the Actor integration to write an actor that > has N instances behind a router, but it feels hacky to have to > use an Actor at all. What is really missing is a Junction that > multiplies the next Flow into N parallel parts that run on > separate threads. > > Another pattern in the cookbook section: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-cookbook.html#Balancing_jobs_to_a_fixed_pool_of_workers Also, you can try mapAsync or mapAsyncUnordered for similar tasks. > > In general, I felt that the documentation was missing any > commentary on concurrency and parallelisation. I was left > wondering what thread everything was happening on. Some initial > questions I have in this area: > > 1. Is everything actually executed in the same thread? What about > when you have a Junction? > This definitely needs careful documentation, but we will also need some extra API that allows the user to define boundaries where processing stages are cut into concurrent pieces. For now, as the default all stream processing element is backed by an actor. So if you have: mySource.map().map().to(mySink) then in the most common case there will be 4 actors each backing one of the parts (1 for the source, 2 for the 2 maps and 1 for the sinks). There is an experimental fusing support that is not yet documented and off by default that would be able to meld the two maps into one actor. In the future we will allow users to add explicit markers where the materializer needs to cut up chains/graphs into concurrent entities. It will be a best effort transformation in the sense that: - elements explicitly marked by the users to be concurrent to each other will be guaranteed to be concurrent - elements inside those segments are attempted to be fused so there is no concurrency, but this will not be always guaranteed. > > 2. Is it possible to be doing work to populate the Source's cache > while work is being executed in a Flow or Sink? > What do you mean by the Source's cache? Maybe there is a misunderstanding here. I recommend to look at this section in the docs: http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-rate.html If you meant buffering you can just use mySource.buffer(100, OverflowStrategy.Backpressure).via(myFlow).to(source) which will introduce a buffer element between your source and flow and it will slurp at max 100 elements from the source upfront if the flow next to it is not fast enough to consume elements. > > It would be good to have a section in the documentation that > discusses this in more detail. > > > And, very importantly, it would be good to have the feature of > being able to split a Flow into N parallel parts! > Please be more explicit when you talk about parallellism here. Just to demonstrate, there are two ways to have parallel/concurrent processing in streams: 1. If you have processing stages A, B, C then they can run concurrently, C works on an older element while A already processes the next. See http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala/stream-rate.html 2. parallelizing a stage A to be a pool of A1, A2, A3, ... . I pasted a cookbook recipe for that, but also mapAsync and mapAsyncUnordered can be used that way too. In fact, since those work on Futures, you can combine them with the ask pattern and a routed pool of worker actors. > I recently > learnt how to do this in ScalazStreams but I'm much rather be > able to do it in Akka Streams as I find everything else about the > architecture to be so much easier to understand (plus integration > with Akka Actors is just tremendous). > > PS: I'm also very excited by Slick 3.0 which appears to be aiming toward > Reactive Streams and, I assume, integration with Akka Streams. e.g. produce > a Source[Entity] from a SELECT with pushback on extremely large result sets. > > > [1] https://groups.google.com/d/msg/akka-user/1TlAy-oqOk8/xvJpyVMWytsJ > [2] > http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0-M2/scala.html > [3] > https://github.com/fommil/rx-playground/blob/master/src/main/scala/com/github/fommil/rx/scratch.scala#L204 > > Best regards, > Sam (fommil) > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
