Dear friends, I've read and grok the back-pressure story around Akka streams, which is pull-based. I have also seen examples of this with fixed topologies that tend to be relatively linear or are fan-out: for example, a coordinator farming work out to worker nodes. For these examples, I can see that it would work well. Each worker issues a pull request (haha, no pun intended) for more work from one master.
Now, consider the case where I may have multiple flows or pipelines feeding in to one I/O sink, which does something like a mapAsync (feeds to futures doing I/O). This is much more of a fan-in pattern. The multiplicity of the pipelines changes dynamically. For example, let's say that each pipeline ingests one customer's data, and it is simpler to keep one pipeline per customer. As a new customer comes in, then we want to start a pipeline for them, and it should feed into the same I/O sink. If there are more customers/pipelines, one would expect backpressure so that each flow slows down, which is expected. Implementing a pull backpressure in this case seems much more complicated. Does Akka streams handle dynamic registration / materialization? In order for the sink to know about each pipeline, each pipeline must somehow register itself with the sink, and the sink must then send pull requests to each pipeline. Or, would the sink only send pull requests to some random subset of pipelines, if it is more heavily loaded? I can think of a way to have push, but still have backpressure, even over networks. -Evan -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
