Hi, you could use TCP, chunked HTTP or WebSockets to transfer the stream of data across VMs while preserving backpressure, but what about error handling? How do you deal with VM2 or VM3 crashing after accepting data elements from VM1?
I have a very basic understanding of Kafka, but maybe a partitioned topic could be a suitable mechanism to reliably distribute work? I have no idea though if Kafka could be configured to backpressure the writer if the readers are not keeping up. This may or may not be a deal breaker, depending on your use case. Alternatively, you could use ActorPublisher / ActorSubscriber at the boundaries of your Akka Streams and use Actor-level constructs to distribute the work: Akka remoting / cluster / sharding etc. Cheers, Rafał W dniu czwartek, 27 października 2016 19:19:44 UTC+2 użytkownik auxdx napisał: > > Hi, > > I know that akka stream doesn't support remote actor. But I am trying to > achieve something similar to "balancer" stages of akka stream for remote > actors / remote processes. My user case is a bit complicated but I will use > a simple example to demonstrate it. > > Suppose, I have a reactive source: such as val source = Source(1 to 10) > which run on VM1 (data can be pulled from Kafka constantly). On VM2, and > VM3, I will need to somehow run an expensive computation runnable graph > (such as Source.via(Flow.something).to(Sink.ignore)) which operates on a > subset of VM1's source's elements . For instance, I want VM2 to handle even > number; such as Source(2 to 10 by 2); and VM3 handle odd number such as > Source(1 to 9 by 2); In other words, given that VM1 having a single > publisher which publishes a data stream consistently; I want to somehow > load balance it to VM2 and VM3 (remote PC) to distribute computation; which > VM (2 or 3) it should forward to is based on property of the number such as > if number is odd or even. > > In case the computation runnable graph must be on VM2 and VM3, what is the > best way to achieve it given that the data source is on VM1? Via TCP? If > so, how to make sure it's back-pressure. > > I guess one way to do is somehow run 2 actors (ActorPublishers) on VM2, > VM3; when those 2 are back-pressued from downstream; it will pull data from > source on VM1; It this the correct way to do so. Some code will be helpful. > > Thanks. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
