Hello, I'm trying to make dynamic data stream processing engine which can
- read data from multiple topics on multiple Kafka cluster - apply several steps of data processing logic on data stream (like filtering, transformation of data, join, etc) - one data source / processing logic can be shared among connected processing logic (fan-out) - load of entire engine should be distributed on engine cluster - sink result stream to Kafka cluster, TCP Server, etc - every data stream flow can be dynamically composed and modified - each operation unit can be handled separately (one flow operation unit connected to the output of another operation unit can change the source of source to output of the other operation unit dynamically(without stopping processing of data processing)) I think akka stream meets most of the requirements above but load distribution as akka streams doesn't support load distribution yet. So, to meet the requirement (load distribution) also, I'm planning to connect each end (output) of akka stream flow to simple actor that can communicate with other actors on the cluster. (most of the cases 3 actors which of the first receives data from other actors, the second processes data with akka streams, and the last passing result to other actors) Surely I understand that it'll break end-to-end back-pressure capability of whole data stream processing. But if it's the situation I can bear, is it the acceptable way to go? Or should I find another framework or solution instead of akka streams as it's too dangerous breaking end-to-end back-pressure capability? Any comments / recommendations would be deeply appreciated! -- JH -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
