Hello Patrik! Thank you for your comment!
While the project is at initial stage, I watch it and tested it for a while. It really is cool and I already decided to use reactive-kafka to cover Kafka area of my requirements. But 'on cluster' part still remains unsolved. ;-( Thank you again!!! -- JH 2016년 4월 7일 목요일 오후 7시 44분 55초 UTC+9, Patrik Nordwall 님의 말: > > You might find https://github.com/akka/reactive-kafka interesting > > /Patrik > > tors 7 apr. 2016 kl. 09:11 skrev JongHyok Lee <[email protected] > <javascript:>>: > >> Hello, >> >> I'm trying to make dynamic data stream processing engine which can >> >> - read data from multiple topics on multiple Kafka cluster >> - apply several steps of data processing logic on data stream (like >> filtering, transformation of data, join, etc) >> - one data source / processing logic can be shared among connected >> processing logic (fan-out) >> - load of entire engine should be distributed on engine cluster >> - sink result stream to Kafka cluster, TCP Server, etc >> - every data stream flow can be dynamically composed and modified >> - each operation unit can be handled separately (one flow operation unit >> connected to the output of another operation unit can change the source of >> source to output of the other operation unit dynamically(without stopping >> processing of data processing)) >> >> I think akka stream meets most of the requirements above but load >> distribution as akka streams doesn't support load distribution yet. >> >> So, to meet the requirement (load distribution) also, I'm planning to >> connect each end (output) of akka stream flow to simple actor that can >> communicate with other actors on the cluster. >> (most of the cases 3 actors which of the first receives data from other >> actors, the second processes data with akka streams, and the last passing >> result to other actors) >> Surely I understand that it'll break end-to-end back-pressure capability >> of whole data stream processing. >> >> But if it's the situation I can bear, is it the acceptable way to go? >> Or should I find another framework or solution instead of akka streams as >> it's too dangerous breaking end-to-end back-pressure capability? >> >> Any comments / recommendations would be deeply appreciated! >> >> -- >> JH >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
