I'm looking to compare Kafka Streams vs Akka Streams in two areas: 1. For ingesting between Kafka and HDFS/RDBMS
Requirements are mainly around performance and latency. A Kafka topic can have several million events, each corresponding to a database change capture. When ingesting this topic into HDFS I'm also looking to partition the data by day, typically based on a timestamp field in the event record, aggregations on-the-fly (say by a userid field) and parquetizing (preferably on the fly to optimize performance by reducing two I/O operations). 2. Low-latency processing Experiences around performances of Storm/Flink and Akka Streams would be *very* nice. Typical use-cases are de-duping and enrichment with metrics computation (# duplicate events/records, aggregate metrics etc). Low latency and scalability are main considerations On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang <[email protected]> wrote: > > > On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal <[email protected]> > wrote: > >> Hi, >> >> I have read through multiple articles describing the SMACK stack but I'm >> having difficulty understanding the role of Akka in the stack. How does >> Akka fit in? >> > > Akka is for building the application itself. > > >> >> Also I would like to know of experiences in using Akka Streams vs Kafka >> Connect for ingesting from Kafka into HDFS (Hive) and RDBMS. Has anyone >> used Akka Streams for say, dynamic partitioning of events from a Kafka >> topic into HDFS? >> > > Based on what requirements? > > >> >> TIA >> -Shiva >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >> urrent/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Cheers, > √ > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
