Hi Viktor, On Fri, Apr 28, 2017 at 2:55 PM, Viktor Klang <[email protected]> wrote:
> Hi Shiva, > > On Fri, Apr 28, 2017 at 11:20 AM, Shiva Ramagopal <[email protected]> > wrote: > >> I'm looking to compare Kafka Streams vs Akka Streams in two areas: >> >> 1. For ingesting between Kafka and HDFS/RDBMS >> >> Requirements are mainly around performance and latency. A Kafka topic can >> have several million events, each corresponding to a database change >> capture. When ingesting this topic into HDFS I'm also looking to partition >> the data by day, typically based on a timestamp field in the event record, >> aggregations on-the-fly (say by a userid field) and parquetizing >> (preferably on the fly to optimize performance by reducing two I/O >> operations). >> > > Looking forward to see your benchmark! > Hey, you wanted requirements! :) > > >> >> 2. Low-latency processing >> >> Experiences around performances of Storm/Flink and Akka Streams would be >> *very* nice. Typical use-cases are de-duping and enrichment with metrics >> computation (# duplicate events/records, aggregate metrics etc). Low >> latency and scalability are main considerations >> > > Low latency is not a metric, and scalability is not a profile. :) > In other words: What latency distribution are you targeting and what > scalability curve? > I know latency is a nuanced topic. I'm just looking for broad experiences on performance comparisons if anyone has done that. Currently we have a Storm topology over 5 nodes doing enrichment of events from Kafka that involves a lookup into a db per event. 90th percentile of latency of this processing is under 200 ms and we are happy with this. While Storm is mature, Akka Streams seems more general purpose than Storm. I'd like to use Akka Streams for this reason if performance is comparable to Storm. > > >> >> On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang <[email protected]> >> wrote: >> >>> >>> >>> On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I have read through multiple articles describing the SMACK stack but >>>> I'm having difficulty understanding the role of Akka in the stack. How does >>>> Akka fit in? >>>> >>> >>> Akka is for building the application itself. >>> >>> >>>> >>>> Also I would like to know of experiences in using Akka Streams vs Kafka >>>> Connect for ingesting from Kafka into HDFS (Hive) and RDBMS. Has anyone >>>> used Akka Streams for say, dynamic partitioning of events from a Kafka >>>> topic into HDFS? >>>> >>> >>> Based on what requirements? >>> >>> >>>> >>>> TIA >>>> -Shiva >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>> urrent/additional/faq.html >>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>> p/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Cheers, >>> √ >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>> urrent/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>> p/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >> urrent/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Cheers, > √ > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
