Be careful with Flink. IME it's got a long way to go. The core model is fantastic but there's a lot of low-hanging fruit that needs to be fixed.
On Tuesday, May 2, 2017 at 4:03:12 PM UTC-6, Evan Chan wrote: > > Hi Shiva, > > Spark will likely be too high latency for you. Practical minimal batch > size is a couple seconds. > > Think of Akka as a best fit for if you want to deploy individual apps each > reading from some set of fixed Kafka partitions. Each one could then write > to HDFS. However you would need to handle failover, state, etc. etc. > It is likely that Flink has much more built in for you - HDFS integration, > checkpointing, failover, shuffling/routing of messages, integration with > Kafka, etc. > > You might want to look into Intel Gearpump - this is Akka-based very low > latency dynamic stream processing, and they have handled distribution > already. > > -Evan > > On Friday, April 28, 2017 at 7:52:50 AM UTC-7, Shivakumar Ramagopal wrote: >> >> Viktor, >> >> >> On Fri, Apr 28, 2017 at 5:03 PM, Viktor Klang <viktor...@gmail.com> >> wrote: >> >>> >>> >>> On Fri, Apr 28, 2017 at 1:12 PM, Shiva Ramagopal <tr....@gmail.com> >>> wrote: >>> >>>> Hi Viktor, >>>> >>>> On Fri, Apr 28, 2017 at 2:55 PM, Viktor Klang <viktor...@gmail.com> >>>> wrote: >>>> >>>>> Hi Shiva, >>>>> >>>>> On Fri, Apr 28, 2017 at 11:20 AM, Shiva Ramagopal <tr....@gmail.com> >>>>> wrote: >>>>> >>>>>> I'm looking to compare Kafka Streams vs Akka Streams in two areas: >>>>>> >>>>>> 1. For ingesting between Kafka and HDFS/RDBMS >>>>>> >>>>>> Requirements are mainly around performance and latency. A Kafka topic >>>>>> can have several million events, each corresponding to a database change >>>>>> capture. When ingesting this topic into HDFS I'm also looking to >>>>>> partition >>>>>> the data by day, typically based on a timestamp field in the event >>>>>> record, >>>>>> aggregations on-the-fly (say by a userid field) and parquetizing >>>>>> (preferably on the fly to optimize performance by reducing two I/O >>>>>> operations). >>>>>> >>>>> >>>>> Looking forward to see your benchmark! >>>>> >>>> >>>> Hey, you wanted requirements! :) >>>> >>>>> >>>>> >>>>>> >>>>>> 2. Low-latency processing >>>>>> >>>>>> Experiences around performances of Storm/Flink and Akka Streams would >>>>>> be *very* nice. Typical use-cases are de-duping and enrichment with >>>>>> metrics >>>>>> computation (# duplicate events/records, aggregate metrics etc). Low >>>>>> latency and scalability are main considerations >>>>>> >>>>> >>>>> Low latency is not a metric, and scalability is not a profile. :) >>>>> In other words: What latency distribution are you targeting and what >>>>> scalability curve? >>>>> >>>> >>>> I know latency is a nuanced topic. I'm just looking for broad >>>> experiences on performance comparisons if anyone has done that. Currently >>>> we have a Storm topology over 5 nodes doing enrichment of events from >>>> Kafka >>>> that involves a lookup into a db per event. 90th percentile of latency of >>>> this processing is under 200 ms and we are happy with this. While Storm is >>>> mature, Akka Streams seems more general purpose than Storm. I'd like to >>>> use >>>> Akka Streams for this reason if performance is comparable to Storm. >>>> >>> >>> There's one important architectural difference here tho: Akka Streams >>> are local-only (as in materialization). You can of course materialize Akka >>> Streams on multiple nodes and use a transport to coordinate data >>> processing. Interestingly it doesn't lock you in to a particular backend >>> such that Kafka Streams or even Storm would do. >>> >> >> The materialization part was something I was not aware of. Thanks for >> pointing it out, really appreciate it. >> >>> >>> All of this boils down to requirements. Something like Flink or Google >>> Beam could be viable options here as well. In your situation I'd look at >>> the requirements and make a couple of prototypes before picking a winner. >>> >> >> I was also looking for something to unify batch and streaming - which is >> how I came to look at SMACK. I'd really like to keep the tech stack small >> and have the parts integrate tightly with each other. Guess I have to pick >> between Spark, Flink, Beam and Storm. >> >>> >>> >>>> >>>> >>>>> >>>>> >>>>>> >>>>>> On Thu, Apr 27, 2017 at 8:31 PM, Viktor Klang <viktor...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Apr 27, 2017 at 10:39 AM, Shiva Ramagopal <tr....@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have read through multiple articles describing the SMACK stack >>>>>>>> but I'm having difficulty understanding the role of Akka in the stack. >>>>>>>> How >>>>>>>> does Akka fit in? >>>>>>>> >>>>>>> >>>>>>> Akka is for building the application itself. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Also I would like to know of experiences in using Akka Streams vs >>>>>>>> Kafka Connect for ingesting from Kafka into HDFS (Hive) and RDBMS. Has >>>>>>>> anyone used Akka Streams for say, dynamic partitioning of events from >>>>>>>> a >>>>>>>> Kafka topic into HDFS? >>>>>>>> >>>>>>> >>>>>>> Based on what requirements? >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> TIA >>>>>>>> -Shiva >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>> >>>>>>>>>> Search the archives: >>>>>>>> https://groups.google.com/group/akka-user >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Akka User List" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Cheers, >>>>>>> √ >>>>>>> >>>>>>> -- >>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>> >>>>>>>>>> Check the FAQ: >>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>> >>>>>>>>>> Search the archives: >>>>>>> https://groups.google.com/group/akka-user >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Akka User List" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to akka-user+...@googlegroups.com. >>>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: >>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: >>>>>> https://groups.google.com/group/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to akka-user+...@googlegroups.com. >>>>>> To post to this group, send email to akka...@googlegroups.com. >>>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> √ >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: >>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>> >>>>>>>>>> Search the archives: >>>>> https://groups.google.com/group/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to akka-user+...@googlegroups.com. >>>>> To post to this group, send email to akka...@googlegroups.com. >>>>> Visit this group at https://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to akka-user+...@googlegroups.com. >>>> To post to this group, send email to akka...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Cheers, >>> √ >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: >>> http://doc.akka.io/docs/akka/current/additional/faq.html >>> >>>>>>>>>> Search the archives: >>> https://groups.google.com/group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to akka-user+...@googlegroups.com. >>> To post to this group, send email to akka...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
