Patrik, the fusion stages would be used if the developer explicitly call them? Btw, a little question, each stage is materialized into a single actor instance or can be materialized into multiple instances?
Em sexta-feira, 23 de outubro de 2015 12:47:56 UTC-3, Patrik Nordwall escreveu: > > > > On Fri, Oct 23, 2015 at 4:00 PM, Jeeva K <[email protected] > <javascript:>> wrote: > >> Hi Patrik, >> >> There is an other part of actor, which I haven't specified. It connects >> to elasticsearch. >> >> The count of data which I got in elasticsearch and the count which I >> mentioned in that actor didn't match up. Later I figured out that there >> were lot of errors from elasticsearch side. So the problem I mentioned is >> nonexistent. >> >> We have a system which publishes data to kafka in a very large scale(Say >> 20Million messages/second). I'll have to consume in such a large scale and >> do json transformation and push it to elasticsearch. For that I need to >> know how parallelism is getting handled in streams. Can you please explain >> that? >> > > Currently, each stage is executed by an actor, i.e. each stage process the > stream elements sequentially but different stages may run concurrently. > This is a great read: > http://doc.akka.io/docs/akka-stream-and-http-experimental/1.0/scala/stream-parallelism.html > > We are working on fusing of stages so that several stages are executed by > one actor. That will give better performance when the stream consists of > many stages that are quick, since it reduces the overhead of the > asynchronous boundaries. It will be possible to demarcate which parts of a > flow/graph that should not be fused. > > /Patrik > > >> >> Thanks, >> Jeeva >> >> On Friday, October 23, 2015 at 3:48:06 PM UTC+5:30, Patrik Nordwall wrote: >>> >>> >>> >>> On Fri, Oct 23, 2015 at 11:45 AM, Jeeva K <[email protected]> wrote: >>> >>>> Looks like the `count` variable is actor is thread safe. >>>> >>> >>> Yes, what you showed is an Actor. >>> >>> Please elaborate what you mean with "count doesn't match up". What is >>> the interaction between the kafkasource and the Sink(Subscriber)? >>> >>> /Patrik >>> >>> >>>> >>>> Can anyone please explain how parallelism is achieved in streams? >>>> >>>> >>>> On Thursday, October 22, 2015 at 10:54:35 PM UTC+5:30, Jeeva K wrote: >>>>> >>>>> >>>>> Hello, >>>>> >>>>> I'm having an akka stream like, >>>>> >>>>> kafkasource ~> transformer ~> Sink(Subscriber) >>>>> >>>>> This is the subscriber class I'm using: >>>>> >>>>> class Subscriber extends ActorSubscriber { >>>>> import context.dispatcher >>>>> context.system.scheduler.schedule(0.milli, 1.minute, self, >>>>> "getcount") >>>>> var count: Long = 0 >>>>> var previousSum: Long = 0 >>>>> >>>>> def receive = { >>>>> case OnNext(data: GenModel) => >>>>> count = count + 1 >>>>> >>>>> case OnError(t: Throwable) => onError(t) >>>>> >>>>> case OnComplete => shutdownIfAllAcked() >>>>> >>>>> case "getcount" => >>>>> log.info("Number of requests/min : " + (count - previousSum) + >>>>> ":: " + count) >>>>> previousSum = count >>>>> >>>>> case _ => >>>>> } >>>>> } >>>>> >>>>> I'm trying to get the count of messages processed by the transformer. >>>>> Somehow, the count doesn't match up. >>>>> I'm using inflight request strategy with number of inflight request as >>>>> zero always(Just for tuning). >>>>> >>>>> These are my queries: >>>>> >>>>> - Is the `count` variable threadsafe? If not, how do I achieve >>>>> the same? >>>>> - I'm not sure how streams spawn the threads to achieve >>>>> parallelism. Can anyone explain that a bit? >>>>> >>>>> As, I'm trying to tune the system, I can not afford to use another >>>>> actor/synchronous block to get the count. Any help is appreciated. >>>>> >>>>> Thanks, >>>>> Jeeva >>>>> >>>> -- >>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>> >>>>>>>>>> Check the FAQ: >>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>> >>>>>>>>>> Search the archives: >>>> https://groups.google.com/group/akka-user >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Akka User List" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/akka-user. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> >>> Patrik Nordwall >>> Typesafe <http://typesafe.com/> - Reactive apps on the JVM >>> Twitter: @patriknw >>> >>> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
