Hi Patrik, There is an other part of actor, which I haven't specified. It connects to elasticsearch.
The count of data which I got in elasticsearch and the count which I mentioned in that actor didn't match up. Later I figured out that there were lot of errors from elasticsearch side. So the problem I mentioned is nonexistent. We have a system which publishes data to kafka in a very large scale(Say 20Million messages/second). I'll have to consume in such a large scale and do json transformation and push it to elasticsearch. For that I need to know how parallelism is getting handled in streams. Can you please explain that? Thanks, Jeeva On Friday, October 23, 2015 at 3:48:06 PM UTC+5:30, Patrik Nordwall wrote: > > > > On Fri, Oct 23, 2015 at 11:45 AM, Jeeva K <[email protected] > <javascript:>> wrote: > >> Looks like the `count` variable is actor is thread safe. >> > > Yes, what you showed is an Actor. > > Please elaborate what you mean with "count doesn't match up". What is the > interaction between the kafkasource and the Sink(Subscriber)? > > /Patrik > > >> >> Can anyone please explain how parallelism is achieved in streams? >> >> >> On Thursday, October 22, 2015 at 10:54:35 PM UTC+5:30, Jeeva K wrote: >>> >>> >>> Hello, >>> >>> I'm having an akka stream like, >>> >>> kafkasource ~> transformer ~> Sink(Subscriber) >>> >>> This is the subscriber class I'm using: >>> >>> class Subscriber extends ActorSubscriber { >>> import context.dispatcher >>> context.system.scheduler.schedule(0.milli, 1.minute, self, "getcount") >>> var count: Long = 0 >>> var previousSum: Long = 0 >>> >>> def receive = { >>> case OnNext(data: GenModel) => >>> count = count + 1 >>> >>> case OnError(t: Throwable) => onError(t) >>> >>> case OnComplete => shutdownIfAllAcked() >>> >>> case "getcount" => >>> log.info("Number of requests/min : " + (count - previousSum) + >>> ":: " + count) >>> previousSum = count >>> >>> case _ => >>> } >>> } >>> >>> I'm trying to get the count of messages processed by the transformer. >>> Somehow, the count doesn't match up. >>> I'm using inflight request strategy with number of inflight request as >>> zero always(Just for tuning). >>> >>> These are my queries: >>> >>> - Is the `count` variable threadsafe? If not, how do I achieve the >>> same? >>> - I'm not sure how streams spawn the threads to achieve >>> parallelism. Can anyone explain that a bit? >>> >>> As, I'm trying to tune the system, I can not afford to use another >>> actor/synchronous block to get the count. Any help is appreciated. >>> >>> Thanks, >>> Jeeva >>> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > Patrik Nordwall > Typesafe <http://typesafe.com/> - Reactive apps on the JVM > Twitter: @patriknw > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
