The thing is, under the load, the actor will have always messages so it will never process other messages. That may fit the user case anyway, but may also not.
Btw it was one of my initial idea. On 14 April 2014 22:59, Scott Clasen <[email protected]> wrote: > Hey man! > > So far so good, under load you dont have to worry about the timeout ever > happening really. > > The client is structured as a supervisor actor that is responsible for > committing the offsets periodically or after N messages, which spins up a > number of child Stream actors, each of which manage a single > KafkaStream/MessageIterator. > > I have tested with a topic with 32 partitions, in 2 cases. One where all > topics have a few million messages, and one where only a single partiton of > the 32 has a few million messages, and both cases are working just fine. > > Cheers! > > > > On Monday, April 14, 2014 2:35:03 PM UTC-7, mighdoll wrote: > >> Nice idea with the timeout. I wrote a kafka -> rx.observable wrapper the >> other week too (KafkaReader.scala). >> >> I'll ponder adding your timeout trick to that interface too. Have you >> tried it under loaded conditions? Our ops folks have been in the habit of >> tuning up the kafka consumer timeout - I'm not sure if a short timeout will >> create other problems. >> >> Hopefully a truly async interface will come around from the kafka core >> team. >> >> Cheers, >> Lee >> >> On Mon, Apr 14, 2014 at 1:14 PM, Scott Clasen <[email protected]>wrote: >> >>> Im actually working on an actor based kafka consumer as we speak >>> >>> You will have to configure the consumer to not block indefinitely >>> >>> "consumer.timeout.ms" -> "500" >>> >>> then you can have a non blocking hasNext() >>> >>> def hasNext() = try { >>> msgIterator.hasNext() >>> } catch { >>> case cte: ConsumerTimeoutException => false >>> } >>> >>> once you have these its down to modeling how you want the actors to >>> react to messages being available (or not). FSMs are great here but not >>> knowing what you want to do its harder to offer more advice >>> >>> >>> On Monday, April 14, 2014 1:02:11 PM UTC-7, Filippo De Luca wrote >>>> >>>> Yes polling messages and send >>>> On Apr 14, 2014 7:47 PM, "√iktor Ҡlang" <[email protected]> wrote: >>>> >>>>> Do you want to read & write etc? >>>>> >>>>> >>>>> On Mon, Apr 14, 2014 at 7:59 PM, Filippo De Luca < >>>>> [email protected]> wrote: >>>>> >>>>>> I think I will need something very similar to zeromq extension, what >>>>>> do you think? >>>>>> On Apr 14, 2014 5:54 PM, "Filippo De Luca" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Yes, it should. >>>>>>> On Apr 14, 2014 4:28 PM, "√iktor Ҡlang" <[email protected]> wrote: >>>>>>> >>>>>>>> Should it poll if nobody is interested in reading anything? >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 14, 2014 at 5:15 PM, Filippo De Luca < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> No one can help or maybe the question is too silly? >>>>>>>>> On Apr 12, 2014 10:20 AM, "Filippo De Luca" < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> I need to have actors reacting to some events arriving from some >>>>>>>>>> kafka topics. I would like leverage on the akka supervisor mechanism >>>>>>>>>> to >>>>>>>>>> keep the kafka consumer app and running. >>>>>>>>>> >>>>>>>>>> I know that an actor should never block waiting for some IO, but >>>>>>>>>> the kafka consumer is blocking so having an actor running a kafka >>>>>>>>>> consumer >>>>>>>>>> doesn't seem a good option, also because it will never release the >>>>>>>>>> thread. >>>>>>>>>> >>>>>>>>>> by other hand if I will have a thread, not controlled by akka, to >>>>>>>>>> run the kafka consumer, it will not leverage on the akka supervision. >>>>>>>>>> >>>>>>>>>> My idea is to use an actor, with a dedicated dispatcher, to run >>>>>>>>>> the kafka consumer. The problem is: if this actor is blocked by the >>>>>>>>>> kafka >>>>>>>>>> stream, how can I make this actor reacting to some messages? I am >>>>>>>>>> thinking >>>>>>>>>> using a timeout on kafka consumer and resent a message like Consume >>>>>>>>>> when >>>>>>>>>> the timeout is elapsed, so I can consume other messages. It works if >>>>>>>>>> the >>>>>>>>>> kafka stream doesn't have always data available (in the timeout >>>>>>>>>> window), >>>>>>>>>> but doesn't work otherwise. >>>>>>>>>> >>>>>>>>>> Another option may implement a mailbox using kafka. >>>>>>>>>> >>>>>>>>>> What is the best approach in your opinion? Thanks. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>>>>> urrent/additional/faq.html >>>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>>>>> p/akka-user >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Akka User List" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>> >>>>>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>>>> urrent/additional/faq.html >>>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>>>> p/akka-user >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Akka User List" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>> >>>>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Cheers, >>>>>>>> √ >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>>>> urrent/additional/faq.html >>>>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>>>> p/akka-user >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Akka User List" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> >>>>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>>> urrent/additional/faq.html >>>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>>> p/akka-user >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Akka User List" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> >>>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Cheers, >>>>> √ >>>>> >>>>> -- >>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/c >>>>> urrent/additional/faq.html >>>>> >>>>>>>>>> Search the archives: https://groups.google.com/grou >>>>> p/akka-user >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Akka User List" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> >>>>> Visit this group at http://groups.google.com/group/akka-user. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- *Filippo De Luca* --------------------- WWW: http://filippodeluca.com IM: [email protected] -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
