Hi Christian,
On Tue, Aug 26, 2014 at 11:34 AM, Christian Kreutzfeldt <[email protected]> wrote: > Hi Viktor, > > thanks for the quick reply. What I am concerned about is the way how to > read from websocket > Read it from the websocket and store it into Kafka, and have the Kafka-consumer actor reused for the work-distribution? > .. kafka is easy as I simply pull data and it does not matter when I do > that, I always receive the next message. But when it comes to websockets, > the server pushes data and I need to forward the data somehow. When I do > not consume the message, it is lost. I use the WebSocketClient provided by > the Jetty framework but that one internally spawns a thread reading from > the socket forwarding data to a callback provided by me. As far as I > understood, spawning threads inside actors isn't a great idea :-) ...or am > I wrong? Do you have any suggestion how to solve this or point me to a > resource which provides any information about how to handle similar > problems ... unfortunately google does not help in that case :-( > You could use the Play WebSocket API. > > Regarding the kafka consumer, actually I have an implementation which does > exactly what you described: read single message from kafka, forward > message, issue next read request via scheduler / directly > getSelf().tell(..), wait for read request, read single message from kafka > .... I could keep on reading forever from kafka but that would block the > actor. But unfortunately, I have to wait for the next read request to > access kafka. Somehow it is a nice approach especially if a processing > actor issues that message since it ensures that it has resources available > for processing the next message. On the other hand it slows down the whole > process and leaves gaps where nothing is processed. From a software > architecture aspect this is a clean solution but I need to process data > quite fast ;-) > You could have the Kafka-actor batch-read up to X messages to not be fully lock-step with the consumer-actor. > > Kind regards, > Christian > > Am Dienstag, 26. August 2014 11:22:17 UTC+2 schrieb √: >> >> Hi Christian, >> >> Reading from websockets or kafka is not something that should be done by >> the mailbox, it's something an actor should do. Then your consumer-actor >> (the one which you wanted to switch mailboxes on) can ask the >> websocket/kafka actor for the next message (pull) or have the >> websocket/kafka actor send the messages to the consumer-actor. >> >> Does that help? >> >> >> On Tue, Aug 26, 2014 at 11:14 AM, Christian Kreutzfeldt <[email protected] >> > wrote: >> >>> Hi >>> >>> I am working on a streaming project where I need to consume different >>> source types on demand. To comply with the akka model I decided to >>> implement dedicated mailbox types which >>> establish a connection with the desired source and read data from it, >>> eg. kafka topics or websockets. As I do not know which source to consume >>> before starting the application I cannot >>> provide the full configuration parameters but need to set them during >>> runtime. >>> >>> What I would like to know is, if there is a chance to get configuration >>> data into a mailbox when it is instantiated. The data must not be read from >>> a configuration file but may be provided >>> by the caller on demand. >>> >>> The only solution I actually thought of was to ramp up the actor + >>> mailbox without any configuration and send a message holding the >>> configuration right after its instantiation. The mailbox >>> sees the message, extracts the information and configures the source >>> listener as required. The message might be removed from the mailbox or be >>> provided to the actor - depends on >>> spec. >>> >>> In some way I dislike this solution as it makes me react to messages on >>> two different layers: the mailbox and the actor. >>> >>> I tried to establish the source connection from within an actor, >>> scheduled READ MESSAGES to itself and on receiving this message type the >>> actor takes a single message from >>> the source. The drawback of this solution is a reduced throughput as >>> each READ MESSAGE must be awaited and then the source must be consumed. It >>> allows to throttle the reading >>> speed but it does not allow to speed things up arbitrarily as the READ >>> MESSAGE must be issued .. which takes time :-( >>> >>> Aside from that I must provide another managing layer which handles the >>> reading actors. Therefore I'd love to use the mailbox approach but find it >>> quite annoying to send in control messages. >>> >>> With kind regards, >>> Christian >>> >>> -- >>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>> >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ >>> current/additional/faq.html >>> >>>>>>>>>> Search the archives: https://groups.google.com/ >>> group/akka-user >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Akka User List" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> >>> Visit this group at http://groups.google.com/group/akka-user. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Cheers, >> √ >> > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Cheers, √ -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
