You could also have 1 non-blocking worker (actor) per queue and 1 blocking worker per queue (actor), Akka has a nice message ordering policy so 1 actor will only process one message at a time and order will be kept.
Even if you have millions of queues, such actors can still run with a dispatcher with few threads. It might be difficult to understand why you don't need to lock anything but think again and read Akka message ordering, you will realize it is simpler than what you think. HTH, Guido. On Tuesday, June 28, 2016 at 9:30:38 AM UTC+1, Guido Medina wrote: > > There are community contributions that can poll messages from a JMS queue, > then you could send these messages to a worker actor, > such actor then can forward that message to another set of worker running > on a blocking dispatcher that do the job, then answer the processed message > back to the original worker and > let such worker acknowledge the original JMS message (if you want to ack) > > 1st set of worker can be something between 1 to N where N can be for > example amount of CPUs, which makes it a router. > 2nd set of worker can be the same as 1st set but running a different > dispatcher. > > Later one today I will give you sample code of how to define separate > dispatchers and create N instances of an actor and then create a router > with it or ... > you can read the documentation, it is very easy to do all that, each > worker can be created with a Creator (read in the docs as creators are good > to pass constructor parameters) > > If you still have doubts when I get home I'mm paste some sample code. > > HTH, > > Guido. > > On Tuesday, June 28, 2016 at 8:58:37 AM UTC+1, Guido Medina wrote: >> >> I try to understand the problem he is describing, but when someone >> suggest a solution it confuses me, >> specially when the solution shows little knowledge of the framework being >> used. >> >> His question is akka related, he wants to resolve his problem with Akka, >> >> he doesn't need to use an ESB to resolve a problem that the key circle >> around three things: >> >> 1. Queues (Which actors have) >> 2. Thread pools (Dispatchers) for specific actions (Which Akka >> provides) >> 3. Coordination (Which Akka provides with many other things) >> >> The problem is not the problem that he is trying to resolve, it is the >> little knowledge of Akka most people have before trying to tackle a complex >> problem. >> I myself came from Java too, 1 year ago I knew nothing about Akka but the >> first thing I read was the manual, then the solution to the system I had to >> develop from scratch and by myself came along. >> >> It is just a matter of most people being lazy and thinking than a simply >> copy & paste will solve the problem, desperation is a bad thing ;-) >> one week reading will be the best spent time, the rest will just come >> easily. >> >> HTH, >> >> Guido. >> >> On Tuesday, June 28, 2016 at 12:51:44 AM UTC+1, Gavin Baumanis wrote: >>> >>> I can't answer your akka questions, sorry... >>> >>> And I realise it isn't Akka based... >>> But considering you are using Java already... you might like to consider >>> http://open-esb.net/ >>> It has the features you require. >>> >>> We use it extensively in health because of its HL7 Messaging >>> capabilities, but it isn't a HL7 solution - as much as it is an integration >>> / messaging / SOA platform. >>> >>> -Gavin. >>> >>> On Tuesday, 28 June 2016 01:24:54 UTC+10, benoit heinrich wrote: >>>> >>>> Hi all, >>>> >>>> I've been playing with akka a few years ago, and I'm just back to it, >>>> trying to implement some kind of internal queueing mechanism on top of a >>>> very big java8 application which perform a lot of message processing. >>>> >>>> >>>> *Background:* >>>> >>>> The current application is a message processing application which >>>> receive JMS messages (using AMQ) and that need to process them. >>>> The application has a notion of a subscriber for a given JMS message >>>> type, and when a message is received, all the subscribers which are >>>> interested in that message type are executed one after the other. >>>> The problem is that some of those subscribers require some kind of >>>> pessimistic locks on some resources, and that causes a lot of threads to >>>> just wait for the resource to be acquired before anything else happen. >>>> >>>> >>>> *Solution:* >>>> >>>> The change I've made (which uses actor system) is that when the JMS >>>> message is received, it sends the message to an actor (a >>>> MessageConsumerActor) based on the resource that needs to be locked, >>>> and then the actor sends the work to a child (a MessageExecutorActor) >>>> one at a time to lock the resource, and execute the actual subscriber code. >>>> The MessageConsumerActor is then managing the queue of messages to be >>>> processed, and given work to the MessageExecutorActor one at a time, >>>> and the MessageExecutorActor is in charge to acquire the lock. >>>> Because the MessageExecutorActor / MessageConsumerActor actor couple >>>> is unique for that specific locked resource, no other processes try to >>>> lock >>>> that resource, and then acquiring the lock is very quick as no other >>>> threads try to acquire that lock a the same time. With this the messages >>>> get executed nicely from the queue, one after the other. >>>> >>>> The class which receive the message from JMS, is something which >>>> unfortunately doesn't support asynchronous calls (it needs to fulfill an >>>> existing API). >>>> The API consider that once the method returns, then the JMS message is >>>> marked as processed, and if it throws an exception, then the transaction >>>> which is around the JMS message delivery is failed. >>>> In order to make sure that I don't lose messages, when a JMS message is >>>> received, I return only when I get confirmation that the message have been >>>> queued by the MessageConsumerActor. >>>> Unfortunately, the only way I've found to achieve this is to use >>>> Await.result() method. >>>> >>>> >>>> *Problem:* >>>> >>>> All this works greatly until I've got too many messages to process, at >>>> that time I receive timeouts when calling the Await.result() from the >>>> MessageConsumerActor, even though that actor should be very fast to >>>> acknowledge the result. >>>> >>>> I think the problem might be due to those subscribers which are in >>>> general quite slow (from a few seconds to sometime a minute) to execute. >>>> When a message is received on the MessageExecutorActor, it then calls >>>> the subscriber call in the same thread, and so that thread is getting busy >>>> with some very complex computation for a few seconds. >>>> When there is a lot of messages received, then I've got the feeling >>>> that all the threads get busy running into those subscribers, and if >>>> another JMS message is received during that time, then I get a timeout due >>>> to the MessageConsumerActor not replying quick enough. >>>> >>>> >>>> *What I tried:* >>>> >>>> I've been googling on this, and I've watched a few (very interesting) >>>> video including the >>>> http://boldradius.com/blog-post/U-jexSsAACwA_8nr/dos-and-donts-when-deploying-akka-in-production >>>> >>>> . >>>> What I get from this is that already, I should never call the >>>> Await.result() method, but then how can I get the feedback that my >>>> message has been queued properly from the JMS thread (thread which has a >>>> transaction context associated to it)? >>>> >>>> Then the next thing I tried is to use a different dispatcher for the >>>> MessageExecutorActor, and another one for the MessageConsumerActor. >>>> I was hoping this would just work by magic, but unfortunately it didn't >>>> ;) >>>> >>>> The way I've used it is by adding two new dispatcher configuration in >>>> my application.conf and then referencing them in the props when >>>> creating my actors. >>>> I could see using JVisualVM (and watching logs) that the threads that >>>> now performed my application were named by the name of the dispatchers I >>>> gave in the configuration. >>>> >>>> I've tried lot of different configurations, but none worked, and I'm >>>> starting to desperate here. >>>> >>>> Here is the last configuration I tried: >>>> >>>> # Dispatcher used by the ActorSystemPublisher and MessageConsumerActor to >>>> allow messages to be queued >>>> queuing-dispatcher { >>>> type = Dispatcher >>>> executor = "fork-join-executor" >>>> fork-join-executor { >>>> parallelism-min = 1 >>>> parallelism-max = 1 >>>> } >>>> # Because a single message in general doesn't have more much >>>> subscribers, a value of 20 should be enough >>>> throughput = 1 >>>> } >>>> >>>> # Dispatcher used by the MessageExecutorActor to execute messages by the >>>> subscribers >>>> executing-dispatcher { >>>> type = Dispatcher >>>> executor = "thread-pool-executor" >>>> thread-pool-executor { >>>> parallelism-min = 1 >>>> parallelism-max = 4 >>>> } >>>> # Because a single locked resource could have lot of messages to be >>>> processed, and because processing a message >>>> # could be time consuming, we want to allow as much fairness for each >>>> resource to be executed. >>>> throughput = 1 >>>> } >>>> >>>> >>>> The queuing-dispatcher is used by the MessageConsumerActor, and the >>>> executing-dispatcher is used by the MessageExecutorActor. >>>> >>>> >>>> As you can see I'm trying to use fork-join-executor and >>>> thread-pool-executor combinations, and all the possible variations, >>>> but none worked. >>>> I tried different parallelism, but none work neither. >>>> I tried to force a single thread for the MessageExecutorActor using >>>> parallelism-max=1 but that didn't work neither :( >>>> >>>> Could someone please let me know how this kind of issue is being solved? >>>> Am I on the right track with the dispatcher configuration? >>>> Do I need to use some kind of routers, and if so, how? >>>> >>>> >>>> Thanks in advance for all the suggestions :) >>>> >>>> Cheers, >>>> /Benoit >>>> >>> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
