I'm quite sure that inspecting the mbox will be costlier than reprocessing at those sizes.
Come up with two different solutions that you could perform between humans having mailboxes. Pick the best of those. -- Cheers, √ On May 8, 2016 5:15 PM, "kraythe" <[email protected]> wrote: > I have a process that has to manage a large amount of data. I want to make > the process reactive but it has to process every data element. The data is > stored in Hazelcast in a map (which is backed by a database but that detail > is irrelevant) and the data is stageful. At each state change something has > to be done either to the data or related data. So if we go from State A to > State B we might have to do something to another object in the process in a > transactional manner. When the data is in state A a process finds the data > and submits it to a map. Right now I have another thread reading from the > map on intervals that are timed and if there is data in the map it > processes the next entry in the map and so on. > > I would like to turn this process into an Akka actor process but the main > stumbling block is to know what is already in the queue. Say I have 1m > objects to process. At each interval the objects are checked if they can > change state and if they can then they are put in the map to process. The > problem is there could be a ton of these objects and they might take longer > to process than the check interval. Furthermore, although it would not be > damaging to the data it would be immensely wasteful to put them into the > queue to process twice. Finally if the server crashed or something happened > I would want to put them back into the queue if they are still in state A > and should move to state B. Right now I can get the key set of the map and > not submit them to the process if they are already in the map. If, instead, > I change the system to Akka, then that ability changes. Whenever an object > needs to change state, I would put it in a message inbox to an actor to > process but I have no way to know what is already in that inbox so it makes > the processing of the messages less durable. If a transaction fails or node > fails I won't know that certain objects need to be processed again. Right > now I can search the store, get the objects to process, remove all the ones > in the queue and then put only the missing ones in the queue. I don't know > how I could architect this with akka. > > What I would be looking for is some means to inspect an inbox to know if a > message has already been enqueued and should not be enqueued again. > > Any suggestions on how I could architect a solution to this problem? > > -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
