Hi, What about the following solution:
Create cache: IgniteCache<Integer, Event>, where key is growing integer. Asssign keys using IgniteAtomicSequence [1] Listen for cache put events When put is done and event group id is "next", process all entries from cache where id < event.key [1] https://apacheignite.readme.io/docs/id-generator 2016-06-05 15:19 GMT+03:00 zshamrock <[email protected]>: > Are there features in Ignite which would support running an infinite (while > the cluster is up and running) job? For example, continuously reading > values > from the distributed queue? So to implement producer/consumer pattern, > where > there could be multiple producers, but I want to limit number of consumers, > ideally per specific key/group or if it is not possible, just to have one > consumer per queue. > > If I asynchronously submit affinity ignite job with `queue.affinityRun()` > what is the implication of the this job never to finish? Will it consume > the > thread from the ExecutorService thread pool on the running node forever > then? > > To give a better a context, this is the problem I am trying to solve (maybe > there are even other approaches to solve it, and I am looking into the > completely wrong direction?): > - there are application events coming periodically (based on the > application > state changes) > - I have to accumulate these events until the block of the events is > "complete" (completion is defined by an application rule), as until the > group is complete nothing can be done/processed > - when the group is complete I have to process all of the events in the > group (as one complete chunk), while still accepting new events coming for > now another "incomplete" group > - and repeat since the beginning > > So, far I came with the following solution: > - collect and keep all the events in the distributed IgniteQueue > - when the application notifies the completion of the group, I trigger > `queue.affinityRun()` (as I have to do a peek before removing the event > from > the queue, so I want to run the execution logic on the node where the queue > is stored, they are small and run in collocated mode, and so peek will not > do an unnecessary network call) > [the reason for a peek, is that even if I receive the application event of > the group completion, due to the way events are stored (in the queue), I > don't know where the group ends, only where it starts (head of the queue), > but looking into the event itself, I can detect whether it is still from > the > same group, or already from a new incomplete group, this is why I have to > do > peek, as if I do poll/take first then I have to the put the element back > into the head of the queue (which obviously is not possible, as it is a > queue and not a dequeue), then I have to store this element/event somewhere > else, and on the next job submitted start with this stored event as a > "head" > of the queue, and only then switch back to the real queue. As I don't want > this extra complexity, I am ready to pay a price for an extra peek before > the take] > - implement custom CollisionSpi which will understand whether there is > already a running job for the given queue, and if so, keeps the newly > submitted job in the waiting list > [here again due to the fact how events are stored (in the queue) I don't > allow multiple jobs running against same queue at the same time, as taking > the element from the middle of one group already processing group is > obviously an error, so I have to limit (to 1) the number of parallel jobs > against the given queue] > - it also requires to submit a new ignite job (distributed closure) on the > queue every time the application triggers/generates a completion group > event, which requires/should schedule a queue processing (also see above on > the overall number of the simultaneous jobs) > > I thought about other alternative solutions, but all of them turned out to > be more complex, and involve more moving parts (as for example, for the > distributed queue Ignite manages atomicity, and consistency, with other > approaches I have to do it all manually, which I just want to minimize) and > more logic to maintain and ensure correctness. > > Is there any other suitable alternative for the problem described above? > > > > > > -- > View this message in context: > http://apache-ignite-users.70518.x6.nabble.com/Running-an-infinite-job-use-case-inside-or-alternatives-tp5430.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Best regards, Alexei Scherbakov
