Transactions only cover data mutation. Your transaction scheme seems like it wants to include the contents of a message queue.
What you want is not really a queue. What you want is more like a reservation list. Workers can claim units of work by "reserving" an item in a collection, but that reservation is revocable by a recovery feature. What's not clear is why a first attempt to process a queued item would fail and another attempt would succeed. In my experience it is better to catch the transaction rolling back and re-queue the failed item in a separate exception handling work queue so that bad data trouble isn't passed around the cluster like a beach ball in a stadium. You might look into this if you haven't already... https://ignite.apache.org/docs/latest/data-structures/queue-and-set Remember Ignite is open source, so you are encouraged to experiment and collaborate and submit PRs for anything the community might benefit from. Maybe you can get the Ignite distributed data structures to do what you want? On Wed, Mar 5, 2025 at 11:52 AM Alex Roytman <roytm...@gmail.com> wrote: > Use case is for extended time processing of the messages which guarantee > that a message is not lost from the queue if a client taking it fails to > complete its processing and is not able to return it to the queue (ex > crash, process termination, bug). > It is how AWS defines its queue API > https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html > but it might be simpler for them because I think their queue is not a > blocking queue and relies on polling > > - There are multiple consumers of a queue > - Message posted to the queue > - Consumer A take message off the queue with Visibility Timeout 60 sec > (message is not removed from queue but instead it becomes invisible for 60 > sec so that subsequent messages can be taken) > - Other consumers do not see the message and can take subsequent messages > - Consumer A deletes the message before 60 sec are over and the workflow > is done > - Consumer A failed to delete the message within 60 seconds and it > reappears on the queue and will be taken by the next waiting consumer > - Consumer A may need to monitor processing and extend visibility timeout > for the message in processing > > Without something like that (a two step queue processing - logical removal > from queue (visibility) followed by physical removal upon completion of > processing) it is necessary to have some sort of recovery on node startup > and/or timer should a process take an entry from a queue and fail to > complete processing or return it to the queue > > BTW how ignite transactions work if multiple queues are modified within > transaction > - start transaction > - take from A > - copy to B > - remove or create an entry to C > - commit > > if c mutation fails and I rollback transaction everything will be rolled > back? > > Is there a more detailed description of transactional behavior > particularly with a queue where order of entries is also involved? I only > see https://ignite.apache.org/docs/latest/key-value-api/transactions > > > > On Wed, Mar 5, 2025 at 10:50 AM Jeremy McMillan <j...@gridgain.com> wrote: > >> I don't think it's clear what you're trying to accomplish. Can you >> describe the desired behavior as a narrative? >> >> >> On Tue, Mar 4, 2025 at 5:42 PM Alex Roytman <roytm...@gmail.com> wrote: >> >>> Hello I was wondering if there is anything in Ignite queue to provide >>> behaviour similar to Queue Visibility Timeout or Processing Completion to >>> support long running processing of queue items >>> >>> When item is taken of a queue it becomes "invisible" to subsequent queue >>> reads until its configurable visibility times out. By that time is >>> should be processed and removed by the consumer or its timeout extended or >>> if consumer failed it should reappear on the queue >>> >>> Perhaps it can be achieved with transactions but I never fully >>> understood how transactions would work for queue consumed concurrently. >>> Perhaps transactions with main queue and journaling queue/list where >>> picked items copied transactionally upon taking them off the main queue >>> (take and push happen in the same transaction) and then pushed to end of >>> main queue upon journaling queue expiration >>> >>> I would appreciate your take on it and would like to encourage >>> implementing it when you get to work on distributed data structures in >>> Ignite 3 (and when it may be? :-) >>> >>
