First of all, I think the removal of ³get message[s] by ID² is a great change, it moves Marconi APIs closer to a typical messaging semantics. However, I still see the ³list messages² API in the spec (https://wiki.openstack.org/wiki/Marconi/specs/api/v1.1#List_Messages). Is it the plan to leave this API in, or is it also going to be removed? If the motivation for removing the ³get message[s] by ID² was to make it easier to support different store backends (e.g. AMQP), I would expect the same argument to apply to the ³list messages² API which allows random access to messages in a queue.
Regarding deleting claimed messages, I think it should be possible to claim multiple messages but then delete any of them individually. For reference, that is the semantics of a ³batch claim² that SQS, Azure, and IronMQ have - during batch claim a number of messages can be claimed, but each of them is assigned a unique ³token² than can later be used to delete just that message. I believe there is a good reason it is organized that way: even if a batch of messages is claimed, their processing can be completely unrelated, and executed in different time frames. It does not make sense to make logical completion of message A conditional on the success of processing of message B within a batch. It also does not make sense to hold up completion of all messages in a batch until the completion of the message that takes most time to process. Put it another way, a batch is merely an optimization over the atomic operation of claiming and deleting a single message. The optimization can allow multiple messages to be claimed at once; it can also allow multiple messages to be deleted at once (I believe AMQP has that semantics); but it should not prevent the basic use case of claiming or deleting of a single message. On 5/29/14, 1:22 AM, "Flavio Percoco" <[email protected]> wrote: >On 28/05/14 17:01 +0000, Kurt Griffiths wrote: >>Crew, as discussed in the last team meeting, I have updated the API v1.1 >>spec >>to remove the ability to get one or more messages by ID. This was done to >>remove unnecessary complexity from the API, and to make it easier to >>support >>different types of message store backends. >> >>However, this now leaves us with asymmetric semantics. On the one hand, >>we do >>not allow retrieving messages by ID, but we still support deleting them >>by ID. >>It seems to me that deleting a message only makes sense in the context >>of a >>claim or pop operation. In the case of a pop, the message is already >>deleted by >>the time the client receives it, so I don¹t see a need for including a >>message >>ID in the response. When claiming a batch of messages, however, the >>client >>still needs some way to delete each message after processing it. In this >>case, >>we either need to allow the client to delete an entire batch of messages >>using >>the claim ID, or we still need individual message IDs (hrefs) that can be >>DELETEd. >> >>Deleting a batch of messages can be accomplished in V1.0 using ³delete >>multiple >>messages by ID². Regardless of how it is done, I¹ve been wondering if it >>is >>actually an anti-pattern; if a worker crashes after processing N >>messages, but >>before deleting those same N messages, the system is left with several >>messages >>that another worker will pick up and potentially reprocess, although the >>work >>has already been done. If the work is idempotent, this isn¹t a big deal. >>Otherwise, the client will have to have a way to check whether a message >>has >>already been processed, ignoring it if it has. But whether it is 1 >>message or N >>messages left in a bad state by the first worker, the other worker has to >>follow the same logic, so perhaps it would make sense after all to >>simply allow >>deleting entire batches of claimed messages by claim ID, and not >>worrying about >>providing individual message hrefs/IDs for deletion. > >There are some risks related to claiming a set of messages and process >them in batch rather than processing 1 message at a time. However, >some of those risks are valid for both scenarios. For instance, if a >worker claims just 1 message and dies before deleting it, the server >will be left with an already processed message. > >I believe this is very specific to the each use-case. Based on their >needs, users will have to choose between 'pop'ng' messages out of the >queue or caliming them. One way to provide more info to the user is by >keeping track of how many times (or even the last time) a message has >been claimed. I'm not a big fan of this because it'll add more >complexity and more importantly we won't be able to support this on >the AMQP driver. > >It's common to have this kind of 'tolerance' implemented in the >client-side. The server must guarantee the delivery mechanism whereas >the client must be tolerant enough based on the use-case. > >> >>With all this in mind, I¹m starting to wonder if I should revert my >>changes to >>the spec, and wait to address these changes in the v2.0 API, since it >>seems >>that to do this right, we need to make some changes that are anything but >>³minor² (for a minor release). >> >>What does everyone think? Should we postpone this work to 2.0? > >I think this is quite a big change to have in a minor release. I vote >for doing this in v2.0 and I'd also like us to put more thoughts on >it. For example, accessing messages by-id seems to be an important >feature in SQS. I'm not saying the decision must be based on that but >since both Marconi's and SQS's targets are very similar, we should >probably take a deeper look at the utility. Unfortunately, as >mentioned above, it'll be hard to support this on the AMQP driver. > >Flavio > >-- >@flaper87 >Flavio Percoco >_______________________________________________ >OpenStack-dev mailing list >[email protected] >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
