On 28/05/14 17:01 +0000, Kurt Griffiths wrote:
Crew, as discussed in the last team meeting, I have updated the API v1.1 spec to remove the ability to get one or more messages by ID. This was done to remove unnecessary complexity from the API, and to make it easier to supportdifferent types of message store backends.However, this now leaves us with asymmetric semantics. On the one hand, we do not allow retrieving messages by ID, but we still support deleting them by ID. It seems to me that deleting a message only makes sense in the context of a claim or pop operation. In the case of a pop, the message is already deleted by the time the client receives it, so I don’t see a need for including a message ID in the response. When claiming a batch of messages, however, the client still needs some way to delete each message after processing it. In this case, we either need to allow the client to delete an entire batch of messages using the claim ID, or we still need individual message IDs (hrefs) that can beDELETEd.Deleting a batch of messages can be accomplished in V1.0 using “delete multiple messages by ID”. Regardless of how it is done, I’ve been wondering if it is actually an anti-pattern; if a worker crashes after processing N messages, but before deleting those same N messages, the system is left with several messages that another worker will pick up and potentially reprocess, although the work has already been done. If the work is idempotent, this isn’t a big deal. Otherwise, the client will have to have a way to check whether a message has already been processed, ignoring it if it has. But whether it is 1 message or N messages left in a bad state by the first worker, the other worker has to follow the same logic, so perhaps it would make sense after all to simply allow deleting entire batches of claimed messages by claim ID, and not worrying about providing individual message hrefs/IDs for deletion.
There are some risks related to claiming a set of messages and process them in batch rather than processing 1 message at a time. However, some of those risks are valid for both scenarios. For instance, if a worker claims just 1 message and dies before deleting it, the server will be left with an already processed message. I believe this is very specific to the each use-case. Based on their needs, users will have to choose between 'pop'ng' messages out of the queue or caliming them. One way to provide more info to the user is by keeping track of how many times (or even the last time) a message has been claimed. I'm not a big fan of this because it'll add more complexity and more importantly we won't be able to support this on the AMQP driver. It's common to have this kind of 'tolerance' implemented in the client-side. The server must guarantee the delivery mechanism whereas the client must be tolerant enough based on the use-case.
With all this in mind, I’m starting to wonder if I should revert my changes to the spec, and wait to address these changes in the v2.0 API, since it seems that to do this right, we need to make some changes that are anything but “minor” (for a minor release).What does everyone think? Should we postpone this work to 2.0?
I think this is quite a big change to have in a minor release. I vote for doing this in v2.0 and I'd also like us to put more thoughts on it. For example, accessing messages by-id seems to be an important feature in SQS. I'm not saying the decision must be based on that but since both Marconi's and SQS's targets are very similar, we should probably take a deeper look at the utility. Unfortunately, as mentioned above, it'll be hard to support this on the AMQP driver. Flavio -- @flaper87 Flavio Percoco
pgpWtYRog7Qj_.pgp
Description: PGP signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
