Thanks Flavio, inline. On 6/13/14, 1:37 AM, "Flavio Percoco" <[email protected]> wrote:
>On 11/06/14 18:01 +0000, Janczuk, Tomasz wrote: >>Thanks Flavio, some comments inline below. >> >>On 6/11/14, 5:15 AM, "Flavio Percoco" <[email protected]> wrote: >> >>>> >>>> 1. Marconi exposes HTTP APIs that allow messages to be listed >>>>without >>>>consuming them. This API cannot be implemented on top of AMQP 0.9 which >>>>implements a strict queueing semantics. >>> >>>I believe this is quite an important endpoint for Marconi. It's not >>>about listing messages but getting batch of messages. Wether it is >>>through claims or not doesn't really matter. What matters is giving >>>the user the opportunity to get a set of messages, do some work and >>>decide what to do with those messages afterwards. >> >>The sticky point here is that this Marconi¹s endpoint allows messages to >>be obtained *without* consuming them in the traditional messaging system >>sense: the messages remain visible to other consumers. It could be argued >>that such semantics can be implemented on top of AMQP by first getting >>the >>messages and then immediately releasing them for consumption by others, >>before the Marconi call returns. However, even that is only possible for >>messages that are at the front of the queue - the "paging" mechanism >>using >>markers cannot be supported. > >What matters is whether the listing functionality is useful or not. >Lets not think about it as "listing" or "paging" but about getting >batches of messages that are still available for others to process in >parallel. As mentioned in my previous email, AMQP has been a good way >to analyze the extra set of features Marconi exposes in the API but I >don't want to make the choice of usability based on whether >traditional messaging systems support it and how it could be >implemented there. This functionality is very useful in a number of scenarios. It has traditionally been the domain of database systems - flexible access to data is what DBs excel in (select top 1000 * from X order by create_date). Vast majority of existing messaging systems has a much more constrained and prescriptive way of accessing data than a database. Why does this functionality need to be part of Marconi? What are the benefits of listing messages in Marconi that cannot be realized with a plain database? In other words, if I need to access my data in that way, why would I use Marconi rather than a DB? > >>>> 5. Marconi message consumption API creates a ³claim ID² for a set of >>>>consumed messages, up to a limit. In the AMQP 0.9 model (as well as SQS >>>>and Azure Queues), ³claim ID² maps onto the concept of ³delivery tag² >>>>which has a 1-1 relationship with a message. Since there is no way to >>>>represent the 1-N mapping between claimID and messages in the AMQP 0.9 >>>>model, it effectively restrict consumption of messages to one per >>>>claimID. This in turn prevents batch consumption benefits. >>>> >>>> 6. Marconi message consumption acknowledgment requires both claimID >>>>and messageID to be provided. MessageID concept is missing in AMQP 0.9. >>>>In order to implement this API, assuming the artificial 1-1 restriction >>>>of claim-message mapping from #5 above, this API could be implemented >>>>by >>>>requiring that messageID === claimID. This is really a workaround. >>>> >>> >>>These 2 points represent quite a change in the way Marconi works and a >>>trade-off in terms of batch consumption (as you mentioned). I believe >>>we can have support for both things. For example, claimID+suffix where >>>suffix point to a specific claimed messages. >>> >>>I don't want to start an extended discussion about this here but lets >>>keep in mind that we may be able to support both. I personally think >>>Marconi's claim's are reasonable as they are, which means I currently >>>like them better than SQS's. >> >>What are the advantages of the Marconi model for claims over the SQS and >>Azure Queue model for acknowledgements? >> >>I think the SQS and Azure Queue model is both simpler and more flexible. >>But the key advantage is that it has been around for a while, has been >>proven to work, and people understand it. >> >>1. SQS and Azure require only one concept to acknowledge a message >>(receipt handle/pop receipt) as opposed to Marconi¹s two concepts >>(message >>ID + claim ID). SQS/Azure model is simpler. > >TBH, I'm not exactly sure where you're going with this. I mean, the >model may look simpler but it's not necessarily better nor easier to >implement. Keeping both, messages and claims, separate in terms of IDs >and management is flexible and powerful enough, IMHO. But I'm probably >missing your point. > >I don't believe requiring the messageID+ClaimID to delete a specific, >claimed, messages is hard. It may not be hard. It is just more complex than it needs to be to accomplish the same task. > >> >>2. Similarly to Marconi, SQS and Azure allow individual claimed messages >>to be deleted. This is a wash. > >Calling it a wash is neither helpful nor friendly. Why do you think it >is a wash? > >Claiming a message does not delete the message, which means consumers >may want to delete it before the claim is released. Do you have a >better way to do it? By "wash" I mean neither SQS/Azure or Marconi offers something the other does not related to this aspect. I don't see a better way of doing it, this is pretty standard. > >> >>3. SQS and Azure allow a batch of messages *up to a particular receipt >>handle/pop receit* to be deleted. This is more flexible than Marconi¹s >>mechanism or deleting all messages associated with a particular claim, >>and >>works very well for the most common scenario of in-order message >>delivery. > >Pop semantic is on its way to the codebase. Limited claim deletes >sounds like an interesting thing, lets talk about it. Want to submit a >new spec? I will be happy to propose a spec, but I'd rather take a more comprehensive approach than addressing this single issue. We have discussed a number of problems both on this thread and outside of it. Perhaps all this learning should come together as a V2 of the HTTP API. > >>>> 7. RabbitMQ message acknowledgment MUST be sent over the same AMQP >>>>channel instance on which the message was originally received. This >>>>requires that the two Marconi HTTP calls that receive and acknowledge a >>>>message are affinitized to the same Marconi backend. It either >>>>substantially complicates driver implementation (server-side reverse >>>>proxing of requests) or adds new requirements onto the Marconi >>>>deployment (server affinity through load balancing). >>>> >>> >>>Nothing to do here. I guess a combination with a persistent-transport >>>protocol would help but we don't want to make the store driver depend >>>on the transport. >> >>The bigger issue here is that the storage layer abstraction has been >>design with a focus on a set of atomic, *independent* operations, with no >>way to correlate them or share any state between them. This works well >>for >>transport layers that are operation-centered, like HTTP. This abstraction >>is likely to be insufficient when one wants to use a connection-oriented >>transport (e.g. WebSocket, AMQP, MQTT), or a storage driver based on a >>connection-oriented protocol (e.g. AMQP). Without explicit modeling of a >>state that spans several atomic calls to the storage driver, in >>particular >>around the lifetime of the connection, current storage abstraction cannot >>satisfy these configurations well. > >This is the current status, True. However, it was not always like >that. We decided to go this way until we actually needed to support >other transports (YAGNI). The time to revisit this is, perhaps, >coming. > >That said, I agree that some things should change in the storage to >support other transports. But this is a different discussion from the >changes in the API to support other storage drivers. Yes, this is a separate discussion. I will start a new thread to talk about this. > >Flavio > >-- >@flaper87 >Flavio Percoco >_______________________________________________ >OpenStack-dev mailing list >[email protected] >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
