Re: [openstack-dev] [oslo.messaging] [mistral] Acknowledge feature of RabbitMQ in oslo.messaging

[Gordon Sim]

On 07/07/2015 05:48 PM, Clint Byrum wrote:

all of the call sites I checked _do not appear to resend_, they
simply explode on timeout waiting for reply. This is how calling code
should work and I'm ok with code in nova, cinder, et. al. being
written this way, because I'd expect my messaging layer to be at
least somewhat reliable


In my opinion, the calling code has better context for determining 
whether or not to retry. Tackling reliability issues end-to-end is often 
much more efficient also.


[...]

I think you'll find that once you try to make oslo.messaging handle the
retrying, that with the broker simply being ack'd all the time, you risk
duplicating RPC calls if you retry in a loop.


Resending the request will always risk duplicating the call (unless the 
caller can verify that the previous request was not executed in some 
call specific way). Whether or not you acknowledge the request (and 
whether you do it before or after the processing of the request), the 
response can still get lost (neither requests nor responses are 
currently confirmed by the broker).


There is a message id 'cache' used to try and detect (and then ignore) 
duplicates. It's not clear to me how effective that is in practice as it 
only tracks the last 16 ids for a given listener. In any case if the 
listener process is restarted, or if the call is redelivered to a 
different server in a group, then the id cache would be of no use.



The pattern is well
established in RabbitMQ that acks should happen _AFTER_ the message has
been consumed and thus should not be duplicated, not before.


That is the pattern for at-least-once delivery, where either processing 
is able to detect that a resent message was already processed or where 
reprocessing it is preferable to not processing it at all.


I *believe* olso.messaging (or impl_rabbit at least) was aiming for an 
at-most-once guarantee (i.e. avoiding duplication at the expense of 
dropped messages). That may be why the acknowledgement is done before 
processing, though since the acknowledgement is asynchronous, that only 
narrows the window it doesn't eliminate it.


I may of course be wrong. It would be great to have some one more 
qualified to comment on the intentions of the design provide some clarity.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [qpid] [zmq] [RabbitMQ] [oslo] Pending deprecation of driver(s).

[Gordon Sim]

On 06/19/2015 07:14 AM, Flavio Percoco wrote:

The qpid family is a set of tools that provide messaging capabilities.
Among those you find qpidd (broker daemon), qpid-proton (amqp1.0
library), qpid-dispatch (message router). It's confusing indeed.


Apache Qpid is an Apache Software Foundation project aiming to help 
promote the AMQP protocol by open, collaborative development of 
libraries, services and tools that use it. It includes many different 
components now. So the 'qpid' in the name of the different components 
denotes the community from which they originated. Hopefully that makes 
it less confusing.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Gordon Sim]

On 06/16/2015 08:51 PM, Alec Hothan (ahothan) wrote:

I saw Sean Dague mention in another email that RabbitMQ is used by 95% of
OpenStack users - and therefore does it make sense to invest in ZMQ (legit
question).


I believe it's used by 95% of users because there is as yet no 
compelling alternative.


The approach taken with the qpid driver was to retain as much of the 
design of the rabbit driver, both in terms of the architecture of the 
driver code itself, and in the underlying broker model it relied on. The 
library used however was based on a different threading model from that 
used for rabbit and deliberately abstracted away from that broker model 
which had a negative effect on the ability to reason about the resulting 
code.


More fundamentally, the qpid driver didn't offer a different design to 
the rabbit driver. It just used a different broker. The broker wasn't 
actually the problem with the rabbit driver though. There was no real 
benefit against which pain encountered during hardening of a less mature 
solution could be offset.


Unfortunately the failure of that effort has left its scars on many in 
the community and continues to colour opinion. I agree that the maturity 
of different solutions needs to be made very clear to user, that there 
must be effective testing under CI, that stale, unmaintained code has to 
be continually removed. There are valuable lessons in that failure which 
we should not ignore. But I don't believe that failure is a reason to 
stifle the emergence of alternative approaches (as described above, the 
qpid driver was not a different approach anyway).


I think store-and-forward is the wrong tool for RPC and end-to-end 
acknowledgement would be better. I think it is better to focus on the 
availability of the communication channel than on consistent replication 
of every single request- and response- message. So I think investing in 
different approaches does make sense.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Gordon Sim]

On 06/12/2015 09:41 PM, Alec Hothan (ahothan) wrote:

One long standing issue I can see is the fact that the oslo messaging API
documentation is sorely lacking details on critical areas such as API
behavior during fault conditions, load conditions and scale conditions.


I very much agree, particularly on the contract/expectations in the face 
of different failure conditions. Even for those who are critical of the 
pluggability of oslo.messaging, greater clarity here would be of benefit.


As I understand it, the intention is that RPC calls are invoked on a 
server at-most-once, meaning that in the event of any failure, the call 
will only be retried by the olso.messaging layer if it believes it can 
ensure the invocation is not made twice.


If that is correct, stating so explicitly and prominently would be 
worthwhile. The expectation for services using the API would then be to 
decide on any retry themselves. An idempotent call could retry for a 
configured number of attempts perhaps. A non-idempotent call might be 
able to check the result via some other call and decide based on that 
whether to retry. Giving up would then be a last resort. This would help 
increase robustness of the system overall.


Again if the assumption of at-most-once is correct, and explicitly 
stated, the design of the code can be reviewed to ensure it logically 
meets that guarantee and of course it can also be explicitly tested for 
in stress tests at the oslo.messaging level, ensuring there are no 
unintended duplicate invocations. An explicit contract also allows 
different approaches to be assessed and compared.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead

[Gordon Sim]

On 05/28/2015 07:20 PM, Fox, Kevin M wrote:

I don't know AMQP very well. At the protocol level, are Exchanges, Queues, etc 
things? Are fanout, durability, and other things required or optional to be 
implemented?


AMQP 1.0 is very different from the pre 1.0 versions. Where 0.8/0.9 
focus on defining the internals of a particular broker implementation 
(queues, exchanges, bindings etc) and 1.0 does not dictate any of that 
and focuses instead on the protocol for transferring messages between 
processes.


So supporting AMQP 1.0 wouldn't require zaqar to adopt any new model. 
It's just an alternative protocol/encoding on the wire for getting 
messages into (and out of) the zaqar service.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead

[Gordon Sim]

On 05/28/2015 05:14 PM, Fox, Kevin M wrote:

Thats like saying you should implement a sql engine inside of
Berkeley DB since so many folks like sql... Not a good fit. If you
want AMQP, get an AMQP server. Zaqar's intentionally lighter weight
then that. Hardly any OpenStack services use but a fraction of AMQP
though, so getting oslo.messaging to support that simple subset of
AMQP on top of Zaqar is much more reasonable then getting Zaqar to
support all of AMQP.


AMQP 1.0 is just a protocol for asynchronously transferring messages 
between processes. Supporting it doesn't require that zaqar provide any 
functionality that it doesn't want to. It is simply another protocol for 
accessing exactly the same service.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar][all] Zaqar will stay... Lots of work ahead

[Gordon Sim]

On 05/27/2015 05:08 PM, Hayes, Graham wrote:

It was agreed that Zaqar would look at a oslo_messaging driver for the
services that did not agree with the 3 options presented.


Another option there would be to add amqp 1.0 support to zaqar, and then 
you could use the amqp 1.0 driver with zaqar as the back-end.


(Support for an existing protocol would be beneficial for the general 
messaging as a service use case also.)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging] Performance testing. Initial steps.

[Gordon Sim]

On 01/27/2015 06:31 PM, Doug Hellmann wrote:

On Tue, Jan 27, 2015, at 12:28 PM, Denis Makogon wrote:

I'd like to build tool that would be able to profile messaging over
various deployments. This "tool" would give me an ability to compare
results of performance testing produced by native tools and
oslo.messaging-based tool, eventually it would lead us into digging into
code and trying to figure out where "bad things" are happening (that's
the
actual place where we would need to profile messaging code). Correct me
if
i'm wrong.


It would be interesting to have recommendations for deployment of rabbit
or qpid based on performance testing with oslo.messaging. It would also
be interesting to have recommendations for changes to the implementation
of oslo.messaging based on performance testing. I'm not sure you want to
do full-stack testing for the latter, though.

Either way, I think you would be able to start the testing without any
changes in oslo.messaging.


I agree. I think the first step is to define what to measure and then 
construct an application using olso.messaging that allows the data of 
interest to be captured using different drivers and indeed different 
configurations of a given driver.


I wrote a very simple test application to test one aspect that I felt 
was important, namely the scalability of the RPC mechanism as you 
increase the number of clients and servers involved. The code I used is 
https://github.com/grs/ombt, its probably stale at the moment, I only 
link to it as an example of approach.


Using that test code I was then able to compare performance in this one 
aspect across drivers (the 'rabbit', 'qpid' and new amqp 1.0 based 
drivers _ I wanted to try zmq, but couldn't figure out how to get it 
working at the time), and for different deployment options using a given 
driver (amqp 1.0 using qpidd or qpid dispatch router in either 
standalone or with multiple connected routers).


There are of course several other aspects that I think would be 
important to explore: notifications, more specific variations in the RPC 
'topology' i.e. number of clients on given server number of servers in 
single group etc, and a better tool (or set of tools) would allow all of 
these to be explored.


From my experimentation, I believe the biggest differences in 
scalability are going to come not from optimising the code in 
oslo.messaging so much as choosing different patterns for communication. 
Those choices may be constrained by other aspects as well of course, 
notably approach to reliability.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] The horse is dead. Long live the horse.

[Gordon Sim]

On 09/24/2014 06:07 PM, Clint Byrum wrote:

I just wanted to commend Flavio Percoco and the Zaqar team for
maintaining poise and being excellent citizens of OpenStack whilst
being questioned intensely by the likes of me, and others.


+1

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

Apologies in advance for possible repetition and pedantry...

On 09/24/2014 02:48 AM, Devananda van der Veen wrote:

2. Single Delivery - each message must be processed *exactly* once
   Example: Using a queue to process votes. Every vote must be counted only 
once.


It is also important to consider the ability of the publisher to 
reliable publish a message exactly once. If that can't be done, there 
may need to be de-duplication even if there is an exactly-once delivery 
guarantee of messages from the queue (because there could exist two 
copies of the same logical message).



5. Aggregate throughput
   Example: Ad banner processing. Remember when sites could get
slash-dotted? I need a queue resilient to truly massive spikes in
traffic.


A massive spike in traffic can be handled also by allowing the queue to 
grow, rather than increasing the throughput. This is obviously only 
effective if it is indeed a spike and the rate of ingress drops again to 
allow the backlog to be processed.


So scaling up aggregate throughput is certainly an important requirement 
for some. However the example illustrates another, which is scaling the 
size of the queue (because the bottleneck for throughput may be in the 
application processing or this processing may be temporarily 
unavailable). The latter is something that both Zaqar and SQS I suspect 
would do quite well at.



6. FIFO - When ordering matters
   Example: I can't "stop" a job that hasn't "start"ed yet.


I think FIFO is insufficiently precise.

The most extreme requirement is total ordering, i.e. all messages are 
assigned a place in a fixed sequence and the order in which they are 
seen is the same for all receivers.


The example you give above is really causal ordering. Since the need to 
stop a job is caused by the starting of that job, the stop request must 
come after the start request. However the ordering of the stop request 
for task A with respect to a stop request for task B may not be defined 
(e.g. if they are triggered concurrently).


The pattern in use is also relevant. For multiple competing consumers, 
if there are ordering requirements such as the one in your example, it 
is not sufficient to *deliver* the messages in order, they must also be 
*processed* in order.


If I have two consumers processing task requests, and give the 'start A' 
message to one, and then the 'stop A' message to another it is possible 
that the second, though dispatched by the messaging service after the 
first message, is still processed before it.


One way to avoid that would be to have the application use a separate 
queue for processing consumer, and ensure causally related messages are 
sent through the same queue. The downside is less adaptive load 
balancing and resiliency. Another option is to have the messaging 
service recognise message groupings and ensure that a group in which a 
previously delivered message has not been acknowledged are delivered 
only to the same consumer as that previous message.


[...]

Zaqar relies on a store-and-forward architecture, which is not
amenable to low-latency message processing (4).


I don't think store-and-forward precludes low-latency ('low' is of 
course subjective). Polling however is not a good fit for latency 
sensitive applications.



Again, as with SQS, it is not a wire-level protocol,


It is a wire-level protocol, but as it is based on HTTP it doesn't 
support asynchronous delivery of messages from server to client at present.



so I don't believe low-latency connectivity (3) was a design goal.


Agreed (and that is the important thing, so sorry for the nitpicking!).

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

On 09/22/2014 05:58 PM, Zane Bitter wrote:

On 22/09/14 10:11, Gordon Sim wrote:

As I understand it, pools don't help scaling a given queue since all the
messages for that queue must be in the same pool. At present traffic
through different Zaqar queues are essentially entirely orthogonal
streams. Pooling can help scale the number of such orthogonal streams,
but to be honest, that's the easier part of the problem.


But I think it's also the important part of the problem. When I talk
about scaling, I mean 1 million clients sending 10 messages per second
each, not 10 clients sending 1 million messages per second each.


I wasn't really talking about high throughput per producer (which I 
agree is not going to be a good fit), but about e.g. a large number of 
subscribers for the same set of messages, e.g. publishing one message 
per second to 10,000 subscribers.


Even at much smaller scale, expanding from 10 subscribers to say 100 
seems relatively modest but the subscriber related load would increase 
by a factor of 10. I think handling these sorts of changes is also an 
important part of the problem (though perhaps not a part that Zaqar is 
focused on).



When a user gets to the point that individual queues have massive
throughput, it's unlikely that a one-size-fits-all cloud offering like
Zaqar or SQS is _ever_ going to meet their needs. Those users will want
to spin up and configure their own messaging systems on Nova servers,
and at that kind of size they'll be able to afford to. (In fact, they
may not be able to afford _not_ to, assuming per-message-based pricing.)


[...]

If scaling the number of communicants on a given communication channel
is a goal however, then strict ordering may hamper that. If it does, it
seems to me that this is not just a policy tweak on the underlying
datastore to choose the desired balance between ordering and scale, but
a more fundamental question on the internal structure of the queue
implementation built on top of the datastore.


I agree with your analysis, but I don't think this should be a goal.


I think it's worth clarifying that alongside the goals since scaling can 
mean different things to different people. The implication then is that 
there is some limit in the number of producers and/or consumers on a 
queue beyond which the service won't scale and applications need to 
design around that.



Note that the user can still implement this themselves using
application-level sharding - if you know that in-order delivery is not
important to you, then randomly assign clients to a queue and then poll
all of the queues in the round-robin. This yields _exactly_ the same
semantics as SQS.


You can certainly leave the problem of scaling in this dimension to the 
application itself by having them split the traffic into orthogonal 
streams or hooking up orthogonal streams to provide an aggregated stream.


A true distributed queue isn't entirely trivial, but it may well be that 
most applications can get by with a much simpler approximation.


Distributed (pub-sub) topic semantics are easier to implement, but if 
the application is responsible for keeping the partitions connected, 
then it also takes on part of the burden for availability and redundancy.



The reverse is true of SQS - if you want FIFO then you have to implement
re-ordering by sequence number in your application. (I'm not certain,
but it also sounds very much like this situation is ripe for losing
messages when your client dies.)

So the question is: in which use case do we want to push additional
complexity into the application? The case where there are truly massive
volumes of messages flowing to a single point?  Or the case where the
application wants the messages in order?


I think the first case is more generally about increasing the number of 
communicating parties (publishers or subscribers or both).


For competing consumers ordering isn't usually a concern since you are 
processing in parallel anyway (if it is important you need some notion 
of message grouping within which order is preserved and some stickiness 
between group and consumer).


For multiple non-competing consumers the choice needn't be as simple as 
total ordering or no ordering at all. Many systems quite naturally only 
define partial ordering which can be guaranteed more scalably.


That's not to deny that there are indeed cases where total ordering may 
be required however.



I'd suggest both that the former applications are better able to handle
that extra complexity and that the latter applications are probably more
common. So it seems that the Zaqar team made a good decision.


If that was a deliberate decision it would be worth clarifying in the 
goals. It seems to be a different conclusion from that reached by SQS 
and as such is part of the answer to the question that began the thread.



(Aside: it follows that Zaqar probably should hav

Re: [openstack-dev] Oslo messaging vs zaqar

[Gordon Sim]

On 09/22/2014 03:40 PM, Geoff O'Callaghan wrote:

That being said I'm not sure why a well constructed zaqar with an rpc
interface couldn't meet the requirements of oslo.messsaging and much more.


What Zaqar is today and what it might become may of course be different 
things but as it stands today, Zaqar relies on polling which in my 
opinion is not a natural fit for RPC[1]. Though using an intermediary 
for routing/addressing can be of benefit, store and forward is not 
necessary and in my opinion even gets in the way[2].


Notifications on the other hand can benefit from store and forward and 
may be less latency sensitive, alleviating the polling concerns.


One of the use cases I've heard cited for Zaqar is as an inbox for 
recording certain sets of relevant events sent out by other open stack 
services. In my opinion using oslo.messaging's notification API on the 
openstack service side of this would seem - to me at least - quite 
sensible, even if the events are then stored in (or forwarded to) Zaqar 
and accessed by users through Zaqar's own protocol.




[1] The latency of an RPC call as perceived by the client is going to 
depend heavily on the polling frequency; to get lower latency, you'll 
need to pool more frequently both on the server and on the client. 
However polling more frequently results in increased load even when no 
requests are being made.


[2] I am of the view that reliable RPC is best handled by replaying the 
request from the client when needed, rather than trying to make the 
request and reply messages durably recorded, replicated and reliably 
delivered. Doing so is more scalable and simpler. An end-to-end 
acknowledgement for the request (rather than a broker taking 
responsibility and acknowledging the request independent of delivery 
status) makes it easier to detect failures and trigger a resend.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

On 09/19/2014 09:13 PM, Zane Bitter wrote:

SQS offers very, very limited guarantees, and it's clear that the reason
for that is to make it massively, massively scalable in the way that
e.g. S3 is scalable while also remaining comparably durable (S3 is
supposedly designed for 11 nines, BTW).

Zaqar, meanwhile, seems to be promising the world in terms of
guarantees. (And then taking it away in the fine print, where it says
that the operator can disregard many of them, potentially without the
user's knowledge.)

On the other hand, IIUC Zaqar does in fact have a sharding feature
("Pools") which is its answer to the massive scaling question.


There are different dimensions to the scaling problem.

As I understand it, pools don't help scaling a given queue since all the 
messages for that queue must be in the same pool. At present traffic 
through different Zaqar queues are essentially entirely orthogonal 
streams. Pooling can help scale the number of such orthogonal streams, 
but to be honest, that's the easier part of the problem.


There is also the possibility of using the sharding capabilities of the 
underlying storage. But the pattern of use will determine how effective 
that can be.


So for example, on the ordering question, if order is defined by a 
single sequence number held in the database and atomically incremented 
for every message published, that is not likely to be something where 
the databases sharding is going to help in scaling the number of 
concurrent publications.


Though sharding would allow scaling the total number messages on the 
queue (by distributing them over multiple shards), the total ordering of 
those messages reduces it's effectiveness in scaling the number of 
concurrent getters (e.g. the concurrent subscribers in pub-sub) since 
they will all be getting the messages in exactly the same order.


Strict ordering impacts the competing consumers case also (and is in my 
opinion of limited value as a guarantee anyway). At any given time, the 
head of the queue is in one shard, and all concurrent claim requests 
will contend for messages in that same shard. Though the unsuccessful 
claimants may then move to another shard as the head moves, they will 
all again try to access the messages in the same order.


So if Zaqar's goal is to scale the number of orthogonal queues, and the 
number of messages held at any time within these, the pooling facility 
and any sharding capability in the underlying store for a pool would 
likely be effective even with the strict ordering guarantee.


If scaling the number of communicants on a given communication channel 
is a goal however, then strict ordering may hamper that. If it does, it 
seems to me that this is not just a policy tweak on the underlying 
datastore to choose the desired balance between ordering and scale, but 
a more fundamental question on the internal structure of the queue 
implementation built on top of the datastore.


I also get the impression, perhaps wrongly, that providing the strict 
ordering guarantee wasn't necessarily an explicit requirement, but was 
simply a property of the underlying implementation(?).


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Oslo messaging vs zaqar

[Gordon Sim]

On 09/22/2014 10:56 AM, Flavio Percoco wrote:

What I meant is that oslo.messaging is an rpc library and it depends on
few very specific message delivery patterns that are somehow tight/based
on AMQP semantics.


RPC at it's core is the request-response pattern which is directly 
supported by many messaging technologies and implementable over almost 
any form of communication (it's mostly just convention for addressing).


In addition the oslo.messaging model offers the ability to invoke on one 
of a group of servers, which is the task-queue pattern and again is very 
general.


One-way messages are also supported either unicast (to a specific 
server) or broadcast (to all servers in a group).


I don't think any of these patterns are in any way tightly bound or 
specific to AMQP.


In fact Zaqar offers (mostly) the same patterns (there is no way to 
indicate the 'address' to reply to defined by the spec itself, but that 
could be easily added).



Implementing Zaqar's API in oslo.messaging would be
like trying to add an AMQP driver to Zaqar.


I don't think it would be. AMQP and Zaqar are wire protocols offering 
very similar levels of abstraction (sending, consuming, browsing, 
acknowledging messages). By contrast oslo.messaging is a language level 
API, generally at a slightly higher level of abstraction, mapping method 
invocation to particular messaging patterns between processes.


Implementing Zaqar's model over oslo.messaging would be like 
implementing AMQP's model over oslo.messaging, i.e. reimplementing a 
general purpose message-oriented abstraction on top of an API intended 
to hide such a model behind message invocation. Though technically 
possible it seems a little pointless (and I don't think anyone is 
suggesting otherwise).


Zaqar drivers are really providing different implementations of 
(distributed) message storage. AMQP (and I'm talking primarily about 
version 1.0) is not intended for that purpose. It's intended to control 
the transfer of messages between processes. Exposing AMQP as an 
alternative interface for publishing/receiving/consuming messages 
through Zaqar on the other hand would be simple.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Oslo messaging vs zaqar

[Gordon Sim]

On 09/22/2014 08:00 AM, Flavio Percoco wrote:

oslo messaging is highly tight to AMQP semantics whereas Zaqar is not.


The API for oslo.messaging is actually quite high level and could easily 
be mapped to different messaging technologies.


There is some terminology that comes from older versions of AMQP, e.g. 
the use of 'exchanges' as essentially a namespace, but I don't myself 
believe these tie the API in anyway to the original implementation from 
which the naming arose.


In what way do you see olso.messaging as being tied to AMQP semantics?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

On 09/19/2014 08:53 AM, Flavio Percoco wrote:

On 09/18/2014 09:25 PM, Gordon Sim wrote:

I don't see that the claim mechanism brings any stronger guarantee, it
just offers a competing consumer behaviour where browsing is
non-competing (non-destructive). In both cases you require the client to
be able to remember which messages it had processed in order to ensure
exactly once. The claim reduces the scope of any doubt, but the client
still needs to be able to determine whether it has already processed any
message in the claim already.


The client needs to remember which messages it had processed if it
doesn't delete them (ack) after it has processed them. It's true the
client could also fail after having processed the message which means it
won't be able to ack it.

That said, being able to prevent other consumers to consume a specific
message can bring a stronger guarantee depending on how messages are
processed. I mean, claiming a message guarantees that throughout the
duration of that claim, no other client will be able to consume the
claimed messages, which means it allows messages to be consumed only once.


I think 'exactly once' means different things when used for competing 
consumers and non-competing consumers. For the former it means the 
message is processed by only one consumer, and only once. For the latter 
it means every consumer processes the message exactly once.


Using a claim provides the competing consumer behaviour. To me this is a 
'different' guarantee from non-competing consumer rather than a 
'stronger' one, and it is orthogonal to the reliability of the delivery.


However we only differ on terminology used; I believe we are on the same 
page as far as the semantics go :-)



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

On 09/18/2014 03:45 PM, Flavio Percoco wrote:

On 09/18/2014 04:09 PM, Gordon Sim wrote:

Is the replication synchronous or asynchronous with respect to client
calls? E.g. will the response to a post of messages be returned only
once the replication of those messages is confirmed? Likewise when
deleting a message, is the response only returned when replicas of the
message are deleted?


It depends on the driver implementation and/or storage configuration.
For example, in the mongodb driver, we use the default write concern
called "acknowledged". This means that as soon as the message gets to
the master node (note it's not written on disk yet nor replicated) zaqar
will receive a confirmation and then send the response back to the
client.


So in that mode it's unreliable. If there is failure right after the 
response is sent the message may be lost, but the client believes it has 
been confirmed so will not resend.



This is also configurable by the deployer by changing the
default write concern in the mongodb uri using `?w=SOME_WRITE_CONCERN`[0].

[0] http://docs.mongodb.org/manual/reference/connection-string/#uri.w


So you could change that to majority to get reliable publication 
(at-least-once).


[...]

 From what I can see, pop provides unreliable delivery (i.e. its similar
to no-ack). If the delete call using pop fails while sending back the
response, the messages are removed but didn't get to the client.


Correct, pop works like no-ack. If you want to have pop+ack, it is
possible to claim just 1 message and then delete it.


Right, claim+delete is ack (and if the claim is replicated and 
recoverable etc you can verify whether deletion occurred to ensure 
message is processed only once). Using delete-with-pop is no-ak, i.e. 
at-most-once.



What do you mean by 'streaming messages'?


I'm sorry, that went out wrong. I had the "browsability" term in my head
and went with something even worse. By streaming messages I meant
polling messages without claiming them. In other words, at-least-once is
guaranteed by default, whereas once-and-only-once is guaranteed just if
claims are used.


I don't see that the claim mechanism brings any stronger guarantee, it 
just offers a competing consumer behaviour where browsing is 
non-competing (non-destructive). In both cases you require the client to 
be able to remember which messages it had processed in order to ensure 
exactly once. The claim reduces the scope of any doubt, but the client 
still needs to be able to determine whether it has already processed any 
message in the claim already.


[...]

That marker is a sequence number of some kind that is used to provide
ordering to queries? Is it generated by the database itself?


It's a sequence number to provide ordering to queries, correct.
Depending on the driver, it may be generated by Zaqar or the database.
In mongodb's case it's generated by Zaqar[0].


Zaqar increments a counter held within the database, am I reading that 
correctly? So mongodb is responsible for the ordering and atomicity of 
multiple concurrent requests for a marker?



[0]
https://github.com/openstack/zaqar/blob/master/zaqar/queues/storage/mongodb/queues.py#L103-L185




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar and SQS Properties of Distributed Queues

[Gordon Sim]

On 09/18/2014 12:31 PM, Flavio Percoco wrote:

On 09/17/2014 10:36 PM, Joe Gordon wrote:

My understanding of Zaqar is that it's like SQS. SQS uses distributed
queues, which have a few unusual properties [0]:


 Message Order

Amazon SQS makes a best effort to preserve order in messages, but due to
the distributed nature of the queue, we cannot guarantee you will
receive messages in the exact order you sent them. If your system
requires that order be preserved, we recommend you place sequencing
information in each message so you can reorder the messages upon receipt.



Zaqar guarantees FIFO. To be more precise, it does that relying on the
storage backend ability to do so as well. Depending on the storage used,
guaranteeing FIFO may have some performance penalties.


Would it be accurate to say that at present Zaqar does not use 
distributed queues, but holds all queue data in a storage mechanism of 
some form which may internally distribute that data among servers but 
provides Zaqar with a consistent data model of some form?


[...]

As of now, Zaqar fully relies on the storage replication/clustering
capabilities to provide data consistency, availability and fault
tolerance.


Is the replication synchronous or asynchronous with respect to client 
calls? E.g. will the response to a post of messages be returned only 
once the replication of those messages is confirmed? Likewise when 
deleting a message, is the response only returned when replicas of the 
message are deleted?



However, as far as consuming messages is concerned, it can
guarantee once-and-only-once and/or at-least-once delivery depending on
the message pattern used to consume messages. Using pop or claims
guarantees the former whereas streaming messages out of Zaqar guarantees
the later.


From what I can see, pop provides unreliable delivery (i.e. its similar 
to no-ack). If the delete call using pop fails while sending back the 
response, the messages are removed but didn't get to the client.


What do you mean by 'streaming messages'?

[...]

Based on our short conversation on IRC last night, I understand you're
concerned that FIFO may result in performance issues. That's a valid
concern and I think the right answer is that it depends on the storage.
If the storage has a built-in FIFO guarantee then there's nothing Zaqar
needs to do there. In the other hand, if the storage does not have a
built-in support for FIFO, Zaqar will cover it in the driver
implementation. In the mongodb driver, each message has a marker that is
used to guarantee FIFO.


That marker is a sequence number of some kind that is used to provide 
ordering to queries? Is it generated by the database itself?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Gordon Sim]

On 09/16/2014 08:55 AM, Flavio Percoco wrote:

pub/sub doesn't necessarily guarantees messages delivery, it really
depends on the implementation.


As I understand it, the model for pub-sub in Zaqar is to have multiple 
subscribers polling the queue with gets, and have the messages removed 
from the queue only when they expire. Is that right?


If the ttl of the messages is long enough, a subscriber can start 
getting the queue from where they left off (if they have or can recover 
their last used marker) or from the head of the queue.


So although not acknowledged, subscribers can retry on failover 
providing they do so before the message expires.



That said, there are ways to guarantee
that depending on the method used. For example, if the subscriber is a
webhook, we can use the response status code to ack the message. if it
has a persistent connection like websocket or even (long|short)-poll an
ack may be needed.


In the pub-sub case, to remove a message based on acks you need to wait 
until all known subscribers have acked it. With the current model there 
is no explicit concept of subscriber (nor of ack in the non-competing 
consumer case). Without changing that I don't think you can use the 
response of a webhook anyway (unless of course there are not get style 
subscribers on the queue).


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Gordon Sim]

On 09/11/2014 07:46 AM, Flavio Percoco wrote:

On 09/10/2014 03:18 PM, Gordon Sim wrote:

On 09/10/2014 09:58 AM, Flavio Percoco wrote:

 Other OpenStack components can integrate with Zaqar to surface events
to end users and to communicate with guest agents that run in the
"over-cloud" layer.


I may be misunderstanding the last sentence, but I think *direct*
integration of other OpenStack services with Zaqar would be a bad idea.

Wouldn't this be better done through olso.messaging's notifications in
some way? and/or through some standard protocol (and there's more than
one to choose from)?

Communicating through a specific, fixed messaging system, with its own
unique protocol is actually a step backwards in my opinion, especially
for things that you want to keep as loosely coupled as possible. This is
exactly why various standard protocols emerged.



Yes and no. The answer is yes most of the time but there are use cases,
like the ones mentioned here[0], that make zaqar a good tool for the job.


I certainly wasn't saying that Zaqar is not a good tool. I was merely 
stating that - in my opinion - wiring it in as the only tool would be a 
mistake.



[0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases


Again, Zaqar might be great for those cases, but none of them describe 
features that are unique to Zaqar, so other solutions could also fit.


All I'm saying is that if the channel between openstack services and 
users is configurable, that will give users more choice (as well as 
operators) and that - in my opinion - would be a good thing.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

On 09/12/2014 09:50 AM, Flavio Percoco wrote:

Zaqar supports once and only once delivery.


For the transfer from Zaqar to consumers it does (providing the claim id 
can be recovered). For transfer from producers to Zaqar I believe it is 
more limited.


If the connection to Zaqar fails during a post, the sender can't tell 
whether the message was successfully enqueued or not.


It could try to determine this is by browsing the entire queue looking 
for a matching body. However thatt would be awkward and in any case the 
absence of the message could mean that it wasn't enqueued or that it was 
already consumed and deleted.


One way of handling this is to have the post return a unique url to 
which the message(s) are put or posted. The sender can then repost 
(re-put) to this in the event of failure and the server can determine 
whether it already processed the publications. Alternatively the client 
can be required to generate a unique id on which the server can 
de-deduplicate.


The ActiveMQ REST interface supports both of these approaches: 
http://activemq.apache.org/restful-queue.html


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Gordon Sim]

On 09/10/2014 07:29 PM, Clint Byrum wrote:

Excerpts from Gordon Sim's message of 2014-09-10 06:18:52 -0700:

On 09/10/2014 09:58 AM, Flavio Percoco wrote:

 Other OpenStack components can integrate with Zaqar to surface events
to end users and to communicate with guest agents that run in the
"over-cloud" layer.


I may be misunderstanding the last sentence, but I think *direct*
integration of other OpenStack services with Zaqar would be a bad idea.

Wouldn't this be better done through olso.messaging's notifications in
some way? and/or through some standard protocol (and there's more than
one to choose from)?



It's not direct, nobody is suggesting that.

What people are suggesting is that a user would be able to tell Nova
to put any messages that would want to deliver in a _user_ focused
queue/inbox.

This has nothing to do with oslo.messaging. Users don't want many options
for backends. They want a simple message passing interface so they don't
have to babysit one and choose one.

Certainly the "undercloud" Zaqar API could be based on the existing
oslo.messaging notifications. A simple daemon that sits between the oslo
notifications firehose and Zaqar's user queues would be quite efficient.


Right, that's what I meant. The interface applications have to it would 
of course not be oslo.messaging notifications.



However, putting the whole burden of talking directly to a notification
bus on the users is unnecessarily complex... especially if they use Java
and have no idea what oslo is.


Agreed. I actually think for applications, making them accessible 
through standard protocols would be an advantage. E.g. for java as you 
mention, there are JMS clients for AMQP or STOMP. Thats not to say that 
Zaqar's protocol as built on top of HTTP is not also an option, just 
that - in my opinion - it shouldn't be the only possible one. (I.e. Nova 
shouldn't be directly tied to Zaqar as the only way for users to get 
relevant message from it).



Communicating through a specific, fixed messaging system, with its own
unique protocol is actually a step backwards in my opinion, especially
for things that you want to keep as loosely coupled as possible. This is
exactly why various standard protocols emerged.



You're thinking like an operator. Think like an application developer.
They're asking you "how do I subscribe to notifications about _just my
instances_ from Nova?", not "how do I pump 40,000 messages per second
through a message bus that I fully control?"


I'm not sure why you think that from my statement. I wasn't talking 
about performance. I was talking about choice both for deployment *and* 
application developers.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

On 09/10/2014 12:47 AM, Devananda van der Veen wrote:

As was pointed out in the TC meeting today, Zaqar is (was?) actually
aiming to provide Messaging-as-a-Service -- not queueing as a service!
This is another way of saying "it's more like email and less like
AMQP"


The glossary[1] describes a model that is much more like 
messaging-oriented-middleware (AMQP and other things like it) than email.


To me, messaging-as-a-service is perhaps a more general purpose term. 
The concept of a 'queue' might not be exposed in such a system (e.g. 
MQTT), whereas the term queueing-as-a-service implies that perhaps it 
is. I don't see them as necessarily distinct things though. The model 
exposed by Zaqar certainly has 'queues' featured prominently.


[1] http://docs.openstack.org/developer/marconi/glossary.html



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Gordon Sim]

On 09/10/2014 01:51 PM, Thierry Carrez wrote:

I think we do need, as Samuel puts it, "some sort of durable
message-broker/queue-server thing". It's a basic application building
block. Some claim it's THE basic application building block, more useful
than database provisioning. It's definitely a layer above pure IaaS, so
if we end up splitting OpenStack into layers this clearly won't be in
the inner one. But I think "IaaS+" basic application building blocks
belong in OpenStack one way or another. That's the reason I supported
Designate ("everyone needs DNS") and Trove ("everyone needs DBs").

With that said, I think yesterday there was a concern that Zaqar might
not fill the "some sort of durable message-broker/queue-server thing"
role well. The argument goes something like: if it was a queue-server
then it should actually be built on top of Rabbit; if it was a
message-broker it should be built on top of postfix/dovecot; the current
architecture is only justified because it's something in between, so
it's broken.


What is the distinction between a message broker and a queue server? To 
me those terms both imply something broadly similar (message broker 
perhaps being a little bit more generic). I could see Zaqar perhaps as 
somewhere between messaging and data-storage.


There are of course quite a lot of "durable message-broker/queue-server 
things" around already. I understood Zaqar to have been created to 
address perceived limitations in existing solutions (e.g. requiring less 
'babysitting', being 'designed for the cloud' etc). All solutions 
certainly have their limitations. Zaqar has limitations with respect to 
existing solutions also.


So while I agree that there is great value in a basic building block for 
'messaging as a service' I think the ideal solution would allow 
different variations, tailored to different patterns of use with a 
common API for provisioning, managing and monitoring coupled with 
support for standard protocols.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Zaqar graduation (round 2) [was: Comments on the concerns arose during the TC meeting]

[Gordon Sim]

On 09/10/2014 09:58 AM, Flavio Percoco wrote:

To clarify the doubts of what Zaqar is or it's not, let me quote what's
written in the project's overview section[0]:

"Zaqar is a multi-tenant cloud messaging service for web developers.


How are different tenants isolated from each other? Can different 
tenants access the same queue? If so, what does Zaqar do to prevent one 
tenant from negatively affecting the other? If not, how is communication 
with other tenants achieved.


Most messaging systems allow authorisation to be used to restrict what a 
particular user can access and quotas to restrict their resource 
consumption. What does Zaqar do differently?



It
combines the ideas pioneered by Amazon's SQS product with additional
semantics to support event broadcasting.

The service features a fully RESTful API, which developers can use to
send messages between various components of their SaaS and mobile
applications, by using a variety of communication patterns. Underlying
this API is an efficient messaging engine designed with scalability and
security in mind.

Other OpenStack components can integrate with Zaqar to surface events
to end users and to communicate with guest agents that run in the
"over-cloud" layer.


I may be misunderstanding the last sentence, but I think *direct* 
integration of other OpenStack services with Zaqar would be a bad idea.


Wouldn't this be better done through olso.messaging's notifications in 
some way? and/or through some standard protocol (and there's more than 
one to choose from)?


Communicating through a specific, fixed messaging system, with its own 
unique protocol is actually a step backwards in my opinion, especially 
for things that you want to keep as loosely coupled as possible. This is 
exactly why various standard protocols emerged.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

On 09/04/2014 01:14 PM, Sean Dague wrote:

https://dague.net/2014/08/26/openstack-as-layers/


Just wanted to say that I found this article very useful indeed and 
agree with the points you make in it.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

On 09/04/2014 09:44 PM, Kurt Griffiths wrote:

Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably.


I think something like that would be valuable, especially in conjunction 
with some application layer proxying and mapping between 'virtual' 
addresses/endpoints and specific queues/virtual hosts/brokers.


That would allow people to use the brokers, protocols and semantics they 
are already familiar with, combine different systems even, and have 
self-provisioning and scalability on top of it, with varying degrees of 
isolation for tenants.



Would
such a service totally overlap in terms of use-cases with Zaqar? Community
feedback suggests otherwise.


I'd love to read the feedback if you have any links. While I agree 
overlap is rarely total, I suspect there will be quite a bit and that it 
will grow as each approach evolves. That emphatically does not mean I 
think Zaqar should in any way be held back though, just that I think the 
openstack community should anticipate and indeed encourage other 
approaches also.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

On 09/04/2014 09:44 PM, Kurt Griffiths wrote:

Thanks for your comments Gordon. I appreciate where you are coming from
and I think we are actually in agreement on a lot of things.

I just want to make it clear that from the very beginning of the project
the team has tried to communicate (but perhaps could have done a better
job at it) that we aren’t trying to displace other messaging systems that
are clearly delivering a lot of value today.

In fact, I personally have long been a proponent of using the best tool
for the job. The Zaqar project was kicked off at an unconference session
several summits ago because the community saw a need that was not covered
by other messaging systems. Does that mean those other systems are “bad”
or “wrong”? Of course not. It simply means that there are some cases where
those other systems aren’t the best tool for the job, and another tool is
needed (and vice versa).


I think communicating that unmet need, those use-cases not best served 
by other systems, would help a lot in clarifying Zaqar's intended role.



Does that other tool look *exactly* like Zaqar? Probably not. But a lot of
people have told us Zaqar--in its current form--already delivers a lot of
value that they can’t get from other messaging systems that are available
to them. Zaqar, like any open source project, is a manifestation of lots
of peoples' ideas, and will evolve over time to meet the needs of the
community.

Does a Qpid/Rabbit/Kafka provisioning service make sense? Probably. Would
such a service totally overlap in terms of use-cases with Zaqar? Community
feedback suggests otherwise. Will there be some other kind of thing that
comes out of the woodwork? Possibly. (Heck, if something better comes
along I for one have no qualms in shifting resources to the more elegant
solution--again, use the best tool for the job.) This process happens all
the time in the broader open-source world. But this process takes a
healthy amount of time, plus broad exposure and usage, which is something
that you simply don’t get as a non-integrated project in the OpenStack
ecosystem.

In any case, it’s pretty clear to me that Zaqar graduating should not be
viewed as making it "the officially blessed messaging service for the
cloud” and nobody is allowed to have any other ideas, ever.


Indeed, and to be clear, I wasn't really commenting on the graduation at 
all. I was really just responding to the statements on scope and 
differentiation; 'messaging service for the cloud' is a very broad 
problem space and as you rightly point out there may be different tools 
that best serve different parts of that problem space.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Zaqar] Comments on the concerns arose during the TC meeting

[Gordon Sim]

Hi Flavio,

On 09/04/2014 08:08 AM, Flavio Percoco wrote:

- Concern on should we really reinvent a queue system rather than
piggyback on one

As mentioned in the meeting on Tuesday, Zaqar is not reinventing message
brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack
flavor on top. [0]


On 09/04/2014 01:13 PM, Flavio Percoco wrote:

If we put both features, multi-tenancy and multi-protocol, aside for a
bit, we can simplify Zaqars goal down to a "messaging service for the
cloud".  I believe this is exactly where the line between Zaqar and other
*queuing*  technologies should be drawn. Zaqar is, at the very end, a
messaging service thought for the cloud whereas existing queuing
technologies were not designed for it.


Isn't this the real answer to the concern above? Zaqar *is* a 
reimplementation of a messaging service, but you believe it is 
*necessary* to do this because existing implementations would not lend 
themselves to being as well integrated into the 'cloud', or do not 
provide sufficient support for multi-tenancy.


I suspect if you asked, you would find that many existing projects 
implementing a messaging service would see their solution as very much 
suitable for the cloud. You disagree and probably with good reason. I'm 
just pointing out the dividing line is I think not as clear to everyone 
as it may be to you. If you add support for STOMP, MQTT, AMQP or any 
other protocol the overlap with other messaging services will be ever 
greater (and will bring in new requirements and new expectations around 
performance).


This is not intended as criticism of Zaqar in anyway. In my opinion, 
'reinvention' is not necessarily a bad thing. It's just another way of 
saying innovative approach and/or original thinking, both of which are 
good and both of which I think are important in the context of 
communicating in the cloud.


My concern would be more that Zaqar's privileged status as the 
officially 'blessed' "messaging service for the cloud" - which is a 
broad space, much broader than just an SQS equivalent - would make it 
harder for other approaches to gain traction.


--Gordon.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] [Oslo.messaging] RPC failover handling in rabbitmq driver

[Gordon Sim]

On 07/28/2014 09:20 AM, Bogdan Dobrelya wrote:

Hello.
I'd like to bring your attention to major RPC failover issue in
impl_rabbit.py [0]. There are several *related* patches and a number of
concerns should be considered as well:
- Passive exchanges fix [1] (looks like the problem is much deeper than
it seems though).
- the first version of the fix [2] which makes the producer to declare a
queue and bind it to exchange as well as consumer does.
- Making all RPC involved reply_* queues durable in order to preserve
them in RabbitMQ after failover (there could be a TTL for such a queues
as well)
- RPC throughput tuning patch [3]

I believe the issue [0] should be at least prioritized and assigned to
some milestone.


I think the real issue is the lack of clarity around what guarantees are 
made by the API.


Is it the case that an RPC call should never fail (i.e. never time out) 
due to failover? Either way, the answer to this should be very clear.


If failures may occur, then the calling code needs to handle that. If 
eliminating failures is part of the 'contract' then the library should 
have a clear strategy for ensuring (and testing) this.


Another possible scenario is that the connection is lost immediately 
after writing the request message to the socket (but before it is 
processed by the rabbit broker). In this case the issue is that the 
request is not confirmed, so it can complete before it is 'safe'. In 
other words requests are unreliable.


My own view is that if you want to avoid time outs on failover, the best 
approach is to have olso.messaging retry the entire request regardless 
of the point it had reached in the previous attempt. I.e. rather than 
trying to make delivery of responses reliable, assume that both requests 
and responses are unreliable and re-issue the request immediately on 
failover. (The retry logic could even be made independent of any driver 
if desired).


This is perhaps a bigger change, but I think it is more easy to get 
right and will also be more scalable and performant since it doesn't 
require replication of every queue and every message.





[0] https://bugs.launchpad.net/oslo.messaging/+bug/1338732
[1] https://review.openstack.org/#/c/109373/
[2]
https://github.com/noelbk/oslo.messaging/commit/960fc26ff050ca3073ad90eccbef1ca95712e82e
[3] https://review.openstack.org/#/c/109143/




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

[Gordon Sim]

On 07/07/2014 07:18 PM, Mark McLoughlin wrote:

I'd expect us to add e.g.

   @asyncio.coroutine
   def call_async(self, ctxt, method, **kwargs):
   ...

to RPCClient. Perhaps we'd need to add an AsyncRPCClient in a separate
module and only add the method there - I don't have a good sense of it
yet.

However, the key thing is that I don't anticipate us needing to change
the current API in a backwards incompatible way.


Agreed, and that is a good thing. You would be *adding* to the API to 
support async behaviour, though right? Perhaps it would be worth being 
more explicit about the asynchronicity in that case, e.g. return a 
promise/future or allow an on-completion callback?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Marconi] RFC AMQP 1.0 Transport Driver for Marconi

[Gordon Sim]

On 07/07/2014 09:53 PM, Victoria Martínez de la Cruz wrote:

During the last couple of weeks I've been working on adding support for
AMQP 1.0 in Marconi transport. For that, I've based my research on
current Marconi WSGI API, Apache Proton API [0] and Azure Service Bus
docs [1].

There are several things to decide here in order to take full profit of
AMQP's performance while keeping things consistent.

I'd like to know your opinions about the following.

 > Marconi and AMQP operations mapping

Marconi doesn't follow a strict queue semantic. It provides some more
operations that are not supported in other queues implementations, as
messages listing, message/s retrieval by id and message/s deletion by id.


Several messaging system provide for queue browsing, which is 
effectively the same as Marconi's 'message listing'.


The fact that deletion is done by id using the HTTP interface to Marconi 
doesn't mean that the same approach needs to be taken using a different 
protocol.



Also, we don't have different entities to provide publish/subscribe and
producer/consumer as other messaging implementations (e.g. in Azure
topics are for pub/sub and queues are for prod/cons)


That needn't be an issue. In AMQP the receiving link (i.e. the 
subscriber/consumer) can specify a 'distribution mode'. If the mode 
specified is 'copy' then the link is essentially a non-destructive 
reader of messages, (i.e. a sub in the pub/sub pattern). If it is move 
then messages allocated to that link should not be sent to other similar 
receiving links (i.e. competing consumers, in the producer/consumer 
pattern). For a distribution mode of 'move' the link claims the message 
sent to it. It can then also be required to acknowledge those messages 
(at which point they are permanently removed).


I.e. the distribution mode specified by the client when establishing the 
receiving link allows them to chose between the two behaviours, exactly 
as users of the HTTP interface do now.



Marconi support prod/cons through claims. We rely on clients to send an
ack of the message being consumed in order to delete the message and
avoid other clients to consume it later.


AMQP is not that different here. The main difference being that in AMQP 
the message are identified through session scoped identifiers rather 
than global identifiers.



So, the thing is... *should we add support for those additional
features?


Assuming I understand the question - i.e. should Marconi try and support 
retrieval and deletion of messages by id over AMQP - I would argue, no, 
it should not.


Using AMQP as the transport, it becomes an alternate interface to 
Marconi. The fact that global ids may better suit the HTTP interface 
doesn't mean the same applies to other interfaces.


Someone using AMQP as the interface would, in my opinion, want the 
'normal' behaviour with that protocol.



how we can do that in order to be consistent with AMQP and
avoid confusion?*

I drafted two different possibilities

e.g. Delete a message

- Specify the operation in the URI

./client.py amqp://127.0.0.1:/myqueue/messages/id/delete


- Specify the operation in the message suject

./client.py amqp://127.0.0.1:/myqueue
 {subject: 'DELETE', id: 'id1'}


As above, I would use AMQP directly and naturally. To acknowledge a 
message you must have had it delivered to you (with a distribution mode 
of 'move'), and you must the accept it (i.e. acknowledge it). There is 
no need to layer a special purpose command to the Marconi system on top 
of an AMQP message.


--Gordon.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

[Gordon Sim]

On 07/07/2014 03:12 PM, Victor Stinner wrote:

The first step is to patch endpoints to add @trollius.coroutine to the methods,
and add yield From(...) on asynchronous tasks.


What are the 'endpoints' here? Are these internal to the oslo.messaging 
library, or external to it?



Later we may modify Oslo Messaging to be able to call an RPC method
asynchronously, a method which would return a Trollius coroutine or task
directly. The problem is that Oslo Messaging currently hides "implementation"
details like eventlet.


I guess my question is how effectively does it hide it? If the answer to 
the above is that this change can be contained within the oslo.messaging 
implementation itself, then that would suggest its hidden reasonably well.


If, as I first understood (perhaps wrongly) it required changes to every 
use of the oslo.messaging API, then it wouldn't really be hidden.



Returning a Trollius object means that Oslo Messaging
will use explicitly Trollius. I'm not sure that OpenStack is ready for that
today.


The oslo.messaging API could evolve/expand to include explicitly 
asynchronous methods that did not directly expose Trollius.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

[Gordon Sim]

On 07/03/2014 04:27 PM, Mark McLoughlin wrote:

Ceilometer's code is run in response to various I/O events like REST API
requests, RPC calls, notifications received, etc. We eventually want the
asyncio event loop to be what schedules Ceilometer's code in response to
these events. Right now, it is eventlet doing that.

Now, because we're using eventlet, the code that is run in response to
these events looks like synchronous code that makes a bunch of
synchronous calls. For example, the code might do some_sync_op() and
that will cause a context switch to a different greenthread (within the
same native thread) where we might handle another I/O event (like a REST
API request) while we're waiting for some_sync_op() to return:

   def foo(self):
   result = some_sync_op()  # this may yield to another greenlet
   return do_stuff(result)

Eventlet's infamous monkey patching is what make this magic happen.

When we switch to asyncio's event loop, all of this code needs to be
ported to asyncio's explicitly asynchronous approach. We might do:

   @asyncio.coroutine
   def foo(self):
   result = yield from some_async_op(...)
   return do_stuff(result)

or:

   @asyncio.coroutine
   def foo(self):
   fut = Future()
   some_async_op(callback=fut.set_result)
   ...
   result = yield from fut
   return do_stuff(result)

Porting from eventlet's implicit async approach to asyncio's explicit
async API will be seriously time consuming and we need to be able to do
it piece-by-piece.


Am I right in saying that this implies a change to the effective API for 
oslo.messaging[1]? I.e. every invocation on the library, e.g. a call or 
a cast, will need to be changed to be explicitly asynchronous?


[1] Not necessarily a change to the signature of functions, but a change 
to the manner in which they are invoked.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Gordon Sim]

On 07/01/2014 04:14 PM, Alexei Kornienko wrote:

A lot of driver details leak outside the API and it makes it hard to
improve driver without changing the API.


I agree that some aspects of specific driver implementations leak into 
the public API for the messaging library as a whole. There are also some 
ambiguities around the intended and actual contracts.


However, I do also think that the existing driver abstraction, while not 
perfect, can be used to implement drivers using quite different approaches.


The current impl_rabbit and impl_qpid drivers may be hard(er) to change, 
but that's not because of the API (in my view).


[...]

This would allow us to change drivers without touching the API and test
their performance separately


I think you can do this already. It is true that there are some semantic 
differences when switching drivers, and it would be ideal to minimise 
those (or at least make them more explicit) at some point.


I did some experiments measuring the scalability of rpc calls for 
different drivers - mainly as an initial stress test for the proposed 
AMQP 1.0 driver -which showed significant differences even where the 
same broker was used, just by changing the driver.


I'm not arguing that there will never be a good reason to change either 
the driver API or indeed the public API, but I think a lot can be 
accomplished within the current framework if desired. (With more 
specific changes to APIs then proposed with some context as needed).


--Gordon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Gordon Sim]

On 07/01/2014 03:52 PM, Ihar Hrachyshka wrote:

On 01/07/14 15:55, Alexei Kornienko wrote:

And in addition I've provided some links to existing
implementation with places that IMHO cause bottlenecks. From my
point of view that code is doing obviously stupid things (like
closing/opening sockets for each message sent).


That indeed sounds bad.


That is enough for me to rewrite it even without additional
proofs that it's wrong.


[Full disclosure: I'm not as involved into oslo.messaging internals as
you probably are, so I may speak out dumb things.]

I wonder whether there are easier ways to fix that particular issue
without rewriting everything from scratch. Like, provide a pool of
connections and make send() functions use it instead of creating new
connections (?)


I did a little investigation, using the test scripts from Alexei. 
Looking at the protocol trace with wireshark, you can certainly see lots 
of redundant stuff. However I did *not* see connections being opened and 
closed.


What I did see was the AMQP (0-9-1) channel being opened and closed, and 
the exchange being declared for every published message. Both these 
actions are synchronous. This clearly impacts throughput and, in my view 
more importantly, will limit scalability by giving the broker redundant 
work.


However, I think this could be avoided (at least to the extreme extent 
it occurs at present) by a simple patch. I posted this up to gerrit in 
case anyone is interested: https://review.openstack.org/#/c/104194/


With this in place the ingress improves significantly[1]. The bottleneck 
then appears to be the acking of messages. This means the queue depth 
tends to get higher than before, and the traffic appears more 'bursty'. 
Each message is acked individually, whereas some degree of batching 
might be possible which would likely improve the efficiency there a 
little bit.


Anyway, just an observation I thought it would be interesting to share.

--Gordon.

[1] From less than 400 msgs/sec to well over 1000 msgs/sec on my laptop

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Gordon Sim]

On 06/30/2014 12:22 PM, Ihar Hrachyshka wrote:

Alexei Kornienko wrote:

Some performance tests may be introduced but they would be more
like functional tests since they require setup of actual
messaging server (rabbit, etc.).


Yes. I think we already have some. F.e.
tests/drivers/test_impl_qpid.py attempts to use local Qpid server
(backing up to fake server if it's not available).


I always get failures when there is a real qpidd service listening on 
the expected port. Does anyone else see this?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] [messaging] 'retry' option

[Gordon Sim]

On 06/28/2014 10:49 PM, Mark McLoughlin wrote:

On Fri, 2014-06-27 at 17:02 +0100, Gordon Sim wrote:

A question about the new 'retry' option. The doc says:

  By default, cast() and call() will block until the
  message is successfully sent.

What does 'successfully sent' mean here?


Unclear, ambiguous, probably driver dependent etc.

The 'blocking' we're talking about here is establishing a connection
with the broker. If the connection has been lost, then cast() will block
until the connection has been re-established and the message 'sent'.


Understood, but to my mind, that is really an implementation detail.


  Does it mean 'written to the wire' or 'accepted by the broker'?

For the impl_qpid.py driver, each send is synchronous, so it means
accepted by the broker[1].

What does the impl_rabbit.py driver do? Does it just mean 'written to
the wire', or is it using RabbitMQ confirmations to get notified when
the broker accepts it (standard 0-9-1 has no way of doing this).


I don't know, but it would be nice if someone did take the time to
figure it out and document it :)


Having googled around a bit, it appears that kombu v3.* has a 
'confirm_publish' transport option when using the 'pyamqp' transport. 
That isn't available in the 2.* versions, which appear to be what is 
used in oslo.messaging, and I can't find that option specified anywhere 
either in the oslo.messaging codebase.


Running a series of casts using the latest impl_rabbit.py driver and 
examining the data on the wire also shows no confirms being sent.


So for impl_rabbit, the send is not acknowledged, but the delivery to 
consumers is. For impl_qpid its the other way round; the send is 
acknowledged but the delivery to consumers is not (though a prefetch of 
1 is used limiting the loss to one message).



Seriously, some docs around the subtle ways that the drivers differ from
one another would be helpful ... particularly if it exposed incorrect
assumptions API users are currently making.


I'm happy to try and contribute to that.


If the intention is to block until accepted by the broker that has
obvious performance implications. On the other hand if it means block
until written to the wire, what is the advantage of that? Was that a
deliberate feature or perhaps just an accident of implementation?

The use case for the new parameter, as described in the git commit,
seems to be motivated by wanting to avoid the blocking when sending
notifications. I can certainly understand that desire.

However, notifications and casts feel like inherently asynchronous
things to me, and perhaps having/needing the synchronous behaviour is
the real issue?


It's not so much about sync vs async, but a failure mode. By default, if
we lose our connection with the broker, we wait until we can
re-establish it rather than throwing exceptions (requiring the API
caller to have its own retry logic) or quietly dropping the message.


Even when you have no failure, your calling thread has to wait until the 
point the send is deemed successful before returning. So it is 
synchronous with respect to whatever that success criteria is.


In the case where success is deemed to be acceptance by the broker 
(which is the case for the impl_qpid.py driver at present, whether 
intentional or not), the call is fully synchronous.


If on the other hand success is merely writing the message to the wire, 
then any failure may well cause message loss regardless of the retry 
option. The reconnect and retry in this case is only of limited value. 
It can avoid certain losses, but not others.



The use case for ceilometer is to allow its RPCPublisher to have a
publishing policy - block until the samples have been sent, queue (in an
in-memory, fixed-length queue) if we don't have a connection to the
broker, or drop it if we don't have a connection to the broker.

   https://review.openstack.org/77845

I do understand the ambiguity around what message delivery guarantees
are implicit in cast() isn't ideal, but that's not what adding this
'retry' parameter was about.


Sure, I understand that. The retry option is necessitated by an 
(existing) implicit behaviour. However in my view that behaviour is 
implementations specific and of limited value in terms of the semantic 
contract of the call.


--Gordon.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] 'retry' option

[Gordon Sim]

I do apologise for omitting the subject qualifiers in my previous mail!

On 06/27/2014 05:02 PM, Gordon Sim wrote:

A question about the new 'retry' option. The doc says:

 By default, cast() and call() will block until the
 message is successfully sent.

What does 'successfully sent' mean here? Does it mean 'written to the
wire' or 'accepted by the broker'?

For the impl_qpid.py driver, each send is synchronous, so it means
accepted by the broker[1].

What does the impl_rabbit.py driver do? Does it just mean 'written to
the wire', or is it using RabbitMQ confirmations to get notified when
the broker accepts it (standard 0-9-1 has no way of doing this).

If the intention is to block until accepted by the broker that has
obvious performance implications. On the other hand if it means block
until written to the wire, what is the advantage of that? Was that a
deliberate feature or perhaps just an accident of implementation?

The use case for the new parameter, as described in the git commit,
seems to be motivated by wanting to avoid the blocking when sending
notifications. I can certainly understand that desire.

However, notifications and casts feel like inherently asynchronous
things to me, and perhaps having/needing the synchronous behaviour is
the real issue? Calls by contrast, are inherently synchronous, but at
present the retry controls only the sending of the request. If the
server fails, the call may timeout regardless of the value of 'retry'.

Just in passing, I'd suggest that renaming the new parameter
max_reconnects, would make it's current behaviour and values clearer.
The name 'retry' sounds like a yes/no type value, and retry=0 v. retry=1
is the reverse of what I would intuitively expect.

--Gordon.

[1] I've personally considered that somewhat unnecessary.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] 'retry' option

[Gordon Sim]

A question about the new 'retry' option. The doc says:

By default, cast() and call() will block until the
message is successfully sent.

What does 'successfully sent' mean here? Does it mean 'written to the 
wire' or 'accepted by the broker'?


For the impl_qpid.py driver, each send is synchronous, so it means 
accepted by the broker[1].


What does the impl_rabbit.py driver do? Does it just mean 'written to 
the wire', or is it using RabbitMQ confirmations to get notified when 
the broker accepts it (standard 0-9-1 has no way of doing this).


If the intention is to block until accepted by the broker that has 
obvious performance implications. On the other hand if it means block 
until written to the wire, what is the advantage of that? Was that a 
deliberate feature or perhaps just an accident of implementation?


The use case for the new parameter, as described in the git commit, 
seems to be motivated by wanting to avoid the blocking when sending 
notifications. I can certainly understand that desire.


However, notifications and casts feel like inherently asynchronous 
things to me, and perhaps having/needing the synchronous behaviour is 
the real issue? Calls by contrast, are inherently synchronous, but at 
present the retry controls only the sending of the request. If the 
server fails, the call may timeout regardless of the value of 'retry'.


Just in passing, I'd suggest that renaming the new parameter 
max_reconnects, would make it's current behaviour and values clearer. 
The name 'retry' sounds like a yes/no type value, and retry=0 v. retry=1 
is the reverse of what I would intuitively expect.


--Gordon.

[1] I've personally considered that somewhat unnecessary.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Gordon Sim]

On 06/26/2014 09:38 PM, Alexei Kornienko wrote:

Benchmark for oslo.messaging is really simple:
You create a client that sends messages infinitively and a server that
processes them. After you can use rabbitmq management plugin to see
average throughput in the queue.
Simple example can be found here -
https://github.com/andreykurilin/profiler-for-oslo-messaging


Just fyi, I also recently did some testing, though focused on scaling of 
rpc calls rather than throughput of a single client-server pair. The 
test I used is here in case anyone is interested: 
https://github.com/grs/ombt.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Gordon Sim]

On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote:

On 10/06/14 15:40, Alexei Kornienko wrote:

On 06/10/2014 03:59 PM, Gordon Sim wrote:

>>>

I think there could be a lot of work required to significantly
improve that driver, and I wonder if that would be better spent
on e.g. the AMQP 1.0 driver which I believe will perform much
better and will offer more choice in deployment.

>>

I agree with you on this. However I'm not sure that we can do such
a decision. If we focus on amqp driver only we should mention it
explicitly and deprecate qpid driver completely. There is no point
in keeping driver that is not really functional.


The driver is functional. It may be not that efficient as
alternatives, but that's not a valid reason to deprecate it.


The question in my view is what the plan is for ongoing development.

Will the driver get better over time, or is it likely to remain as is at 
best (or even deteriorate)?


Choice is good, but every choice adds to the maintenance burden, in 
testing against regressions if nothing else.


I think an explicit decision about the future is beneficial, whatever 
the decision may be.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/11/2014 06:05 PM, Janczuk, Tomasz wrote:

Process level isolation is more costly than sub-process level isolation
primarily due to larger memory consumption. For example, CGI has worse
cost characteristics than FastCGI when scaled out. But the example closer
to Marconi¹s use case is database systems: I can¹t put my finger on a
single one that would isolate queries executed by its users using
processes.


That is certainly a good point. However not all database systems would 
entirely isolate users from each other. In some a user could overload 
the system with lots of slow queries and impact the responsiveness of 
the database to other users. In extreme cases you could cause deadlocks 
in the database.



Sub-process multi-tenancy is a not an easy problem, but it is not one that
cannot be solved, especially in the context of a messaging system. A
messaging system exposes a set of operations that is very well defined and
constrained. The cost of these operations can be relatively easily
assessed a priori and managed accordingly to provide a secure multi-tenant
platform.


I agree it can be done. Not sure whether I agree that its relatively 
easy to do so - but then it is all relative! :-)


Thanks for the responses though, they have been helpful in improving my 
understanding.


--Gordon.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/11/2014 01:28 PM, Flavio Percoco wrote:

There are three things that I consider really valuable:

1. Sharding support: It allows admins to have separate clusters/nodes
of stores and configure marconi to use them all based on
pre-configured rules. It's similar to what qpid-dispatch does but the
supported rules in marconi are currently very limited.


How do the API servers coordinate on which messages/queues belong to 
which which shard(s)? Is there a shared database? Or some coordination 
protocol?



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/11/2014 12:31 PM, Flavio Percoco wrote:

On 06/10/2014 09:59 PM, Mark McLoughlin wrote:

Now why is there a desire to implement these requirements using
traditional message brokers?

[...]


There are 2 main reasons that I don't believe are strong enough to
change the way Marconi works right now:

1. Being able to support technologies that many deployments have
already deployed. This will facilitate Marconi's adoption but it also
brings in Marconi's scaling capabilities to those environments.


Is there somewhere I can read more about the mechanism behind those 
capabilities? (I.e. the mechanisms that Marconi implements independent 
of a given driver to allow scaling). Or can you describe it a little?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/10/2014 09:57 PM, Janczuk, Tomasz wrote:
> Using processes to isolate tenants is certainly possible. There is a range
> of isolation mechanisms that can be used, from VM level isolation
> (basically a separate deployment of the broker per-tenant), to process
> level isolation, to sub-process isolation. The higher the density the
> lower the overall marginal cost of adding a tenant to the system, and
> overall cost of operating it. From the cost perspective it is therefore
> desired to provide sub-process multi-tenancy mechanism; at the same time
> this is the most challenging approach.

Where does the increased cost for process level isolation come from? Is
it simply the extra process required (implying an eventual limit for a
given VM)?

With sub-process isolation you have to consider the fairness of
scheduling between operations for different tenants, i.e. potentially
limiting the processing done on behalf of any given tenant in a given
period. You would also need to limit the memory used on behalf of any
given tenant. Wouldn't you end up reinventing much of what the operating
system does?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/10/2014 09:59 PM, Mark McLoughlin wrote:

Now why is there a desire to implement these requirements using
traditional message brokers?


I would speculate that any desire to utilise a message broker as opposed 
to a database would be to achieve different performance characteristics 
for the service.


E.g. a message broker might be optimised for high throughput and low 
latency, a database for handling very large queues and random access to 
messages.


However if the interface through which the service is used requires 
random access to messages and polling then any perceived benefits of 
using a broker may not be realised anyway.



And what Marconi API semantics are impossible to implement using
traditional message brokers?

Either those semantics are fundamental requirements for this API, or the
requirement to have support for traditional message brokers is the
fundamental requirement. We can't have it both ways.


I suspect is is a question of understanding the expected performance 
metrics, the impact on those that aspects of the API design may have, 
and whether those aspects are intrinsic to the semantic requirements.


Is there an explicit list of requirements for Marconi anywhere? I've 
seen the use cases, which are useful. What would be the expected message 
rates and latencies for these patterns?


--Gordon.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/10/2014 06:33 PM, Janczuk, Tomasz wrote:

 From my perspective the key promise of Marconi is to provide a
*multi-tenant*,*HTTP*  based queuing system. Think an OpenStack equivalent
of SQS or Azure Storage Queues.

As far as I know there are no off-the-shelve message brokers out these
that fit that bill.


Indeed. The brokers I'm familiar with don't have multi-tenancy built 
into them. But rather than have one broker process support multiple 
tenants, wouldn't it be possible to just have separate processes (even 
separate containers) for each tenant?



Note that when I say ³multi-tenant² I don¹t mean just having multi-tenancy
concept reflected in the APIs. The key aspect of the multi-tenancy is
security hardening against a variety of attacks absent in single-tenant
broker deployments. For example, an authenticated DOS attack.


Understood, ensuring that one tenant is completely isolated from being 
impacted by anything another tenant might try to do.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/10/2014 05:27 PM, Kurt Griffiths wrote:

I think the crux of the issue is that Marconi follows the REST
architectural style. As such, the client must track the state of where it
is in the queue it is consuming (to keep the server stateless). So, it
must be given some kind of marker, allowing it to page through messages in
the queue.


Isn't this true both for task distribution and feed semantics?

[...]

To my knowledge, this API can’t be mapped directly to AMQP.


I believe you are right, you could not map the HTTP based interface 
defined for Marconi onto standard AMQP. As well as the explicit paging 
through the queue you mention above, the concepts of claims and direct 
deletion of messages are not exposed through AMQP[1].


AMQP was designed as an alternative interface into a messaging service, 
one tailored for asynchronous behaviour. It isn't intended as a direct 
interface to message storage.



Perhaps there are other types of brokers that can do it?


There are certainly brokers that support a RESTful HTTP based 
interface[2] alongside other protocols. I don't know of any standard 
protocol designed for controlling message storage however.


--Gordon.

[1] You could use a filter (aka a selector) to retrieve/dequeue a 
specific message by id or some other property. You could also define 
behaviour where an acquired message gets automatically released by the 
server after a configurable time. However, in my view, even with such 
extensions AMQP would still not be an ideal interface for message 
*storage* as opposed to message transfer between two processes.


[2] http://activemq.apache.org/rest.html, http://hornetq.jboss.org/rest 
and I'm sure there are others.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

[Gordon Sim]

On 06/10/2014 12:03 PM, Dina Belova wrote:

Hello, stackers!


Oslo.messaging is future of how different OpenStack components
communicate with each other, and really I’d love to start discussion
about how we can make this library even better then it’s now and how can
we refactor it make more production-ready.


As we all remember, oslo.messaging was initially inspired to be created
as a logical continuation of nova.rpc - as a separated library, with
lots of transports supported, etc. That’s why oslo.messaging inherited
not only advantages of now did the nova.rpc work (and it were lots of
them), but also some architectural decisions that currently sometimes
lead to the performance issues (we met some of them while Ceilometer
performance testing [1] during the Icehouse).


For instance, simple testing messaging server (with connection pool and
eventlet) can process 700 messages per second. The same functionality
implemented using plain kombu (without connection pool and eventlet)
driver is processing ten times more - 7000-8000 messages per second.


So we have the following suggestions about how we may make this process
better and quicker (and really I’d love to collect your feedback, folks):


1) Currently we have main loop running in the Executor class, and I
guess it’ll be much better to move it to the Server class, as it’ll make
relationship between the classes easier and will leave Executor only one
task - process the message and that’s it (in blocking or eventlet mode).
Moreover, this will make further refactoring much easier.

2) Some of the drivers implementations (such as impl_rabbit and
impl_qpid, for instance) are full of useless separated classes that in
reality might be included to other ones. There are already some changes
making the whole structure easier [2], and after the 1st issue will be
solved Dispatcher and Listener also will be able to be refactored.

3) If we’ll separate RPC functionality and messaging functionality it’ll
make code base clean and easily reused.

4) Connection pool can be refactored to implement more efficient
connection reusage.


Folks, are you ok with such a plan? Alexey Kornienko already started
some of this work [2], but really we want to be sure that we chose the
correct vector of development here.


For the impl_qpid driver, I think there would need to be quite 
significant changes to make it efficient. At present there are several 
synchronous roundtrips for every RPC call made[1]. Notifications are not 
treated any differently than RPCs (and sending a call is no different to 
sending a cast).


I agree the connection pooling is not efficient. For qpid at least it 
creates too many connections for no real benefit[2].


I think this may be a result of trying to fit the same high-level design 
to two entirely different underlying APIs.


For me at least, this also makes it hard to spot issues by reading the 
code. The qpid specific 'unit' tests for oslo.messaging also fail for me 
everytime when an actual qpidd broker is running (I haven't yet got to 
the bottom of that).


I'm personally not sure that the changes to impl_qpid you linked to have 
much impact on either efficiency or readability, safety of the code. I 
think there could be a lot of work required to significantly improve 
that driver, and I wonder if that would be better spent on e.g. the AMQP 
1.0 driver which I believe will perform much better and will offer more 
choice in deployment.


--Gordon

[1] For both the request and the response, the sender is created every 
time, which results in at least one roundtrip to the broker. Again, for 
both the request and the response, the message is then sent with a 
blocking send, meaning a further synchronous round trip for each. So for 
an RPC call, instead of just one roundtrip, there are at least four.


[2] In my view, what matters more than per-connection throughput for 
olso.messaging, is the scalability of the system as you add many RPC 
clients and servers. Excessive creation of connections by each process 
will have a negative impact on this. I don't believe the current code 
gets anywhere close to the limits of the underlying connection and 
suspect it would be more efficient and faster to multiplex different 
streams down the same connection. This would be especially true where 
using eventlet I suspect.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/09/2014 08:31 PM, Kurt Griffiths wrote:

Lately we have been talking about writing drivers for traditional
message brokers that will not be able to support the message feeds part
of the API.


Could you elaborate a little on this point? In some sense of the term at 
least, handling message feeds is what 'traditional' message brokers are 
all about. What are 'message feeds' in the Marconi context, in more 
detail? And what aspect of them is it that message brokers don't support?



I’ve started to think that having a huge part of the API
that may or may not “work”, depending on how Marconi is deployed, is not
a good story for users


I agree, that certainly doesn't sound good.

--Gordon.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [marconi] Reconsidering the unified API model

[Gordon Sim]

On 06/10/2014 09:48 AM, Mark McLoughlin wrote:

Perhaps the first point to get super clear on is why drivers for
traditional message brokers are needed. What problems would such drivers
address? Who would the drivers help? Would the Marconi team recommend
using any of those drivers for a production queuing service? Would the
subset of Marconi's API which is implementable by these drivers really
be useful for application developers?

I'd like to understand that in more detail because I worry the Marconi
team is being pushed into adding these drivers without truly believing
they will be useful.


Your questions are all good ones to ask, and I would certainly agree 
that doing anything without truly believing it to be useful is not a 
recipe for success.


To lay my cards on the table, my background is in message brokers, 
however I would certainly not want to appear to be pushing anyone into 
anything.


My question (in addition to those above) would be in what way is Marconi 
different to 'traditional message brokers' (which after all have been 
providing 'a production queuing service' for some time)?


I understand that having HTTP as the protocol used by clients is of 
central importance. However many 'traditional message brokers' have 
offered that as well. Will Marconi only support HTTP as a transport, or 
will it add other protocols as well?


Scalability is another focus as I understand it. There is more than one 
dimension to scaling when it comes to messaging however. Can anyone 
describe what is unique about the Marconi design with respect to 
scalability?


I sincerely hope these question don't sound argumentative, since that is 
not my intent. My background may blind me to what is obvious to others, 
and my questions may not be helpful. Please ignore them if that is the 
case! :-)


--Gordon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] using transport_url disables connection pooling?

[Gordon Sim]

This may be a known and accepted fact, but it was a surprise to me so 
thought it was worth a mention on this list.


If you use the transport_url config option (or pass a url directly to 
get_transport()), then the amqp connection pooling appears to be 
disabled[1]. That means a new connection is created for every request 
send as well as for every response.


So I guess rpc_backend is the recommended approach to selecting the 
transport at present(?).


[1] I think this is because if a transport url is specified then the url 
will always have at least the hosts set, so the AMQPDriverBase will 
always have a non-empty map for server_params, which overrides any use 
of pooling.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo]: some functional tests for the olso.messaging API

[Gordon Sim]

On 03/31/2014 04:11 PM, Doug Hellmann wrote:




On Mon, Mar 31, 2014 at 8:54 AM, Gordon Sim mailto:g...@redhat.com>> wrote:

I have recently been trying to get some API level functional tests
for the olso.messaging library. The idea is that these tests could
be run with any driver and configured 'backend' and would test the
basic functional guarantees that the API makes.


This sounds like a good idea, thanks for tackling it!


Initial changeset for review available here: 
https://review.openstack.org/#/c/84164/


I put these tests in their own module just to highlight that they are 
somewhat different in nature than the unit tests, requiring explicit 
selection of a transport url to use.


Any comments or feedback welcome!

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo]: some functional tests for the olso.messaging API

[Gordon Sim]

On 03/31/2014 04:41 PM, Doug Hellmann wrote:




On Mon, Mar 31, 2014 at 11:36 AM, Gordon Sim mailto:g...@redhat.com>> wrote:

On 03/31/2014 04:11 PM, Doug Hellmann wrote:

On Mon, Mar 31, 2014 at 8:54 AM, Gordon Sim mailto:g...@redhat.com>
<mailto:g...@redhat.com <mailto:g...@redhat.com>>> wrote:
 Another slight annoyance from the testing pov is that the API
 provides no way of determining when a server is 'ready'
i.e. when
 its subscriptions are in place. I had to work around this
for qpid
 and rabbit with a short sleep, which is always a bit nasty.


Are you saying that creating the subscription doesn't block
until it is
ready?


With the blocking executor, if I call start() on the server returned
by get_rpc_server(), then the start call blocks (until it detects
stop has been called). It does look like this is just an issue for
that executor though, since the eventlet executor will return after
the listen has been issued.


That's how I would expect them to work. Is the eventlet executor not
"ready" then?


Yes, as I said (in a rather clumsy way), it is just an issue for the 
blocking executor.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] ordering of notification 'events' in oslo.messaging

[Gordon Sim]

On 03/31/2014 03:36 PM, Sandy Walsh wrote:

If they're on different queues, the order they appear depends on the
consumer(s). It's not really an oslo.messaging issue.


Well, they are on two queues because that's how the oslo.messaging 
notification listener is implemented. However...


[...]

I think we have to assume that ordering cannot be guaranteed and it's
the consumers responsibility to handle it.


... that is indeed a perfectly reasonable position.

I was certainly arguing against that, I was merely pointing out one 
aspect that I found (a little) unexpected. Since it appears to be well 
understood, I apologise for the noise!



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo]: some functional tests for the olso.messaging API

[Gordon Sim]

On 03/31/2014 04:11 PM, Doug Hellmann wrote:

On Mon, Mar 31, 2014 at 8:54 AM, Gordon Sim mailto:g...@redhat.com>> wrote:
Another slight annoyance from the testing pov is that the API
provides no way of determining when a server is 'ready' i.e. when
its subscriptions are in place. I had to work around this for qpid
and rabbit with a short sleep, which is always a bit nasty.


Are you saying that creating the subscription doesn't block until it is
ready?


With the blocking executor, if I call start() on the server returned by 
get_rpc_server(), then the start call blocks (until it detects stop has 
been called). It does look like this is just an issue for that executor 
though, since the eventlet executor will return after the listen has 
been issued.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] ordering of notification 'events' in oslo.messaging

[Gordon Sim]

I believe that ordering of notifications at different levels is not 
guaranteed when receiving those notifications using a notification 
listener in olso.messaging.


I.e. with something like:

notifier = notifier.Notifier(get_transport(CONF), 'compute')
notifier.info(ctxt, event_type, payload_1)
notifier.warn(ctxt, event_type, payload_2)

its possible that payload_1 is received after payload_2. The root cause 
is that a different queue is used for events of each level.


In practice this is easier to observe with rabbit than qpid, as the qpid 
driver send every message synchronously which reduces the likelihood of 
there being more than one message on the listeners queues from the same 
notifier. Even for rabbit it takes a couple of thousand events before it 
usually occurs. Load on either the receiving client or the broker could 
increase the likelihood of out of order deliveries.


Not sure if this is intended, but as it isn't immediately obvious, I 
thought it would be worth a note to the list.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo]: some functional tests for the olso.messaging API

[Gordon Sim]

I have recently been trying to get some API level functional tests for 
the olso.messaging library. The idea is that these tests could be run 
with any driver and configured 'backend' and would test the basic 
functional guarantees that the API makes.


I have an initial set of such tests now. The url use is set via an 
environment variable, so e.g.


TRANSPORT_URL=rabbit://127.0.0.1 python -m testtools.run 
functional_tests.test_functional.NotifyTests


would run the notify tests against a local rabbit broker on the standard 
port. If attached the tests here, but if there is interest in including 
these in the repo, I'll create a proper changeset for review and comments.


I've run these tests against the qpid and rabbit drivers as well as the 
new AMQP 1.0 based driver available for initial review[1]. For rabbit 
and qpid there is one failure, where the exchange specified in the 
target is ignored. I'll be running them against the 0mq driver also.


At present, against rabbit and qpid, the tests work only if each test is 
run in isolation. This is because there appears to be no way through the 
API to have subscriptions created by the driver to be dropped, short of 
killing the process[2].


There may be some step my tests aren't doing that would trigger proper 
cleanup. This may also not be that important to real world uses of the 
library(?) and is only a minor irritation from the testing pov. However 
if there is interest, and assuming it is indeed an issue and not just a 
misunderstanding on my part,I can try and get a patch up for review to 
address it.


Another slight annoyance from the testing pov is that the API provides 
no way of determining when a server is 'ready' i.e. when its 
subscriptions are in place. I had to work around this for qpid and 
rabbit with a short sleep, which is always a bit nasty.


The last issue to mention here is that stop() implementation doesn't 
signal the driver in any way. So to actually get it to stop, at least 
for the blocking executor, you have to send a dummy message after 
calling stop() in order to have the poll() on the listener return and 
allow the detection of the stop flag. This is easy enough to work around 
in the tests and is a lot less nasty than the sleep.


--Gordon.

[1] https://review.openstack.org/#/c/75815/

[2] The cleanup() impl for both drivers simply closes any free 
connections in the pool
#
#Licensed under the Apache License, Version 2.0 (the "License"); you may
#not use this file except in compliance with the License. You may obtain
#a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
#Unless required by applicable law or agreed to in writing, software
#distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#License for the specific language governing permissions and limitations
#under the License.

import fixtures
import os
import random
import threading
import time

from oslo.config import cfg
from oslo import messaging
from oslo.messaging.notify import notifier
from six import moves
from tests import utils as test_utils
from testtools import matchers


class TestServer(object):
def __init__(self):
self.ival = 0
self.sval = ''

def add(self, ctxt, increment):
self.ival += increment
return self.ival

def subtract(self, ctxt, increment):
if self.ival < increment:
raise ValueError("ival can't go negative!")
self.ival -= increment
return self.ival

def append(self, ctxt, text):
self.sval += text
return self.sval


class TransportFixture(fixtures.Fixture):
def __init__(self, url):
self.url = url

def setUp(self):
super(TransportFixture, self).setUp()
self.transport = messaging.get_transport(cfg.CONF, url=self.url)

def cleanUp(self):
self.transport.cleanup()
super(TransportFixture, self).cleanUp()

def wait(self):
if self.url.startswith("rabbit")or self.url.startswith("qpid"):
time.sleep(0.5)


class RpcServerFixture(fixtures.Fixture):
def __init__(self, transport, target, obj=None, ctrl_target=None):
super(RpcServerFixture, self).__init__()
self.transport = transport
self.target = target
self.instance = obj or TestServer()
self.syncq = moves.queue.Queue()
self.ctrl_target = ctrl_target or self.target

def setUp(self):
super(RpcServerFixture, self).setUp()
instances = [self.instance, self]
self.server = messaging.get_rpc_server(self.transport,
   self.target,
   instances)
self._ctrl = messaging.RPCClient(self.transport, self.ctrl_target)
self._start()

def cleanUp(self):
self._stop()
super(RpcServerFixture, self)

[openstack-dev] [oslo] AMQP 1.0 based messaging driver available for review

[Gordon Sim]

I have been working with Ken Giusti on an AMQP 1.0 messaging driver for 
olso.messaging. The code for this is now available for review at

https://review.openstack.org/#/c/75815/

As previously mentioned on this list, this uses the Apache Qpid Proton 
'protocol engine' library. This is not a standalone client library, it 
includes no IO or threading, and simply provides the encoding and 
protocol rules. This was chosen to give full flexibility in other 
aspects of the client, allowing it to be tailored to best suit the 
olso.messaging usecase. It does mean however that the driver has a 
little more code than might be expected had a full client library been used.


The code uses a new directory (package) layout for this driver, based on 
suggestions from Flavio Percoco (thanks Flavio!). There is a protocols 
directory under _drivers, with amqp being used for this one. This name 
was preferred to proton to make it clear that the intention is to speak 
clean, neutral AMQP 1.0 and to avoid tying the driver code to specific 
intermediaries.


With that new amqp package, there is a subdirectory called engine which 
contains some generic wrappers around the lower level proton APIs. It 
may be that at some point this layer, or something similar, is available 
either directly in proton or in some supplementary library at which 
point we would have the option of dropping some of the code from within 
oslo.messaging itself.


The driver.py module implements the defined driver API, mostly by making 
requests on the controller.py module where most of the protocol mapping 
logic lies. The io is driven by a separate thread and the event loop for 
this is defined in eventloop.py. The threads calling on the driver 
communicate with the io thread (which uses non-blocking io), using Queues.


So far the testing has largely been through ad hoc clients and servers 
for the oslo.messaging API. I have also used it with nova under 
devstack. My knowledge of openstack is still very low. I followed the 
very helpful advice offered by Russell Bryant here: 
https://ask.openstack.org/en/question/28/openstack-api-mocker-or-simulator/


However I am eager to learn more and any suggestions for things to carry 
out with real openstack service will be greatly appreciated. I also plan 
to work on some functional tests at the olso.messaging API level that 
could be used to test any of the drivers. This would allow the lack of 
covergae with the current qpid driver to be addressed as welll as 
providing more confidence with this new driver (and would I think be 
useful for any other driver implementer who lacks sufficient knowledge 
of the other openstack services, or simply as a way of catching more 
issues earlier).


The current driver available for review requires an intermediary of some 
form, whether a 'broker' that supports AMQP 1.0 or something slightly 
different such as the Qpid Dispatch Router previously mentioned on this 
list.


I have tested successfully with qpidd and qpid dispatch router. At 
present RabbitMQ does not support 'dynamic nodes' necessary for 
temporary reply queues in 1.0, but I have a workaround planned for that.


This email is already growing rather long, so I'll leave it at that for 
now but would be delighted to answer any questions or address nay 
feedback whether here or through the review request above, or the 
associated blueprint: 
https://blueprints.launchpad.net/oslo.messaging/+spec/amqp10-driver-implementation


I will communicate regarding enhancements and additions to the code.

--Gordon

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

[Gordon Sim]

On 01/02/2014 10:46 PM, Herndon, John Luke wrote:



On 1/2/14, 11:36 AM, "Gordon Sim"  wrote:


On 12/20/2013 09:26 PM, Herndon, John Luke wrote:


On Dec 20, 2013, at 12:13 PM, Gordon Sim  wrote:


On 12/20/2013 05:27 PM, Herndon, John Luke wrote:


Other protocols may support bulk consumption. My one concern with
this approach is error handling. Currently the executors treat
each notification individually. So let¹s say the broker hands
100 messages at a time. When client is done processing the
messages, the broker needs to know if message 25 had an error or
not. We would somehow need to communicate back to the broker
which messages failed. I think this may take some refactoring of
executors/dispatchers. What do you think?

[...]

(2) What would you want the broker to do with the failed messages?
What sort of things might fail? Is it related to the message
content itself? Or is it failures suspected to be of a temporal
nature?


There will be situations where the message can¹t be parsed, and those
messages can¹t just be thrown away. My current thought is that
ceilometer could provide some sort of mechanism for sending messages
that are invalid to an external data store (like a file, or a
different topic on the amqp server) where a living, breathing human
can look at them and try to parse out any meaningful information.


Right, in those cases simply requeueing probably is not the right thing
and you really want it dead-lettered in some way. I guess the first
question is whether that is part of the notification systems function,
or if it is done by the application itself (e.g. by storing it or
republishing it). If it is the latter you may not need any explicit
negative acknowledgement.


Exactly, I¹m thinking this is something we¹d build into ceilometer and not
oslo, since ceilometer is where the event parsing knowledge lives. From an
oslo point of view, the message would be 'acked¹.




Other errors might be ³database not available², in which case
re-queing the message is probably the right way to go.


That does mean however that the backlog of messages starts to grow on
the broker, so some scheme for dealing with this if the database outage
goes on for a bit is probably important. It also means that the messages
will keep being retried without any 'backoff' waiting for the database
to be restored which could increase the load.


This is a problem we already have :(


Agreed, it is a property of reliable (i.e. acknowledged) transfer from 
the broker, rather than batching. And of course, some degree of 
buffering here is exactly what message queues are supposed to provide. 
The point is simply to provide some way of configuring things so that 
this can be bounded, or prevented from taking down the entire broker. 
(And perhaps some way of altering the unfortunate someone!)



https://github.com/openstack/ceilometer/blob/master/ceilometer/notification
.py#L156-L158
Since notifications cannot be lost, overflow needs to be detected and the
messages need to be saved. I¹m thinking the database being down is a rare
occurrence that will be worthy of waking someone up in the middle of the
night. One possible solution: flip the collector into an emergency mode
and save notifications to disc until the issue is resolved. Once the db is
up and running, the collector inserts all of these saved messages (as one
big batch!). Thoughts?

I¹m not sure I understand what you are saying about retrying without a
backoff. Can you explain?


I mean that if the messages are explicitly requeued and the original 
subscription is still active, they will be immediately redelivered and 
will thus keep cycling from broker to client, back to broker, back to 
client etc etc until the database is available again.


Pulling messages off continually like this without actually being able 
to dequeue them may reduce the brokers effectiveness at e.g. paging out, 
and in any event involves some unnecessary load on top of the expanding 
queue.


It might be better, just as an example, to abort the connection to the 
broker (implicitly requeueing all unacked messages), and only reconnect 
when the database becomes available (and that can be tried after 1 
second, then 2, then 4 etc up to some maximum retry interval).


Or another alternative would be to leave the connection to the broker, 
but by not requeing or acking ensure that once the prefetch has been 
reached, no further messages will be delivered. Then locally, on the 
client, retry the processing for the prefetched messages until the 
database is back again.


The basic point I'm trying to make is that it seems to me there is 
little value in simply handing the messages back to the broker for 
immediate redelivery back to the client. It delays the retry certainly, 
but at unnecessary expense.


More generally I wonder whether an explicit negative acknowledgement is 
actually needed in the notify API at all. If it isn't, tha

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

[Gordon Sim]

On 12/20/2013 09:26 PM, Herndon, John Luke wrote:


On Dec 20, 2013, at 12:13 PM, Gordon Sim  wrote:


On 12/20/2013 05:27 PM, Herndon, John Luke wrote:


Other protocols may support bulk consumption. My one concern with
this approach is error handling. Currently the executors treat
each notification individually. So let’s say the broker hands
100 messages at a time. When client is done processing the
messages, the broker needs to know if message 25 had an error or
not. We would somehow need to communicate back to the broker
which messages failed. I think this may take some refactoring of
executors/dispatchers. What do you think?

[...]

(2) What would you want the broker to do with the failed messages?
What sort of things might fail? Is it related to the message
content itself? Or is it failures suspected to be of a temporal
nature?

>

There will be situations where the message can’t be parsed, and those
messages can’t just be thrown away. My current thought is that
ceilometer could provide some sort of mechanism for sending messages
that are invalid to an external data store (like a file, or a
different topic on the amqp server) where a living, breathing human
can look at them and try to parse out any meaningful information.


Right, in those cases simply requeueing probably is not the right thing 
and you really want it dead-lettered in some way. I guess the first 
question is whether that is part of the notification systems function, 
or if it is done by the application itself (e.g. by storing it or 
republishing it). If it is the latter you may not need any explicit 
negative acknowledgement.



Other errors might be “database not available”, in which case
re-queing the message is probably the right way to go.


That does mean however that the backlog of messages starts to grow on 
the broker, so some scheme for dealing with this if the database outage 
goes on for a bit is probably important. It also means that the messages 
will keep being retried without any 'backoff' waiting for the database 
to be restored which could increase the load.





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

[Gordon Sim]

On 12/20/2013 07:13 PM, Gordon Sim wrote:

AMQP (in all it's versions) allows for a subscription with a
configurable amount of 'prefetch', which means the broker can send lots
of messages without waiting for the client to request them one at a time.


Just as an aside, the impl_qpid.py driver currently explicitly restricts 
the broker to sending one at a time. Probably not what we want for the 
notifications at any rate (more justifiable perhaps for the 'invoke on 
one of a group of servers' case).


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

[Gordon Sim]

On 12/20/2013 05:27 PM, Herndon, John Luke wrote:


On Dec 20, 2013, at 8:48 AM, Julien Danjou 
wrote:

Anyway, my main concern here is that I am not very enthusiast
about using the executor to do that. I wonder if there is not a way
to ask the broker to get as many as message as it has up to a
limit?

You would have 100 messages waiting in the notifications.info
queue, and you would be able to tell to oslo.messaging that you
want to read up to 10 messages at a time. If the underlying
protocol (e.g. AMQP) can support that too, it would be more
efficient too.


Yeah, I like this idea. As far as I can tell, AMQP doesn’t support
grabbing more than a single message at a time, but we could
definitely have the broker store up the batch before sending it
along.


AMQP (in all it's versions) allows for a subscription with a 
configurable amount of 'prefetch', which means the broker can send lots 
of messages without waiting for the client to request them one at a time.


That's not quite the same as the batching I think you are looking for, 
but it does allow the broker to do its own batching. My guess is the 
rabbit driver is already using basic.consume rather than basic.get 
anyway(?), so the broker is free to batch as it sees fit.  (I haven't 
actually dug into the kombu code to verify that however, perhaps someone 
else here can confirm?)


However you still need the client to have some way of batching up the 
messages and then processing them together.



Other protocols may support bulk consumption. My one concern
with this approach is error handling. Currently the executors treat
each notification individually. So let’s say the broker hands 100
messages at a time. When client is done processing the messages, the
broker needs to know if message 25 had an error or not. We would
somehow need to communicate back to the broker which messages failed.
I think this may take some refactoring of executors/dispatchers. What
do you think?


I've have some related questions, that I haven't yet satisfactorily 
answered yet. The extra context here may be useful in doing so.


(1) What are the expectations around message delivery guarantees for 
insertion into a store? I.e. if there is a failure, is it ok to get 
duplicate entries for notifications? (I'm assuming losing notifications 
is not acceptable).


(2) What would you want the broker to do with the failed messages? What 
sort of things might fail? Is it related to the message content itself? 
Or is it failures suspected to be of a temporal nature?


(3) How important is ordering ? If a failure causes some notifications 
to be inserted out of order is that a problem at all?


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo]: implementing olso.messaging over amqp 1.0

[Gordon Sim]

On 12/17/2013 01:53 PM, Flavio Percoco wrote:

On 16/12/13 10:37 +, Gordon Sim wrote:

On 12/12/2013 02:14 PM, Flavio Percoco wrote:

I've a draft in my head of how the amqp 1.0 driver could be
implemented and how to map the current expectations of the messaging
layer to the new protocol.

I think a separate thread to discuss this mapping is worth it. There
are some critical areas that definitely need more discussion


I have also been looking at this, and trying to write up a simple
design notes. Some of the questions that occurred to me while doing so
are:

* Use one link for all sends, with 'to' field set, or use a link for
each target?


I like this proposal. It keeps messages atomic and isolated from the
rest of the environment. The only I can think OTO is: What happens if
the node that the reply should go to dies before the reply is sent?

Is this something we should be worrying about? I mean, if the node
that was waiting for the response dies, I think we'd have bigger
problems than just a 'missed response' :D. However, this doesn't mean
we couldn't take care of that.


I'm afraid I'm not following you here. To clarify the original point, in 
AMQP 1.0 all messages are sent over a sending link (this is like a 
subscription, but for senders). You can also set an address per-message. 
However my view is that using a link per target is more interoperable at 
present. The spec doesn't really require the routing of messages by 'to' 
field and consequently not all implementations support it.


The point you are making seems to be around reliability(?). I would like 
to see some definitive statement about expectations in that regard for 
the API users and for transport implementers, but I think its a separate 
issue from whether or not to use a single link. (Perhaps the term 'link' 
is overloaded here, causing the confusion. In AMQP 1.0 a link is 
something like a subscription, but its a symmetric concept so also 
covers sending of messages).



* How to handle calls to one of a group of servers?


Could you elaborate more what issues you see around this?


In the 0-9-1 and 0-10 drivers, the pattern here is implemented by a 
shared subscription queue.


AMQP 1.0 doesn't define any mechanism to create a shared queue or a 
shared subscription. However most of the 1.0 implementations do allow 
queues to be created on demand. In the notes I wrote, I suggested using 
configurable prefixes for the addresses used, in order to accommodate a 
variety of implementations.



* Use a distinct response address per request, or allow an address to
be shared by multiple requests in conjunction with correlation id on
responses?


mmh, this is an interesting one. Using a distinct address for
responses will make the implementation less error prone. However, I
don't really think we need to have different addresses since there's
already a way to distinguish multiple requests. So, I'd prefer the
later.


Yes, I lean towards the latter option also. I'm actually not sure it is 
any more error prone either.



* Support both intermediated and direct communication? For all patterns?


Besides fanout, I'd say yes. We need to support intermediated and
direct communication.


The aim in my view should be to have the driver support as many
alternatives in deployment as possible without overcomplicating
things, distorting the mapping or introducing server specific extensions.

I have some notes to share if anyone is interested. I can send them to
this list or put them upon the wiki or an etherpad or something.


I think this questions are worth raising - thanks for that - and I
agree with you about not overcomplicating things. I think we could
start working something out and then we'll face different issues that
we'll need to discuss further on this list.

Cheers,
FF



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo]: implementing olso.messaging over amqp 1.0

[Gordon Sim]

On 12/16/2013 10:37 AM, Gordon Sim wrote:

On 12/12/2013 02:14 PM, Flavio Percoco wrote:

I've a draft in my head of how the amqp 1.0 driver could be
implemented and how to map the current expectations of the messaging
layer to the new protocol.

I think a separate thread to discuss this mapping is worth it. There
are some critical areas that definitely need more discussion


I have also been looking at this, and trying to write up a simple design
notes. Some of the questions that occurred to me while doing so are:

* Use one link for all sends, with 'to' field set, or use a link for
each target?

* How to handle calls to one of a group of servers?

* Use a distinct response address per request, or allow an address to be
shared by multiple requests in conjunction with correlation id on
responses?

* Support both intermediated and direct communication? For all patterns?

The aim in my view should be to have the driver support as many
alternatives in deployment as possible without overcomplicating things,
distorting the mapping or introducing server specific extensions.

I have some notes to share if anyone is interested. I can send them to
this list or put them upon the wiki or an etherpad or something.


I've pasted these into an etherpad[1] for anyone interested. Please feel 
free to edit/augment etc, or even to query anything on this list. It's 
really just an initial draft to get the ball rolling.


--Gordon

[1] https://etherpad.openstack.org/p/olso.messaging_amqp_1.0


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo]: implementing olso.messaging over amqp 1.0

[Gordon Sim]

On 12/12/2013 02:14 PM, Flavio Percoco wrote:

I've a draft in my head of how the amqp 1.0 driver could be
implemented and how to map the current expectations of the messaging
layer to the new protocol.

I think a separate thread to discuss this mapping is worth it. There
are some critical areas that definitely need more discussion


I have also been looking at this, and trying to write up a simple design 
notes. Some of the questions that occurred to me while doing so are:


* Use one link for all sends, with 'to' field set, or use a link for 
each target?


* How to handle calls to one of a group of servers?

* Use a distinct response address per request, or allow an address to be 
shared by multiple requests in conjunction with correlation id on responses?


* Support both intermediated and direct communication? For all patterns?

The aim in my view should be to have the driver support as many 
alternatives in deployment as possible without overcomplicating things, 
distorting the mapping or introducing server specific extensions.


I have some notes to share if anyone is interested. I can send them to 
this list or put them upon the wiki or an etherpad or something.


--Gordon.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] TransportURL and virtualhost/exchnage (was Re: [Oslo] Layering olso.messaging usage of config)

[Gordon Sim]

Hi Mark,

Thanks for the response! Some further followup inline...

On 12/09/2013 09:53 PM, Mark McLoughlin wrote:

It's not a completely unreasonable approach to take, but my thinking was
that a transport URL connects you to a conduit which multiple
applications could be sharing so you need the application to specify its
own application namespace.


I agree with the general picture, I was just wondering whether the 
application is best placed to specify its own namespace.



e.g. you can have 'scheduler' topics for both Nova and Cinder, and you
need each application to specify its exchange whereas the administrator
is in full control of the transport URL and doesn't need to worry about
application namespacing on the transport.


I can see the value in simplifying things for the administrator by default.


There are three ways the exchange appears in the API:

   1) A way for an application to set up the default exchange it
  operates in:

  messaging.set_transport_defaults(control_exchange='nova')

  
http://docs.openstack.org/developer/oslo.messaging/transport.html#oslo.messaging.set_transport_defaults

   2) The server can explicitly say what exchange its listening on:

  target = messaging.Target(exchange='nova',
topic='scheduler',
server='myhost')
  server = messaging.get_rpc_server(transport, target, endpoints)

  
http://docs.openstack.org/developer/oslo.messaging/server.html#oslo.messaging.get_rpc_server

   3) The client can explicitly say what exchange to connect to:

  target = messaging.Target(exchange='nova',
topic='scheduler')
  client = messaging.RPCClient(transport, target)

  
http://docs.openstack.org/developer/oslo.messaging/rpcclient.html#oslo.messaging.RPCClient

But also the admin can override the default exchange so that e.g. you
could put two instances of the application on the same transport, but
with different exchanges.


I guess I'm saying (thinking aloud?) that if you have this ability (and 
indeed this need), then why not take the responsibility out of the 
application altogether?


Where an application chooses the exchange itself, is that usually 
hardcoded? or would that also be a configuration item, albeit of the 
application and not the transport? I assume the override applies on to 
item (1) above?



Now, in saying all that, we know that fanout topics of the same name
will conflict even if the exchange name is different:

   https://bugs.launchpad.net/oslo.messaging/+bug/1173552

So that means the API doesn't work quite as intended yet, but I think
the idea makes sense.

I'm guessing you have a concern about how transports might implement
this application namespacing? Could you elaborate if so?


Not really, though the mechanisms available may vary from one transport 
to another. I don't really have any real concerns with things as they are.


My motivation, if that is the right word, was really just around 
simplicity. I was originally confused by what appeared to be multiple 
different ways to disambiguate names and thought this suggestion might 
simplify things overall a little. However as you point out it make it 
less simple for the administrator.


--Gordon



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] First steps towards amqp 1.0

[Gordon Sim]

On 12/10/2013 12:36 AM, Mike Wilson wrote:

This is the first time I've heard of the dispatch router, I'm really
excited now that I've looked at it a bit. Thx Gordon and Russell for
bringing this up. I'm very familiar with the scaling issues associated
with any kind of brokered messaging solution. We grew an Openstack
installation to about 7,000 nodes and started having significant scaling
issues with the qpid broker. We've talked about our problems at a couple
summits in a fair amount of detail[1][2]. I won't bother repeating the
information in this thread.

I really like the idea of separating the logic of routing away from the
the message emitter. Russell mentioned the 0mq matchmaker, we
essentially ditched the qpid broker for direct communication via 0mq and
it's matchmaker. It still has a lot of problems which dispatch seems to
address. For example, in ceilometer we have store-and-forward behavior
as a requirement. This kind of communication requires a broker but 0mq
doesn't really officially support one, which means we would probably end
up with some broker as part of OpenStack. Matchmaker is also a fairly
basic implementation of what is essentially a directory. For any sort of
serious production use case you end up sprinkling JSON files all over
the place or maintaining a Redis backend. I feel like the matchmaker
needs a bunch more work to make modifying the directory simpler for
operations. I would rather put that work into a separate project like
dispatch than have to maintain essentially a one off in Openstack's
codebase.

I wonder how this fits into messaging from a driver perspective in
Openstack or even how this fits into oslo.messaging? Right now we have
topics for binaries(compute, network, consoleauth, etc),
hostname.service_topic for nodes, fanout queue per node (not sure if
kombu also has this) and different exchanges per project. If we can
abstract the routing from the emission of the message all we really care
about is emitter, endpoint, messaging pattern (fanout, store and
forward, etc). Also not sure if there's a dispatch analogue in the
rabbit world, if not we need to have some mapping of concepts etc
between impls.

So many questions, but in general I'm really excited about this and
eager to contribute. For sure I will start playing with this in
Bluehost's environments that haven't been completely 0mqized. I also
have some lingering concerns about qpid in general. Beyond scaling
issues I've run into some other terrible bugs that motivated our move
away from it. Again, these are mentioned in our presentations at summits
and I'd be happy to talk more about them in a separate discussion. I've
also been able to talk to some other qpid+openstack users who have seen
the same bugs. Another large installation that comes to mind is Qihoo
360 in China. They run a few thousand nodes with qpid for messaging and
are familiar with the snags we run into.

Gordon,

I would really appreciate if you could watch those two talks and
comment. The bugs are probably separate from the dispatch router
discussion, but it does dampen my enthusiasm a bit not knowing how to
fix issues beyond scale :-(.


Mike (and others),

First, as a Qpid developer, let me apologise for the frustrating 
experience you have had.


The qpid components used here are not the most user friendly, it has to 
be said. They work well for the paths most usually taken, but there can 
be some unanticipated problems outside that.


The main failing I think is that we in the Qpid community did not get 
involved in OpenStack to listen, understand the use cases and to help 
diagnose and address problems earlier. I joined this list specifically 
to try and rectify that failing.


The specific issues I gleaned from the presentations were:

(a) issues with eventlet and qpid.messaging integration

The qpid.messaging library does some perhaps quirky things that made the 
monkey patched solution more awkward. The openstack rpc implementation 
over qpid was heavily driven by the kombu base rabbitmq implementation, 
although the client libraries are quite different in design. The 
addressing syntax for the qpid.messaging library is not always the most 
intuitive.


As suggested in another mail on this thread, for an AMQP 1.0 based 
driver I would pick an approach that allows olso.messaging to retain 
control over threading choices etc to avoid some of these sorts of 
integration issues, and program more directly to the protocol.


(b) general scaling issues with standalone qpidd instance,

As you point out very clearly, a single broker is always going to be a 
bottleneck. Further there are some aspects of the integration code that 
I think unnecessarily reduce performance. E.g. each call and cast is 
synchronous, only a single message of prefetch is enabled for any 
subscription (forcing more roundtrips), senders and receivers are 
created for every request and reply  etc.


(c) message loss,

The code I have studied doesn't enable acknowledgements

Re: [openstack-dev] [Oslo] First steps towards amqp 1.0

[Gordon Sim]

On 12/09/2013 11:29 PM, Mark McLoughlin wrote:

On Mon, 2013-12-09 at 16:05 +0100, Flavio Percoco wrote:

Greetings,

As $subject mentions, I'd like to start discussing the support for
AMQP 1.0[0] in oslo.messaging. We already have rabbit and qpid drivers
for earlier (and different!) versions of AMQP, the proposal would be
to add an additional driver for a _protocol_ not a particular broker.
(Both RabbitMQ and Qpid support AMQP 1.0 now).

By targeting a clear mapping on to a protocol, rather than a specific
implementation, we would simplify the task in the future for anyone
wishing to move to any other system that spoke AMQP 1.0. That would no
longer require a new driver, merely different configuration and
deployment. That would then allow openstack to more easily take
advantage of any emerging innovations in this space.


Sounds sane to me.

To put it another way, assuming all AMQP 1.0 client libraries are equal,
all the operator cares about is that we have a driver that connect into
whatever AMQP 1.0 messaging topology they want to use.

Of course, not all client libraries will be equal, so if we don't offer
the choice of library/driver to the operator, then the onus is on us to
pick the best client library for this driver.


That is a fair point. One thing to point out about Qpid proton is that 
it is in fact two different things in the one library. On the one hand 
it is a fully fledged client library with its own IO and model of use. 
On the other hand it has a passive protocol engine that is agnostic as 
to the IO/threading approach used and merely encapsulates the encoding 
and protocol rules. This allows it to be integrated into different 
environments without imposing architectural restrictions.


My suggestion would be to use the protocol engine, and design the IO and 
threading to work well with the rest of the oslo.messaging code (e.g. 
with eventlet or asynchio or whatever). In some ways this makes 
oslo.messaging a client library in its own right, with and RPC and 
notify based API and ensuring that other choices fit in well with the 
overall codebase.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] First steps towards amqp 1.0

[Gordon Sim]

On 12/09/2013 10:37 PM, Russell Bryant wrote:

On 12/09/2013 05:16 PM, Gordon Sim wrote:

On 12/09/2013 07:15 PM, Russell Bryant wrote:
Understood. The Dispatch Router was indeed created from an understanding
of the limitations and drawbacks of the 'federation' feature of qpidd
(which was the primary mechanism for scaling beyond one broker) as well
learning lessons around the difficulties of message replication and
storage.


Cool.  To make the current situation worse, AFAIK, we've never been able
to make Qpid federation work at all for OpenStack.  That may be due to
the way we use Qpid, though.


The federation in qpidd requires path specific configuration. I.e. for 
each topic or queu or whatever you need to explicitly enable the flow of 
messages in each direction between each pair of brokers.


The original uses cases it was designed for were much simpler than 
openstacks needs and this wasn't insurmountable. As it was used in new 
areas however the limitation became apparent. The Dispatch Router 
instances on the other hand communicate with each other to automatically 
setup the internal routing necessary to ensure publishers and 
subscribers communicate regardless of the point at which they are attached.


Another limitation of the original federation was the inability to 
handle redundant routes between brokers without duplicating messages.


[...]

The Dispatch Router can work for pub-sub patterns as well, though not
store and forward directly. In theory, for flows where store-and-forward
is needed, that can be supplied by an additional service e.g. a more
traditional broker, which would take responsibility for replaying over
from the publisher in order that subscribers could if needed have
message replayed even after the original publisher had exited.


Any thoughts on what we would be recommending for notifications?


For basic pub-sub (i.e. no durable/reliable subscriptions that retain 
messages across disconnects), dispatch router itself will work fine.


For those cases where you do need store-and-forward, i.e. where 
something other than the publisher needs to reliably store notifications 
until all interested subscribers have confirmed receipt, you could use a 
broker (rabbitmq, qpidd, activemq or similar).


The plan for dispatch router is to allow such brokers to be hooked into 
the router network also. So you could have the broker(s) accept the 
messages from the publishers, and then have any subscribers needing 
reliability guarantees subscribe to the broker via the router network. 
The aim is to allow the load to be spread across multiple brokers, but 
have dispatch router control the flow of messages in and out of these, 
rather than relying on the brokers to link themselves up. This aspect 
however is not yet built (but is coming soon!).


Durable/reliable subscriptions do of course bring with them the need to 
manage potential growth of backed up, unconfirmed messages, e.g. when a 
subscriber is down or unresponsive for some time. Ensuring that there 
are limits in place that prevent this getting out of hand are essential.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Oslo] First steps towards amqp 1.0

[Gordon Sim]

On 12/09/2013 07:15 PM, Russell Bryant wrote:

On 12/09/2013 12:56 PM, Gordon Sim wrote:

In the case of Nova (and others that followed Nova's messaging
patterns), I firmly believe that for scaling reasons, we need to move
toward it becoming the norm to use peer-to-peer messaging for most
things.  For example, the API and conductor services should be talking
directly to compute nodes instead of through a broker.


Is scale the only reason for preferring direct communication? I don't
think an intermediary based solution _necessarily_ scales less
effectively (providing it is distributed in nature, which for example is
one of the central aims of the dispatch router in Qpid).

That's not to argue that peer-to-peer shouldn't be used, just trying to
understand all the factors.


Scale is the primary one.  If the intermediary based solution is easily
distributed to handle our scaling needs, that would probably be fine,
too.  That just hasn't been our experience so far with both RabbitMQ and
Qpid.


Understood. The Dispatch Router was indeed created from an understanding 
of the limitations and drawbacks of the 'federation' feature of qpidd 
(which was the primary mechanism for scaling beyond one broker) as well 
learning lessons around the difficulties of message replication and storage.



One other pattern that can benefit from intermediated message flow is in
load balancing. If the processing entities are effectively 'pulling'
messages, this can more naturally balance the load according to capacity
than when the producer of the workload is trying to determine the best
balance.


Yes, that's another factor.  Today, we rely on the message broker's
behavior to equally distribute messages to a set of consumers.


Sometimes you even _want_ message distribution to be 'unequal', if the
load varies by message or the capacity by consumer. E.g. If one consumer
is particularly slow (or is given a particularly arduous task), it may 
not be optimal for it to receive the same portion of subsequent messages 
as other less heavily loaded or more powerful consumers.



One example is how Nova components talk to the nova-scheduler service.
All instances of the nova-scheduler service are reading off a single
'scheduler' queue, so messages hit them round-robin.

In the case of the zeromq driver, this logic is embedded in the client.
  It has to know about all consumers and handles choosing where each
message goes itself.  See references to the 'matchmaker' code for this.

Honestly, using a distributed more lightweight router like Dispatch
sounds *much* nicer.


  The exception
to that is cases where we use a publish-subscribe model, and a broker
serves that really well.  Notifications and notification consumers
(such as Ceilometer) are the prime example.


The 'fanout' RPC cast would perhaps be another?


Good point.

In Nova we have been working to get rid of the usage of this pattern.
In the latest code the only place it's used AFAIK is in some code we
expect to mark deprecated (nova-network).


Interesting. Is that because of problems in scaling the messaging 
solution or for other reasons?


[...]

I'm very interested in diving deeper into how Dispatch would fit into
the various ways OpenStack is using messaging today.  I'd like to get
a better handle on how the use of Dispatch as an intermediary would
scale out for a deployment that consists of 10s of thousands of
compute nodes, for example.

Is it roughly just that you can have a network of N Dispatch routers
that route messages from point A to point B, and for notifications we
would use a traditional message broker (qpidd or rabbitmq) ?


For scaling the basic idea is that not all connections are made to the 
same process and therefore not all messages need to travel through a 
single intermediary process.


So for N different routers, each have a portion of the total number of 
publishers and consumers connected to them. Though client can 
communicate even if they are not connected to the same router, each 
router only needs to handle the messages sent by the publishers directly 
attached, or sent to the consumer directly attached. It never needs to 
see messages between publishers and consumer that are not directly attached.


To address your example, the 10s of thousands of compute nodes would be 
spread across N routers. Assuming these were all interconnected, a 
message from the scheduler would only travel through at most two of 
these N routers (the one the scheduler was connected to and the one the 
receiving compute node was connected to). No process needs to be able to 
handle 10s of thousands of connections itself (as contrasted with full 
direct, non-intermediated communication, where the scheduler would need 
to manage connections to each of the compute nodes).


This basic pattern is the same as networks of brokers, but Dispatch 
router has been

Re: [openstack-dev] [Oslo] First steps towards amqp 1.0

[Gordon Sim]

On 12/09/2013 04:10 PM, Russell Bryant wrote:

From looking it appears that RabbitMQ's support is via an experimental
plugin.  I don't know any more about it.  Has anyone looked at it in
detail?


I believe initial support was added in 3.1.0: 
http://www.rabbitmq.com/release-notes/README-3.1.0.txt


I have certainly successfully tested basic interaction with RabbitMQ 
over 1.0.



 https://www.rabbitmq.com/specification.html

As I understand it, "Qpid" supports it, in that it's a completely new
implementation as a library (Proton) under the Qpid project umbrella.
  There's also a message router called Dispatch.  This is not to be
confused with the existing Qpid libraries, or the existing Qpid broker
(qpidd).


Yes, proton is a library that encapsulates the AMQP 1.0 encoding and 
protocol rules. It is used in the existing native broker (i.e. qpidd) to 
offer 1.0 support (as well as the qpid::messaging c++ client library).


In addition there is as you mention the 'Dispatch Router'. This is an 
alternative architecture for an intermediary to address some of the 
issues that can arise with qpidd or other brokers (distributed in nature 
to scale better, end-to-end reliability rather than store and forward etc).


So the Qpid project offers both new components as well as 1.0 support 
and smooth transition for existing components. (Disclosure, I'm a 
developer in the Qpid community also).


(There are of course other implementations also e.g., ActiveMQ, 
ApolloMQ, HornetQ, Microsoft ServiceBus, SwiftMQ.)



 http://qpid.apache.org/proton/
 http://qpid.apache.org/components/dispatch-router/



[...]


All of this sounds fine to me.  Surely a single driver for multiple
systems is an improvement.  What's not really mentioned though is why
we should care about AMQP 1.0 beyond that.  Why is it architecturally
superior?  It has been discussed on this list some before, but I
figure it's worth re-visiting if some code is going to show up soon.


Personally I think there is benefit to having a standardised, open 
wire-protocol as the basis for communication in systems like OpenStack, 
rather than having the driver tied to a particular implementation 
throughout (and having more of the key details of the interaction as 
details of the implementation of the driver). The bytes over the wire 
are another level of interface and having that tightly specified can be 
valuable.


Having one driver that still offers choice with regard to intermediaries 
used (I avoid the term broker in case it is implies particular 
approaches), is I think an advantage. Hypothetically for example it 
would have been an advantage had the same driver been usable against 
both RabbitMQ and Qpid previously. The (bumpy!) evolution  of AMQP meant 
that wasn't quite possible since they both spoke different versions 
ofthe early protocol. AMQP 1.0 might in the future avoid needing new 
drivers in such cases however, making it easier to adopt alternative or 
emerging solutions.


AMQP is not the only messaging protocol of course, However its general 
purpose nature and the fact that both RabbitMQ and Qpid really came 
about through AMQP makes it a reasonable choice.



In the case of Nova (and others that followed Nova's messaging
patterns), I firmly believe that for scaling reasons, we need to move
toward it becoming the norm to use peer-to-peer messaging for most
things.  For example, the API and conductor services should be talking
directly to compute nodes instead of through a broker.


Is scale the only reason for preferring direct communication? I don't 
think an intermediary based solution _necessarily_ scales less 
effectively (providing it is distributed in nature, which for example is 
one of the central aims of the dispatch router in Qpid).


That's not to argue that peer-to-peer shouldn't be used, just trying to 
understand all the factors.


One other pattern that can benefit from intermediated message flow is in 
load balancing. If the processing entities are effectively 'pulling' 
messages, this can more naturally balance the load according to capacity 
than when the producer of the workload is trying to determine the best 
balance.



 The exception
to that is cases where we use a publish-subscribe model, and a broker
serves that really well.  Notifications and notification consumers
(such as Ceilometer) are the prime example.


The 'fanout' RPC cast would perhaps be another?


In terms of existing messaging drivers, you could accomplish this with
a combination of both RabbitMQ or Qpid for brokered messaging and
ZeroMQ for the direct messaging cases.  It would require only a small
amount of additional code to allow you to select a separate driver for
each case.

Based on my understanding, AMQP 1.0 could be used for both of these
patterns.  It seems ideal long term to be able to use the same
protocol for everything we need.


That is certainly true. AMQP 1.0 is fully symmetric so it can be used 
directly peer-to-peer as well as

[openstack-dev] TransportURL and virtualhost/exchnage (was Re: [Oslo] Layering olso.messaging usage of config)

[Gordon Sim]

On 11/18/2013 04:44 PM, Mark McLoughlin wrote:

On Mon, 2013-11-18 at 11:29 -0500, Doug Hellmann wrote:

IIRC, one of the concerns when oslo.messaging was split out was
maintaining support for existing deployments with configuration files that
worked with oslo.rpc. We had said that we would use URL query parameters
for optional configuration values (with the required values going into
other places in the URL)

[...]

I hadn't ever considered exposing all configuration options via the URL.
We have a lot of fairly random options, that I don't think you need to
configure per-connection if you have multiple connections in the one
application.


I certainly agree that not all configuration options may make sense in a 
URL. However if you will forgive me for hijacking this thread 
momentarily on a related though tangential question/suggestion...


Would it make sense to (and/or even be possible to) take the 'exchange' 
option out of the API, and let transports deduce their implied 
scope/namespace purely from the transport URL in perhaps transport 
specific ways?


E.g. you could have rabbit://my-host/my-virt-host/my-exchange or 
rabbit://my-host/my-virt-host or rabbit://my-host//my-exchange, and the 
rabbit driver would ensure that the given virtualhost and or exchange 
was used.


Alternatively you could have zmq://my-host:9876 or zmq://my-host:6789 
to 'scope' 0MQ communication channels. and hypothetically 
something-new://my-host/xyz, where xyz would be interpreted by the 
driver in question in a relevant way to scope the interactions over that 
transport.


Applications using RPC would then assume they were using a namespace 
free from the danger of collisions with other applications, but this 
would all be driven through transport specific configuration.


Just a suggestion based on my initial confusion through ignorance on the 
different scoping mechanisms described in the API docs. It may not be 
feasible or may have negative consequences I have not in my naivety 
foreseen.


--Gordon.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][messaging]: some questions on the 'driver' API

[Gordon Sim]

The BaseDriver - which I take to be the API that must be fulfilled by 
any transport implementation - has a listen() method taking a target, 
but no way to stop listening on that specific target.


Is this because there is no need to ever cancel a subscription?

There is a cleanup() method on the BaseDriver and the Transport object 
that contains it. However if more than one Transport object exists (e.g. 
because there are more than one Targets) then they will share the same 
driver, so calling cleanup on one would affect them all.


On the topic of relationship between Transport (and therefore RPCClient) 
and driver instances, it would be nice if there was some more explicit 
association.  For example, some transport mechanisms may require some 
setup for sending to each distinct address, and would perhaps want to 
avoid doing that on every individual call. They could of course cache 
their own context against the target instance. However it seems like it 
might be generally useful for drivers to be aware of the lifecycle of 
the different Transports or RPCClient instances that use them (it might 
help inform other caching or pooling choices for example).


This is admittedly coming from a very limited understanding  of the 
anticipated use cases, and perhaps assuming it is a more general purpose 
rpc mechanism than is actually intended.


--Gordon.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo][messaging]: expected delivery guarantees & handling failure

[Gordon Sim]

What are the expected delivery guarantees for messages for rpc 
transports in oslo.messaging?


Are failures to be handled by the applications using the rpc interface 
(e.g. by retrying on timeout)? Or is the transport expected to provide 
the necessary reliability of messages? Or is this something that can 
vary by transport (i.e. with different qos for different transports)?


I see acks seem to be used for consumer with rabbit, but there is no 
means in AMQP 0-9-1 to confirm published messages (Rabbit have an 
extension, but I can't see this being used either). So though the 
delivery from broker to consumers is reliable (assuming durable or 
replicated storage of the message by the broker), the delivery from 
producers to the broker is not.


For qpid the queues are always autodelete and no reliability is set for 
consumers (i.e. messages are assumed delivered without waiting for 
acknowledgement). As the reconnect is handled by the driver rather than 
the underlying qpid.messaging client, in doubt published messages aren't 
replayed either.


I'm very new to the code so I may be missing something, but it seems 
like messages could be lost regardless of broker configuration with 
either the rabbit or qpid drivers and just wanted to confirm this is as 
intended.


--Gordon.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev