Re: [openstack-dev] [all] Treating notifications as a contract
There's a review in progress for a generic event format for PaaS-services which is a move with the right spirit: allow various services to join the the notification party without needing special handlers. See: https://review.openstack.org/#/c/101967/ -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Tue, 15 Jul 2014, Sandy Walsh wrote: This looks like a particular schema for one event-type (let's say "foo.sample"). It's hard to extrapolate this one schema to a generic set of common metadata applicable to all events. Really the only common stuff we can agree on is the stuff already there: tenant, user, server, message_id, request_id, timestamp, event_type, etc. This is pretty much what I'm trying to figure out. We can, relatively agree on a small set of stuff (like what you mention). Presumably there are three more sets: * special keys that could be changed to something more generally meaningful if we tried hard enough * special keys that really are special and must be saved as such * special keys that nobody cares about and can be tossed Everybody thinks their own stuff is special[1] but it is often the case that it's not. In your other message you linked to http://paste.openstack.org/show/54140/ which shows some very complicated payloads (but only gets through the first six events). Is there related data (even speculative) for how many of those keys are actually used? And just looking at the paste (and the problem) generally, does it make sense for the accessors in the dictionaries (the keys) to be terms which are specific to the producer? Obviously that will increase the appearance of disjunction between different events. A different representation might not be as problematic. Or maybe I'm completely wrong, just thinking out loud. This way, we can keep important notifications in a priority queue and handle them accordingly (since they hold important data), but let the samples get routed across less-reliable transports (like UDP) via the RoutingNotifier. Presumably a more robust, uh, contract, for notifications, will allow them to be dispatched (and re-dispatched) more effectively. Also, send the samples one-at-a-time and let them either a) drop on the floor (udp) or b) let the aggregator roll them up into something smaller (sliding window, etc). Making these large notifications contain a list of samples means we had to store state somewhere on the server until transmission time. Ideally something we wouldn't want to rely on. I've wondered about this too. Is there history for why some of the notifications which include samples are rolled up lists instead of fired off one at a time. Seems like that will hurt parallelism opportunities? [1] There's vernacular here that I'd prefer to use but this is a family mailing list. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Tue, 15 Jul 2014, Mark McLoughlin wrote: So you're proposing that all payloads should contain something like: [...] a class, type, id, value, unit and a space to put additional metadata. That's the gist of it, but I'm only presenting that as a way to get somebody else to point out what's wrong with it so we can get closer to what's actually needed... On the subject of "notifications as a contract", calling the additional metadata field 'extra' suggests to me that there are no stability promises being made about those fields. Was that intentional? ...and as you point out, if everything that doesn't fit in the "known" fields goes in 'extra' then the goal of contractual stability may be lost. What I think that shows us is that what we probably want is three levels of contract. Currently we have one: * There is a thing called a notification and it has a small number of top-level fields include 'payload' At the second level would be: * There is a list of things _in_ the payload which are events and they have some known general struture that allows injestion (as data) by lots of consumers. And the third level would be: * Each event has an internal structure (below the general structure) which is based on its type. In the simplest cases (some meters for example) a third level could either not be necessary or at least very small[1]. This is the badly named "extras" above. Basically: If people are willing to pay the price (in terms of changes) for contractual stability may as well get some miles out of it. Three layers of abstraction means there can be three distinct levels in applications or tools, each of which are optimized to a different level of the topology: transporting messages, persisting/publishing messages, extracting meaning from messages. That's kind of present already, but it is done in a way that requires a lot of duplication of knowledge between producer and consumer and within different parts of the consumer. Which makes effective testing and scaling more complex that it ought to be. I know from various IRC chatter that this is a perennial topic round these parts, frequently foundering, but perhaps each time we get a bit further, learn a bit more? In any case, as Eoghan and I stated elsewhere in the thread I'm going to try to drive this forward, but I'm not going to rush it as there's no looming deadline. [1] Part of the reason I drove us off into the so-called weeds earlier in the thread is a sense that a large number of events/samples/notification-payloads are capable of being classed as nearly the same thing (except for their name and units) and thus would not warrant their own specific schema. These are "just" events that should have a very similar structure. If that structure is well known and well accepted we will likely find that many existing events can be bent to fit that structure for the sake of less code and more reuse. As part of this process I'll try to figure out if this is true . -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/11/2014 6:08 AM, Chris Dent wrote: > On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: > >> The data format that Ironic will send was part of the spec proposed >> and could have been reviewed. I think there's still time to change it >> tho, if you have a better format talk to Haomeng which is the guys >> responsible for that work in Ironic and see if he can change it (We >> can put up a following patch to fix the spec with the new format as >> well) . But we need to do this ASAP because we want to get it landed >> in Ironic soon. > It was only after doing the work that I realized how it might be an > example for the sake of this discussion. As the architecure of > Ceilometer currently exist there still needs to be some measure of > custom code, even if the notifications are as I described them. > > However, if we want to take this opportunity to move some of the > smarts from Ceilomer into the Ironic code then the paste that I created > might be a guide to make it possible: > > http://paste.openstack.org/show/86071/ > > However on that however, if there's some chance that a large change could > happen, it might be better to wait, I don't know. > Just to give a sense of what we're dealing with, as while back I wrote a little script to dump the schema of all events StackTach collected from Nova. The value fields are replaced with types (or if it was a class object). http://paste.openstack.org/show/54140/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/15/2014 3:51 AM, Mark McLoughlin wrote:
> On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote:
>> On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote:
>>
>>> The data format that Ironic will send was part of the spec proposed
>>> and could have been reviewed. I think there's still time to change it
>>> tho, if you have a better format talk to Haomeng which is the guys
>>> responsible for that work in Ironic and see if he can change it (We
>>> can put up a following patch to fix the spec with the new format as
>>> well) . But we need to do this ASAP because we want to get it landed
>>> in Ironic soon.
>> It was only after doing the work that I realized how it might be an
>> example for the sake of this discussion. As the architecure of
>> Ceilometer currently exist there still needs to be some measure of
>> custom code, even if the notifications are as I described them.
>>
>> However, if we want to take this opportunity to move some of the
>> smarts from Ceilomer into the Ironic code then the paste that I created
>> might be a guide to make it possible:
>>
>> http://paste.openstack.org/show/86071/
> So you're proposing that all payloads should contain something like:
>
> 'events': [
> # one or more dicts with something like
> {
> # some kind of identifier for the type of event
> 'class': 'hardware.ipmi.temperature',
> 'type': '#thing that indicates threshold, discrete,
> cumulative',
> 'id': 'DIMM GH VR Temp (0x3b)',
> 'value': '26',
> 'unit': 'C',
> 'extra': {
> ...
> }
> }
>
> i.e. a class, type, id, value, unit and a space to put additional metadata.
This looks like a particular schema for one event-type (let's say
"foo.sample"). It's hard to extrapolate this one schema to a generic
set of common metadata applicable to all events. Really the only common
stuff we can agree on is the stuff already there: tenant, user, server,
message_id, request_id, timestamp, event_type, etc.
Side note on using notifications for sample data:
1. you should generate a proper notification when the rules of a sample
change (limits, alarms, sources, etc) ... but no actual measurements.
This would be something like a "ironic.alarm-rule-change" notification
or something
2. you should generate a minimal event for the actual samples "CPU-xxx:
70%" that relates to the previous rule-changing notification. And do
this on a queue something like "foo.sample".
This way, we can keep important notifications in a priority queue and
handle them accordingly (since they hold important data), but let the
samples get routed across less-reliable transports (like UDP) via the
RoutingNotifier.
Also, send the samples one-at-a-time and let them either a) drop on the
floor (udp) or b) let the aggregator roll them up into something smaller
(sliding window, etc). Making these large notifications contain a list
of samples means we had to store state somewhere on the server until
transmission time. Ideally something we wouldn't want to rely on.
> On the subject of "notifications as a contract", calling the additional
> metadata field 'extra' suggests to me that there are no stability
> promises being made about those fields. Was that intentional?
>
>> However on that however, if there's some chance that a large change could
>> happen, it might be better to wait, I don't know.
> Unlikely that a larger change will be made in Juno - take small window
> of opportunity to rationalize Ironic's payload IMHO.
>
> Mark.
___ OpenStack-dev mailing
list OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Thu, 2014-07-10 at 16:21 -0400, Eoghan Glynn wrote: > > > One of the issues that has been raised in the recent discussions with > > > the QA team about branchless Tempest relates to some legacy defects > > > in the OpenStack notification system. > > > > Got links to specifics? I thought the consensus was that there was a > > contract here which we need to maintain, so I'd be curious where that > > broke down. > > Well I could go digging in the LP fossil-record for specific bugs, but > it's late, so for now I'll simply appeal to anecdata and tribal memory > of ceilometer being broken by notification changes on the nova side. > > > Versioning and ability to newer contract versions would be good too, but > > in the absence of such things we should maintain backwards compat. > > Yes, I think that was the aspiration, but not always backed up by practice > in reality. The reason I ask about specifics is to figure out which is more important - versioned payloads, or automated testing of payload format. i.e. have we been accidentally or purposefully changing the format? If the latter, would the change have warranted a new incompatible version of the payload format? Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote:
> On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote:
>
> > The data format that Ironic will send was part of the spec proposed
> > and could have been reviewed. I think there's still time to change it
> > tho, if you have a better format talk to Haomeng which is the guys
> > responsible for that work in Ironic and see if he can change it (We
> > can put up a following patch to fix the spec with the new format as
> > well) . But we need to do this ASAP because we want to get it landed
> > in Ironic soon.
>
> It was only after doing the work that I realized how it might be an
> example for the sake of this discussion. As the architecure of
> Ceilometer currently exist there still needs to be some measure of
> custom code, even if the notifications are as I described them.
>
> However, if we want to take this opportunity to move some of the
> smarts from Ceilomer into the Ironic code then the paste that I created
> might be a guide to make it possible:
>
> http://paste.openstack.org/show/86071/
So you're proposing that all payloads should contain something like:
'events': [
# one or more dicts with something like
{
# some kind of identifier for the type of event
'class': 'hardware.ipmi.temperature',
'type': '#thing that indicates threshold, discrete, cumulative',
'id': 'DIMM GH VR Temp (0x3b)',
'value': '26',
'unit': 'C',
'extra': {
...
}
}
i.e. a class, type, id, value, unit and a space to put additional metadata.
On the subject of "notifications as a contract", calling the additional
metadata field 'extra' suggests to me that there are no stability
promises being made about those fields. Was that intentional?
> However on that however, if there's some chance that a large change could
> happen, it might be better to wait, I don't know.
Unlikely that a larger change will be made in Juno - take small window
of opportunity to rationalize Ironic's payload IMHO.
Mark.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Thu, Jul 10, 2014 at 1:48 AM, Eoghan Glynn wrote: > > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? > > Folks, > > One of the issues that has been raised in the recent discussions with > the QA team about branchless Tempest relates to some legacy defects > in the OpenStack notification system. > > Now, I don't personally subscribe to the PoV that ceilometer, or > indeed any other consumer of these notifications (e.g. StackTach), was > at fault for going ahead and depending on this pre-existing mechanism > without first fixing it. > > But be that as it may, we have a shortcoming here that needs to be > called out explicitly, and possible solutions explored. > > In many ways it's akin to the un-versioned RPC that existed in nova > before the versioned-rpc-apis BP[1] was landed back in Folsom IIRC, > except that notification consumers tend to be at arms-length from the > producer, and the effect of a notification is generally more advisory > than actionable. > > A great outcome would include some or all of the following: > > 1. more complete in-tree test coverage of notification logic on the > producer side > I't would be great if we can do this for the existing notifications, so that we don't regress on these while working on improving notifications. To that end, it would great if someone could work on a proof of concept on how we can do this testing, so we can sort out the details, so we can potentially start merging these tests Juno. > > 2. versioned notification payloads to protect consumers from breaking > changes in payload format > > 3. external discoverability of which event types a service is emitting > > 4. external discoverability of which event types a service is consuming > > If you're thinking that sounds like a substantial chunk of cross-project > work & co-ordination, you'd be right :) > > So the purpose of this thread is simply to get a read on the appetite > in the community for such an effort. At the least it would require: > > * trashing out the details in say a cross-project-track session at >the K* summit > > * buy-in from the producer-side projects (nova, glance, cinder etc.) >in terms of stepping up to make the changes > Yup, it would be great if we can finalize a plan for improved notifications at the K cycle. For that to happen it would be great to have a fairly concrete detailed proposal going into the summit. > > * acquiescence from non-integrated projects that currently consume >these notifications > >(we shouldn't, as good citizens, simply pull the rug out from under >projects such as StackTach without discussion upfront) > > * dunno if the TC would need to give their imprimatur to such an >approach, or whether we could simply self-organize and get it done >without the need for governance resolutions etc. > > Any opinions on how desirable or necessary this is, and how the > detailed mechanics might work, would be welcome. > > Apologies BTW if this has already been discussed and rejected as > unworkable. I see a stalled versioned-notifications BP[2] and some > references to the CADF versioning scheme in the LP fossil-record. > Also an inconclusive ML thread from 2012[3], and a related grizzly > summit design session[4], but it's unclear to me whether these > aspirations got much traction in the end. > > Cheers, > Eoghan > > [1] https://blueprints.launchpad.net/nova/+spec/versioned-rpc-apis > [2] https://blueprints.launchpad.net/nova/+spec/versioned-notifications > [3] http://osdir.com/ml/openstack/2012-10/msg3.html > [4] https://etherpad.openstack.org/p/grizzly-common-messaging > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> > So what we need to figure out is how exactly this common structure can be > > accommodated without reverting back to what Sandy called the "wild west" > > in another post. > > I got the impression that "wild west" is what we've already got > (within the payload)? Yeah, exactly, that was my interpretation too. So basically just to ensure that the lightweight-schema/common-structure notion doesn't land us back not too far beyond square one (if there are too many degrees-of-freedom in that declaration of "a list of dicts with certain required fields" that you had envisaged in an earlier post). > > For example you could write up a brief wiki walking through how an > > existing widely-consumed notification might look under your vision, > > say compute.instance.start.end. Then post a link back here as an RFC. > > > > Or, possibly better, maybe submit up a strawman spec proposal to one > > of the relevant *-specs repos and invite folks to review in the usual > > way? > > Would oslo-specs (as in messaging) be the right place for that? That's a good question. Another approach would be to hone in on the producer-side that's currently the heaviest user of notifications, i.e. nova, and propose the strawman to nova-specs given that (a) that's where much of the change will be needed, and (b) many of the notification patterns originated in nova and then were subsequently aped by other projects as they were spun up. > My thinking is the right thing to do is bounce around some questions > here (or perhaps in a new thread if this one has gone far enough off > track to have dropped people) and catch up on some loose ends. Absolutely! > For example: It appears that CADF was designed for this sort of thing and > was considered at some point in the past. It would be useful to know > more of that story if there are any pointers. > > My initial reaction is that CADF has the stank of enterprisey all over > it rather than "less is more" and "worse is better" but that's a > completely uninformed and thus unfair opinion. TBH I don't know enough about CADF, but I know a man who does ;) (gordc, I'm looking at you!) > Another question (from elsewhere in the thread) is if it is worth, in > the Ironic notifications, to try and cook up something generic or to > just carry on with what's being used. Well, my gut instinct is that the content of the Ironic notifications is perhaps on the outlier end of the spectrum compared to the more traditional notifications we see emitted by nova, cinder etc. So it may make better sense to concentrate initially on how contractizing these more established notifications might play out. > > This feels like something that we should be thinking about with an eye > > to the K* cycle - would you agree? > > Yup. > > Thanks for helping to tease this all out and provide some direction on > where to go next. Well thank *you* for picking up the baton on this and running with it :) Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Sat, 12 Jul 2014, Eoghan Glynn wrote: So what we need to figure out is how exactly this common structure can be accommodated without reverting back to what Sandy called the "wild west" in another post. I got the impression that "wild west" is what we've already got (within the payload)? For example you could write up a brief wiki walking through how an existing widely-consumed notification might look under your vision, say compute.instance.start.end. Then post a link back here as an RFC. Or, possibly better, maybe submit up a strawman spec proposal to one of the relevant *-specs repos and invite folks to review in the usual way? Would oslo-specs (as in messaging) be the right place for that? My thinking is the right thing to do is bounce around some questions here (or perhaps in a new thread if this one has gone far enough off track to have dropped people) and catch up on some loose ends. For example: It appears that CADF was designed for this sort of thing and was considered at some point in the past. It would be useful to know more of that story if there are any pointers. My initial reaction is that CADF has the stank of enterprisey all over it rather than "less is more" and "worse is better" but that's a completely uninformed and thus unfair opinion. Another question (from elsewhere in the thread) is if it is worth, in the Ironic notifications, to try and cook up something generic or to just carry on with what's being used. This feels like something that we should be thinking about with an eye to the K* cycle - would you agree? Yup. Thanks for helping to tease this all out and provide some direction on where to go next. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> > A notification of compute.instance.create.start is naturally going to > > carry different types of data than a volume.snapshot.delete.end for > > example, but of course we'd seek to accommodate that difference within > > a generic structure as far as possible. > > Is it going to carry different types of data or different values on > identical keys. The goal would be that the "meaning" of a notification > for compute.instance.create.start and volume.snapshot.delete.end > should be found on the same set of keys. That's all "the proposal" is. OK, got it. :) So what we need to figure out is how exactly this common structure can be accommodated without reverting back to what Sandy called the "wild west" in another post. (i.e. the current scenario of a small set of common fields such as priority or timestamp, followed by a free-form payload dict that differs from one event type to another, or may even change from the old form of one event type to the new form of the same notification) > >> It may be, they have the same basic idea: "stuff that > >> happens" can be represented by a sequence of relatively stupid event- > >> like things. > > > > OK, I might be missing something here, but we seem to have a close > > approximation of that already: > > > > stuff happens ==> events pop out on the bus > > > > So is your point that our events aren't dumb enough? > > Events are too different in from, from event to event. > > > (e.g. encode too much structure, or carry too much data, or require > > too much interpretation?) > > Of these three, "require too much interpretation" is perhaps the best > match. OK, I think I can agree with the spirit of that. > > But overall, it doesn't look like too radical a change in format. > > No, it's not. What it is is a minor change that "everyone" could adopt > and thus be able to play in the party. Agreed, so I think we just need to tie down the benefits of such a non- radical change, and compare those to the potential costs and benefits of more locked-down schemata (i.e. more specific to individual event types). > > The metric-oriented view is subset/specialized aspect of something > > more generic. > > Okay, so we agree on that. Can we agree that it might be possible to > represent that generic thing with a common grammar? Yes, I think we can :) > >> No, you are seeing the evolution of my thinking: as I write about this > >> stuff and try to understand it, it becomes more clear (to me) that > >> samples ought to be treated as event-like thingies. > > > > OK, so that shift makes it a little difficult to follow and engage with. > > This is a mailing list for a distributed project, it is the sole place > where we can get some reasonable simulation of "having a chat" with a > wide body of interested parties. The point of having a chat is to > explore, think, and exchange and hone ideas. I certainly hope you > don't expect me and everyone else only to write to the list when we > have a fully researched proposal. This (apart from summits and > mid-cycles) is the most accessible place we have to which we can go to > create the information that allows us to do the compare and contrast > the eventually leads to creating proposals. Fully agreed that the ML is exactly the place for discussing and honing ideas, outside of the design summit and mid-cycle meetups. And I agree that there is no requirement for these ideas to be fully formed before bringing them to the list. > > OK, so perhaps I misread your earlier comments on samples and assumed > > a predisposition to ceilometer's requirements was being proposed? > > It seems that way. Throughout I've been saying notifications should be > easy for anyone to create and anyone to consume and have confidence > they are doing it right. Yep ... after discussing this on IRC, it seems that the confusion arose from me reading too much into your prior statement of: "to me ceilometer is a service for gathering metrics and then being able to do things with those metrics. anything with the proper creds ought to be able to send out a notification which is to be treated as a metric and ceilometer should consume it" i.e. assuming this to indicate a stronger equivalence between notifications and metrics/samples than you had intended. > > Can you start fleshing out exactly what you mean by a standard not > > necessarily predisposed to ceilometer, still sufficiently close to > > eliminate the need for a mapping layer on the ceilometer side? > > That's what I'm trying to do here in this discussion here on this list > because as far as I'm able to tell that's the one way to ensure that > the idea has some validity and I'm not missing important details. I > thought I was saying "how about this general idea, any merit? Help me > flesh it out?" but apparently it came across as something different. Apologies, I should have wrote: can you *continue* fleshing out ... Poor choice of words on my part. > That's a shame. If I we
Re: [openstack-dev] [all] Treating notifications as a contract
On Fri, 11 Jul 2014, Eoghan Glynn wrote: A notification of compute.instance.create.start is naturally going to carry different types of data than a volume.snapshot.delete.end for example, but of course we'd seek to accommodate that difference within a generic structure as far as possible. Is it going to carry different types of data or different values on identical keys. The goal would be that the "meaning" of a notification for compute.instance.create.start and volume.snapshot.delete.end should be found on the same set of keys. That's all "the proposal" is. Elsewhere there's been some discussion of ActivityStreams as a possible model. Is that the social media syndication format you're talking about? That's been discussed elsewhere as possible model for openstack notifications? I missed that, can you provide a link? I'm making something of a leap from http://lists.openstack.org/pipermail/openstack-dev/2014-July/039965.html which links to https://wiki.openstack.org/wiki/NotificationSystem which mentions PUSH, which, in the time since that NotificationSystem document was written has evolved to be the transport for ActivityStreams, where activities are events. It may be, they have the same basic idea: "stuff that happens" can be represented by a sequence of relatively stupid event- like things. OK, I might be missing something here, but we seem to have a close approximation of that already: stuff happens ==> events pop out on the bus So is your point that our events aren't dumb enough? Events are too different in from, from event to event. (e.g. encode too much structure, or carry too much data, or require too much interpretation?) Of these three, "require too much interpretation" is perhaps the best match. The intial versions of ActivityStreams was based on atom+xml but later people realized "this would be a ton easier if it was some well known and well formed JSON". Are you suggesting here we adopt atom+json? No, I'm just charting the path from a complex over-schematized format (atom+xml) through to one which is easier to consume (atom+json) to one which is even easier to consume (a well known dictionary). Has the notification lost any expressive power? Not much by the looks of it. Is it a good idea to drop for example the error range on the floor because ceilometer doesn't know what to do with it? The error range isn't being dropped: it would be in the extras because it is of its own special meaning. If there's agreement that "error_range" is a useful general key, then that would be a part of the standard somewhere. Dunno, that would depend on whether (a) the delta value reported by IPMI is actually meaningful and (b) some other consumer can do something smart with it. As I worried before, this concrete example seems to have diverted you to focussing on the details of what is a complete strawman made in a few minutes. Whether the delta value is dropped is an unknown at this time. It might be that it could make sense for the producer to emit multiple events where it now emits one. I don't really think that's that important to the discussion, right now. I don't know enough about IPMI to comment on (a). My instinct would be to leave the door open for (b). I don't reckon the door is being shut in any way. But overall, it doesn't look like too radical a change in format. No, it's not. What it is is a minor change that "everyone" could adopt and thus be able to play in the party. The metric-oriented view is subset/specialized aspect of something more generic. Okay, so we agree on that. Can we agree that it might be possible to represent that generic thing with a common grammar? No, you are seeing the evolution of my thinking: as I write about this stuff and try to understand it, it becomes more clear (to me) that samples ought to be treated as event-like thingies. OK, so that shift makes it a little difficult to follow and engage with. This is a mailing list for a distributed project, it is the sole place where we can get some reasonable simulation of "having a chat" with a wide body of interested parties. The point of having a chat is to explore, think, and exchange and hone ideas. I certainly hope you don't expect me and everyone else only to write to the list when we have a fully researched proposal. This (apart from summits and mid-cycles) is the most accessible place we have to which we can go to create the information that allows us to do the compare and contrast the eventually leads to creating proposals. OK, so perhaps I misread your earlier comments on samples and assumed a predisposition to ceilometer's requirements was being proposed? It seems that way. Throughout I've been saying notifications should be easy for anyone to create and anyone to consume and have confidence they are doing it right. Can you start fleshing out exactly what you mean by a standard not necessarily predisposed to ceilometer, still sufficiently close to el
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/11/2014 6:04 AM, Chris Dent wrote: > On Fri, 11 Jul 2014, Eoghan Glynn wrote: > >> But I guess you're suggesting not only that we version/schematize >> individual notification payloads, but that we do so in a way that's >> global across event types and emitters? > That's partially correct. I'm suggesting the we consider standardizing a > general format for notification payloads across emitters. _Not_ > individual notification payloads. > > Schematizing individual payloads strikes me as heavyweight[1] when it > _might_[2] be easier to declare that "a list of dicts with certain > required fields" is sufficient to represent a multiplicity of value- > oriented events which are consumed as any of metric, event, > visualization data-point, etc. This is the model of notifications [1] (see elsewhere in this thread) The payload is the wild-west and loosely spec'ed on the wiki. Above the payload we have strict fields. [1] https://wiki.openstack.org/wiki/NotificationSystem ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/10/2014 12:10 PM, Chris Dent wrote: > On Thu, 10 Jul 2014, Julien Danjou wrote: > >> My initial plan was to leverage a library like voluptuous to do schema >> based validation on the sender side. That would allow for receiver to >> introspect schema and know the data structure to expect. I didn't think >> deeply on how to handle versioning, but that should be doable too. > It's not clear to me in this discussion what it is that is being > versioned, contracted or standardized. > > Is it each of the many different notifications that various services > produce now? > > Is it the general concept of a notification which can be considered > a "sample" that something like Ceilometer or StackTack might like to > consume? > The only real differences between a sample and an event are: 1. the size of the context. "Host X CPU = 70%" tells you nearly everything you need to know. But "compute.scheduler.host_selected" will require lots of information to tell you why and how "host X" was selected. The event payload should be atomic and not depend on previous events for context. With samples, the context is sort of implied by the key or queue name. 2. The handling of Samples can be sloppy. If you miss a CPU sample, just wait for the next one. But if you drop an Event, a billing report is going to be wrong or a dependent system loses sync. 3. There are a *lot* more samples emitted than events. Samples are a shotgun blast while events are registered mail. This is why samples don't usually have the schema problems of events. They are so tiny, there's not much to change. Putting a lot of metadata in a sample is generally a bad idea. Leave it to the queue or key name. That said, Monasca is doing some really cool stuff with high-speed sample processing such that the likelihood of dropping a sample is so low that event support should be able to come from the same framework. The difference is simply the size of the payload and if the system can handle it at volume (quickly and reliably). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> > But I guess you're suggesting not only that we version/schematize > > individual notification payloads, but that we do so in a way that's > > global across event types and emitters? > > That's partially correct. I'm suggesting the we consider standardizing a > general format for notification payloads across emitters. _Not_ > individual notification payloads. So I'm open to correction, but I don't think anyone suggested using egregiously different formats for different notifications. A notification of compute.instance.create.start is naturally going to carry different types of data than a volume.snapshot.delete.end for example, but of course we'd seek to accommodate that difference within a generic structure as far as possible. > Schematizing individual payloads strikes me as heavyweight[1] when it > _might_[2] be easier to declare that "a list of dicts with certain > required fields" is sufficient to represent a multiplicity of value- > oriented events which are consumed as any of metric, event, > visualization data-point, etc. OK, so lightweight schemas are lighter in weight that heavyweight schemas. Yep, agreed. > Elsewhere there's been some discussion of ActivityStreams as a > possible model. Is that the social media syndication format you're talking about? That's been discussed elsewhere as possible model for openstack notifications? I missed that, can you provide a link? > It may be, they have the same basic idea: "stuff that > happens" can be represented by a sequence of relatively stupid event- > like things. OK, I might be missing something here, but we seem to have a close approximation of that already: stuff happens ==> events pop out on the bus So is your point that our events aren't dumb enough? (e.g. encode too much structure, or carry too much data, or require too much interpretation?) > The intial versions of ActivityStreams was based on atom+xml but later > people realized "this would be a ton easier if it was some well known > and well formed JSON". Are you suggesting here we adopt atom+json? > > So for the purposes of grounding the discussion, can you give an > > example of what the Ironic notification payload might look like in > > a perfect world? > > I can't give you a perfect example because I haven't had an > opportunity to understand all the needs and issues but a rough quickie > to try to illustrate the point is here: > > http://paste.openstack.org/show/86071/ > > The main difference there is that the value and the unit are provided > in known fields at a known descent into the data structure, rather than > needing to be extracted from strings that are values at custom keys. OK, so in this case, can we sum it up with: * provide an ID field * parse out the Sensor Reading into value and units, dropping the delta * push the concept of cumulative versus delta versus gauge into the producer * dump the stuff that ceilometer is not so interested into an 'extras; dict So, cool, we've made our job a little easier on the ceilometer side. Has the notification lost any expressive power? Not much by the looks of it. Is it a good idea to drop for example the error range on the floor because ceilometer doesn't know what to do with it? Dunno, that would depend on whether (a) the delta value reported by IPMI is actually meaningful and (b) some other consumer can do something smart with it. I don't know enough about IPMI to comment on (a). My instinct would be to leave the door open for (b). But overall, it doesn't look like too radical a change in format. > I'd rather we not get hung up on the details of the representation as > that's the way rabbits go. If the concept has merit the representation > can follow. > > >> Yes, this moves the onus of creating well-formed metrics to notifying > >> systems, but that is good: It is providing exactly the sort of clean > >> and easy to test contract at the boundaries that leads to good > >> neighbors and easy testing. > > > > OK, so the thing to note is that the set of potential consumers is > > not limited to services with a metrics-oriented view of the world. > > I'm of the opinion that it _may_ be possible that "metrics-oriented" > is a subclass of something more generic that might be representable. > The CADF folk seem to think so (although I'm not suggesting we go down > that road, c.f. "heavyweight"). And again, unless I'm missing something, that's pretty much what I was saying. Re-using your phraseology: "creating well-formed metrics" is not necessarily what we want the producer to do, because "metrics-oriented is a subclass of something more generic" > > * services with a metrics-oriented PoV (e.g. Ceilometer) > > * services with an events-oriented PoV (e.g. StackTach) > > * services with a UX-oriented PoV (e.g. Horizon) > > * services with an entitlement-enforcement PoV (e.g. Katello) > > These all sound like there are in at least the same ballbark. I think > the common term w
Re: [openstack-dev] [all] Treating notifications as a contract
On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if you have a better format talk to Haomeng which is the guys responsible for that work in Ironic and see if he can change it (We can put up a following patch to fix the spec with the new format as well) . But we need to do this ASAP because we want to get it landed in Ironic soon. It was only after doing the work that I realized how it might be an example for the sake of this discussion. As the architecure of Ceilometer currently exist there still needs to be some measure of custom code, even if the notifications are as I described them. However, if we want to take this opportunity to move some of the smarts from Ceilomer into the Ironic code then the paste that I created might be a guide to make it possible: http://paste.openstack.org/show/86071/ However on that however, if there's some chance that a large change could happen, it might be better to wait, I don't know. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Fri, 11 Jul 2014, Eoghan Glynn wrote: But I guess you're suggesting not only that we version/schematize individual notification payloads, but that we do so in a way that's global across event types and emitters? That's partially correct. I'm suggesting the we consider standardizing a general format for notification payloads across emitters. _Not_ individual notification payloads. Schematizing individual payloads strikes me as heavyweight[1] when it _might_[2] be easier to declare that "a list of dicts with certain required fields" is sufficient to represent a multiplicity of value- oriented events which are consumed as any of metric, event, visualization data-point, etc. Elsewhere there's been some discussion of ActivityStreams as a possible model. It may be, they have the same basic idea: "stuff that happens" can be represented by a sequence of relatively stupid event- like things. The intial versions of ActivityStreams was based on atom+xml but later people realized "this would be a ton easier if it was some well known and well formed JSON". Take that line of evolution a little bit further and you've got "a list of dicts with certain required fields". (more below) [1] A useful turn of phrase learned in this message: http://lists.openstack.org/pipermail/openstack-dev/2014-July/039941.html [2] Emphasis on the might, meaning might be worthy of consideration at least as a strawman to help illuminate concerns and benefits. So for the purposes of grounding the discussion, can you give an example of what the Ironic notification payload might look like in a perfect world? I can't give you a perfect example because I haven't had an opportunity to understand all the needs and issues but a rough quickie to try to illustrate the point is here: http://paste.openstack.org/show/86071/ The main difference there is that the value and the unit are provided in known fields at a known descent into the data structure, rather than needing to be extracted from strings that are values at custom keys. I'd rather we not get hung up on the details of the representation as that's the way rabbits go. If the concept has merit the representation can follow. Yes, this moves the onus of creating well-formed metrics to notifying systems, but that is good: It is providing exactly the sort of clean and easy to test contract at the boundaries that leads to good neighbors and easy testing. OK, so the thing to note is that the set of potential consumers is not limited to services with a metrics-oriented view of the world. I'm of the opinion that it _may_ be possible that "metrics-oriented" is a subclass of something more generic that might be representable. The CADF folk seem to think so (although I'm not suggesting we go down that road, c.f. "heavyweight"). * services with a metrics-oriented PoV (e.g. Ceilometer) * services with an events-oriented PoV (e.g. StackTach) * services with a UX-oriented PoV (e.g. Horizon) * services with an entitlement-enforcement PoV (e.g. Katello) These all sound like there are in at least the same ballbark. I think the common term would be "events" and a metric is a type of event? But in an case, I'm bit confused by where this line of reasoning is headed. We seem to have gone from the notifying service directly emitting well-formed samples that ceilometer can just directly persist, to more generic "event-like thingies". Have I misread that point about *directly* emitting samples? No, you are seeing the evolution of my thinking: as I write about this stuff and try to understand it, it becomes more clear (to me) that samples ought to be treated as event-like thingies. Having a standard notification payload format would of course mean change, but we know that flexible metering/auditing is very important for the OpenStack universe. Your argument seems to be that having such a standard, predisposed to ceilometer, would limit flexibility and lose capability. Yep, that is exactly my point. Predisposition of the notification format to ceilometer's needs is what concerns me. As opposed to the notion of standardization/schematization/versioning which is the explicit goal of the discussion. Okay, and my proposal is not to have a standard predisposed to ceilometer. It is to make a limited standard for a large class of notifications and _not_ schematize individual notification types (more on this aspect below) and then change ceilometer so it can use the plan. Take my example from above, the processing of Ironic notifications. I think it is weird that I had to write code for that. Does it not seem odd to you? OK, so have we strayed into an orthogonal concern here? I don't think so, it was supposed to be an example of the cost of there not being an existing general standard. If there were such a standard I wouldn't have had to write any code, only the Ironic folk would and I would have had the free time to help them. Less code == good! Similarly if there are individual sc
Re: [openstack-dev] [all] Treating notifications as a contract
> I just started the code for processing of notifications from Ironic. > Conceptually they are the same as notifications from Nova but the > actual form of the payload is completely different. This means I have to > write a different processor for that payload. And now so does StackTach > if they want to handle it. The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if you have a better format talk to Haomeng which is the guys responsible for that work in Ironic and see if he can change it (We can put up a following patch to fix the spec with the new format as well) . But we need to do this ASAP because we want to get it landed in Ironic soon. Cheers, Lucas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> tl;dr: Having a minimum payload standard enables more and more robust > services on both sides of notification bus. Yes, that's exactly the point of this thread. > We do not have a standard format for the payload. I think we should > (more on that below). Again, such standardization is exactly the point of this thread. But I guess you're suggesting not only that we version/schematize individual notification payloads, but that we do so in a way that's global across event types and emitters? > > Well, the notifications emitted are intended to be general-purpose > > in nature. > > The general purpose is that they are notifications of some info, > somewhat like described above. Well, yeah. > If the payload had a guaranteed base form with known structure and keys > then consumption becomes easier. The structure doesn't have to limit > what _can_ be there, it should describe what must be there to be > adequate: if there are multiple samples, how is that sequence > represented (a list); in each sample what are the required fields (name, > time, unit, volume to guess at a few); in the parent framing what are > the required fields (e.g source, type). > > I just started the code for processing of notifications from Ironic. > Conceptually they are the same as notifications from Nova but the > actual form of the payload is completely different. This means I have to > write a different processor for that payload. And now so does StackTach > if they want to handle it. So for the purposes of grounding the discussion, can you give an example of what the Ironic notification payload might look like in a perfect world? (Just a link to a paste of a hand-crafted example would suffice) > Yes, this moves the onus of creating well-formed metrics to notifying > systems, but that is good: It is providing exactly the sort of clean > and easy to test contract at the boundaries that leads to good > neighbors and easy testing. OK, so the thing to note is that the set of potential consumers is not limited to services with a metrics-oriented view of the world. So for example it's been proposed, or is actually a reality, that at least the following different types of beasts are interested in consuming notifications: * services with a metrics-oriented PoV (e.g. Ceilometer) * services with an events-oriented PoV (e.g. StackTach) * services with a UX-oriented PoV (e.g. Horizon) * services with an entitlement-enforcement PoV (e.g. Katello) I'm not sure that "moving the onus of creating well-formed metrics to notifying systems" is feasible, when many of the actual and potential consumers of these data don't actually have a strictly metrics-oriented perspective. > > Problem is, the requirements of Ceilometer, StackTack, and indeed > > NewShinyMeteringShiz, are probably gonna be different. > > Is that really a problem or do we fear it is? Don't they all want a > thing that has the same basic form and info (an event-like thingie > with a value). Well I think it is a real problem, given the breath of consumers as described above. But in an case, I'm bit confused by where this line of reasoning is headed. We seem to have gone from the notifying service directly emitting well-formed samples that ceilometer can just directly persist, to more generic "event-like thingies". Have I misread that point about *directly* emitting samples? > Having a standard notification payload format would of course mean change, > but we know that flexible metering/auditing is very important for the > OpenStack universe. Your argument seems to be that having such a > standard, predisposed to ceilometer, would limit flexibility and lose > capability. Yep, that is exactly my point. Predisposition of the notification format to ceilometer's needs is what concerns me. As opposed to the notion of standardization/schematization/versioning which is the explicit goal of the discussion. > Take my example from above, the processing of Ironic notifications. > I think it is weird that I had to write code for that. Does it not > seem odd to you? OK, so have we strayed into an orthogonal concern here? i.e. not that ceilometer requires some translation to be done, but that this translation must be hand-craft in Python code as opposed to being driven declaratively via some configured mapping rules? Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
tl;dr: Having a minimum payload standard enables more and more robust services on both sides of notification bus. Expanded earnest optimism below. On Thu, 10 Jul 2014, Eoghan Glynn wrote: So it's the standardization of notification payload formats that's being proposed. Generally that payload can be represented as a sequence of dictionaries of key value pairs that, in the Ceilometer context, is translated into a Sample. In a general sense is an event with an actor, a volume, a unit, and some associated metadata. As clarified elsewhere in the thread we already, in a sort of squishy way, have a standard format for the envelope of a notification. We do not have a standard format for the payload. I think we should (more on that below). Well, the notifications emitted are intended to be general-purpose in nature. The general purpose is that they are notifications of some info, somewhat like described above. Ceilometer, StackTack OTOH put those notifications to a specific (and slightly different) purpose. Having the services emit exactly the sample format that ceilometer desired would save ceilometer itself doing the transformation, but would make the notifications less useful to other consumers. I'm afraid I'll have to respectfully disagree on "less useful" and in the spirit of good discussion present my case a bit more fully: Having a large body of sevices emit a _known_ and well described sample format (i.e. a "protocol") will make it possible for _many_ services, including but not limited to Ceilometer and StackTach, to consume those samples. Enabling many services to publish samples notifications and many other services to consume those samples (especially those services and samples we haven't thought of or even imagined yet) is exactly why I'm arguing this point. If the payload had a guaranteed base form with known structure and keys then consumption becomes easier. The structure doesn't have to limit what _can_ be there, it should describe what must be there to be adequate: if there are multiple samples, how is that sequence represented (a list); in each sample what are the required fields (name, time, unit, volume to guess at a few); in the parent framing what are the required fields (e.g source, type). I just started the code for processing of notifications from Ironic. Conceptually they are the same as notifications from Nova but the actual form of the payload is completely different. This means I have to write a different processor for that payload. And now so does StackTach if they want to handle it. That seems completely ridiculous when all that is happening with the data is that it is getting a bit sliced and diced and stored. If I had a GeneralNotificationProcessor in Ceilo, all I would need to do is, in config, say that I wanted to start accepting sample notifications of the type that Ironic produces and wham, they're in there. StackTach could do much the same (plus whatever else extra they like). Yes, this moves the onus of creating well-formed metrics to notifying systems, but that is good: It is providing exactly the sort of clean and easy to test contract at the boundaries that leads to good neighbors and easy testing. Wouldn't the metering process be a lot easier if there was a standardized package for a "sample" and anyone with the proper credentials could drop a sample on the bus with the right exchange with the right topic and anything (e.g. Ceilometer, StackTack, the NewShinyMeteringShiz) that wants to consider itself a metering store can consume it and hey presto. Problem is, the requirements of Ceilometer, StackTack, and indeed NewShinyMeteringShiz, are probably gonna be different. Is that really a problem or do we fear it is? Don't they all want a thing that has the same basic form and info (an event-like thingie with a value). If people are going to have to write a bunch of new tests and related code to get notifications healthier why not make notifications for metrics _healthy_ and available to any system without needing to write a bunch of code on both sides of the bus. The intent was loose-coupling between the notifier and the notified. Which intent? The current code certainly does not represent my understanding of loose-coupling. It is, however, often a very overloaded term so we may be meaning different things. These notifications pre-exist ceilometer, where never intended for ceilometer's exclusive use, and are in current use by frameworks other than ceilometer, each with differing requirements. Yes, I understand most of that. The part I'm not convinced on is the "differing requirements". If there are standard "events" then lots of different (de-coupled) things can handle them, whatever their requirements might be. If I'm missing something critical about these differing requirements, please enlighten me. Having a standard notification payload format would of course mean change, but we know that flexible metering/auditing is very importa
Re: [openstack-dev] [all] Treating notifications as a contract
-Original Message- From: Eoghan Glynn Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Thursday, July 10, 2014 at 3:30 PM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [all] Treating notifications as a contract > > >> The original implementation of the notification system was never >>intended >> to be the *final* implementation. I think we all identified the need for >> versioning several years ago. As for backwards compatibility, I think >>the >> version field itself, in whatever form it takes, should be optional. If >> not provided, it can be assumed the body of the notification is >>freeform. > >Fair point. > >> Adding fields is easy. Removal of fields should only occur after we can >> comfortably say that all services are only consuming the new field and >> we¹ve left sufficient time for deprecating the old ones. > >Put on the deprecation path, then eventually bump the version when the old >field finally goes away, right? Yessir :-) > >> In some capacity, I think PubSubHubBub has a lot to offer here. The >> publisher has to be able to provide a list of feeds it¹s authoritative >> for, which I think is sufficient for discovery. I think PSHB itself is >>bit >> heavy weight for what we need (and has some scaling problems) but I >>think >> the concept itself is very useful for us. > >Note to self: learn more about PSHB. > >> >> 4. external discoverability of which event types a service is >>consuming >> > >> >Isn't this what the topic queues are for? Consumers should only >> >subscribe to the topics they're interested in >> >> I¹m not sure I understand the value of this one. A service consumes what >> it consumes. If we¹ve versioned correctly, as mentioned above, why does >>it >> matter? > >Yes, as I said in other responses on this thread, the potential usefulness >of that is rapidly receding in my mind. Sorry, I’ve been away and hadn’t read the entire thread yet. Didn’t mean to rehash an existing point. > >Cheers, >Eoghan > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> The original implementation of the notification system was never intended > to be the *final* implementation. I think we all identified the need for > versioning several years ago. As for backwards compatibility, I think the > version field itself, in whatever form it takes, should be optional. If > not provided, it can be assumed the body of the notification is freeform. Fair point. > Adding fields is easy. Removal of fields should only occur after we can > comfortably say that all services are only consuming the new field and > we¹ve left sufficient time for deprecating the old ones. Put on the deprecation path, then eventually bump the version when the old field finally goes away, right? > In some capacity, I think PubSubHubBub has a lot to offer here. The > publisher has to be able to provide a list of feeds it¹s authoritative > for, which I think is sufficient for discovery. I think PSHB itself is bit > heavy weight for what we need (and has some scaling problems) but I think > the concept itself is very useful for us. Note to self: learn more about PSHB. > >> 4. external discoverability of which event types a service is consuming > > > >Isn't this what the topic queues are for? Consumers should only > >subscribe to the topics they're interested in > > I¹m not sure I understand the value of this one. A service consumes what > it consumes. If we¹ve versioned correctly, as mentioned above, why does it > matter? Yes, as I said in other responses on this thread, the potential usefulness of that is rapidly receding in my mind. Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> From my perspective, the requirement is to be able to have a consistent and > predictable format for notifications that are being sent from all services. > This means: > 1. a set of required fields that all events contain and have consistent > meaning > 2. a set of optional fields, you don’t have to include these but if you do > then you follow the same format and meaning > 3. versioning of events: version is updated whenever the required fields are > changed. managing optional fields can be done via a specification Yes, that pretty much sums it up. > Discovery of events would be interesting from an automated testing > perspective, but I am not sure how effective this would be for an > application actually consuming the event.s Yes, the motivating usecase was discoverability for testing. > Not sure how you would use enumerating the consumption of events Ditto, though on reflection I'm thinking discoverablity of the more downstream thing (the gathering of certain meters in ceilometer's case) would be far more meaningful to reason over. Cheers, Eoghan > > > On Jul 10, 2014, at 2:48 AM, Eoghan Glynn wrote: > > > > > TL;DR: do we need to stabilize notifications behind a versioned > > and discoverable contract? > > > > Folks, > > > > One of the issues that has been raised in the recent discussions with > > the QA team about branchless Tempest relates to some legacy defects > > in the OpenStack notification system. > > > > Now, I don't personally subscribe to the PoV that ceilometer, or > > indeed any other consumer of these notifications (e.g. StackTach), was > > at fault for going ahead and depending on this pre-existing mechanism > > without first fixing it. > > > > But be that as it may, we have a shortcoming here that needs to be > > called out explicitly, and possible solutions explored. > > > > In many ways it's akin to the un-versioned RPC that existed in nova > > before the versioned-rpc-apis BP[1] was landed back in Folsom IIRC, > > except that notification consumers tend to be at arms-length from the > > producer, and the effect of a notification is generally more advisory > > than actionable. > > > > A great outcome would include some or all of the following: > > > > 1. more complete in-tree test coverage of notification logic on the > >producer side > > > > 2. versioned notification payloads to protect consumers from breaking > >changes in payload format > > > > 3. external discoverability of which event types a service is emitting > > > > 4. external discoverability of which event types a service is consuming > > > > If you're thinking that sounds like a substantial chunk of cross-project > > work & co-ordination, you'd be right :) > > > > So the purpose of this thread is simply to get a read on the appetite > > in the community for such an effort. At the least it would require: > > > > * trashing out the details in say a cross-project-track session at > > the K* summit > > > > * buy-in from the producer-side projects (nova, glance, cinder etc.) > > in terms of stepping up to make the changes > > > > * acquiescence from non-integrated projects that currently consume > > these notifications > > > > (we shouldn't, as good citizens, simply pull the rug out from under > > projects such as StackTach without discussion upfront) > > > > * dunno if the TC would need to give their imprimatur to such an > > approach, or whether we could simply self-organize and get it done > > without the need for governance resolutions etc. > > > > Any opinions on how desirable or necessary this is, and how the > > detailed mechanics might work, would be welcome. > > > > Apologies BTW if this has already been discussed and rejected as > > unworkable. I see a stalled versioned-notifications BP[2] and some > > references to the CADF versioning scheme in the LP fossil-record. > > Also an inconclusive ML thread from 2012[3], and a related grizzly > > summit design session[4], but it's unclear to me whether these > > aspirations got much traction in the end. > > > > Cheers, > > Eoghan > > > > [1] https://blueprints.launchpad.net/nova/+spec/versioned-rpc-apis > > [2] https://blueprints.launchpad.net/nova/+spec/versioned-notifications > > [3] http://osdir.com/ml/openstack/2012-10/msg3.html > > [4] https://etherpad.openstack.org/p/grizzly-common-messaging > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> > One of the issues that has been raised in the recent discussions with > > the QA team about branchless Tempest relates to some legacy defects > > in the OpenStack notification system. > > Got links to specifics? I thought the consensus was that there was a > contract here which we need to maintain, so I'd be curious where that > broke down. Well I could go digging in the LP fossil-record for specific bugs, but it's late, so for now I'll simply appeal to anecdata and tribal memory of ceilometer being broken by notification changes on the nova side. > Versioning and ability to newer contract versions would be good too, but > in the absence of such things we should maintain backwards compat. Yes, I think that was the aspiration, but not always backed up by practice in reality. Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> It's not clear to me in this discussion what it is that is being > versioned, contracted or standardized. Well the statement in the original mail was pretty explicit: "versioned notification payloads to protect consumers from breaking changes in payload format" So it's the standardization of notification payload formats that's being proposed. > Is it each of the many different notifications that various services > produce now? Its the event types emitted and the version of each payload. > Is it the general concept of a notification which can be considered > a "sample" that something like Ceilometer or StackTack might like to > consume? Well, the notifications emitted are intended to be general-purpose in nature. Ceilometer, StackTack OTOH put those notifications to a specific (and slightly different) purpose. Having the services emit exactly the sample format that ceilometer desired would save ceilometer itself doing the transformation, but would make the notifications less useful to other consumers. > If it not the latter, why isn't it the latter? For the reason state above. > Here's some semi-random > noodling: > > Wouldn't the metering process be a lot easier if there was a > standardized package for a "sample" and anyone with the proper > credentials could drop a sample on the bus with the right exchange with > the right topic and anything (e.g. Ceilometer, StackTack, the > NewShinyMeteringShiz) that wants to consider itself a metering store can > consume it and hey presto. Problem is, the requirements of Ceilometer, StackTack, and indeed NewShinyMeteringShiz, are probably gonna be different. > If people are going to have to write a bunch of new tests and > related code to get notifications healthier why not make > notifications for metrics _healthy_ and available to any system > without needing to write a bunch of code on both sides of the bus. The intent was loose-coupling between the notifier and the notified. These notifications pre-exist ceilometer, where never intended for ceilometer's exclusive use, and are in current use by frameworks other than ceilometer, each with differing requirements. > Currently Ceilometer is required to know far too much about the > notifications it receives and that knowledge is being represented is > code. That is a BadThing™. I'm sure there are plenty of reasons for > why it has turned out that way, but if there is an opportunity for > change...? Not really, I don't think, TBH. Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> Glad to see this coming up again. I've been pushing that since the > Icehouse summit in Hong-Kong, where we had a Oslo session about that > subject and where everybody nod about my initial plan. :) > > I've created a blueprint and started a very primitive implementation > back then: > > https://blueprints.launchpad.net/oslo.messaging/+spec/notification-structured > > but had to stop by lacking time to push that effort further. > > My initial plan was to leverage a library like voluptuous to do schema > based validation on the sender side. That would allow for receiver to > introspect schema and know the data structure to expect. I didn't think > deeply on how to handle versioning, but that should be doable too. > > To summarize, I got the feeling that there's more a need of resources to > work on that than any problem. Same old, same old. Yes, this is something that would require some effort across a range of producer-side and consumer-side projects, so it would be completely dependent on the fine aspirations being backed up by actual resources. Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
> Thanks for dusting this off. Versioning and published schemas for > notifications are important to the StackTach team. It would be nice to > get this resolved. We're happy to help out. Great! > > A great outcome would include some or all of the following: > > > > 1. more complete in-tree test coverage of notification logic on the > > producer side > > Ultimately this is the core problem. A breaking change in the > notifications caused tests to fail in other systems. Should we be adding > more tests or simply add version checking at the lower levels (like the > first pass of RPC versioning did)? Yes, that would definitely help. > > 2. versioned notification payloads to protect consumers from breaking > > changes in payload format > Yep, like RPC the biggies are: > 1. removal of fields from notifications > 2. change in semantics of a particular field > 3. addition of new fields (not a biggie) True, true and true. > The urgency for notifications is a little different than RPC where there > is a method on the other end expecting a certain format. Notifications > consumers have to be a little more forgiving when things don't come in > as expected. Yes some tolerance can be expected, but whole point would be get away from the current free-fir-all. > This isn't a justification for breaking changes. Just stating that we > have some leeway. Sure. > I guess it really comes down to systems that are using notifications for > critical synchronization vs. purely informational. Agreed. That's what I was getting at with the advisory versus actionable distinction. > > 3. external discoverability of which event types a service is emitting > These questions can be saved for later, but ... > > Is the use-case that a downstream system can learn which queue to > subscribe to programmatically? > > Is this a nice-to-have? > > Would / should this belong in a metadata service? Possibly. The idea was to facilitate the drive towards discoverability for testing purposes. > > 4. external discoverability of which event types a service is consuming > > Isn't this what the topic queues are for? Consumers should only > subscribe to the topics they're interested in. Yeah, that one is questionable all right. Again it relates back to the motivating discussion with QA around branchless Tempest. The idea was to make it externally discoverable which notifications that ceilometer was currently capable of and configured to consume. But on reflection it would be far more meaningful to surface which meters ceilo is currently capable of and configured to *produce*. > > If you're thinking that sounds like a substantial chunk of cross-project > > work & co-ordination, you'd be right :) > > Perhaps notification schemas should be broken out into a separate > repo(s)? That way we can test independent of the publishing system. For > example, our notigen event simulator [5] could use it. > > These could just be dependent libraries/plugins to oslo.messaging. That is an interesting idea, it would allow for the schemas to be easily depended upon on the consumer side also for validation purposes. > > * acquiescence from non-integrated projects that currently consume > >these notifications > > > >(we shouldn't, as good citizens, simply pull the rug out from under > >projects such as StackTach without discussion upfront) > We'll adapt StackTach.v2 accordingly. StackTach.v3 is far less impacted > by notification changes since they are offloaded and processed in a > secondary step. Breaking changes will just stall the processing. I > suspect .v3 will be in place before .v2 is affected. > > Adding version handling to Stack-Distiller (our notification->event > translator) should be pretty easy (and useful) [6] Cool, sounds like this would not be disruptive to StackTach. > > Any opinions on how desirable or necessary this is, and how the > > detailed mechanics might work, would be welcome. > > A published set of schemas would be very useful for StackTach, we'd love > to help out in any way possible. In the near-term we have to press on > under the assumption notification definitions are fragile. Yes, the time horizon for this would be beyond the near-time. > > Apologies BTW if this has already been discussed and rejected as > > unworkable. I see a stalled versioned-notifications BP[2] and some > > references to the CADF versioning scheme in the LP fossil-record. > > Also an inconclusive ML thread from 2012[3], and a related grizzly > > summit design session[4], but it's unclear to me whether these > > aspirations got much traction in the end. > > I'd really like to see the CADF work adopted vs. other, hand-rolled, > schema solutions. Yes, I need to learn more about CADF, but the general principle of avoiding bicycle-reinvention applies. Cheers, Eoghan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/m
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/10/2014 2:59 PM, Daniel Dyer wrote: > From my perspective, the requirement is to be able to have a consistent and > predictable format for notifications that are being sent from all services. > This means: > 1. a set of required fields that all events contain and have consistent > meaning > 2. a set of optional fields, you don’t have to include these but if you do > then you follow the same format and meaning That is the design of notifications [7]. I guess we're debating the schema of the Payload section on a per-event basis. (as opposed to the somewhat loose definitions we have for those sections currently [8]) [7] https://wiki.openstack.org/wiki/NotificationSystem [8] https://wiki.openstack.org/wiki/SystemUsageData > 3. versioning of events: version is updated whenever the required fields are > changed. managing optional fields can be done via a specification > > Discovery of events would be interesting from an automated testing > perspective, but I am not sure how effective this would be for an application > actually consuming the event.s > Not sure how you would use enumerating the consumption of events > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
>From my perspective, the requirement is to be able to have a consistent and >predictable format for notifications that are being sent from all services. >This means: 1. a set of required fields that all events contain and have consistent meaning 2. a set of optional fields, you don’t have to include these but if you do then you follow the same format and meaning 3. versioning of events: version is updated whenever the required fields are changed. managing optional fields can be done via a specification Discovery of events would be interesting from an automated testing perspective, but I am not sure how effective this would be for an application actually consuming the event.s Not sure how you would use enumerating the consumption of events On Jul 10, 2014, at 2:48 AM, Eoghan Glynn wrote: > > TL;DR: do we need to stabilize notifications behind a versioned > and discoverable contract? > > Folks, > > One of the issues that has been raised in the recent discussions with > the QA team about branchless Tempest relates to some legacy defects > in the OpenStack notification system. > > Now, I don't personally subscribe to the PoV that ceilometer, or > indeed any other consumer of these notifications (e.g. StackTach), was > at fault for going ahead and depending on this pre-existing mechanism > without first fixing it. > > But be that as it may, we have a shortcoming here that needs to be > called out explicitly, and possible solutions explored. > > In many ways it's akin to the un-versioned RPC that existed in nova > before the versioned-rpc-apis BP[1] was landed back in Folsom IIRC, > except that notification consumers tend to be at arms-length from the > producer, and the effect of a notification is generally more advisory > than actionable. > > A great outcome would include some or all of the following: > > 1. more complete in-tree test coverage of notification logic on the >producer side > > 2. versioned notification payloads to protect consumers from breaking >changes in payload format > > 3. external discoverability of which event types a service is emitting > > 4. external discoverability of which event types a service is consuming > > If you're thinking that sounds like a substantial chunk of cross-project > work & co-ordination, you'd be right :) > > So the purpose of this thread is simply to get a read on the appetite > in the community for such an effort. At the least it would require: > > * trashing out the details in say a cross-project-track session at > the K* summit > > * buy-in from the producer-side projects (nova, glance, cinder etc.) > in terms of stepping up to make the changes > > * acquiescence from non-integrated projects that currently consume > these notifications > > (we shouldn't, as good citizens, simply pull the rug out from under > projects such as StackTach without discussion upfront) > > * dunno if the TC would need to give their imprimatur to such an > approach, or whether we could simply self-organize and get it done > without the need for governance resolutions etc. > > Any opinions on how desirable or necessary this is, and how the > detailed mechanics might work, would be welcome. > > Apologies BTW if this has already been discussed and rejected as > unworkable. I see a stalled versioned-notifications BP[2] and some > references to the CADF versioning scheme in the LP fossil-record. > Also an inconclusive ML thread from 2012[3], and a related grizzly > summit design session[4], but it's unclear to me whether these > aspirations got much traction in the end. > > Cheers, > Eoghan > > [1] https://blueprints.launchpad.net/nova/+spec/versioned-rpc-apis > [2] https://blueprints.launchpad.net/nova/+spec/versioned-notifications > [3] http://osdir.com/ml/openstack/2012-10/msg3.html > [4] https://etherpad.openstack.org/p/grizzly-common-messaging > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
-Original Message- From: Sandy Walsh Reply-To: "OpenStack Development Mailing List (not for usage questions)" Date: Thursday, July 10, 2014 at 9:31 AM To: "OpenStack Development Mailing List (not for usage questions)" Subject: Re: [openstack-dev] [all] Treating notifications as a contract >Ultimately this is the core problem. A breaking change in the >notifications caused tests to fail in other systems. Should we be adding >more tests or simply add version checking at the lower levels (like the >first pass of RPC versioning did)? > >(more on this below) > >> 2. versioned notification payloads to protect consumers from breaking >> changes in payload format >Yep, like RPC the biggies are: >1. removal of fields from notifications >2. change in semantics of a particular field >3. addition of new fields (not a biggie) The original implementation of the notification system was never intended to be the *final* implementation. I think we all identified the need for versioning several years ago. As for backwards compatibility, I think the version field itself, in whatever form it takes, should be optional. If not provided, it can be assumed the body of the notification is freeform. Adding fields is easy. Removal of fields should only occur after we can comfortably say that all services are only consuming the new field and we¹ve left sufficient time for deprecating the old ones. >> >> 3. external discoverability of which event types a service is emitting >These questions can be saved for later, but ... > >Is the use-case that a downstream system can learn which queue to >subscribe to programmatically? > >Is this a nice-to-have? > >Would / should this belong in a metadata service? In some capacity, I think PubSubHubBub has a lot to offer here. The publisher has to be able to provide a list of feeds it¹s authoritative for, which I think is sufficient for discovery. I think PSHB itself is bit heavy weight for what we need (and has some scaling problems) but I think the concept itself is very useful for us. > >> 4. external discoverability of which event types a service is consuming > >Isn't this what the topic queues are for? Consumers should only >subscribe to the topics they're interested in I¹m not sure I understand the value of this one. A service consumes what it consumes. If we¹ve versioned correctly, as mentioned above, why does it matter? >> If you're thinking that sounds like a substantial chunk of cross-project >> work & co-ordination, you'd be right :) > >Perhaps notification schemas should be broken out into a separate >repo(s)? That way we can test independent of the publishing system. For >example, our notigen event simulator [5] could use it. > >These could just be dependent libraries/plugins to oslo.messaging. +1 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Thu, 2014-07-10 at 04:48 -0400, Eoghan Glynn wrote: > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? > > Folks, > > One of the issues that has been raised in the recent discussions with > the QA team about branchless Tempest relates to some legacy defects > in the OpenStack notification system. Got links to specifics? I thought the consensus was that there was a contract here which we need to maintain, so I'd be curious where that broke down. Versioning and ability to newer contract versions would be good too, but in the absence of such things we should maintain backwards compat. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Thu, 10 Jul 2014, Julien Danjou wrote: My initial plan was to leverage a library like voluptuous to do schema based validation on the sender side. That would allow for receiver to introspect schema and know the data structure to expect. I didn't think deeply on how to handle versioning, but that should be doable too. It's not clear to me in this discussion what it is that is being versioned, contracted or standardized. Is it each of the many different notifications that various services produce now? Is it the general concept of a notification which can be considered a "sample" that something like Ceilometer or StackTack might like to consume? If it not the latter, why isn't it the latter? Here's some semi-random noodling: Wouldn't the metering process be a lot easier if there was a standardized package for a "sample" and anyone with the proper credentials could drop a sample on the bus with the right exchange with the right topic and anything (e.g. Ceilometer, StackTack, the NewShinyMeteringShiz) that wants to consider itself a metering store can consume it and hey presto. If people are going to have to write a bunch of new tests and related code to get notifications healthier why not make notifications for metrics _healthy_ and available to any system without needing to write a bunch of code on both sides of the bus. Currently Ceilometer is required to know far too much about the notifications it receives and that knowledge is being represented is code. That is a BadThing™. I'm sure there are plenty of reasons for why it has turned out that way, but if there is an opportunity for change...? -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/10/2014 5:52 AM, Eoghan Glynn wrote: > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? Thanks for dusting this off. Versioning and published schemas for notifications are important to the StackTach team. It would be nice to get this resolved. We're happy to help out. > Folks, > > One of the issues that has been raised in the recent discussions with > the QA team about branchless Tempest relates to some legacy defects > in the OpenStack notification system. > > Now, I don't personally subscribe to the PoV that ceilometer, or > indeed any other consumer of these notifications (e.g. StackTach), was > at fault for going ahead and depending on this pre-existing mechanism > without first fixing it. > > But be that as it may, we have a shortcoming here that needs to be > called out explicitly, and possible solutions explored. > > In many ways it's akin to the un-versioned RPC that existed in nova > before the versioned-rpc-apis BP[1] was landed back in Folsom IIRC, > except that notification consumers tend to be at arms-length from the > producer, and the effect of a notification is generally more advisory > than actionable. > > A great outcome would include some or all of the following: > > 1. more complete in-tree test coverage of notification logic on the > producer side Ultimately this is the core problem. A breaking change in the notifications caused tests to fail in other systems. Should we be adding more tests or simply add version checking at the lower levels (like the first pass of RPC versioning did)? (more on this below) > 2. versioned notification payloads to protect consumers from breaking > changes in payload format Yep, like RPC the biggies are: 1. removal of fields from notifications 2. change in semantics of a particular field 3. addition of new fields (not a biggie) The urgency for notifications is a little different than RPC where there is a method on the other end expecting a certain format. Notifications consumers have to be a little more forgiving when things don't come in as expected. This isn't a justification for breaking changes. Just stating that we have some leeway. I guess it really comes down to systems that are using notifications for critical synchronization vs. purely informational. > > 3. external discoverability of which event types a service is emitting These questions can be saved for later, but ... Is the use-case that a downstream system can learn which queue to subscribe to programmatically? Is this a nice-to-have? Would / should this belong in a metadata service? > 4. external discoverability of which event types a service is consuming Isn't this what the topic queues are for? Consumers should only subscribe to the topics they're interested in. > If you're thinking that sounds like a substantial chunk of cross-project > work & co-ordination, you'd be right :) Perhaps notification schemas should be broken out into a separate repo(s)? That way we can test independent of the publishing system. For example, our notigen event simulator [5] could use it. These could just be dependent libraries/plugins to oslo.messaging. > > So the purpose of this thread is simply to get a read on the appetite > in the community for such an effort. At the least it would require: > > * trashing out the details in say a cross-project-track session at >the K* summit > > * buy-in from the producer-side projects (nova, glance, cinder etc.) >in terms of stepping up to make the changes > > * acquiescence from non-integrated projects that currently consume >these notifications > >(we shouldn't, as good citizens, simply pull the rug out from under >projects such as StackTach without discussion upfront) We'll adapt StackTach.v2 accordingly. StackTach.v3 is far less impacted by notification changes since they are offloaded and processed in a secondary step. Breaking changes will just stall the processing. I suspect .v3 will be in place before .v2 is affected. Adding version handling to Stack-Distiller (our notification->event translator) should be pretty easy (and useful) [6] > * dunno if the TC would need to give their imprimatur to such an >approach, or whether we could simply self-organize and get it done >without the need for governance resolutions etc. > > Any opinions on how desirable or necessary this is, and how the > detailed mechanics might work, would be welcome. A published set of schemas would be very useful for StackTach, we'd love to help out in any way possible. In the near-term we have to press on under the assumption notification definitions are fragile. > Apologies BTW if this has already been discussed and rejected as > unworkable. I see a stalled versioned-notifications BP[2] and some > references to the CADF versioning scheme in the LP fossil-record. > Also an inconclusive ML thread from 2012[3], and a related grizzly > summit design session[4], but it's unclear
Re: [openstack-dev] [all] Treating notifications as a contract
On 7/10/2014 3:48 AM, Eoghan Glynn wrote: TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the OpenStack notification system. Now, I don't personally subscribe to the PoV that ceilometer, or indeed any other consumer of these notifications (e.g. StackTach), was at fault for going ahead and depending on this pre-existing mechanism without first fixing it. But be that as it may, we have a shortcoming here that needs to be called out explicitly, and possible solutions explored. In many ways it's akin to the un-versioned RPC that existed in nova before the versioned-rpc-apis BP[1] was landed back in Folsom IIRC, except that notification consumers tend to be at arms-length from the producer, and the effect of a notification is generally more advisory than actionable. A great outcome would include some or all of the following: 1. more complete in-tree test coverage of notification logic on the producer side 2. versioned notification payloads to protect consumers from breaking changes in payload format 3. external discoverability of which event types a service is emitting 4. external discoverability of which event types a service is consuming If you're thinking that sounds like a substantial chunk of cross-project work & co-ordination, you'd be right :) So the purpose of this thread is simply to get a read on the appetite in the community for such an effort. At the least it would require: * trashing out the details in say a cross-project-track session at the K* summit * buy-in from the producer-side projects (nova, glance, cinder etc.) in terms of stepping up to make the changes * acquiescence from non-integrated projects that currently consume these notifications (we shouldn't, as good citizens, simply pull the rug out from under projects such as StackTach without discussion upfront) * dunno if the TC would need to give their imprimatur to such an approach, or whether we could simply self-organize and get it done without the need for governance resolutions etc. Any opinions on how desirable or necessary this is, and how the detailed mechanics might work, would be welcome. Apologies BTW if this has already been discussed and rejected as unworkable. I see a stalled versioned-notifications BP[2] and some references to the CADF versioning scheme in the LP fossil-record. Also an inconclusive ML thread from 2012[3], and a related grizzly summit design session[4], but it's unclear to me whether these aspirations got much traction in the end. Cheers, Eoghan [1] https://blueprints.launchpad.net/nova/+spec/versioned-rpc-apis [2] https://blueprints.launchpad.net/nova/+spec/versioned-notifications [3] http://osdir.com/ml/openstack/2012-10/msg3.html [4] https://etherpad.openstack.org/p/grizzly-common-messaging ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I didn't read all of this, but in nova-land yes we've wanted versioned notifications for a long time because there are things we can't change in the notification payload without that. There have been a few ML threads in the past about this but no one has ever really stepped up to work on it seriously from what I can tell. -- Thanks, Matt Riedemann ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Treating notifications as a contract
On Thu, Jul 10 2014, Eoghan Glynn wrote: Hi Eoghan, > So the purpose of this thread is simply to get a read on the appetite > in the community for such an effort. At the least it would require: > > * trashing out the details in say a cross-project-track session at >the K* summit > > * buy-in from the producer-side projects (nova, glance, cinder etc.) >in terms of stepping up to make the changes > > * acquiescence from non-integrated projects that currently consume >these notifications > >(we shouldn't, as good citizens, simply pull the rug out from under >projects such as StackTach without discussion upfront) > > * dunno if the TC would need to give their imprimatur to such an >approach, or whether we could simply self-organize and get it done >without the need for governance resolutions etc. > > Any opinions on how desirable or necessary this is, and how the > detailed mechanics might work, would be welcome. Glad to see this coming up again. I've been pushing that since the Icehouse summit in Hong-Kong, where we had a Oslo session about that subject and where everybody nod about my initial plan. :) I've created a blueprint and started a very primitive implementation back then: https://blueprints.launchpad.net/oslo.messaging/+spec/notification-structured but had to stop by lacking time to push that effort further. My initial plan was to leverage a library like voluptuous to do schema based validation on the sender side. That would allow for receiver to introspect schema and know the data structure to expect. I didn't think deeply on how to handle versioning, but that should be doable too. To summarize, I got the feeling that there's more a need of resources to work on that than any problem. Same old, same old. Cheers, -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Treating notifications as a contract
TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the OpenStack notification system. Now, I don't personally subscribe to the PoV that ceilometer, or indeed any other consumer of these notifications (e.g. StackTach), was at fault for going ahead and depending on this pre-existing mechanism without first fixing it. But be that as it may, we have a shortcoming here that needs to be called out explicitly, and possible solutions explored. In many ways it's akin to the un-versioned RPC that existed in nova before the versioned-rpc-apis BP[1] was landed back in Folsom IIRC, except that notification consumers tend to be at arms-length from the producer, and the effect of a notification is generally more advisory than actionable. A great outcome would include some or all of the following: 1. more complete in-tree test coverage of notification logic on the producer side 2. versioned notification payloads to protect consumers from breaking changes in payload format 3. external discoverability of which event types a service is emitting 4. external discoverability of which event types a service is consuming If you're thinking that sounds like a substantial chunk of cross-project work & co-ordination, you'd be right :) So the purpose of this thread is simply to get a read on the appetite in the community for such an effort. At the least it would require: * trashing out the details in say a cross-project-track session at the K* summit * buy-in from the producer-side projects (nova, glance, cinder etc.) in terms of stepping up to make the changes * acquiescence from non-integrated projects that currently consume these notifications (we shouldn't, as good citizens, simply pull the rug out from under projects such as StackTach without discussion upfront) * dunno if the TC would need to give their imprimatur to such an approach, or whether we could simply self-organize and get it done without the need for governance resolutions etc. Any opinions on how desirable or necessary this is, and how the detailed mechanics might work, would be welcome. Apologies BTW if this has already been discussed and rejected as unworkable. I see a stalled versioned-notifications BP[2] and some references to the CADF versioning scheme in the LP fossil-record. Also an inconclusive ML thread from 2012[3], and a related grizzly summit design session[4], but it's unclear to me whether these aspirations got much traction in the end. Cheers, Eoghan [1] https://blueprints.launchpad.net/nova/+spec/versioned-rpc-apis [2] https://blueprints.launchpad.net/nova/+spec/versioned-notifications [3] http://osdir.com/ml/openstack/2012-10/msg3.html [4] https://etherpad.openstack.org/p/grizzly-common-messaging ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
