Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-25 Thread Chris Dent
There's a review in progress for a generic event format for PaaS-services which is a move with the right spirit: allow various services to join the the notification party without needing special handlers. See: https://review.openstack.org/#/c/101967/ -- Chris Dent tw:@anticdent freenode:cdent

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-17 Thread Chris Dent
On Tue, 15 Jul 2014, Mark McLoughlin wrote: So you're proposing that all payloads should contain something like: [...] a class, type, id, value, unit and a space to put additional metadata. That's the gist of it, but I'm only presenting that as a way to get somebody else to point out what's

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-17 Thread Chris Dent
On Tue, 15 Jul 2014, Sandy Walsh wrote: This looks like a particular schema for one event-type (let's say foo.sample). It's hard to extrapolate this one schema to a generic set of common metadata applicable to all events. Really the only common stuff we can agree on is the stuff already there:

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-16 Thread Sandy Walsh
On 7/11/2014 6:08 AM, Chris Dent wrote: On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if you have a better format talk to Haomeng which is the guys

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-15 Thread Mark McLoughlin
On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote: On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if you have a better format talk to Haomeng

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-15 Thread Mark McLoughlin
On Thu, 2014-07-10 at 16:21 -0400, Eoghan Glynn wrote: One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the OpenStack notification system. Got links to specifics? I thought the consensus was

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-15 Thread Sandy Walsh
On 7/15/2014 3:51 AM, Mark McLoughlin wrote: On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote: On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-14 Thread Eoghan Glynn
So what we need to figure out is how exactly this common structure can be accommodated without reverting back to what Sandy called the wild west in another post. I got the impression that wild west is what we've already got (within the payload)? Yeah, exactly, that was my

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-14 Thread Joe Gordon
On Thu, Jul 10, 2014 at 1:48 AM, Eoghan Glynn egl...@redhat.com wrote: TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Eoghan Glynn
tl;dr: Having a minimum payload standard enables more and more robust services on both sides of notification bus. Yes, that's exactly the point of this thread. We do not have a standard format for the payload. I think we should (more on that below). Again, such standardization is exactly

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Lucas Alvares Gomes
I just started the code for processing of notifications from Ironic. Conceptually they are the same as notifications from Nova but the actual form of the payload is completely different. This means I have to write a different processor for that payload. And now so does StackTach if they want

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Chris Dent
On Fri, 11 Jul 2014, Eoghan Glynn wrote: But I guess you're suggesting not only that we version/schematize individual notification payloads, but that we do so in a way that's global across event types and emitters? That's partially correct. I'm suggesting the we consider standardizing a

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Chris Dent
On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: The data format that Ironic will send was part of the spec proposed and could have been reviewed. I think there's still time to change it tho, if you have a better format talk to Haomeng which is the guys responsible for that work in Ironic and

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Eoghan Glynn
But I guess you're suggesting not only that we version/schematize individual notification payloads, but that we do so in a way that's global across event types and emitters? That's partially correct. I'm suggesting the we consider standardizing a general format for notification

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Sandy Walsh
On 7/10/2014 12:10 PM, Chris Dent wrote: On Thu, 10 Jul 2014, Julien Danjou wrote: My initial plan was to leverage a library like voluptuous to do schema based validation on the sender side. That would allow for receiver to introspect schema and know the data structure to expect. I didn't

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Chris Dent
On Fri, 11 Jul 2014, Eoghan Glynn wrote: A notification of compute.instance.create.start is naturally going to carry different types of data than a volume.snapshot.delete.end for example, but of course we'd seek to accommodate that difference within a generic structure as far as possible. Is

[openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the OpenStack notification system. Now, I don't

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Julien Danjou
On Thu, Jul 10 2014, Eoghan Glynn wrote: Hi Eoghan, So the purpose of this thread is simply to get a read on the appetite in the community for such an effort. At the least it would require: * trashing out the details in say a cross-project-track session at the K* summit * buy-in from

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Sandy Walsh
On 7/10/2014 5:52 AM, Eoghan Glynn wrote: TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Thanks for dusting this off. Versioning and published schemas for notifications are important to the StackTach team. It would be nice to get this

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Chris Dent
On Thu, 10 Jul 2014, Julien Danjou wrote: My initial plan was to leverage a library like voluptuous to do schema based validation on the sender side. That would allow for receiver to introspect schema and know the data structure to expect. I didn't think deeply on how to handle versioning, but

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Mark McLoughlin
On Thu, 2014-07-10 at 04:48 -0400, Eoghan Glynn wrote: TL;DR: do we need to stabilize notifications behind a versioned and discoverable contract? Folks, One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Matt Dietz
@lists.openstack.org Subject: Re: [openstack-dev] [all] Treating notifications as a contract Ultimately this is the core problem. A breaking change in the notifications caused tests to fail in other systems. Should we be adding more tests or simply add version checking at the lower levels (like

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Daniel Dyer
From my perspective, the requirement is to be able to have a consistent and predictable format for notifications that are being sent from all services. This means: 1. a set of required fields that all events contain and have consistent meaning 2. a set of optional fields, you don’t have to

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Sandy Walsh
On 7/10/2014 2:59 PM, Daniel Dyer wrote: From my perspective, the requirement is to be able to have a consistent and predictable format for notifications that are being sent from all services. This means: 1. a set of required fields that all events contain and have consistent meaning 2. a

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
Thanks for dusting this off. Versioning and published schemas for notifications are important to the StackTach team. It would be nice to get this resolved. We're happy to help out. Great! A great outcome would include some or all of the following: 1. more complete in-tree test

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
It's not clear to me in this discussion what it is that is being versioned, contracted or standardized. Well the statement in the original mail was pretty explicit: versioned notification payloads to protect consumers from breaking changes in payload format So it's the standardization of

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
One of the issues that has been raised in the recent discussions with the QA team about branchless Tempest relates to some legacy defects in the OpenStack notification system. Got links to specifics? I thought the consensus was that there was a contract here which we need to maintain,

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
From my perspective, the requirement is to be able to have a consistent and predictable format for notifications that are being sent from all services. This means: 1. a set of required fields that all events contain and have consistent meaning 2. a set of optional fields, you don’t have to

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Eoghan Glynn
The original implementation of the notification system was never intended to be the *final* implementation. I think we all identified the need for versioning several years ago. As for backwards compatibility, I think the version field itself, in whatever form it takes, should be optional. If

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Matt Dietz
@lists.openstack.org Subject: Re: [openstack-dev] [all] Treating notifications as a contract The original implementation of the notification system was never intended to be the *final* implementation. I think we all identified the need for versioning several years ago. As for backwards compatibility

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Chris Dent
tl;dr: Having a minimum payload standard enables more and more robust services on both sides of notification bus. Expanded earnest optimism below. On Thu, 10 Jul 2014, Eoghan Glynn wrote: So it's the standardization of notification payload formats that's being proposed. Generally that