Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> -Original Message-
> From: Andrew Laski [mailto:and...@lascii.com]
> Sent: November 30, 2015 20:32
> On 11/30/15 at 07:32am, Sean Dague wrote:
> >On 11/24/2015 10:09 AM, John Garbutt wrote:
> >> On 24 November 2015 at 15:00, Balázs Gibizer
> >>  wrote:
>  From: Andrew Laski [mailto:and...@lascii.com]
>  Sent: November 24, 2015 15:35
>  On 11/24/15 at 10:26am, Balázs Gibizer wrote:
> >> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> >> Sent: November 23, 2015 22:33
> >> On 11/23/2015 2:23 PM, Andrew Laski wrote:
> >>> On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
> > From: Andrew Laski [mailto:and...@lascii.com]
> > Sent: November 23, 2015 17:03
> >
> > On 11/23/15 at 08:54am, Ryan Rossiter wrote:
> >>
> >>
> >> On 11/23/2015 5:33 AM, John Garbutt wrote:
> >>> On 20 November 2015 at 09:37, Balázs Gibizer
> >>>  wrote:
>  
>  
> >>
> >>> There is a bit I am conflicted/worried about, and thats when
> >>> we start including verbatim, DB objects into the
> >>> notifications. At least you can now quickly detect if that
> >>> blob is something compatible with your current parsing code.
> >>> My preference is really to keep the Notifications as a
> >>> totally separate object tree, but I am sure there are many
> >>> cases where that ends up being seemingly stupid duplicate
> >>> work. I am not expressing this well in text form :(
> >> Are you saying we don't want to be willy-nilly tossing DB
> >> objects across the wire? Yeah that was part of the
> >> rug-pulling of just having the payload contain an object.
> >> We're automatically tossing everything with the object then,
> >> whether or not some of that was supposed to be a secret. We
> >> could add some sort of property to the field like
> >> dont_put_me_on_the_wire=True (or I guess a
> >> notification_ready() function that helps an object sanitize
> >> itself?) that the notifications will look at to know if it
> >> puts that on the wire-serialized dict, but that's adding a
> >> lot more complexity and work to a pile that's already growing
> rapidly.
> >
> > I don't want to be tossing db objects across the wire.  But I
> > also am not convinced that we should be tossing the current
> > objects over the wire either.
> > You make the point that there may be things in the object that
> > shouldn't be exposed, and I think object version bumps is
> > another thing to watch out for.
> > So far the only object that has been bumped is Instance but in
> > doing so no notifications needed to change.  I think if we
> > just put objects into notifications we're coupling the
> > notification versions to db or RPC changes unnecessarily.
> > Some times they'll move together but other times, like moving
> > flavor into instance_extra, there's no reason to bump
> notifications.
> 
> 
>  Sanitizing existing versioned objects before putting them to
>  the wire is not hard to do.
>  You can see an example of doing it in
> 
> https://review.openstack.org/#/c/245678/8/nova/objects/service.
>  py,
>  cm
>  L382.
>  We don't need extra effort to take care of minor version bumps
>  because that does not break a well written consumer. We do
> have
>  to take care of the major version bumps but that is a rare
>  event and therefore can be handled one by one in a way John
>  suggested, by keep sending the previous major version for a
> while too.
> >>>
> >>> That review is doing much of what I was suggesting.  There is a
> >>> separate notification and payload object.  The issue I have is
> >>> that within the ServiceStatusPayload the raw Service object and
> >>> version is being dumped, with the filter you point out.  But I
> >>> don't think that consumers really care about tracking Service
> >>> object versions and dealing with compatibility there, it would
> >>> be easier for them to track the ServiceStatusPayload version
> >>> which can remain relatively stable even if Service is changing to
> adapt to db/RPC changes.
> >> Not only do they not really care about tracking the Service
> >> object versions, they probably also don't care about what's in that
> filter list.
> >>
> >> But I think you're getting on the right track as to where this
> >> needs to go. We can integrate the filtering into the versioning of the
> payload.
> >> But instead of a blacklist, we turn the filter into a white list.
> >> If the underlying object adds a new field that we 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Andrew Laski]

On 11/30/15 at 07:32am, Sean Dague wrote:

On 11/24/2015 10:09 AM, John Garbutt wrote:

On 24 November 2015 at 15:00, Balázs Gibizer
 wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 24, 2015 15:35
On 11/24/15 at 10:26am, Balázs Gibizer wrote:

From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
Sent: November 23, 2015 22:33
On 11/23/2015 2:23 PM, Andrew Laski wrote:

On 11/23/15 at 04:43pm, Balázs Gibizer wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 23, 2015 17:03

On 11/23/15 at 08:54am, Ryan Rossiter wrote:



On 11/23/2015 5:33 AM, John Garbutt wrote:

On 20 November 2015 at 09:37, Balázs Gibizer
 wrote:






There is a bit I am conflicted/worried about, and thats when we
start including verbatim, DB objects into the notifications. At
least you can now quickly detect if that blob is something
compatible with your current parsing code. My preference is
really to keep the Notifications as a totally separate object
tree, but I am sure there are many cases where that ends up
being seemingly stupid duplicate work. I am not expressing this
well in text form :(

Are you saying we don't want to be willy-nilly tossing DB
objects across the wire? Yeah that was part of the rug-pulling
of just having the payload contain an object. We're
automatically tossing everything with the object then, whether
or not some of that was supposed to be a secret. We could add
some sort of property to the field like
dont_put_me_on_the_wire=True (or I guess a
notification_ready() function that helps an object sanitize
itself?) that the notifications will look at to know if it puts
that on the wire-serialized dict, but that's adding a lot more
complexity and work to a pile that's already growing rapidly.


I don't want to be tossing db objects across the wire.  But I
also am not convinced that we should be tossing the current
objects over the wire either.
You make the point that there may be things in the object that
shouldn't be exposed, and I think object version bumps is another
thing to watch out for.
So far the only object that has been bumped is Instance but in
doing so no notifications needed to change.  I think if we just
put objects into notifications we're coupling the notification
versions to db or RPC changes unnecessarily.  Some times they'll
move together but other times, like moving flavor into
instance_extra, there's no reason to bump notifications.



Sanitizing existing versioned objects before putting them to the
wire is not hard to do.
You can see an example of doing it in
https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
cm
L382.
We don't need extra effort to take care of minor version bumps
because that does not break a well written consumer. We do have to
take care of the major version bumps but that is a rare event and
therefore can be handled one by one in a way John suggested, by
keep sending the previous major version for a while too.


That review is doing much of what I was suggesting.  There is a
separate notification and payload object.  The issue I have is that
within the ServiceStatusPayload the raw Service object and version
is being dumped, with the filter you point out.  But I don't think
that consumers really care about tracking Service object versions
and dealing with compatibility there, it would be easier for them
to track the ServiceStatusPayload version which can remain
relatively stable even if Service is changing to adapt to db/RPC changes.

Not only do they not really care about tracking the Service object
versions, they probably also don't care about what's in that filter list.

But I think you're getting on the right track as to where this needs
to go. We can integrate the filtering into the versioning of the payload.
But instead of a blacklist, we turn the filter into a white list. If
the underlying object adds a new field that we don't want/care if
people know about, the payload version doesn't have to change. But if
we add something (or if we're changing the existing fields) that we
want to expose, we then assert that we need to update the version of
the payload, so the consumer can look at the payload and say "oh, in
1.x, now I get ___" and can add the appropriate checks/compat.
Granted with this you can get into rebase nightmares ([1] still
haunts me in my sleep), but I don't see us frantically changing the
exposed fields all too often. This way gives us some form of
pseudo-pinning of the subobject. Heck, in this method, we could even
pass the whitelist on the wire right? That way we tell the consumer

explicitly what's available to them (kinda like a fake schema).


I think see your point, and it seems like a good way forward. Let's
turn the black list to a white list. Now I'm thinking about creating a
new Field type something like WhiteListedObjectField which get a type
name (as the ObjectField) but also get a white_list that describes which

fields 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Sean Dague]

On 11/24/2015 10:09 AM, John Garbutt wrote:
> On 24 November 2015 at 15:00, Balázs Gibizer
>  wrote:
>>> From: Andrew Laski [mailto:and...@lascii.com]
>>> Sent: November 24, 2015 15:35
>>> On 11/24/15 at 10:26am, Balázs Gibizer wrote:
> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> Sent: November 23, 2015 22:33
> On 11/23/2015 2:23 PM, Andrew Laski wrote:
>> On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
 From: Andrew Laski [mailto:and...@lascii.com]
 Sent: November 23, 2015 17:03

 On 11/23/15 at 08:54am, Ryan Rossiter wrote:
>
>
> On 11/23/2015 5:33 AM, John Garbutt wrote:
>> On 20 November 2015 at 09:37, Balázs Gibizer
>>  wrote:
>>> 
>>> 
>
>> There is a bit I am conflicted/worried about, and thats when we
>> start including verbatim, DB objects into the notifications. At
>> least you can now quickly detect if that blob is something
>> compatible with your current parsing code. My preference is
>> really to keep the Notifications as a totally separate object
>> tree, but I am sure there are many cases where that ends up
>> being seemingly stupid duplicate work. I am not expressing this
>> well in text form :(
> Are you saying we don't want to be willy-nilly tossing DB
> objects across the wire? Yeah that was part of the rug-pulling
> of just having the payload contain an object. We're
> automatically tossing everything with the object then, whether
> or not some of that was supposed to be a secret. We could add
> some sort of property to the field like
> dont_put_me_on_the_wire=True (or I guess a
> notification_ready() function that helps an object sanitize
> itself?) that the notifications will look at to know if it puts
> that on the wire-serialized dict, but that's adding a lot more
> complexity and work to a pile that's already growing rapidly.

 I don't want to be tossing db objects across the wire.  But I
 also am not convinced that we should be tossing the current
 objects over the wire either.
 You make the point that there may be things in the object that
 shouldn't be exposed, and I think object version bumps is another
 thing to watch out for.
 So far the only object that has been bumped is Instance but in
 doing so no notifications needed to change.  I think if we just
 put objects into notifications we're coupling the notification
 versions to db or RPC changes unnecessarily.  Some times they'll
 move together but other times, like moving flavor into
 instance_extra, there's no reason to bump notifications.
>>>
>>>
>>> Sanitizing existing versioned objects before putting them to the
>>> wire is not hard to do.
>>> You can see an example of doing it in
>>> https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
>>> cm
>>> L382.
>>> We don't need extra effort to take care of minor version bumps
>>> because that does not break a well written consumer. We do have to
>>> take care of the major version bumps but that is a rare event and
>>> therefore can be handled one by one in a way John suggested, by
>>> keep sending the previous major version for a while too.
>>
>> That review is doing much of what I was suggesting.  There is a
>> separate notification and payload object.  The issue I have is that
>> within the ServiceStatusPayload the raw Service object and version
>> is being dumped, with the filter you point out.  But I don't think
>> that consumers really care about tracking Service object versions
>> and dealing with compatibility there, it would be easier for them
>> to track the ServiceStatusPayload version which can remain
>> relatively stable even if Service is changing to adapt to db/RPC changes.
> Not only do they not really care about tracking the Service object
> versions, they probably also don't care about what's in that filter list.
>
> But I think you're getting on the right track as to where this needs
> to go. We can integrate the filtering into the versioning of the payload.
> But instead of a blacklist, we turn the filter into a white list. If
> the underlying object adds a new field that we don't want/care if
> people know about, the payload version doesn't have to change. But if
> we add something (or if we're changing the existing fields) that we
> want to expose, we then assert that we need to update the version of
> the payload, so the consumer can look at the payload and say "oh, in
> 1.x, now I get ___" and can add the appropriate checks/compat.
> Granted with this you can get into 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Ryan Rossiter]

On 11/24/2015 8:35 AM, Andrew Laski wrote:

On 11/24/15 at 10:26am, Balázs Gibizer wrote:






I think see your point, and it seems like a good way forward. Let's 
turn the black list to
a white list. Now I'm thinking about creating a new Field type 
something like
WhiteListedObjectField which get a type name (as the ObjectField) but 
also get
a white_list that describes which fields needs to be used from the 
original type.
Then this new field serializes only the white listed fields from the 
original type
and only forces a version bump on the parent object if one of the 
white_listed field

changed or a new field added to the white_list.
What it does not solve out of the box is the transitive dependency. 
If today we
Have an o.vo object having a filed to another o.vo object and we want 
to put
the first object into a notification payload but want to white_list 
fields from
the second o.vo then our white list needs to be able to handle not 
just first
level fields but subfields too. I guess this is doable but I'm 
wondering if we
can avoid inventing a syntax expressing something like 
'field.subfield.subsubfield'

in the white list.


Rather than a whitelist/blacklist why not just define the schema of 
the notification within the notification object and then have the 
object code handle pulling the appropriate fields, converting formats 
if necessary, from contained objects.  Something like:


class ServicePayloadObject(NovaObject):
SCHEMA = {'host': ('service', 'host'),
  'binary': ('service', 'binary'),
  'compute_node_foo': ('compute_node', 'foo'),
 }

fields = {
'service': fields.ObjectField('Service'),
'compute_node': fields.ObjectField('ComputeNode'),
}

def populate_schema(self):
self.compute_node = self.service.compute_node
notification = {}
for key, (obj, field) in schema.iteritems():
notification[key] = getattr(getattr(self, obj), field)

Then object changes have no effect on the notifications unless there's 
a major version bump in which case a SCHEMA_VNEXT could be defined if 
necessary.
To be fair, that is basically a whitelist ;) [1]. But if we use this 
method, don't we lose a lot of o.vo's usefulness? When we serialize, we 
have to specifically *not* use the fields because that is the master 
sheet of information that we don't want to expose all of. Either that or 
we have to do the transform as part of the serialization using the 
schema, which you may be aiming at, I might just be looking at the 
snippet too literally.



[1] http://www.smbc-comics.com/index.php?id=3907

--
Thanks,

Ryan Rossiter (rlrossit)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Andrew Laski]

On 11/24/15 at 11:19am, Ryan Rossiter wrote:



On 11/24/2015 8:35 AM, Andrew Laski wrote:

On 11/24/15 at 10:26am, Balázs Gibizer wrote:






I think see your point, and it seems like a good way forward. 
Let's turn the black list to
a white list. Now I'm thinking about creating a new Field type 
something like
WhiteListedObjectField which get a type name (as the ObjectField) 
but also get
a white_list that describes which fields needs to be used from 
the original type.
Then this new field serializes only the white listed fields from 
the original type
and only forces a version bump on the parent object if one of the 
white_listed field

changed or a new field added to the white_list.
What it does not solve out of the box is the transitive 
dependency. If today we
Have an o.vo object having a filed to another o.vo object and we 
want to put
the first object into a notification payload but want to 
white_list fields from
the second o.vo then our white list needs to be able to handle 
not just first
level fields but subfields too. I guess this is doable but I'm 
wondering if we
can avoid inventing a syntax expressing something like 
'field.subfield.subsubfield'

in the white list.


Rather than a whitelist/blacklist why not just define the schema of 
the notification within the notification object and then have the 
object code handle pulling the appropriate fields, converting 
formats if necessary, from contained objects.  Something like:


class ServicePayloadObject(NovaObject):
   SCHEMA = {'host': ('service', 'host'),
 'binary': ('service', 'binary'),
 'compute_node_foo': ('compute_node', 'foo'),
}

   fields = {
   'service': fields.ObjectField('Service'),
   'compute_node': fields.ObjectField('ComputeNode'),
   }

   def populate_schema(self):
   self.compute_node = self.service.compute_node
   notification = {}
   for key, (obj, field) in schema.iteritems():
   notification[key] = getattr(getattr(self, obj), field)

Then object changes have no effect on the notifications unless 
there's a major version bump in which case a SCHEMA_VNEXT could be 
defined if necessary.
To be fair, that is basically a whitelist ;) [1]. 


Heh, fair point :)

But if we use this 
method, don't we lose a lot of o.vo's usefulness? When we serialize, 
we have to specifically *not* use the fields because that is the 
master sheet of information that we don't want to expose all of. 
Either that or we have to do the transform as part of the 
serialization using the schema, which you may be aiming at, I might 
just be looking at the snippet too literally.


I was thinking along the lines of doing the transform as part of 
serialization.  But really I wasn't thinking that serialization as it's 
done for RPC is needed at all.  I'm not expecting notification consumers 
to hydrate Nova objects from whatever is emitted, just receive a 
standard JSON payload that they can use.  So I wouldn't expect 
notification code to call obj_to_primitive() but perhaps a new emit() 
method which will do the transform to the schema.


I think we're all on a similar page though and perhaps we can continue 
the discussion on a review to nail down details.





[1] http://www.smbc-comics.com/index.php?id=3907

--
Thanks,

Ryan Rossiter (rlrossit)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> From: John Garbutt [mailto:j...@johngarbutt.com]
> Sent: November 24, 2015 16:09
> On 24 November 2015 at 15:00, Balázs Gibizer 
> wrote:
> >> From: Andrew Laski [mailto:and...@lascii.com]
> >> Sent: November 24, 2015 15:35
> >> On 11/24/15 at 10:26am, Balázs Gibizer wrote:
> >> >> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> >> >> Sent: November 23, 2015 22:33
> >> >> On 11/23/2015 2:23 PM, Andrew Laski wrote:
> >> >> > On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
> >> >> >>> From: Andrew Laski [mailto:and...@lascii.com]
> >> >> >>> Sent: November 23, 2015 17:03
> >> >> >>>
> >> >> >>> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
> >> >> >>> >
> >> >> >>> >
> >> >> >>> >On 11/23/2015 5:33 AM, John Garbutt wrote:
> >> >> >>> >>On 20 November 2015 at 09:37, Balázs Gibizer
> >> >> >>> >> wrote:
> >> >> >>> >>>
> >> >> >>> >>>
> >> >> >> >>
> >> >> >>> >>There is a bit I am conflicted/worried about, and thats when
> >> >> >>> >>we start including verbatim, DB objects into the
> >> >> >>> >>notifications. At least you can now quickly detect if that
> >> >> >>> >>blob is something compatible with your current parsing code.
> >> >> >>> >>My preference is really to keep the Notifications as a
> >> >> >>> >>totally separate object tree, but I am sure there are many
> >> >> >>> >>cases where that ends up being seemingly stupid duplicate
> >> >> >>> >>work. I am not expressing this well in text form :(
> >> >> >>> >Are you saying we don't want to be willy-nilly tossing DB
> >> >> >>> >objects across the wire? Yeah that was part of the
> >> >> >>> >rug-pulling of just having the payload contain an object.
> >> >> >>> >We're automatically tossing everything with the object then,
> >> >> >>> >whether or not some of that was supposed to be a secret. We
> >> >> >>> >could add some sort of property to the field like
> >> >> >>> >dont_put_me_on_the_wire=True (or I guess a
> >> >> >>> >notification_ready() function that helps an object sanitize
> >> >> >>> >itself?) that the notifications will look at to know if it
> >> >> >>> >puts that on the wire-serialized dict, but that's adding a
> >> >> >>> >lot more complexity and work to a pile that's already growing
> rapidly.
> >> >> >>>
> >> >> >>> I don't want to be tossing db objects across the wire.  But I
> >> >> >>> also am not convinced that we should be tossing the current
> >> >> >>> objects over the wire either.
> >> >> >>> You make the point that there may be things in the object that
> >> >> >>> shouldn't be exposed, and I think object version bumps is
> >> >> >>> another thing to watch out for.
> >> >> >>> So far the only object that has been bumped is Instance but in
> >> >> >>> doing so no notifications needed to change.  I think if we
> >> >> >>> just put objects into notifications we're coupling the
> >> >> >>> notification versions to db or RPC changes unnecessarily.
> >> >> >>> Some times they'll move together but other times, like moving
> >> >> >>> flavor into instance_extra, there's no reason to bump
> notifications.
> >> >> >>
> >> >> >>
> >> >> >> Sanitizing existing versioned objects before putting them to
> >> >> >> the wire is not hard to do.
> >> >> >> You can see an example of doing it in
> >> >> >> https://review.openstack.org/#/c/245678/8/nova/objects/service.
> >> >> >> py,
> >> >> >> cm
> >> >> >> L382.
> >> >> >> We don't need extra effort to take care of minor version bumps
> >> >> >> because that does not break a well written consumer. We do have
> >> >> >> to take care of the major version bumps but that is a rare
> >> >> >> event and therefore can be handled one by one in a way John
> >> >> >> suggested, by keep sending the previous major version for a while
> too.
> >> >> >
> >> >> > That review is doing much of what I was suggesting.  There is a
> >> >> > separate notification and payload object.  The issue I have is
> >> >> > that within the ServiceStatusPayload the raw Service object and
> >> >> > version is being dumped, with the filter you point out.  But I
> >> >> > don't think that consumers really care about tracking Service
> >> >> > object versions and dealing with compatibility there, it would
> >> >> > be easier for them to track the ServiceStatusPayload version
> >> >> > which can remain relatively stable even if Service is changing to 
> >> >> > adapt
> to db/RPC changes.
> >> >> Not only do they not really care about tracking the Service object
> >> >> versions, they probably also don't care about what's in that filter 
> >> >> list.
> >> >>
> >> >> But I think you're getting on the right track as to where this
> >> >> needs to go. We can integrate the filtering into the versioning of the
> payload.
> >> >> But instead of a blacklist, we turn the filter into a white list.
> >> >> If the underlying object adds a new field that we don't want/care
> >> >> if people know about, the payload version doesn't have to change.
> >> >> But if we add something (or 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> Sent: November 23, 2015 22:33
> On 11/23/2015 2:23 PM, Andrew Laski wrote:
> > On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
> >>> From: Andrew Laski [mailto:and...@lascii.com]
> >>> Sent: November 23, 2015 17:03
> >>>
> >>> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
> >>> >
> >>> >
> >>> >On 11/23/2015 5:33 AM, John Garbutt wrote:
> >>> >>On 20 November 2015 at 09:37, Balázs Gibizer
> >>> >> wrote:
> >>> >>>
> >>> >>>
> >> >>
> >>> >>There is a bit I am conflicted/worried about, and thats when we
> >>> >>start including verbatim, DB objects into the notifications. At
> >>> >>least you can now quickly detect if that blob is something
> >>> >>compatible with your current parsing code. My preference is really
> >>> >>to keep the Notifications as a totally separate object tree, but I
> >>> >>am sure there are many cases where that ends up being seemingly
> >>> >>stupid duplicate work. I am not expressing this well in text form
> >>> >>:(
> >>> >Are you saying we don't want to be willy-nilly tossing DB objects
> >>> >across the wire? Yeah that was part of the rug-pulling of just
> >>> >having the payload contain an object. We're automatically tossing
> >>> >everything with the object then, whether or not some of that was
> >>> >supposed to be a secret. We could add some sort of property to the
> >>> >field like dont_put_me_on_the_wire=True (or I guess a
> >>> >notification_ready() function that helps an object sanitize
> >>> >itself?) that the notifications will look at to know if it puts
> >>> >that on the wire-serialized dict, but that's adding a lot more
> >>> >complexity and work to a pile that's already growing rapidly.
> >>>
> >>> I don't want to be tossing db objects across the wire.  But I also
> >>> am not convinced that we should be tossing the current objects over
> >>> the wire either.
> >>> You make the point that there may be things in the object that
> >>> shouldn't be exposed, and I think object version bumps is another
> >>> thing to watch out for.
> >>> So far the only object that has been bumped is Instance but in doing
> >>> so no notifications needed to change.  I think if we just put
> >>> objects into notifications we're coupling the notification versions
> >>> to db or RPC changes unnecessarily.  Some times they'll move
> >>> together but other times, like moving flavor into instance_extra,
> >>> there's no reason to bump notifications.
> >>
> >>
> >> Sanitizing existing versioned objects before putting them to the wire
> >> is not hard to do.
> >> You can see an example of doing it in
> >> https://review.openstack.org/#/c/245678/8/nova/objects/service.py,cm
> >> L382.
> >> We don't need extra effort to take care of minor version bumps
> >> because that does not break a well written consumer. We do have to
> >> take care of the major version bumps but that is a rare event and
> >> therefore can be handled one by one in a way John suggested, by keep
> >> sending the previous major version for a while too.
> >
> > That review is doing much of what I was suggesting.  There is a
> > separate notification and payload object.  The issue I have is that
> > within the ServiceStatusPayload the raw Service object and version is
> > being dumped, with the filter you point out.  But I don't think that
> > consumers really care about tracking Service object versions and
> > dealing with compatibility there, it would be easier for them to track
> > the ServiceStatusPayload version which can remain relatively stable
> > even if Service is changing to adapt to db/RPC changes.
> Not only do they not really care about tracking the Service object versions,
> they probably also don't care about what's in that filter list.
> 
> But I think you're getting on the right track as to where this needs to go. We
> can integrate the filtering into the versioning of the payload.
> But instead of a blacklist, we turn the filter into a white list. If the 
> underlying
> object adds a new field that we don't want/care if people know about, the
> payload version doesn't have to change. But if we add something (or if we're
> changing the existing fields) that we want to expose, we then assert that we
> need to update the version of the payload, so the consumer can look at the
> payload and say "oh, in 1.x, now I get ___" and can add the appropriate
> checks/compat. Granted with this you can get into rebase nightmares ([1]
> still haunts me in my sleep), but I don't see us frantically changing the
> exposed fields all too often. This way gives us some form of pseudo-pinning
> of the subobject. Heck, in this method, we could even pass the whitelist on
> the wire right? That way we tell the consumer explicitly what's available to
> them (kinda like a fake schema).

I think see your point, and it seems like a good way forward. Let's turn the 
black list to 
a white list. Now I'm thinking about creating a new Field type something 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Andrew Laski]

On 11/24/15 at 10:26am, Balázs Gibizer wrote:

From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
Sent: November 23, 2015 22:33
On 11/23/2015 2:23 PM, Andrew Laski wrote:
> On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
>>> From: Andrew Laski [mailto:and...@lascii.com]
>>> Sent: November 23, 2015 17:03
>>>
>>> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
>>> >
>>> >
>>> >On 11/23/2015 5:33 AM, John Garbutt wrote:
>>> >>On 20 November 2015 at 09:37, Balázs Gibizer
>>> >> wrote:
>>> >>>
>>> >>>
>> >>
>>> >>There is a bit I am conflicted/worried about, and thats when we
>>> >>start including verbatim, DB objects into the notifications. At
>>> >>least you can now quickly detect if that blob is something
>>> >>compatible with your current parsing code. My preference is really
>>> >>to keep the Notifications as a totally separate object tree, but I
>>> >>am sure there are many cases where that ends up being seemingly
>>> >>stupid duplicate work. I am not expressing this well in text form
>>> >>:(
>>> >Are you saying we don't want to be willy-nilly tossing DB objects
>>> >across the wire? Yeah that was part of the rug-pulling of just
>>> >having the payload contain an object. We're automatically tossing
>>> >everything with the object then, whether or not some of that was
>>> >supposed to be a secret. We could add some sort of property to the
>>> >field like dont_put_me_on_the_wire=True (or I guess a
>>> >notification_ready() function that helps an object sanitize
>>> >itself?) that the notifications will look at to know if it puts
>>> >that on the wire-serialized dict, but that's adding a lot more
>>> >complexity and work to a pile that's already growing rapidly.
>>>
>>> I don't want to be tossing db objects across the wire.  But I also
>>> am not convinced that we should be tossing the current objects over
>>> the wire either.
>>> You make the point that there may be things in the object that
>>> shouldn't be exposed, and I think object version bumps is another
>>> thing to watch out for.
>>> So far the only object that has been bumped is Instance but in doing
>>> so no notifications needed to change.  I think if we just put
>>> objects into notifications we're coupling the notification versions
>>> to db or RPC changes unnecessarily.  Some times they'll move
>>> together but other times, like moving flavor into instance_extra,
>>> there's no reason to bump notifications.
>>
>>
>> Sanitizing existing versioned objects before putting them to the wire
>> is not hard to do.
>> You can see an example of doing it in
>> https://review.openstack.org/#/c/245678/8/nova/objects/service.py,cm
>> L382.
>> We don't need extra effort to take care of minor version bumps
>> because that does not break a well written consumer. We do have to
>> take care of the major version bumps but that is a rare event and
>> therefore can be handled one by one in a way John suggested, by keep
>> sending the previous major version for a while too.
>
> That review is doing much of what I was suggesting.  There is a
> separate notification and payload object.  The issue I have is that
> within the ServiceStatusPayload the raw Service object and version is
> being dumped, with the filter you point out.  But I don't think that
> consumers really care about tracking Service object versions and
> dealing with compatibility there, it would be easier for them to track
> the ServiceStatusPayload version which can remain relatively stable
> even if Service is changing to adapt to db/RPC changes.
Not only do they not really care about tracking the Service object versions,
they probably also don't care about what's in that filter list.

But I think you're getting on the right track as to where this needs to go. We
can integrate the filtering into the versioning of the payload.
But instead of a blacklist, we turn the filter into a white list. If the 
underlying
object adds a new field that we don't want/care if people know about, the
payload version doesn't have to change. But if we add something (or if we're
changing the existing fields) that we want to expose, we then assert that we
need to update the version of the payload, so the consumer can look at the
payload and say "oh, in 1.x, now I get ___" and can add the appropriate
checks/compat. Granted with this you can get into rebase nightmares ([1]
still haunts me in my sleep), but I don't see us frantically changing the
exposed fields all too often. This way gives us some form of pseudo-pinning
of the subobject. Heck, in this method, we could even pass the whitelist on
the wire right? That way we tell the consumer explicitly what's available to
them (kinda like a fake schema).


I think see your point, and it seems like a good way forward. Let's turn the 
black list to
a white list. Now I'm thinking about creating a new Field type something like
WhiteListedObjectField which get a type name (as the ObjectField) but also get
a white_list that describes which fields 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> From: Andrew Laski [mailto:and...@lascii.com]
> Sent: November 24, 2015 15:35
> On 11/24/15 at 10:26am, Balázs Gibizer wrote:
> >> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> >> Sent: November 23, 2015 22:33
> >> On 11/23/2015 2:23 PM, Andrew Laski wrote:
> >> > On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
> >> >>> From: Andrew Laski [mailto:and...@lascii.com]
> >> >>> Sent: November 23, 2015 17:03
> >> >>>
> >> >>> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
> >> >>> >
> >> >>> >
> >> >>> >On 11/23/2015 5:33 AM, John Garbutt wrote:
> >> >>> >>On 20 November 2015 at 09:37, Balázs Gibizer
> >> >>> >> wrote:
> >> >>> >>>
> >> >>> >>>
> >> >> >>
> >> >>> >>There is a bit I am conflicted/worried about, and thats when we
> >> >>> >>start including verbatim, DB objects into the notifications. At
> >> >>> >>least you can now quickly detect if that blob is something
> >> >>> >>compatible with your current parsing code. My preference is
> >> >>> >>really to keep the Notifications as a totally separate object
> >> >>> >>tree, but I am sure there are many cases where that ends up
> >> >>> >>being seemingly stupid duplicate work. I am not expressing this
> >> >>> >>well in text form :(
> >> >>> >Are you saying we don't want to be willy-nilly tossing DB
> >> >>> >objects across the wire? Yeah that was part of the rug-pulling
> >> >>> >of just having the payload contain an object. We're
> >> >>> >automatically tossing everything with the object then, whether
> >> >>> >or not some of that was supposed to be a secret. We could add
> >> >>> >some sort of property to the field like
> >> >>> >dont_put_me_on_the_wire=True (or I guess a
> >> >>> >notification_ready() function that helps an object sanitize
> >> >>> >itself?) that the notifications will look at to know if it puts
> >> >>> >that on the wire-serialized dict, but that's adding a lot more
> >> >>> >complexity and work to a pile that's already growing rapidly.
> >> >>>
> >> >>> I don't want to be tossing db objects across the wire.  But I
> >> >>> also am not convinced that we should be tossing the current
> >> >>> objects over the wire either.
> >> >>> You make the point that there may be things in the object that
> >> >>> shouldn't be exposed, and I think object version bumps is another
> >> >>> thing to watch out for.
> >> >>> So far the only object that has been bumped is Instance but in
> >> >>> doing so no notifications needed to change.  I think if we just
> >> >>> put objects into notifications we're coupling the notification
> >> >>> versions to db or RPC changes unnecessarily.  Some times they'll
> >> >>> move together but other times, like moving flavor into
> >> >>> instance_extra, there's no reason to bump notifications.
> >> >>
> >> >>
> >> >> Sanitizing existing versioned objects before putting them to the
> >> >> wire is not hard to do.
> >> >> You can see an example of doing it in
> >> >> https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
> >> >> cm
> >> >> L382.
> >> >> We don't need extra effort to take care of minor version bumps
> >> >> because that does not break a well written consumer. We do have to
> >> >> take care of the major version bumps but that is a rare event and
> >> >> therefore can be handled one by one in a way John suggested, by
> >> >> keep sending the previous major version for a while too.
> >> >
> >> > That review is doing much of what I was suggesting.  There is a
> >> > separate notification and payload object.  The issue I have is that
> >> > within the ServiceStatusPayload the raw Service object and version
> >> > is being dumped, with the filter you point out.  But I don't think
> >> > that consumers really care about tracking Service object versions
> >> > and dealing with compatibility there, it would be easier for them
> >> > to track the ServiceStatusPayload version which can remain
> >> > relatively stable even if Service is changing to adapt to db/RPC changes.
> >> Not only do they not really care about tracking the Service object
> >> versions, they probably also don't care about what's in that filter list.
> >>
> >> But I think you're getting on the right track as to where this needs
> >> to go. We can integrate the filtering into the versioning of the payload.
> >> But instead of a blacklist, we turn the filter into a white list. If
> >> the underlying object adds a new field that we don't want/care if
> >> people know about, the payload version doesn't have to change. But if
> >> we add something (or if we're changing the existing fields) that we
> >> want to expose, we then assert that we need to update the version of
> >> the payload, so the consumer can look at the payload and say "oh, in
> >> 1.x, now I get ___" and can add the appropriate checks/compat.
> >> Granted with this you can get into rebase nightmares ([1] still
> >> haunts me in my sleep), but I don't see us frantically changing the
> >> exposed fields all too often. This way 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[John Garbutt]

On 24 November 2015 at 15:00, Balázs Gibizer
 wrote:
>> From: Andrew Laski [mailto:and...@lascii.com]
>> Sent: November 24, 2015 15:35
>> On 11/24/15 at 10:26am, Balázs Gibizer wrote:
>> >> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
>> >> Sent: November 23, 2015 22:33
>> >> On 11/23/2015 2:23 PM, Andrew Laski wrote:
>> >> > On 11/23/15 at 04:43pm, Balázs Gibizer wrote:
>> >> >>> From: Andrew Laski [mailto:and...@lascii.com]
>> >> >>> Sent: November 23, 2015 17:03
>> >> >>>
>> >> >>> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
>> >> >>> >
>> >> >>> >
>> >> >>> >On 11/23/2015 5:33 AM, John Garbutt wrote:
>> >> >>> >>On 20 November 2015 at 09:37, Balázs Gibizer
>> >> >>> >> wrote:
>> >> >>> >>>
>> >> >>> >>>
>> >> >> >>
>> >> >>> >>There is a bit I am conflicted/worried about, and thats when we
>> >> >>> >>start including verbatim, DB objects into the notifications. At
>> >> >>> >>least you can now quickly detect if that blob is something
>> >> >>> >>compatible with your current parsing code. My preference is
>> >> >>> >>really to keep the Notifications as a totally separate object
>> >> >>> >>tree, but I am sure there are many cases where that ends up
>> >> >>> >>being seemingly stupid duplicate work. I am not expressing this
>> >> >>> >>well in text form :(
>> >> >>> >Are you saying we don't want to be willy-nilly tossing DB
>> >> >>> >objects across the wire? Yeah that was part of the rug-pulling
>> >> >>> >of just having the payload contain an object. We're
>> >> >>> >automatically tossing everything with the object then, whether
>> >> >>> >or not some of that was supposed to be a secret. We could add
>> >> >>> >some sort of property to the field like
>> >> >>> >dont_put_me_on_the_wire=True (or I guess a
>> >> >>> >notification_ready() function that helps an object sanitize
>> >> >>> >itself?) that the notifications will look at to know if it puts
>> >> >>> >that on the wire-serialized dict, but that's adding a lot more
>> >> >>> >complexity and work to a pile that's already growing rapidly.
>> >> >>>
>> >> >>> I don't want to be tossing db objects across the wire.  But I
>> >> >>> also am not convinced that we should be tossing the current
>> >> >>> objects over the wire either.
>> >> >>> You make the point that there may be things in the object that
>> >> >>> shouldn't be exposed, and I think object version bumps is another
>> >> >>> thing to watch out for.
>> >> >>> So far the only object that has been bumped is Instance but in
>> >> >>> doing so no notifications needed to change.  I think if we just
>> >> >>> put objects into notifications we're coupling the notification
>> >> >>> versions to db or RPC changes unnecessarily.  Some times they'll
>> >> >>> move together but other times, like moving flavor into
>> >> >>> instance_extra, there's no reason to bump notifications.
>> >> >>
>> >> >>
>> >> >> Sanitizing existing versioned objects before putting them to the
>> >> >> wire is not hard to do.
>> >> >> You can see an example of doing it in
>> >> >> https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
>> >> >> cm
>> >> >> L382.
>> >> >> We don't need extra effort to take care of minor version bumps
>> >> >> because that does not break a well written consumer. We do have to
>> >> >> take care of the major version bumps but that is a rare event and
>> >> >> therefore can be handled one by one in a way John suggested, by
>> >> >> keep sending the previous major version for a while too.
>> >> >
>> >> > That review is doing much of what I was suggesting.  There is a
>> >> > separate notification and payload object.  The issue I have is that
>> >> > within the ServiceStatusPayload the raw Service object and version
>> >> > is being dumped, with the filter you point out.  But I don't think
>> >> > that consumers really care about tracking Service object versions
>> >> > and dealing with compatibility there, it would be easier for them
>> >> > to track the ServiceStatusPayload version which can remain
>> >> > relatively stable even if Service is changing to adapt to db/RPC 
>> >> > changes.
>> >> Not only do they not really care about tracking the Service object
>> >> versions, they probably also don't care about what's in that filter list.
>> >>
>> >> But I think you're getting on the right track as to where this needs
>> >> to go. We can integrate the filtering into the versioning of the payload.
>> >> But instead of a blacklist, we turn the filter into a white list. If
>> >> the underlying object adds a new field that we don't want/care if
>> >> people know about, the payload version doesn't have to change. But if
>> >> we add something (or if we're changing the existing fields) that we
>> >> want to expose, we then assert that we need to update the version of
>> >> the payload, so the consumer can look at the payload and say "oh, in
>> >> 1.x, now I get ___" and can add the appropriate checks/compat.
>> >> 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Andrew Laski]

On 11/23/15 at 04:43pm, Balázs Gibizer wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 23, 2015 17:03

On 11/23/15 at 08:54am, Ryan Rossiter wrote:
>
>
>On 11/23/2015 5:33 AM, John Garbutt wrote:
>>On 20 November 2015 at 09:37, Balázs Gibizer
>> wrote:
>>>
>>>

>>

>>There is a bit I am conflicted/worried about, and thats when we start
>>including verbatim, DB objects into the notifications. At least you
>>can now quickly detect if that blob is something compatible with your
>>current parsing code. My preference is really to keep the
>>Notifications as a totally separate object tree, but I am sure there
>>are many cases where that ends up being seemingly stupid duplicate
>>work. I am not expressing this well in text form :(
>Are you saying we don't want to be willy-nilly tossing DB objects
>across the wire? Yeah that was part of the rug-pulling of just having
>the payload contain an object. We're automatically tossing everything
>with the object then, whether or not some of that was supposed to be a
>secret. We could add some sort of property to the field like
>dont_put_me_on_the_wire=True (or I guess a notification_ready()
>function that helps an object sanitize itself?) that the notifications
>will look at to know if it puts that on the wire-serialized dict, but
>that's adding a lot more complexity and work to a pile that's already
>growing rapidly.

I don't want to be tossing db objects across the wire.  But I also am not
convinced that we should be tossing the current objects over the wire either.
You make the point that there may be things in the object that shouldn't be
exposed, and I think object version bumps is another thing to watch out for.
So far the only object that has been bumped is Instance but in doing so no
notifications needed to change.  I think if we just put objects into
notifications we're coupling the notification versions to db or RPC changes
unnecessarily.  Some times they'll move together but other times, like
moving flavor into instance_extra, there's no reason to bump notifications.



Sanitizing existing versioned objects before putting them to the wire is not 
hard to do.
You can see an example of doing it in
https://review.openstack.org/#/c/245678/8/nova/objects/service.py,cm L382.
We don't need extra effort to take care of minor version bumps because that 
does not
break a well written consumer. We do have to take care of the major version 
bumps
but that is a rare event and therefore can be handled one by one in a way John
suggested, by keep sending the previous major version for a while too.


That review is doing much of what I was suggesting.  There is a separate 
notification and payload object.  The issue I have is that within the 
ServiceStatusPayload the raw Service object and version is being dumped, 
with the filter you point out.  But I don't think that consumers really 
care about tracking Service object versions and dealing with 
compatibility there, it would be easier for them to track the 
ServiceStatusPayload version which can remain relatively stable even if 
Service is changing to adapt to db/RPC changes.






Note that I'm not against using objects for notifications and versioning, but I
picture having something like an InstanceNotification object which can
handle converting an Instance object into a suitable notification
version/format.


Converting between two object models is doable what I'm afraid of is that it
means we have to maintain two object models. Which also means If a new
field added to an internal object which needs to be in the related notification
then we have to add them in the notification model as well. So one field but
two places. That seems to be a duplication for me. I think we have to guess 
which
case will be more frequent
a) adding a new field to an internal object that does not need to be added to 
the
related notification. In this case having two separate models is better.
OR b) adding a new field to an internal object that needs to be added to the 
related
notification. In this case having one model and a filtering mechanism is better.



>>
>>Thanks,
>>John
>>
>>>Cheers,
>>>Gibi
--

Thanks,

Matt Riedemann
>>>
>>>___
___
>>>OpenStack Development Mailing List (not for usage questions)
>>>Unsubscribe: OpenStack-dev-
requ...@lists.openstack.org?subject:unsubscribe
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
__
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-
requ...@lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>--
>Thanks,
>
>Ryan Rossiter (rlrossit)
>
>
>_
_
>OpenStack Development Mailing List (not for usage 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Alexis Lee]

gord chung said on Fri, Nov 20, 2015 at 01:32:02PM -0500:
> On 20/11/15 11:33 AM, Alexis Lee wrote:
> >why would a producer spit out non-useful datapoints? If no-one cares
> >or will ever care, it simply shouldn't be included.
> 
> ... right now the
> producer is just sending out a grab bag of data that it thinks is
> important but doesn't define who the audience is. ...
>
> ... whoever the producer of notifications is, it should know it's
> audience.

Here I, with cdent, have to disagree. The criterion has to be
"potentially useful to someone", rather than tailoring production to the
known audience.

> >The problem is knowing what each consumer thinks is interesting and that
> >isn't something that can be handled by the producer.

^ this is important.

> the notification consumption service in ceilometer is essentially
> just a pipeline that normalises select incoming notifications into a
> data model(s) and pushes that model to whoever wants it (a known
> consumer is the storage service but it's configurable to allow other
> consumers).

It sounds like Ceilometer is performing a mapping function then and has
to be responsible for knowing how to do that. IE Ceilometer has to
express an opinion on relevance and presentation. This is inevitably
going to be a pain in the butt to maintain, but it's Ceilometer's job,
not the services, since it's Ceilometer trying to express an opinion.


Alexis (lxsli)
-- 
Nova developer, Hewlett-Packard Limited.
Registered Office: Cain Road, Bracknell, Berkshire RG12 1HN.
Registered Number: 00690597 England
VAT number: GB 314 1496 79

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[John Garbutt]

On 20 November 2015 at 09:37, Balázs Gibizer
 wrote:
>> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
>> Sent: November 19, 2015 23:29
>> On 11/19/2015 4:05 PM, Ryan Rossiter wrote:
>> > Reading through [1] I started getting worries in the back of my head
>> > about versioning these notifications. The main concern being how can
>> > the consumer know about the versions and what's different between
>> them?
>> > Because these versioned notification payloads hold live nova objects,
>> > there can be a lot of rug-pulling going on underneath these
>> > notifications. If the payload doesn't pin itself to a certain level of
>> > the object, a consumer can never be guaranteed the version of the
>> > payload's object they will be receiving. I ran through a few of the
>> > scenarios about irregular versions in the notifications subteam
>> > meeting on Tuesday [2].
>> >
>> > My question is do we care about the consumer? Or is it a case of
>> > "the consumer is always right" so we need to make sure we hand them
>> > super consistent, well-defined blobs across the wire? Consumers will
>> > have no idea of nova object internals, unless they feel like `python
>> > -c import nova`. I do think we get one piece of help from o.vo though.
>> > When the object is serialized, it hands the version with the object.
>> > So consumers can look at the object and say "oh, I got 1.4 I know what
>> > to do with this". But... they will have to implement their own compat
>> > logic. Everyone will have to implement their own compat logic.
>> >
>> > We could expose a new API for getting the schema for a specific
>> > version of a notification, so a consumer will know what they're
>> > getting with their notifications. But I think that made mriedem
>> > nauseous. We could make an oslo library that stuffs a shim in between
>> > o.vo and nova's notifications to help out with compat/versioning, but
>> > that sounds like a lot of work, especially because the end goal is still 
>> > not
>> clearly defined.
>>
>> The term is 'nauseated'. To be nauseous, you nauseate others. Which I might
>> do from time to time.
>>
>> Sorry, I'm channeling one of my wives' pet peeves because I say the same
>> thing. Sometimes just to annoy her.
>>
>> But yeah, I was made physically ill from that idea.
>>
>> >
>> > Thoughts?
>> >
>> > [1] https://review.openstack.org/#/c/247024
>> > [2]
>> > http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-
>> alt/%23ope
>> > nstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29
>> >
>> >
>>
>> One idea I had was (and maybe this would be in a separate library like you're
>> talking about, i.e. nova-notifications), but if nova emits the notification 
>> with
>> the version and the consumer calls into the library that translates it into a
>> version they want, then get that transformed thing back.
>>
>> However, how does the consumer know what they want or what they can
>> handle? Do they pin a version in configuration somewhere? I could see
>> something like how we have upgrade levels pinned in nova so newer
>> conductor can backlevel things for older computes.
>>
>> I was also thinking about microversions and novaclient, but in that case
>> novaclient knows what max microversion it can handle and only requests up
>> to that version, and then nova-api handles the compat work. In the case of
>> notifications, nova is just broadcasting those so it's not doing any compat
>> work, but it's the only thing that knows *how* to do the compat work...
>>
>> So yeah, I'm lost.
>
> Minor version change shall not cause any problem for the consumer as the 
> payload is backward compatible between minor versions. So if the consumer 
> does not need the new field then he/she does not need to change anything in 
> his/her parser. As far as I know we had a single major object version in nova 
> so far so this is not a frequent event.  In case of major change I think we 
> can offer version pinning from nova via configuration as a future step.
>
> The library idea has the problem that it would be python lib and consumers 
> can be in any language. For me lib would be used for compat code but as I 
> mentioned above incompatibility is not that frequent. In the other hand 
> discoverability of notifications are more important for me. For that I can 
> suggest providing notification samples as a first step so the consumer can 
> see in the source tree what notifications are provided by the nova. As a 
> natural next step would be to provide not just samples but schemas for the 
> notifications. I haven't looked it too deep but I feel if we provide json 
> schema for the notifications then the consumer can generate an object model 
> from that schema without too much effort. It might not help directly with 
> backporting the payload but at least automate things around it. The json 
> schema has the benefit that it is language independent too.
>

So, on re-reading, I think we are missing some 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[gord chung]

On 20/11/2015 5:12 PM, Chris Dent wrote:

On Fri, 20 Nov 2015, gord chung wrote:

i think a lot of the complexity we have in versioning is that the 
projects are too silo'd. i think some of the versioning issues would 
be irrelevant if the producer knew it's consumers before sending 
rather than producers just tossing out a chunk of data (versioned 
schema or not) and considering their job complete once it leaves it's 
own walls. the producer doesn't necessarily have to be the individual 
project teams but whoever the producer of notifications is, it should 
know it's audience.


To me this is entirely contrary to the point of having notifications
as a generic concept in the OpenStack environment. To me the point is
to ensure that it is possible for the audience to be and do _anything_.


you can still do anything and add anything to notifications but you also 
know that "attributeA/B/C are probably useful to consumerX; 
attributeD/E/F are related and all the attributes can be used by anyone."




We've become so accustomed to some of the misfeatures in the messaging
architecture that we've lost track of the fact that it could be an
event pool on which we have diverse listeners that the producers have
no requirement to know anything about. We could have nova-compute 
spinning

along shouting "I made a VM. I made another VM. Hey, I made another
VM" and "This VM is hot. Halp, I am oversubscribed." All sorts of
tools and devices need to be able to hear that stuff and choose for
themselves what they might do with it.

(This is similar to the reason we have well-formed HTTP APIs: It is so
we can have unexpected clients that do unexpected things.)


i actually view notifications different from 'well-formed HTTP APIs' as 
the initiator is not the consumer but the producer but i agree with 
"unexpected clients that do unexpected things."




It is certainly the case that if we're going to have schematized
and versioned notifications it is critical that the schema are
discoverable in a language independent fashion.

Sometimes it is hard, though, to be convinced that such formalisms are
quite what we really need. From the consumers end the concern is "do
you have these several keys that I care about?" and, as has been said,
the rest is noise. It sounds like microversioned notifications which
almost never version on the major axis might be able to provide this.

We can't allow the introduction of either the formalisms or
discoverability thereof to grant license to change stuff willy nilly.


agree, following this makes versioning pretty much moot. it's the major 
version changes i'm thinking of. how does producer know not to remove 
attributeX? how does consumer know attributeX is now attributeX.Y.Z if 
the name/location is different?



Nor should we be building formalisms that are defenses against an
architecture that's sub-optimal. We need to evolve the formalisms and
the architecture toward the ideal.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
gord

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[gord chung]

given John's reply, i think we should start a new thread? apologies for 
hijacking log thread.


On 23/11/2015 5:01 AM, Alexis Lee wrote:

gord chung said on Fri, Nov 20, 2015 at 01:32:02PM -0500:

On 20/11/15 11:33 AM, Alexis Lee wrote:

why would a producer spit out non-useful datapoints? If no-one cares
or will ever care, it simply shouldn't be included.

... right now the
producer is just sending out a grab bag of data that it thinks is
important but doesn't define who the audience is. ...

... whoever the producer of notifications is, it should know it's
audience.

Here I, with cdent, have to disagree. The criterion has to be
"potentially useful to someone", rather than tailoring production to the
known audience.


so i'm not suggesting producers need to know the entire audience or for 
it to be tailored to a single consumer. i do think whatever we put out 
as a notification should have at least one potential recipient in mind 
(see: useful data to someone [not a hypothetical]). i'm all for 
flexibility and multiple/any consumers, but right now, to be honest, our 
notification criteria seems to be similar to "that guy outside your 
local mall yelling stuff and hoping he makes a convert."


this is just me trying to view it from another approach -- trying to vet 
our current process :)





The problem is knowing what each consumer thinks is interesting and that
isn't something that can be handled by the producer.

^ this is important.


the notification consumption service in ceilometer is essentially
just a pipeline that normalises select incoming notifications into a
data model(s) and pushes that model to whoever wants it (a known
consumer is the storage service but it's configurable to allow other
consumers).

It sounds like Ceilometer is performing a mapping function then and has
to be responsible for knowing how to do that. IE Ceilometer has to
express an opinion on relevance and presentation. This is inevitably
going to be a pain in the butt to maintain, but it's Ceilometer's job,
not the services, since it's Ceilometer trying to express an opinion.
fair enough, i'm just re-raising common feedback i receive when projects 
want to add metrics for ceilometer to collect: 'why do i need to edit 
ceilometer, it's my data.' so there is obviously a divide in the 
community as to whose responsibility it is for maintaining the data.


--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Ryan Rossiter]

On 11/23/2015 5:33 AM, John Garbutt wrote:

On 20 November 2015 at 09:37, Balázs Gibizer
 wrote:


Minor version change shall not cause any problem for the consumer as the 
payload is backward compatible between minor versions. So if the consumer does 
not need the new field then he/she does not need to change anything in his/her 
parser. As far as I know we had a single major object version in nova so far so 
this is not a frequent event.  In case of major change I think we can offer 
version pinning from nova via configuration as a future step.

The library idea has the problem that it would be python lib and consumers can 
be in any language. For me lib would be used for compat code but as I mentioned 
above incompatibility is not that frequent. In the other hand discoverability 
of notifications are more important for me. For that I can suggest providing 
notification samples as a first step so the consumer can see in the source tree 
what notifications are provided by the nova. As a natural next step would be to 
provide not just samples but schemas for the notifications. I haven't looked it 
too deep but I feel if we provide json schema for the notifications then the 
consumer can generate an object model from that schema without too much effort. 
It might not help directly with backporting the payload but at least automate 
things around it. The json schema has the benefit that it is language 
independent too.


So, on re-reading, I think we are missing some of the context from the
summit session in this spec:
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/versioned-notification-api.html

The aim of notifications is to focus more on structured logging, for
the operator.

It is not trying to provide an end-user async API, such as:
https://etherpad.openstack.org/p/liberty-cross-project-user-notifications

As Gibi mentions, the idea was to add a version, so you can tell when
the content has changed, generally in a backwards compatible way. The
other big parts, like properly testing the notifications to ensure
they stay compatible.

It uses o.vo, with the intent we avoid a major version bump. Should we
need one, its likely to end up with a similar transition phase to the
current un-versioned->versioned transition phase (effectively the
deployer decides which major versions will be emitted at any one
time).

The interface is a stable JSON structure, with some handy version and
identification metadata. It happens to use o.vo to generate that
format, and we happen to have python code to read that format, but
thats not the aim.
This is a stupid question but... who's the object versioning designed 
for? Is it for the consumer, so they know what they're getting? Or is it 
for Nova, so it has a contract to fill, instead of tossing a big bag of 
dicts across the wire? Those sound the same, but, suddenly in my head, 
they aren't.


In the case of the former, if we add a new field, we bump the version, 
so the consumer knows when it can start looking for new things when it 
gets that version.


In the case of the latter, the fields defined in version 1.0 is the only 
guaranteed law until the end of the universe. Every field added after 
1.0 needs to be nullable to allow backversioned computes to still work, 
right? We're still handcuffed on what we can actually truly guarantee 
the consumer.


There is a bit I am conflicted/worried about, and thats when we start
including verbatim, DB objects into the notifications. At least you
can now quickly detect if that blob is something compatible with your
current parsing code. My preference is really to keep the
Notifications as a totally separate object tree, but I am sure there
are many cases where that ends up being seemingly stupid duplicate
work. I am not expressing this well in text form :(
Are you saying we don't want to be willy-nilly tossing DB objects across 
the wire? Yeah that was part of the rug-pulling of just having the 
payload contain an object. We're automatically tossing everything with 
the object then, whether or not some of that was supposed to be a 
secret. We could add some sort of property to the field like 
dont_put_me_on_the_wire=True (or I guess a notification_ready() function 
that helps an object sanitize itself?) that the notifications will look 
at to know if it puts that on the wire-serialized dict, but that's 
adding a lot more complexity and work to a pile that's already growing 
rapidly.


Thanks,
John


Cheers,
Gibi

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Kevin L. Mitchell]

On Mon, 2015-11-23 at 10:01 +, Alexis Lee wrote:
> gord chung said on Fri, Nov 20, 2015 at 01:32:02PM -0500:
> > On 20/11/15 11:33 AM, Alexis Lee wrote:
> > >why would a producer spit out non-useful datapoints? If no-one cares
> > >or will ever care, it simply shouldn't be included.
> > 
> > ... right now the
> > producer is just sending out a grab bag of data that it thinks is
> > important but doesn't define who the audience is. ...
> >
> > ... whoever the producer of notifications is, it should know it's
> > audience.
> 
> Here I, with cdent, have to disagree. The criterion has to be
> "potentially useful to someone", rather than tailoring production to the
> known audience.
> 
> > >The problem is knowing what each consumer thinks is interesting and that
> > >isn't something that can be handled by the producer.
> 
> ^ this is important.

I think there is a minor disconnect here; when Gord talks about
tailoring to a specific audience, there seems to be a jump to the
conclusion that the audience must be ceilometer.  I think Gord is
instead suggesting what you could think of as an audience *class*—this
notification is intended for metrics consumers, this one is intended for
log monitors, etc.  That kind of data could be included in the actual
notification, so you still have a single notification stream.  (I could
envision later refinements allowing filtering, maybe by using multiple
queues…)
-- 
Kevin L. Mitchell 
Rackspace


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
> Sent: November 23, 2015 15:54
> On 11/23/2015 5:33 AM, John Garbutt wrote:
> > On 20 November 2015 at 09:37, Balázs Gibizer
> >  wrote:
> >> 
> >> Minor version change shall not cause any problem for the consumer as
> the payload is backward compatible between minor versions. So if the
> consumer does not need the new field then he/she does not need to
> change anything in his/her parser. As far as I know we had a single major
> object version in nova so far so this is not a frequent event.  In case of 
> major
> change I think we can offer version pinning from nova via configuration as a
> future step.
> >>
> >> The library idea has the problem that it would be python lib and
> consumers can be in any language. For me lib would be used for compat code
> but as I mentioned above incompatibility is not that frequent. In the other
> hand discoverability of notifications are more important for me. For that I 
> can
> suggest providing notification samples as a first step so the consumer can see
> in the source tree what notifications are provided by the nova. As a natural
> next step would be to provide not just samples but schemas for the
> notifications. I haven't looked it too deep but I feel if we provide json 
> schema
> for the notifications then the consumer can generate an object model from
> that schema without too much effort. It might not help directly with
> backporting the payload but at least automate things around it. The json
> schema has the benefit that it is language independent too.
> >>
> > So, on re-reading, I think we are missing some of the context from the
> > summit session in this spec:
> > http://specs.openstack.org/openstack/nova-
> specs/specs/mitaka/approved/
> > versioned-notification-api.html
> >
> > The aim of notifications is to focus more on structured logging, for
> > the operator.
> >
> > It is not trying to provide an end-user async API, such as:
> > https://etherpad.openstack.org/p/liberty-cross-project-user-notificati
> > ons
> >
> > As Gibi mentions, the idea was to add a version, so you can tell when
> > the content has changed, generally in a backwards compatible way. The
> > other big parts, like properly testing the notifications to ensure
> > they stay compatible.
> >
> > It uses o.vo, with the intent we avoid a major version bump. Should we
> > need one, its likely to end up with a similar transition phase to the
> > current un-versioned->versioned transition phase (effectively the
> > deployer decides which major versions will be emitted at any one
> > time).
> >
> > The interface is a stable JSON structure, with some handy version and
> > identification metadata. It happens to use o.vo to generate that
> > format, and we happen to have python code to read that format, but
> > thats not the aim.
> This is a stupid question but... who's the object versioning designed for? Is 
> it
> for the consumer, so they know what they're getting? Or is it for Nova, so it
> has a contract to fill, instead of tossing a big bag of dicts across the wire?
> Those sound the same, but, suddenly in my head, they aren't.
> 
> In the case of the former, if we add a new field, we bump the version, so the
> consumer knows when it can start looking for new things when it gets that
> version.
Currently if we add a field we bump the minor version, so if the consumer want 
to look into the new content he or she can detect that there is new content.

> 
> In the case of the latter, the fields defined in version 1.0 is the only
> guaranteed law until the end of the universe. Every field added after
> 1.0 needs to be nullable to allow backversioned computes to still work, right?
> We're still handcuffed on what we can actually truly guarantee the
> consumer.
I don't fully understand this comment. If the consumer prepared to consume a 
payload with two fields as version 1.0, when we add a new field and bump the 
version to 1.1 the consumer's old parser can still work without change (if it 
is written well) and it will found all the data the 1.0 view needs.  If we 
decides to remove a field from the payload and therefore bump the version to 
2.0 then the consumer's parser will fail so we have to avoid that. In this case 
it is nova responsibility to offer a solution and I agree with John that a 
transition period with keeping the old 1.1 version will work. 

> >
> > There is a bit I am conflicted/worried about, and thats when we start
> > including verbatim, DB objects into the notifications. At least you
> > can now quickly detect if that blob is something compatible with your
> > current parsing code. My preference is really to keep the
> > Notifications as a totally separate object tree, but I am sure there
> > are many cases where that ends up being seemingly stupid duplicate
> > work. I am not expressing this well in text form :(
> Are you saying we don't want to be willy-nilly tossing DB objects 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> From: Andrew Laski [mailto:and...@lascii.com]
> Sent: November 23, 2015 17:03
> 
> On 11/23/15 at 08:54am, Ryan Rossiter wrote:
> >
> >
> >On 11/23/2015 5:33 AM, John Garbutt wrote:
> >>On 20 November 2015 at 09:37, Balázs Gibizer
> >> wrote:
> >>>
> >>>Minor version change shall not cause any problem for the consumer as
> the payload is backward compatible between minor versions. So if the
> consumer does not need the new field then he/she does not need to
> change anything in his/her parser. As far as I know we had a single major
> object version in nova so far so this is not a frequent event.  In case of 
> major
> change I think we can offer version pinning from nova via configuration as a
> future step.
> >>>
> >>>The library idea has the problem that it would be python lib and
> consumers can be in any language. For me lib would be used for compat code
> but as I mentioned above incompatibility is not that frequent. In the other
> hand discoverability of notifications are more important for me. For that I 
> can
> suggest providing notification samples as a first step so the consumer can see
> in the source tree what notifications are provided by the nova. As a natural
> next step would be to provide not just samples but schemas for the
> notifications. I haven't looked it too deep but I feel if we provide json 
> schema
> for the notifications then the consumer can generate an object model from
> that schema without too much effort. It might not help directly with
> backporting the payload but at least automate things around it. The json
> schema has the benefit that it is language independent too.
> >>>
> >>So, on re-reading, I think we are missing some of the context from the
> >>summit session in this spec:
> >>http://specs.openstack.org/openstack/nova-
> specs/specs/mitaka/approved/
> >>versioned-notification-api.html
> >>
> >>The aim of notifications is to focus more on structured logging, for
> >>the operator.
> >>
> >>It is not trying to provide an end-user async API, such as:
> >>https://etherpad.openstack.org/p/liberty-cross-project-user-notificati
> >>ons
> >>
> >>As Gibi mentions, the idea was to add a version, so you can tell when
> >>the content has changed, generally in a backwards compatible way. The
> >>other big parts, like properly testing the notifications to ensure
> >>they stay compatible.
> >>
> >>It uses o.vo, with the intent we avoid a major version bump. Should we
> >>need one, its likely to end up with a similar transition phase to the
> >>current un-versioned->versioned transition phase (effectively the
> >>deployer decides which major versions will be emitted at any one
> >>time).
> >>
> >>The interface is a stable JSON structure, with some handy version and
> >>identification metadata. It happens to use o.vo to generate that
> >>format, and we happen to have python code to read that format, but
> >>thats not the aim.
> >This is a stupid question but... who's the object versioning designed
> >for? Is it for the consumer, so they know what they're getting? Or is
> >it for Nova, so it has a contract to fill, instead of tossing a big bag
> >of dicts across the wire? Those sound the same, but, suddenly in my
> >head, they aren't.
> >
> >In the case of the former, if we add a new field, we bump the version,
> >so the consumer knows when it can start looking for new things when it
> >gets that version.
> >
> >In the case of the latter, the fields defined in version 1.0 is the
> >only guaranteed law until the end of the universe. Every field added
> >after 1.0 needs to be nullable to allow backversioned computes to still
> >work, right? We're still handcuffed on what we can actually truly
> >guarantee the consumer.
> >>
> >>There is a bit I am conflicted/worried about, and thats when we start
> >>including verbatim, DB objects into the notifications. At least you
> >>can now quickly detect if that blob is something compatible with your
> >>current parsing code. My preference is really to keep the
> >>Notifications as a totally separate object tree, but I am sure there
> >>are many cases where that ends up being seemingly stupid duplicate
> >>work. I am not expressing this well in text form :(
> >Are you saying we don't want to be willy-nilly tossing DB objects
> >across the wire? Yeah that was part of the rug-pulling of just having
> >the payload contain an object. We're automatically tossing everything
> >with the object then, whether or not some of that was supposed to be a
> >secret. We could add some sort of property to the field like
> >dont_put_me_on_the_wire=True (or I guess a notification_ready()
> >function that helps an object sanitize itself?) that the notifications
> >will look at to know if it puts that on the wire-serialized dict, but
> >that's adding a lot more complexity and work to a pile that's already
> >growing rapidly.
> 
> I don't want to be tossing db objects across the wire.  But I also am not
> convinced that we should be 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Andrew Laski]

On 11/23/15 at 08:54am, Ryan Rossiter wrote:



On 11/23/2015 5:33 AM, John Garbutt wrote:

On 20 November 2015 at 09:37, Balázs Gibizer
 wrote:


Minor version change shall not cause any problem for the consumer as the 
payload is backward compatible between minor versions. So if the consumer does 
not need the new field then he/she does not need to change anything in his/her 
parser. As far as I know we had a single major object version in nova so far so 
this is not a frequent event.  In case of major change I think we can offer 
version pinning from nova via configuration as a future step.

The library idea has the problem that it would be python lib and consumers can 
be in any language. For me lib would be used for compat code but as I mentioned 
above incompatibility is not that frequent. In the other hand discoverability 
of notifications are more important for me. For that I can suggest providing 
notification samples as a first step so the consumer can see in the source tree 
what notifications are provided by the nova. As a natural next step would be to 
provide not just samples but schemas for the notifications. I haven't looked it 
too deep but I feel if we provide json schema for the notifications then the 
consumer can generate an object model from that schema without too much effort. 
It might not help directly with backporting the payload but at least automate 
things around it. The json schema has the benefit that it is language 
independent too.


So, on re-reading, I think we are missing some of the context from the
summit session in this spec:
http://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/versioned-notification-api.html

The aim of notifications is to focus more on structured logging, for
the operator.

It is not trying to provide an end-user async API, such as:
https://etherpad.openstack.org/p/liberty-cross-project-user-notifications

As Gibi mentions, the idea was to add a version, so you can tell when
the content has changed, generally in a backwards compatible way. The
other big parts, like properly testing the notifications to ensure
they stay compatible.

It uses o.vo, with the intent we avoid a major version bump. Should we
need one, its likely to end up with a similar transition phase to the
current un-versioned->versioned transition phase (effectively the
deployer decides which major versions will be emitted at any one
time).

The interface is a stable JSON structure, with some handy version and
identification metadata. It happens to use o.vo to generate that
format, and we happen to have python code to read that format, but
thats not the aim.
This is a stupid question but... who's the object versioning designed 
for? Is it for the consumer, so they know what they're getting? Or is 
it for Nova, so it has a contract to fill, instead of tossing a big 
bag of dicts across the wire? Those sound the same, but, suddenly in 
my head, they aren't.


In the case of the former, if we add a new field, we bump the 
version, so the consumer knows when it can start looking for new 
things when it gets that version.


In the case of the latter, the fields defined in version 1.0 is the 
only guaranteed law until the end of the universe. Every field added 
after 1.0 needs to be nullable to allow backversioned computes to 
still work, right? We're still handcuffed on what we can actually 
truly guarantee the consumer.


There is a bit I am conflicted/worried about, and thats when we start
including verbatim, DB objects into the notifications. At least you
can now quickly detect if that blob is something compatible with your
current parsing code. My preference is really to keep the
Notifications as a totally separate object tree, but I am sure there
are many cases where that ends up being seemingly stupid duplicate
work. I am not expressing this well in text form :(
Are you saying we don't want to be willy-nilly tossing DB objects 
across the wire? Yeah that was part of the rug-pulling of just having 
the payload contain an object. We're automatically tossing everything 
with the object then, whether or not some of that was supposed to be 
a secret. We could add some sort of property to the field like 
dont_put_me_on_the_wire=True (or I guess a notification_ready() 
function that helps an object sanitize itself?) that the 
notifications will look at to know if it puts that on the 
wire-serialized dict, but that's adding a lot more complexity and 
work to a pile that's already growing rapidly.


I don't want to be tossing db objects across the wire.  But I also am 
not convinced that we should be tossing the current objects over the 
wire either.  You make the point that there may be things in the object 
that shouldn't be exposed, and I think object version bumps is another 
thing to watch out for.  So far the only object that has been bumped is 
Instance but in doing so no notifications needed to change.  I think if 
we just put objects 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Ryan Rossiter]

On 11/23/2015 2:23 PM, Andrew Laski wrote:

On 11/23/15 at 04:43pm, Balázs Gibizer wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 23, 2015 17:03

On 11/23/15 at 08:54am, Ryan Rossiter wrote:
>
>
>On 11/23/2015 5:33 AM, John Garbutt wrote:
>>On 20 November 2015 at 09:37, Balázs Gibizer
>> wrote:
>>>
>>>

>>

>>There is a bit I am conflicted/worried about, and thats when we start
>>including verbatim, DB objects into the notifications. At least you
>>can now quickly detect if that blob is something compatible with your
>>current parsing code. My preference is really to keep the
>>Notifications as a totally separate object tree, but I am sure there
>>are many cases where that ends up being seemingly stupid duplicate
>>work. I am not expressing this well in text form :(
>Are you saying we don't want to be willy-nilly tossing DB objects
>across the wire? Yeah that was part of the rug-pulling of just having
>the payload contain an object. We're automatically tossing everything
>with the object then, whether or not some of that was supposed to be a
>secret. We could add some sort of property to the field like
>dont_put_me_on_the_wire=True (or I guess a notification_ready()
>function that helps an object sanitize itself?) that the notifications
>will look at to know if it puts that on the wire-serialized dict, but
>that's adding a lot more complexity and work to a pile that's already
>growing rapidly.

I don't want to be tossing db objects across the wire.  But I also 
am not
convinced that we should be tossing the current objects over the 
wire either.
You make the point that there may be things in the object that 
shouldn't be
exposed, and I think object version bumps is another thing to watch 
out for.
So far the only object that has been bumped is Instance but in doing 
so no

notifications needed to change.  I think if we just put objects into
notifications we're coupling the notification versions to db or RPC 
changes

unnecessarily.  Some times they'll move together but other times, like
moving flavor into instance_extra, there's no reason to bump 
notifications.



Sanitizing existing versioned objects before putting them to the wire 
is not hard to do.

You can see an example of doing it in
https://review.openstack.org/#/c/245678/8/nova/objects/service.py,cm 
L382.
We don't need extra effort to take care of minor version bumps 
because that does not
break a well written consumer. We do have to take care of the major 
version bumps
but that is a rare event and therefore can be handled one by one in a 
way John

suggested, by keep sending the previous major version for a while too.


That review is doing much of what I was suggesting.  There is a 
separate notification and payload object.  The issue I have is that 
within the ServiceStatusPayload the raw Service object and version is 
being dumped, with the filter you point out.  But I don't think that 
consumers really care about tracking Service object versions and 
dealing with compatibility there, it would be easier for them to track 
the ServiceStatusPayload version which can remain relatively stable 
even if Service is changing to adapt to db/RPC changes.
Not only do they not really care about tracking the Service object 
versions, they probably also don't care about what's in that filter list.


But I think you're getting on the right track as to where this needs to 
go. We can integrate the filtering into the versioning of the payload. 
But instead of a blacklist, we turn the filter into a white list. If the 
underlying object adds a new field that we don't want/care if people 
know about, the payload version doesn't have to change. But if we add 
something (or if we're changing the existing fields) that we want to 
expose, we then assert that we need to update the version of the 
payload, so the consumer can look at the payload and say "oh, in 1.x, 
now I get ___" and can add the appropriate checks/compat. Granted 
with this you can get into rebase nightmares ([1] still haunts me in my 
sleep), but I don't see us frantically changing the exposed fields all 
too often. This way gives us some form of pseudo-pinning of the 
subobject. Heck, in this method, we could even pass the whitelist on the 
wire right? That way we tell the consumer explicitly what's available to 
them (kinda like a fake schema).


I think I can whip a PoC up for this (including the tests, since I'm so 
intimately familiar with them now that I'm THE nova-objects guy) if we 
want to see where this goes.




[1] https://review.openstack.org/#/c/198730/

--
Thanks,

Ryan Rossiter (rlrossit)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Balázs Gibizer]

> From: Matt Riedemann [mailto:mrie...@linux.vnet.ibm.com]
> Sent: November 19, 2015 23:29
> On 11/19/2015 4:05 PM, Ryan Rossiter wrote:
> > Reading through [1] I started getting worries in the back of my head
> > about versioning these notifications. The main concern being how can
> > the consumer know about the versions and what's different between
> them?
> > Because these versioned notification payloads hold live nova objects,
> > there can be a lot of rug-pulling going on underneath these
> > notifications. If the payload doesn't pin itself to a certain level of
> > the object, a consumer can never be guaranteed the version of the
> > payload's object they will be receiving. I ran through a few of the
> > scenarios about irregular versions in the notifications subteam
> > meeting on Tuesday [2].
> >
> > My question is do we care about the consumer? Or is it a case of
> > "the consumer is always right" so we need to make sure we hand them
> > super consistent, well-defined blobs across the wire? Consumers will
> > have no idea of nova object internals, unless they feel like `python
> > -c import nova`. I do think we get one piece of help from o.vo though.
> > When the object is serialized, it hands the version with the object.
> > So consumers can look at the object and say "oh, I got 1.4 I know what
> > to do with this". But... they will have to implement their own compat
> > logic. Everyone will have to implement their own compat logic.
> >
> > We could expose a new API for getting the schema for a specific
> > version of a notification, so a consumer will know what they're
> > getting with their notifications. But I think that made mriedem
> > nauseous. We could make an oslo library that stuffs a shim in between
> > o.vo and nova's notifications to help out with compat/versioning, but
> > that sounds like a lot of work, especially because the end goal is still not
> clearly defined.
> 
> The term is 'nauseated'. To be nauseous, you nauseate others. Which I might
> do from time to time.
> 
> Sorry, I'm channeling one of my wives' pet peeves because I say the same
> thing. Sometimes just to annoy her.
> 
> But yeah, I was made physically ill from that idea.
> 
> >
> > Thoughts?
> >
> > [1] https://review.openstack.org/#/c/247024
> > [2]
> > http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-
> alt/%23ope
> > nstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29
> >
> >
> 
> One idea I had was (and maybe this would be in a separate library like you're
> talking about, i.e. nova-notifications), but if nova emits the notification 
> with
> the version and the consumer calls into the library that translates it into a
> version they want, then get that transformed thing back.
> 
> However, how does the consumer know what they want or what they can
> handle? Do they pin a version in configuration somewhere? I could see
> something like how we have upgrade levels pinned in nova so newer
> conductor can backlevel things for older computes.
> 
> I was also thinking about microversions and novaclient, but in that case
> novaclient knows what max microversion it can handle and only requests up
> to that version, and then nova-api handles the compat work. In the case of
> notifications, nova is just broadcasting those so it's not doing any compat
> work, but it's the only thing that knows *how* to do the compat work...
> 
> So yeah, I'm lost.

Minor version change shall not cause any problem for the consumer as the 
payload is backward compatible between minor versions. So if the consumer does 
not need the new field then he/she does not need to change anything in his/her 
parser. As far as I know we had a single major object version in nova so far so 
this is not a frequent event.  In case of major change I think we can offer 
version pinning from nova via configuration as a future step.

The library idea has the problem that it would be python lib and consumers can 
be in any language. For me lib would be used for compat code but as I mentioned 
above incompatibility is not that frequent. In the other hand discoverability 
of notifications are more important for me. For that I can suggest providing 
notification samples as a first step so the consumer can see in the source tree 
what notifications are provided by the nova. As a natural next step would be to 
provide not just samples but schemas for the notifications. I haven't looked it 
too deep but I feel if we provide json schema for the notifications then the 
consumer can generate an object model from that schema without too much effort. 
It might not help directly with backporting the payload but at least automate 
things around it. The json schema has the benefit that it is language 
independent too.

Cheers,
Gibi
> 
> --
> 
> Thanks,
> 
> Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Alexis Lee]

gord chung said on Thu, Nov 19, 2015 at 11:59:33PM -0500:
> just to clarify, the idea doesn't involve tailoring the notification
> payload to ceilometer, just that if a producer is producing a
> notification it knows contains a useful datapoint, the producer
> should tell someone explicitly 'this datapoint exists'.

I know very little about Nova notifications or Ceilometer, so stepping
wildly into the unknown here but... why would a producer spit out
non-useful datapoints? If no-one cares or will ever care, it simply
shouldn't be included.

The problem is knowing what each consumer thinks is interesting and that
isn't something that can be handled by the producer. If Ceilometer is
just a pipeline that has no opinion on what's relevant and what isn't,
that's a special case easily implemented by an identity function.


Alexis (lxsli)
-- 
Nova developer, Hewlett-Packard Limited.
Registered Office: Cain Road, Bracknell, Berkshire RG12 1HN.
Registered Number: 00690597 England
VAT number: GB 314 1496 79

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[gord chung]

On 20/11/15 11:33 AM, Alexis Lee wrote:

gord chung said on Thu, Nov 19, 2015 at 11:59:33PM -0500:

just to clarify, the idea doesn't involve tailoring the notification
payload to ceilometer, just that if a producer is producing a
notification it knows contains a useful datapoint, the producer
should tell someone explicitly 'this datapoint exists'.

I know very little about Nova notifications or Ceilometer, so stepping
wildly into the unknown here but... why would a producer spit out
non-useful datapoints? If no-one cares or will ever care, it simply
shouldn't be included.

fully agree.

it seems like even before addressing versioning, that the notification 
paradigm itself should be discussed. right now the producer is just 
sending out a grab bag of data that it thinks is important but doesn't 
define who the audience is. while that makes it extremely flexible so 
that anyone can consume the message, it also guarantees nothing (not 
even that it's being consumed). you can version a payload or make a 
schema accessible as much as you like but if no one is listening or the 
data published isn't useful to those listening, it's just noise.


i think a lot of the complexity we have in versioning is that the 
projects are too silo'd. i think some of the versioning issues would be 
irrelevant if the producer knew it's consumers before sending rather 
than producers just tossing out a chunk of data (versioned schema or 
not) and considering their job complete once it leaves it's own walls. 
the producer doesn't necessarily have to be the individual project teams 
but whoever the producer of notifications is, it should know it's audience.




The problem is knowing what each consumer thinks is interesting and that
isn't something that can be handled by the producer. If Ceilometer is
just a pipeline that has no opinion on what's relevant and what isn't,
that's a special case easily implemented by an identity function.
the notification consumption service in ceilometer is essentially just a 
pipeline that normalises select incoming notifications into a data 
model(s) and pushes that model to whoever wants it (a known consumer is 
the storage service but it's configurable to allow other consumers).


cheers,

--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Chris Dent]

On Fri, 20 Nov 2015, gord chung wrote:

i think a lot of the complexity we have in versioning is that the projects 
are too silo'd. i think some of the versioning issues would be irrelevant if 
the producer knew it's consumers before sending rather than producers just 
tossing out a chunk of data (versioned schema or not) and considering their 
job complete once it leaves it's own walls. the producer doesn't necessarily 
have to be the individual project teams but whoever the producer of 
notifications is, it should know it's audience.


To me this is entirely contrary to the point of having notifications
as a generic concept in the OpenStack environment. To me the point is
to ensure that it is possible for the audience to be and do _anything_.

We've become so accustomed to some of the misfeatures in the messaging
architecture that we've lost track of the fact that it could be an
event pool on which we have diverse listeners that the producers have
no requirement to know anything about. We could have nova-compute spinning
along shouting "I made a VM. I made another VM. Hey, I made another
VM" and "This VM is hot. Halp, I am oversubscribed." All sorts of
tools and devices need to be able to hear that stuff and choose for
themselves what they might do with it.

(This is similar to the reason we have well-formed HTTP APIs: It is so
we can have unexpected clients that do unexpected things.)

It is certainly the case that if we're going to have schematized
and versioned notifications it is critical that the schema are
discoverable in a language independent fashion.

Sometimes it is hard, though, to be convinced that such formalisms are
quite what we really need. From the consumers end the concern is "do
you have these several keys that I care about?" and, as has been said,
the rest is noise. It sounds like microversioned notifications which
almost never version on the major axis might be able to provide this.

We can't allow the introduction of either the formalisms or
discoverability thereof to grant license to change stuff willy nilly.
Nor should we be building formalisms that are defenses against an
architecture that's sub-optimal. We need to evolve the formalisms and
the architecture toward the ideal.

--
Chris Dent   (�s°□°)�s�喋擤ォ�http://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[gord chung]

ceilometer cares. we listen to all notifications and build Measurement 
and Event data from (some of) them. to be honest, i don't know if 
changes in nova notifications have/are broken in ceilometer because 
quite frankly there are far too many notifications to test and track.


as a consumer of data, we don't really care what the format of the 
entire payload is. we just care that the select attributes we do care 
about are present and in some known position.


in ceilometer, we have two mapping files which define the notifications 
we are interested in and how we to normalise them[1][2]. currently, they 
are two files contained within ceilometer which we load at runtime. that 
said, at the summit we talked about how the projects should own their 
own definitions rather than ceilometer[3]. essentially, producer and 
consumer schemas are tightly coupled.


the basic premise was that we as consumers have no idea what is getting 
produced, but the projects do know and they know exactly what is useful 
in what they are producing. so if nova owned their own meters.yaml and 
events.yaml, nova devs could easily edit the appropriate file themselves 
when they add/modify/remove notification payloads. if a new metric is 
added to a notification, it can easily be added to meters.yaml; if a 
metric value is moved, meters.yaml can be updated for old and new locations.


using each projects definition files, ceilometer would load them all in 
and as a consumer, not have to worry about what every single change that 
happens outside of its domain. it's a different take on it: instead of 
having ownership on consumer to interpret accurately, the ownership is 
on producer to ensure consumer can understand.


we are working on a POC currently as there some logistics to be thought 
out still(ie. how ceilometer knows where to look for each projects 
definition files) but using this idea, we wouldn't even need to know 
what version of notification we were getting, just that we are told that 
we should look "either here or here...". if there's interest in 
exploring this idea or if there are any glaring issues please let us know.


[1] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml
[2] 
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml

[3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

On 19/11/2015 5:05 PM, Ryan Rossiter wrote:
Reading through [1] I started getting worries in the back of my head 
about versioning these notifications. The main concern being how can 
the consumer know about the versions and what's different between 
them? Because these versioned notification payloads hold live nova 
objects, there can be a lot of rug-pulling going on underneath these 
notifications. If the payload doesn't pin itself to a certain level of 
the object, a consumer can never be guaranteed the version of the 
payload's object they will be receiving. I ran through a few of the 
scenarios about irregular versions in the notifications subteam 
meeting on Tuesday [2].


My question is do we care about the consumer? Or is it a case of 
"the consumer is always right" so we need to make sure we hand them 
super consistent, well-defined blobs across the wire? Consumers will 
have no idea of nova object internals, unless they feel like `python 
-c import nova`. I do think we get one piece of help from o.vo though. 
When the object is serialized, it hands the version with the object. 
So consumers can look at the object and say "oh, I got 1.4 I know what 
to do with this". But... they will have to implement their own compat 
logic. Everyone will have to implement their own compat logic.


We could expose a new API for getting the schema for a specific 
version of a notification, so a consumer will know what they're 
getting with their notifications. But I think that made mriedem 
nauseous. We could make an oslo library that stuffs a shim in between 
o.vo and nova's notifications to help out with compat/versioning, but 
that sounds like a lot of work, especially because the end goal is 
still not clearly defined.


Thoughts?

[1] https://review.openstack.org/#/c/247024
[2] 
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29




--
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Matt Riedemann]

On 11/19/2015 5:52 PM, gord chung wrote:

ceilometer cares. we listen to all notifications and build Measurement
and Event data from (some of) them. to be honest, i don't know if
changes in nova notifications have/are broken in ceilometer because
quite frankly there are far too many notifications to test and track.

as a consumer of data, we don't really care what the format of the
entire payload is. we just care that the select attributes we do care
about are present and in some known position.

in ceilometer, we have two mapping files which define the notifications
we are interested in and how we to normalise them[1][2]. currently, they
are two files contained within ceilometer which we load at runtime. that
said, at the summit we talked about how the projects should own their
own definitions rather than ceilometer[3]. essentially, producer and
consumer schemas are tightly coupled.

the basic premise was that we as consumers have no idea what is getting
produced, but the projects do know and they know exactly what is useful
in what they are producing. so if nova owned their own meters.yaml and
events.yaml, nova devs could easily edit the appropriate file themselves
when they add/modify/remove notification payloads. if a new metric is
added to a notification, it can easily be added to meters.yaml; if a
metric value is moved, meters.yaml can be updated for old and new
locations.

using each projects definition files, ceilometer would load them all in
and as a consumer, not have to worry about what every single change that
happens outside of its domain. it's a different take on it: instead of
having ownership on consumer to interpret accurately, the ownership is
on producer to ensure consumer can understand.

we are working on a POC currently as there some logistics to be thought
out still(ie. how ceilometer knows where to look for each projects
definition files) but using this idea, we wouldn't even need to know
what version of notification we were getting, just that we are told that
we should look "either here or here...". if there's interest in
exploring this idea or if there are any glaring issues please let us know.

[1]
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml

[2]
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml

[3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

On 19/11/2015 5:05 PM, Ryan Rossiter wrote:

Reading through [1] I started getting worries in the back of my head
about versioning these notifications. The main concern being how can
the consumer know about the versions and what's different between
them? Because these versioned notification payloads hold live nova
objects, there can be a lot of rug-pulling going on underneath these
notifications. If the payload doesn't pin itself to a certain level of
the object, a consumer can never be guaranteed the version of the
payload's object they will be receiving. I ran through a few of the
scenarios about irregular versions in the notifications subteam
meeting on Tuesday [2].

My question is do we care about the consumer? Or is it a case of
"the consumer is always right" so we need to make sure we hand them
super consistent, well-defined blobs across the wire? Consumers will
have no idea of nova object internals, unless they feel like `python
-c import nova`. I do think we get one piece of help from o.vo though.
When the object is serialized, it hands the version with the object.
So consumers can look at the object and say "oh, I got 1.4 I know what
to do with this". But... they will have to implement their own compat
logic. Everyone will have to implement their own compat logic.

We could expose a new API for getting the schema for a specific
version of a notification, so a consumer will know what they're
getting with their notifications. But I think that made mriedem
nauseous. We could make an oslo library that stuffs a shim in between
o.vo and nova's notifications to help out with compat/versioning, but
that sounds like a lot of work, especially because the end goal is
still not clearly defined.

Thoughts?

[1] https://review.openstack.org/#/c/247024
[2]
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29






I don't think nova is going to maintain it's own meters/events yaml 
files in the form that ceilometer necessarily expects/uses today, since 
there are other things out there that consume nova notifications besides 
ceilometer. But I think you're thinking along the same lines of what 
Ryan was thinking, i.e. somehow nova providing a schema or some way for 
the consumer to digest/interpret what they are getting, but I don't know 
what that would look like or how it would be used by the consumer.


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage 

[openstack-dev] [nova] Versioned notifications... who cares about the version?

[Ryan Rossiter]

Reading through [1] I started getting worries in the back of my head 
about versioning these notifications. The main concern being how can the 
consumer know about the versions and what's different between them? 
Because these versioned notification payloads hold live nova objects, 
there can be a lot of rug-pulling going on underneath these 
notifications. If the payload doesn't pin itself to a certain level of 
the object, a consumer can never be guaranteed the version of the 
payload's object they will be receiving. I ran through a few of the 
scenarios about irregular versions in the notifications subteam meeting 
on Tuesday [2].


My question is do we care about the consumer? Or is it a case of 
"the consumer is always right" so we need to make sure we hand them 
super consistent, well-defined blobs across the wire? Consumers will 
have no idea of nova object internals, unless they feel like `python -c 
import nova`. I do think we get one piece of help from o.vo though. When 
the object is serialized, it hands the version with the object. So 
consumers can look at the object and say "oh, I got 1.4 I know what to 
do with this". But... they will have to implement their own compat 
logic. Everyone will have to implement their own compat logic.


We could expose a new API for getting the schema for a specific version 
of a notification, so a consumer will know what they're getting with 
their notifications. But I think that made mriedem nauseous. We could 
make an oslo library that stuffs a shim in between o.vo and nova's 
notifications to help out with compat/versioning, but that sounds like a 
lot of work, especially because the end goal is still not clearly defined.


Thoughts?

[1] https://review.openstack.org/#/c/247024
[2] 
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29


--
Thanks,

Ryan Rossiter (rlrossit)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[Matt Riedemann]

On 11/19/2015 4:05 PM, Ryan Rossiter wrote:

Reading through [1] I started getting worries in the back of my head
about versioning these notifications. The main concern being how can the
consumer know about the versions and what's different between them?
Because these versioned notification payloads hold live nova objects,
there can be a lot of rug-pulling going on underneath these
notifications. If the payload doesn't pin itself to a certain level of
the object, a consumer can never be guaranteed the version of the
payload's object they will be receiving. I ran through a few of the
scenarios about irregular versions in the notifications subteam meeting
on Tuesday [2].

My question is do we care about the consumer? Or is it a case of
"the consumer is always right" so we need to make sure we hand them
super consistent, well-defined blobs across the wire? Consumers will
have no idea of nova object internals, unless they feel like `python -c
import nova`. I do think we get one piece of help from o.vo though. When
the object is serialized, it hands the version with the object. So
consumers can look at the object and say "oh, I got 1.4 I know what to
do with this". But... they will have to implement their own compat
logic. Everyone will have to implement their own compat logic.

We could expose a new API for getting the schema for a specific version
of a notification, so a consumer will know what they're getting with
their notifications. But I think that made mriedem nauseous. We could
make an oslo library that stuffs a shim in between o.vo and nova's
notifications to help out with compat/versioning, but that sounds like a
lot of work, especially because the end goal is still not clearly defined.


The term is 'nauseated'. To be nauseous, you nauseate others. Which I 
might do from time to time.


Sorry, I'm channeling one of my wives' pet peeves because I say the same 
thing. Sometimes just to annoy her.


But yeah, I was made physically ill from that idea.



Thoughts?

[1] https://review.openstack.org/#/c/247024
[2]
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29




One idea I had was (and maybe this would be in a separate library like 
you're talking about, i.e. nova-notifications), but if nova emits the 
notification with the version and the consumer calls into the library 
that translates it into a version they want, then get that transformed 
thing back.


However, how does the consumer know what they want or what they can 
handle? Do they pin a version in configuration somewhere? I could see 
something like how we have upgrade levels pinned in nova so newer 
conductor can backlevel things for older computes.


I was also thinking about microversions and novaclient, but in that case 
novaclient knows what max microversion it can handle and only requests 
up to that version, and then nova-api handles the compat work. In the 
case of notifications, nova is just broadcasting those so it's not doing 
any compat work, but it's the only thing that knows *how* to do the 
compat work...


So yeah, I'm lost.

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

[gord chung]

On 19/11/15 08:53 PM, Matt Riedemann wrote:



On 11/19/2015 5:52 PM, gord chung wrote:

ceilometer cares. we listen to all notifications and build Measurement
and Event data from (some of) them. to be honest, i don't know if
changes in nova notifications have/are broken in ceilometer because
quite frankly there are far too many notifications to test and track.

as a consumer of data, we don't really care what the format of the
entire payload is. we just care that the select attributes we do care
about are present and in some known position.

in ceilometer, we have two mapping files which define the notifications
we are interested in and how we to normalise them[1][2]. currently, they
are two files contained within ceilometer which we load at runtime. that
said, at the summit we talked about how the projects should own their
own definitions rather than ceilometer[3]. essentially, producer and
consumer schemas are tightly coupled.

the basic premise was that we as consumers have no idea what is getting
produced, but the projects do know and they know exactly what is useful
in what they are producing. so if nova owned their own meters.yaml and
events.yaml, nova devs could easily edit the appropriate file themselves
when they add/modify/remove notification payloads. if a new metric is
added to a notification, it can easily be added to meters.yaml; if a
metric value is moved, meters.yaml can be updated for old and new
locations.

using each projects definition files, ceilometer would load them all in
and as a consumer, not have to worry about what every single change that
happens outside of its domain. it's a different take on it: instead of
having ownership on consumer to interpret accurately, the ownership is
on producer to ensure consumer can understand.

we are working on a POC currently as there some logistics to be thought
out still(ie. how ceilometer knows where to look for each projects
definition files) but using this idea, we wouldn't even need to know
what version of notification we were getting, just that we are told that
we should look "either here or here...". if there's interest in
exploring this idea or if there are any glaring issues please let us 
know.


[1]
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml 



[2]
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml 



[3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

On 19/11/2015 5:05 PM, Ryan Rossiter wrote:

Reading through [1] I started getting worries in the back of my head
about versioning these notifications. The main concern being how can
the consumer know about the versions and what's different between
them? Because these versioned notification payloads hold live nova
objects, there can be a lot of rug-pulling going on underneath these
notifications. If the payload doesn't pin itself to a certain level of
the object, a consumer can never be guaranteed the version of the
payload's object they will be receiving. I ran through a few of the
scenarios about irregular versions in the notifications subteam
meeting on Tuesday [2].

My question is do we care about the consumer? Or is it a case of
"the consumer is always right" so we need to make sure we hand them
super consistent, well-defined blobs across the wire? Consumers will
have no idea of nova object internals, unless they feel like `python
-c import nova`. I do think we get one piece of help from o.vo though.
When the object is serialized, it hands the version with the object.
So consumers can look at the object and say "oh, I got 1.4 I know what
to do with this". But... they will have to implement their own compat
logic. Everyone will have to implement their own compat logic.

We could expose a new API for getting the schema for a specific
version of a notification, so a consumer will know what they're
getting with their notifications. But I think that made mriedem
nauseous. We could make an oslo library that stuffs a shim in between
o.vo and nova's notifications to help out with compat/versioning, but
that sounds like a lot of work, especially because the end goal is
still not clearly defined.

Thoughts?

[1] https://review.openstack.org/#/c/247024
[2]
http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29 








I don't think nova is going to maintain it's own meters/events yaml 
files in the form that ceilometer necessarily expects/uses today, 
since there are other things out there that consume nova notifications 
besides ceilometer. But I think you're thinking along the same lines 
of what Ryan was thinking, i.e. somehow nova providing a schema or 
some way for the consumer to digest/interpret what they are getting, 
but I don't know what that would look like or how it would be used by 
the consumer.




just to clarify, the idea doesn't involve tailoring the notification 
payload to ceilometer, just