Re: [openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-28 Thread Angus Salkeld
On Tue, Apr 28, 2015 at 1:46 AM, Steven Hardy  wrote:

> Hi all,
>
> I've been looking into $subject recently, I raised this bug:
>
> https://bugs.launchpad.net/heat/+bug/1448155
>
> Basically we've got some historically weird and potentially inconsistent
> behavior around updated_at, and I'm trying to figure out the best way to
> proceed.
>
> Now, we selectively update updated_at only on the transition to
> UPDATE_COMPLETE, where we store the time that we moved into
> UPDATE_IN_PROGRESS.  During the update, there's no way to derive the
> time we started the update.
>
> Also, we inconsistently store the time associated with the transition into
> IN_PROGRESS for suspend, resume, snapshot, restore and check actions (even
> though many of these don't modify the stack definition).
>
> The reason I need this is the hook/breakpoint API - the only way to detect
> if you've hit a breakpoint is via events, and to detect you've hit a hook
> during multiple sequential updates (some of which may fail or time out with
> hooks pending), you need to filter the events to only consider those with a
> timestamp newer than the transition of the stack to the update IN_PROGRESS.
>
> AFAICT there's two options:
>
> 1. Update the stack.Stack so we store "now" at every transition (e.g in
> state_set)
>
> 2. Stop trying to explicitly control updated_at, and just allow the oslo
> TimestampMixin to do it's job and update updated_at every time the DB model
> is updated.
>
> What are peoples thoughts?  Either will solve my problem, but I'm leaning
> towards (2) as the cleanest and most technically correct solution.
>

Just beware:
https://github.com/openstack/heat/blob/master/heat/engine/resources/stack_resource.py#L328-L346
and
https://review.openstack.org/#/c/173045/

This is our only current way for knowing if something has changed between 2
updates.

-A


> Similar problems exist for resource.Resource AFAICT.
>
> Steve
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-28 Thread Zane Bitter

On 28/04/15 03:56, Steven Hardy wrote:

On Mon, Apr 27, 2015 at 06:41:52PM -0400, Zane Bitter wrote:

On 27/04/15 13:38, Steven Hardy wrote:

On Mon, Apr 27, 2015 at 04:46:20PM +0100, Steven Hardy wrote:

AFAICT there's two options:

1. Update the stack.Stack so we store "now" at every transition (e.g in
state_set)

2. Stop trying to explicitly control updated_at, and just allow the oslo
TimestampMixin to do it's job and update updated_at every time the DB model
is updated.


Ok, at the risk of answering my own question, there's a third option, which
is to output an event for all stack transitions, not only resource
transitions.  This appears to be the way the CFN event API works AFAICS.


My recollection was that in CFN events were always about a particular
resource. That may have been wrong, or they may have changed it. In any
event (uh, no pun intended), I think this option is preferable to options 1
& 2.


Well from the docs I've been looking at, events are also output for stacks:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-listing-event-history.html

Here we see a stack "myteststack", which generates events of ResourceType
AWS::CloudFormation::Stack, with a LogicalResourceId of "myteststack".


Huh, so it does. So the only difference is that the stack events don't 
have a ResourceProperties key. Ick.



It's a bit confusing because the PhysicalResourceId doesn't match the
StackId, but I'm interpreting this as an event from the stack rather than a
resource inside the stack.  Could be that it's just a bad example though.


When we first implemented this stuff we only operated on one resource at a
time, there was no way to cancel an update, &c. It was a simpler world ;)


Yeah, true - and (with the benefit of hindsight) events are a really bad
interface for hook polling, which is what I'm currently trying to work
around.

Trying to do this has exposed how limited our event API is though, so IMO
it's worth trying to fix this for the benefit of all API consumers.


I guess the event would have a dummy OS::Heat::Stack type and then you


That's too hacky IMHO, I think we should have a more solid way of
distinguishing resource events from stack events. OS::Heat::Stack is a type
of resource already, after all. Arguably they should come from separate
endpoints, to avoid breaking clients until we get to a v2 API.


I disagree about the separate endpoint (not least because it implies hooks
will be unusable for kilo):

Looking more closely at our native event API:

http://developer.openstack.org/api-ref-orchestration-v1.html#stack-events

The path for events is:·

/v1/{tenant_id}/stacks/{stack_name}/events

This, to me (historical resource-ness aside) implies events associated with
a particular stack - IMHO it's fair game to output both events associated
with the stack itself here and the resources contained by the stack.

If we were to use some other endpoint, I don't even know what we would
use, because intuitively the path above is the one which makes sense for
events associated with a stack?


I'm not saying it's the wrong place, but somehow, somewhere, it will 
break some client who is not expecting it.



I'm open to using something other than OS::Heat::Stack, but that to me is
the most obvious option, which fits OK with the current resource-orientated
event API response payload - it is the resource which describes a stack
after all (and it potentially aligns with the AWS interface I mention above.)


For consistency with CloudFormation, I agree that's the obvious choice. 
I withdraw my objection.



could find the most recent transition to e.g UPDATE_IN_PROGRESS in the
events and use that as a marker so you only list results after that event?


Even that is not valid in a distributed system. For convergence we're
planning to have a UUID associated with each update. We should reuse that to
connect events with particular update traversals.


There's still going to be some event (or at least a point in time) where an
API request for update-stack is recieved, and the stack, as a whole, moves
from a stable state (COMPLETE/FAILED) into an in-progress one though, is
there not?

I'm not really sure why distribution of the update workload will affect the
nature of that initial transition, other than that there may be multiple
passes before we reach the final transition back into a stable state (e.g
potentially multiple updates on resources before we stop updating the stack
as a whole)?


Sorry that was far too vague, I should have been more clear: 
establishing the order of events by timestamp is not a valid strategy 
for a distributed system because time is not monotonic in a distributed 
system.


cheers,
Zane.


Anyway, https://review.openstack.org/#/c/177961/2 has been approved now -
I'm happy to follow up if you have specific suggestions on how we can
improve it.

Cheers,

Steve

__
OpenStack Development Maili

Re: [openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-28 Thread Steven Hardy
On Mon, Apr 27, 2015 at 06:41:52PM -0400, Zane Bitter wrote:
> On 27/04/15 13:38, Steven Hardy wrote:
> >On Mon, Apr 27, 2015 at 04:46:20PM +0100, Steven Hardy wrote:
> >>AFAICT there's two options:
> >>
> >>1. Update the stack.Stack so we store "now" at every transition (e.g in
> >>state_set)
> >>
> >>2. Stop trying to explicitly control updated_at, and just allow the oslo
> >>TimestampMixin to do it's job and update updated_at every time the DB model
> >>is updated.
> >
> >Ok, at the risk of answering my own question, there's a third option, which
> >is to output an event for all stack transitions, not only resource
> >transitions.  This appears to be the way the CFN event API works AFAICS.
> 
> My recollection was that in CFN events were always about a particular
> resource. That may have been wrong, or they may have changed it. In any
> event (uh, no pun intended), I think this option is preferable to options 1
> & 2.

Well from the docs I've been looking at, events are also output for stacks:

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-listing-event-history.html

Here we see a stack "myteststack", which generates events of ResourceType
AWS::CloudFormation::Stack, with a LogicalResourceId of "myteststack".

It's a bit confusing because the PhysicalResourceId doesn't match the
StackId, but I'm interpreting this as an event from the stack rather than a
resource inside the stack.  Could be that it's just a bad example though.

> When we first implemented this stuff we only operated on one resource at a
> time, there was no way to cancel an update, &c. It was a simpler world ;)

Yeah, true - and (with the benefit of hindsight) events are a really bad
interface for hook polling, which is what I'm currently trying to work
around.

Trying to do this has exposed how limited our event API is though, so IMO
it's worth trying to fix this for the benefit of all API consumers.

> >I guess the event would have a dummy OS::Heat::Stack type and then you
> 
> That's too hacky IMHO, I think we should have a more solid way of
> distinguishing resource events from stack events. OS::Heat::Stack is a type
> of resource already, after all. Arguably they should come from separate
> endpoints, to avoid breaking clients until we get to a v2 API.

I disagree about the separate endpoint (not least because it implies hooks
will be unusable for kilo):

Looking more closely at our native event API:

http://developer.openstack.org/api-ref-orchestration-v1.html#stack-events

The path for events is:·

/v1/{tenant_id}/stacks/{stack_name}/events

This, to me (historical resource-ness aside) implies events associated with
a particular stack - IMHO it's fair game to output both events associated
with the stack itself here and the resources contained by the stack.

If we were to use some other endpoint, I don't even know what we would
use, because intuitively the path above is the one which makes sense for
events associated with a stack?

I'm open to using something other than OS::Heat::Stack, but that to me is
the most obvious option, which fits OK with the current resource-orientated
event API response payload - it is the resource which describes a stack
after all (and it potentially aligns with the AWS interface I mention above.)

> >could find the most recent transition to e.g UPDATE_IN_PROGRESS in the
> >events and use that as a marker so you only list results after that event?
> 
> Even that is not valid in a distributed system. For convergence we're
> planning to have a UUID associated with each update. We should reuse that to
> connect events with particular update traversals.

There's still going to be some event (or at least a point in time) where an
API request for update-stack is recieved, and the stack, as a whole, moves
from a stable state (COMPLETE/FAILED) into an in-progress one though, is
there not?

I'm not really sure why distribution of the update workload will affect the
nature of that initial transition, other than that there may be multiple
passes before we reach the final transition back into a stable state (e.g
potentially multiple updates on resources before we stop updating the stack
as a whole)?

Anyway, https://review.openstack.org/#/c/177961/2 has been approved now -
I'm happy to follow up if you have specific suggestions on how we can
improve it.

Cheers,

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-27 Thread Zane Bitter

On 27/04/15 13:38, Steven Hardy wrote:

On Mon, Apr 27, 2015 at 04:46:20PM +0100, Steven Hardy wrote:

AFAICT there's two options:

1. Update the stack.Stack so we store "now" at every transition (e.g in
state_set)

2. Stop trying to explicitly control updated_at, and just allow the oslo
TimestampMixin to do it's job and update updated_at every time the DB model
is updated.


Ok, at the risk of answering my own question, there's a third option, which
is to output an event for all stack transitions, not only resource
transitions.  This appears to be the way the CFN event API works AFAICS.


My recollection was that in CFN events were always about a particular 
resource. That may have been wrong, or they may have changed it. In any 
event (uh, no pun intended), I think this option is preferable to 
options 1 & 2.


When we first implemented this stuff we only operated on one resource at 
a time, there was no way to cancel an update, &c. It was a simpler world ;)



I guess the event would have a dummy OS::Heat::Stack type and then you


That's too hacky IMHO, I think we should have a more solid way of 
distinguishing resource events from stack events. OS::Heat::Stack is a 
type of resource already, after all. Arguably they should come from 
separate endpoints, to avoid breaking clients until we get to a v2 API.



could find the most recent transition to e.g UPDATE_IN_PROGRESS in the
events and use that as a marker so you only list results after that event?


Even that is not valid in a distributed system. For convergence we're 
planning to have a UUID associated with each update. We should reuse 
that to connect events with particular update traversals.


cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-27 Thread Steven Hardy
On Mon, Apr 27, 2015 at 04:46:20PM +0100, Steven Hardy wrote:
> AFAICT there's two options:
> 
> 1. Update the stack.Stack so we store "now" at every transition (e.g in
> state_set)
> 
> 2. Stop trying to explicitly control updated_at, and just allow the oslo
> TimestampMixin to do it's job and update updated_at every time the DB model
> is updated.

Ok, at the risk of answering my own question, there's a third option, which
is to output an event for all stack transitions, not only resource
transitions.  This appears to be the way the CFN event API works AFAICS.

I guess the event would have a dummy OS::Heat::Stack type and then you
could find the most recent transition to e.g UPDATE_IN_PROGRESS in the
events and use that as a marker so you only list results after that event?

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-27 Thread Steven Hardy
On Mon, Apr 27, 2015 at 04:45:10PM +, Randall Burt wrote:
> 2 sounds right to me, but does the in-memory representation get updated or 
> are we forced into a refetch at every change?

Yeah, we probably would be, digging some more there's historical context
here:

https://bugs.launchpad.net/heat/+bug/1193269

AFAICT the fix for that bug introduced the behavior I'm now trying to
avoid, but is actually correct, if the objective is to remain consistent
with the CFN "LastUpdatedTime", which says "The time the stack was last
updated".

My comments about suspend/resume etc were actually incorrect, we only store
updated_at on the transistion to UPDATE_COMPLETE.

Any other ideas of where we can expose the timestamp associated with the
transition to UPDATE_IN_PROGRESS, which is the info I need?

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [heat] Stack/Resource updated_at conventions

2015-04-27 Thread Steven Hardy
Hi all,

I've been looking into $subject recently, I raised this bug:

https://bugs.launchpad.net/heat/+bug/1448155

Basically we've got some historically weird and potentially inconsistent
behavior around updated_at, and I'm trying to figure out the best way to
proceed.

Now, we selectively update updated_at only on the transition to
UPDATE_COMPLETE, where we store the time that we moved into
UPDATE_IN_PROGRESS.  During the update, there's no way to derive the
time we started the update.

Also, we inconsistently store the time associated with the transition into
IN_PROGRESS for suspend, resume, snapshot, restore and check actions (even
though many of these don't modify the stack definition).

The reason I need this is the hook/breakpoint API - the only way to detect
if you've hit a breakpoint is via events, and to detect you've hit a hook
during multiple sequential updates (some of which may fail or time out with
hooks pending), you need to filter the events to only consider those with a
timestamp newer than the transition of the stack to the update IN_PROGRESS.

AFAICT there's two options:

1. Update the stack.Stack so we store "now" at every transition (e.g in
state_set)

2. Stop trying to explicitly control updated_at, and just allow the oslo
TimestampMixin to do it's job and update updated_at every time the DB model
is updated.

What are peoples thoughts?  Either will solve my problem, but I'm leaning
towards (2) as the cleanest and most technically correct solution.

Similar problems exist for resource.Resource AFAICT.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev