Re: [openstack-dev] [Nova] State machines in Nova

2016-06-02 Thread Miles Gould 

On 01/06/16 13:50, Andrew Laski wrote:

This is a great point. I think most people have an implicit assumption
that the state machine will be exposed to end users via the API. I would
like to avoid that for exactly the reason you've mentioned. Of course
we'll want to expose something to users but whatever that is should be
loosely coupled with the internal states that actually drive the system.


That would probably help, but think about how you'll handle things when 
you have to make changes to the client-visible representation :-)


Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-02 Thread Murray, Paul (HP Cloud) 


> -Original Message-
> From: Monty Taylor [mailto:mord...@inaugust.com]
> Sent: 01 June 2016 13:54
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Nova] State machines in Nova
> 
> On 06/01/2016 03:50 PM, Andrew Laski wrote:
> >
> >
> > On Wed, Jun 1, 2016, at 05:51 AM, Miles Gould wrote:
> >> On 31/05/16 21:03, Timofei Durakov wrote:
> >>> there is blueprint[1] that was approved during Liberty and
> >>> resubmitted to Newton(with spec[2]).
> >>> The idea is to define state machines for operations as
> >>> live-migration, resize, etc. and to deal with them operation states.
> >>
> >> +1 to introducing an explicit state machine - IME they make complex
> >> logic much easier to reason about. However, think carefully about how
> >> you'll make changes to that state machine later. In Ironic, this is
> >> an ongoing problem: every time we change the state machine, we have
> >> to decide whether to lie to older clients (and if so, what lie to
> >> tell them), or whether to present them with the truth (and if so, how
> >> badly they'll break). AIUI this would be a much smaller problem if
> >> we'd considered this possibility carefully at the beginning.
> >
> > This is a great point. I think most people have an implicit assumption
> > that the state machine will be exposed to end users via the API. I
> > would like to avoid that for exactly the reason you've mentioned. Of
> > course we'll want to expose something to users but whatever that is
> > should be loosely coupled with the internal states that actually drive the
> system.
> 
> +1billion
> 

I think this raises an interesting point.

tl;dr: I am starting to think we should not do the migration state machine spec 
being proposed before the tasks. But we should at least make the states we 
assign something other than arbitrary strings (e.g. constants defined in a 
particular place) and we should use the state names consistently.

Transitions can come from two places: 1) the user invokes the API to change the 
state of an instance, this is a good place to check that the instance is in a 
state to do the externally visible transition, 2) the state of the instance 
changes due to an internal event (host crash, deliberate operation...) this 
implies a change in the externally visible state of the instance, but cannot be 
prevented just because the state machine says this shouldn't happen (usually 
this is captured by the error state, but we can do better sometimes).

I think the state machines that are being defined in these changes are actually 
high level phases of the migration process that are currently observed by the 
user. I'm not sure they are particularly useful for coordinating the migration 
process itself and so are maybe not the right place to enforce internal 
transitions.

Live migration is an oddity in nova. Usually an instance is a single entity 
running on a single host (ignoring clustered hypervisors for the moment). There 
is a host manager responsible for that host that has the best view of the 
actual state of the instance or operations being performed on it. Generally the 
host manager is the natural place to coordinate operations on the instance.

In the case of live migration there are actually two VMs running on different 
hosts at a same time. The migration process involves coordinating transitions 
of those two VMs (attaching disks, plugging networks, starting the target VM, 
starting the migration, rebinding ports, stopping the source VM.). The two 
VMs and their own individual states in this process are not represented 
explicitly. We only have an overall process coordinated by a distributed  
sequence of rpcs. There is a current spec moving that coordination to the 
conductor. When that sequence is interrupted or even completely lost (e.g. by a 
conductor failing or being restarted) we get into trouble. I think this is 
where our real problem lies.

We should sort out the internal process. The external view given to the user 
can be a true reflection the current state of the instance. The transitions of 
the instance should be internally coordinated.

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-02 Thread Miles Gould 

On 01/06/16 16:45, Joshua Harlow wrote:

Do u have any more details (perhaps an 'real-life' example that you can
walk us through) of this and how it played out. It'd be interesting to
hear (I believe it has happened a few times but I've never heard how it
was resolved or the details of it).


The most recent example was IIRC the proposed addition of an ADOPTING state:

https://review.openstack.org/#/c/275766/

Here's the log of the meeting where we argued about how to deal with it:

http://eavesdrop.openstack.org/meetings/ironic/2016/ironic.2016-05-16-17.00.log.html#l-66

Eventually we put it to a vote, and the "tell the client the truth, even 
if they're too old to handle it properly" side won.


Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Joshua Harlow 

Miles Gould wrote:

On 31/05/16 21:03, Timofei Durakov wrote:

there is blueprint[1] that was approved during Liberty and resubmitted
to Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.


+1 to introducing an explicit state machine - IME they make complex
logic much easier to reason about. However, think carefully about how
you'll make changes to that state machine later. In Ironic, this is an
ongoing problem: every time we change the state machine, we have to
decide whether to lie to older clients (and if so, what lie to tell
them), or whether to present them with the truth (and if so, how badly
they'll break). AIUI this would be a much smaller problem if we'd
considered this possibility carefully at the beginning.


Do u have any more details (perhaps an 'real-life' example that you can 
walk us through) of this and how it played out. It'd be interesting to 
hear (I believe it has happened a few times but I've never heard how it 
was resolved or the details of it).




Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Joshua Harlow 
Sounds similar to https://review.openstack.org/#/c/224022/ (which is the 
ironic version of expose state machine transitions over the REST API); 
probably useful to read over the review commentary there and/or talk to 
the ironic folks about that before doing much here (to learn some of the 
pros/cons and such).


Andrew Laski wrote:


On Wed, Jun 1, 2016, at 05:51 AM, Miles Gould wrote:

On 31/05/16 21:03, Timofei Durakov wrote:

there is blueprint[1] that was approved during Liberty and resubmitted
to Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.

+1 to introducing an explicit state machine - IME they make complex
logic much easier to reason about. However, think carefully about how
you'll make changes to that state machine later. In Ironic, this is an
ongoing problem: every time we change the state machine, we have to
decide whether to lie to older clients (and if so, what lie to tell
them), or whether to present them with the truth (and if so, how badly
they'll break). AIUI this would be a much smaller problem if we'd
considered this possibility carefully at the beginning.


This is a great point. I think most people have an implicit assumption
that the state machine will be exposed to end users via the API. I would
like to avoid that for exactly the reason you've mentioned. Of course
we'll want to expose something to users but whatever that is should be
loosely coupled with the internal states that actually drive the system.



Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Monty Taylor 
On 06/01/2016 03:50 PM, Andrew Laski wrote:
> 
> 
> On Wed, Jun 1, 2016, at 05:51 AM, Miles Gould wrote:
>> On 31/05/16 21:03, Timofei Durakov wrote:
>>> there is blueprint[1] that was approved during Liberty and resubmitted
>>> to Newton(with spec[2]).
>>> The idea is to define state machines for operations as live-migration,
>>> resize, etc. and to deal with them operation states.
>>
>> +1 to introducing an explicit state machine - IME they make complex 
>> logic much easier to reason about. However, think carefully about how 
>> you'll make changes to that state machine later. In Ironic, this is an 
>> ongoing problem: every time we change the state machine, we have to 
>> decide whether to lie to older clients (and if so, what lie to tell 
>> them), or whether to present them with the truth (and if so, how badly 
>> they'll break). AIUI this would be a much smaller problem if we'd 
>> considered this possibility carefully at the beginning.
> 
> This is a great point. I think most people have an implicit assumption
> that the state machine will be exposed to end users via the API. I would
> like to avoid that for exactly the reason you've mentioned. Of course
> we'll want to expose something to users but whatever that is should be
> loosely coupled with the internal states that actually drive the system. 

+1billion


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Andrew Laski 
 
 
 
On Wed, Jun 1, 2016, at 06:06 AM, Timofei Durakov wrote:
> From my sight, I concerned proposed transition from option #1 to
> option #2.
> because it would be quite big change. So I wonder, has any
> component team
> implemented such transition. Open questions:
>  * upgrades story potential issues
>  * dealing with clients(?)
>  * promoting state machine from verification of states to conductor of
>the task(success stories)
 
I would also be interested in hearing post mortems from projects that
have done this.
 
It would be a big change to transition from #1 to #2 but I don't think
there's any less work involved to just do #2 first. Formalizing the
states we want and adding logic around that has to take place in either
option. I see option 1 as a small chunk of option 2, not an alternative.
 
 
> Timofey
>
> On Wed, Jun 1, 2016 at 12:51 PM, Miles Gould
>  wrote:
>> On 31/05/16 21:03, Timofei Durakov wrote:
>>> there is blueprint[1] that was approved during Liberty and
>>> resubmitted to Newton(with spec[2]). The idea is to define state
>>> machines for operations as live-migration, resize, etc. and to deal
>>> with them operation states.
>>
>> +1 to introducing an explicit state machine - IME they make complex
>> logic much easier to reason about. However, think carefully about how
>> you'll make changes to that state machine later. In Ironic, this is
>> an ongoing problem: every time we change the state machine, we have
>> to decide whether to lie to older clients (and if so, what lie to
>> tell them), or whether to present them with the truth (and if so, how
>> badly they'll break). AIUI this would be a much smaller problem if
>> we'd considered this possibility carefully at the beginning.
>>
>>  Miles
>>
>>
>> ___-
>> ___
>>  OpenStack Development Mailing List (not for usage questions)
>>  Unsubscribe: OpenStack-dev-
>>  requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> -
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Andrew Laski 


On Wed, Jun 1, 2016, at 05:51 AM, Miles Gould wrote:
> On 31/05/16 21:03, Timofei Durakov wrote:
> > there is blueprint[1] that was approved during Liberty and resubmitted
> > to Newton(with spec[2]).
> > The idea is to define state machines for operations as live-migration,
> > resize, etc. and to deal with them operation states.
> 
> +1 to introducing an explicit state machine - IME they make complex 
> logic much easier to reason about. However, think carefully about how 
> you'll make changes to that state machine later. In Ironic, this is an 
> ongoing problem: every time we change the state machine, we have to 
> decide whether to lie to older clients (and if so, what lie to tell 
> them), or whether to present them with the truth (and if so, how badly 
> they'll break). AIUI this would be a much smaller problem if we'd 
> considered this possibility carefully at the beginning.

This is a great point. I think most people have an implicit assumption
that the state machine will be exposed to end users via the API. I would
like to avoid that for exactly the reason you've mentioned. Of course
we'll want to expose something to users but whatever that is should be
loosely coupled with the internal states that actually drive the system. 


> 
> Miles
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Murray, Paul (HP Cloud) 


> -Original Message-
> From: Andrew Laski [mailto:and...@lascii.com]
> Sent: 31 May 2016 22:34
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [Nova] State machines in Nova
> 
> 
> 
> On Tue, May 31, 2016, at 04:26 PM, Joshua Harlow wrote:
> > Timofei Durakov wrote:
> > > Hi team,
> > >
> > > there is blueprint[1] that was approved during Liberty and
> > > resubmitted to Newton(with spec[2]).

Do you mean the blueprint was approved - I think it was agreed there was no 
need for a spec then. There is a spec now because there is a minor API impact 
(if state names are changed to be cConsistent across types of migration).



> > > The idea is to define state machines for operations as
> > > live-migration, resize, etc. and to deal with them operation states.
> > > The spec PoC patches are overall good. At the same time I think is
> > > will be good to get agreement on the usage of state-machines in Nova.
> > > There are 2 options:
> > >
> > >   * implement proposed change and use state machines to deal with
> states
> > > only
> >
> > I think this is what could be called the ironic equivalent correct?
> >
> > In ironic @
> >
> https://github.com/openstack/ironic/blob/master/ironic/common/states.p
> > y the state machine here is used to ensure proper states are
> > transitioned over and no invalid/unexpected state transitions happen.
> > The code though itself still runs in a implicit fashion and afaik only
> > interacts with the state machine as a side-effect of actions occurring
> > (instead of the reverse where the state machine itself is 'driving'
> > those actions to happen/to completion).
> 
> Yes. This exists in a limited form already in Nova for instances and
> task_states.
> 
> >
> > >   o procs:
> > >   + could be implemented/merged right now
> > >   + cleans up states for migrations
> > >   o cons:
> > >   + state machine only deal with states, and it will be hard to
> > > build on top of it task API, as bp [1] was designed for
> > > another thing.
> > >
> > >   * use state machines in Task API(which I'm going to work on during
> > > next release):
> >
> > So this would be the second model described above, where the state
> > machine (or set of state machines) itself (together could be formed
> > into a action plan, or action workflow or ...) would be the 'entity'
> > realizing a given action and ensuring that it is performed until
> > completed (or tracking where it was paused and such); is that correct?
> >
> > >   o procs:
> > >   + Task API will orchestrate and deal with long running tasks
> > >   + usage state-machines could help with actions
> > > rollbacks/retries/etc.
> > >   o cons:
> > >   + big amount of work
> > >   + requires time.
> > >
> > > I'd like to discuss these options in this thread.
> >
> > It seems like one could progress from the first model to the second
> > one, although that kind of progression would still be large (because
> > if my understanding is correct the control of who runs what has to be
> > given over to something else in the second model, similar to the
> > control a taskflow engine or mistral engine has over what it runs);
> > said control means that certain programming models may not map so well
> > (from what I have seen).
> 
> I think working through this as a progression from the first model to the
> second one would be the best plan. Start with formalizing the states and
> their allowed transitions and add checking and error handling around that.
> Then work towards handing off control to an engine that could drive the
> operation.
> 

I'm inclined to agree with Andrew. I don't see this as inconsistent with the 
tasks API approach although it may become redundant. The spec is actually 
pretty simple so why make it wait for something that may exist in the future.


> > >
> > > [1] -
> > > https://blueprints.launchpad.net/openstack/?searchtext=migration-sta
> > > te-machine [2] - https://review.openstack.org/#/c/320849/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Timofei Durakov 
>From my sight, I concerned proposed transition from option #1 to option #2.
because it would be quite big change. So I wonder, has any component team
implemented such transition. Open questions:

   - upgrades story potential issues
   - dealing with clients(?)
   - promoting state machine from verification of states to conductor of
   the task(success stories)

Timofey

On Wed, Jun 1, 2016 at 12:51 PM, Miles Gould  wrote:

> On 31/05/16 21:03, Timofei Durakov wrote:
>
>> there is blueprint[1] that was approved during Liberty and resubmitted
>> to Newton(with spec[2]).
>> The idea is to define state machines for operations as live-migration,
>> resize, etc. and to deal with them operation states.
>>
>
> +1 to introducing an explicit state machine - IME they make complex logic
> much easier to reason about. However, think carefully about how you'll make
> changes to that state machine later. In Ironic, this is an ongoing problem:
> every time we change the state machine, we have to decide whether to lie to
> older clients (and if so, what lie to tell them), or whether to present
> them with the truth (and if so, how badly they'll break). AIUI this would
> be a much smaller problem if we'd considered this possibility carefully at
> the beginning.
>
> Miles
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-06-01 Thread Miles Gould 

On 31/05/16 21:03, Timofei Durakov wrote:

there is blueprint[1] that was approved during Liberty and resubmitted
to Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.


+1 to introducing an explicit state machine - IME they make complex 
logic much easier to reason about. However, think carefully about how 
you'll make changes to that state machine later. In Ironic, this is an 
ongoing problem: every time we change the state machine, we have to 
decide whether to lie to older clients (and if so, what lie to tell 
them), or whether to present them with the truth (and if so, how badly 
they'll break). AIUI this would be a much smaller problem if we'd 
considered this possibility carefully at the beginning.


Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-05-31 Thread Joshua Harlow 

Andrew Laski wrote:


On Tue, May 31, 2016, at 04:26 PM, Joshua Harlow wrote:

Timofei Durakov wrote:

Hi team,

there is blueprint[1] that was approved during Liberty and resubmitted
to Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.
The spec PoC patches are overall good. At the same time I think is will
be good to get agreement on the usage of state-machines in Nova.
There are 2 options:

   * implement proposed change and use state machines to deal with states
 only

I think this is what could be called the ironic equivalent correct?

In ironic @
https://github.com/openstack/ironic/blob/master/ironic/common/states.py
the state machine here is used to ensure proper states are transitioned
over and no invalid/unexpected state transitions happen. The code though
itself still runs in a implicit fashion and afaik only interacts with
the state machine as a side-effect of actions occurring (instead of the
reverse where the state machine itself is 'driving' those actions to
happen/to completion).


Yes. This exists in a limited form already in Nova for instances and
task_states.


Right, I think I remember some attempts by some redhat folks to try to 
extract that information (I think it was via some complex grep scripts) 
into a state-table; don't quite think that ever got anywhere though 
(that I think was trying to create an equivalent of 
http://docs.openstack.org/developer/ironic/dev/states.html if I recall).


Maybe a first step to this all is to try to extract the task_states into 
an official state machine, ending up with something like 
https://github.com/openstack/ironic/blob/master/ironic/common/states.py 
(which is combined into a state machine at 
https://github.com/openstack/ironic/blob/master/ironic/common/states.py#L197 
...); then associate that machine with an instance and then in all prior 
locations where something like 'instance.task_state=XYZ' was happening 
change that to be instance.task_transition(new_task_state); that would 
use the state-machine for validation for allowed transitions (and would 
likely also help in centralizing what the valid states are, and as a 
side-effect of doing that nova can produce a similar svg diagram as 
ironic has when that is done).


Might be useful to find out some of the pros/cons from the ironic folks 
as they have gone through option #1 already (not many projects, maybe 
outside of heat, cue, octavia,  ? from what I can tell have gone 
with option #2 from the start, although I would have liked them to, ha).





   o procs:
   + could be implemented/merged right now
   + cleans up states for migrations
   o cons:
   + state machine only deal with states, and it will be hard to
 build on top of it task API, as bp [1] was designed for
 another thing.

   * use state machines in Task API(which I'm going to work on during
 next release):

So this would be the second model described above, where the state
machine (or set of state machines) itself (together could be formed into
a action plan, or action workflow or ...) would be the 'entity'
realizing a given action and ensuring that it is performed until
completed (or tracking where it was paused and such); is that correct?


   o procs:
   + Task API will orchestrate and deal with long running tasks
   + usage state-machines could help with actions
 rollbacks/retries/etc.
   o cons:
   + big amount of work
   + requires time.

I'd like to discuss these options in this thread.

It seems like one could progress from the first model to the second one,
although that kind of progression would still be large (because if my
understanding is correct the control of who runs what has to be given
over to something else in the second model, similar to the control a
taskflow engine or mistral engine has over what it runs); said control
means that certain programming models may not map so well (from what I
have seen).


I think working through this as a progression from the first model to
the second one would be the best plan. Start with formalizing the states
and their allowed transitions and add checking and error handling around
that. Then work towards handing off control to an engine that could
drive the operation.


Timofey

[1] -
https://blueprints.launchpad.net/openstack/?searchtext=migration-state-machine
[2] - https://review.openstack.org/#/c/320849/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:

Re: [openstack-dev] [Nova] State machines in Nova

2016-05-31 Thread Andrew Laski 


On Tue, May 31, 2016, at 04:26 PM, Joshua Harlow wrote:
> Timofei Durakov wrote:
> > Hi team,
> >
> > there is blueprint[1] that was approved during Liberty and resubmitted
> > to Newton(with spec[2]).
> > The idea is to define state machines for operations as live-migration,
> > resize, etc. and to deal with them operation states.
> > The spec PoC patches are overall good. At the same time I think is will
> > be good to get agreement on the usage of state-machines in Nova.
> > There are 2 options:
> >
> >   * implement proposed change and use state machines to deal with states
> > only
> 
> I think this is what could be called the ironic equivalent correct?
> 
> In ironic @ 
> https://github.com/openstack/ironic/blob/master/ironic/common/states.py 
> the state machine here is used to ensure proper states are transitioned 
> over and no invalid/unexpected state transitions happen. The code though 
> itself still runs in a implicit fashion and afaik only interacts with 
> the state machine as a side-effect of actions occurring (instead of the 
> reverse where the state machine itself is 'driving' those actions to 
> happen/to completion).

Yes. This exists in a limited form already in Nova for instances and
task_states.

> 
> >   o procs:
> >   + could be implemented/merged right now
> >   + cleans up states for migrations
> >   o cons:
> >   + state machine only deal with states, and it will be hard to
> > build on top of it task API, as bp [1] was designed for
> > another thing.
> >
> >   * use state machines in Task API(which I'm going to work on during
> > next release):
> 
> So this would be the second model described above, where the state 
> machine (or set of state machines) itself (together could be formed into 
> a action plan, or action workflow or ...) would be the 'entity' 
> realizing a given action and ensuring that it is performed until 
> completed (or tracking where it was paused and such); is that correct?
> 
> >   o procs:
> >   + Task API will orchestrate and deal with long running tasks
> >   + usage state-machines could help with actions
> > rollbacks/retries/etc.
> >   o cons:
> >   + big amount of work
> >   + requires time.
> >
> > I'd like to discuss these options in this thread.
> 
> It seems like one could progress from the first model to the second one, 
> although that kind of progression would still be large (because if my 
> understanding is correct the control of who runs what has to be given 
> over to something else in the second model, similar to the control a 
> taskflow engine or mistral engine has over what it runs); said control 
> means that certain programming models may not map so well (from what I 
> have seen).

I think working through this as a progression from the first model to
the second one would be the best plan. Start with formalizing the states
and their allowed transitions and add checking and error handling around
that. Then work towards handing off control to an engine that could
drive the operation.

> 
> >
> > Timofey
> >
> > [1] -
> > https://blueprints.launchpad.net/openstack/?searchtext=migration-state-machine
> > [2] - https://review.openstack.org/#/c/320849/
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] State machines in Nova

2016-05-31 Thread Joshua Harlow 

Timofei Durakov wrote:

Hi team,

there is blueprint[1] that was approved during Liberty and resubmitted
to Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.
The spec PoC patches are overall good. At the same time I think is will
be good to get agreement on the usage of state-machines in Nova.
There are 2 options:

  * implement proposed change and use state machines to deal with states
only


I think this is what could be called the ironic equivalent correct?

In ironic @ 
https://github.com/openstack/ironic/blob/master/ironic/common/states.py 
the state machine here is used to ensure proper states are transitioned 
over and no invalid/unexpected state transitions happen. The code though 
itself still runs in a implicit fashion and afaik only interacts with 
the state machine as a side-effect of actions occurring (instead of the 
reverse where the state machine itself is 'driving' those actions to 
happen/to completion).



  o procs:
  + could be implemented/merged right now
  + cleans up states for migrations
  o cons:
  + state machine only deal with states, and it will be hard to
build on top of it task API, as bp [1] was designed for
another thing.

  * use state machines in Task API(which I'm going to work on during
next release):


So this would be the second model described above, where the state 
machine (or set of state machines) itself (together could be formed into 
a action plan, or action workflow or ...) would be the 'entity' 
realizing a given action and ensuring that it is performed until 
completed (or tracking where it was paused and such); is that correct?



  o procs:
  + Task API will orchestrate and deal with long running tasks
  + usage state-machines could help with actions
rollbacks/retries/etc.
  o cons:
  + big amount of work
  + requires time.

I'd like to discuss these options in this thread.


It seems like one could progress from the first model to the second one, 
although that kind of progression would still be large (because if my 
understanding is correct the control of who runs what has to be given 
over to something else in the second model, similar to the control a 
taskflow engine or mistral engine has over what it runs); said control 
means that certain programming models may not map so well (from what I 
have seen).




Timofey

[1] -
https://blueprints.launchpad.net/openstack/?searchtext=migration-state-machine
[2] - https://review.openstack.org/#/c/320849/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] State machines in Nova

2016-05-31 Thread Timofei Durakov 
Hi team,

there is blueprint[1] that was approved during Liberty and resubmitted to
Newton(with spec[2]).
The idea is to define state machines for operations as live-migration,
resize, etc. and to deal with them operation states.
The spec PoC patches are overall good. At the same time I think is will be
good to get agreement on the usage of state-machines in Nova.
There are 2 options:

   - implement proposed change and use state machines to deal with states
   only
   - procs:
 - could be implemented/merged right now
 - cleans up states for migrations
  - cons:
 - state machine only deal with states, and it will be hard to
 build on top of it task API, as bp [1] was designed for another thing.


   - use state machines in Task API(which I'm going to work on during next
   release):
  - procs:
 - Task API will orchestrate and deal with long running tasks
 - usage state-machines could help with actions
 rollbacks/retries/etc.
  - cons:
 - big amount of work
 - requires time.

I'd like to discuss these options in this thread.

Timofey

[1] -
https://blueprints.launchpad.net/openstack/?searchtext=migration-state-machine
[2] - https://review.openstack.org/#/c/320849/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev