Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-28 Thread Renat Akhmerov
Hi Joshua,

Sorry, I’ve been very busy for the last couple of days and didn’t respond 
quickly enough.

Well, first of all, it’s my bad that I’ve not been following TaskFlow progress 
for a while and, honestly, I just need to get more info on the current TaskFlow 
status. So I’ll do that and get back to you soon. As you know, there were 
reasons why we decided to go this path (without using TaskFlow) but I’ve always 
thought we will be able to align our efforts as we move forward once 
requirements and design of Mistral become more clear. I really want to use 
TaskFlow for Mistral implementation. We just need to make sure that it will 
bring more value than pain (sorry if it sounds harsh).

Thanks for your feedback and this info. We’ll get in touch with you soon.

Renat Akhmerov
@ Mirantis Inc.



On 27 Feb 2014, at 03:22, Joshua Harlow harlo...@yahoo-inc.com wrote:

 So this design is starting to look pretty familiar to a what we have in 
 taskflow.
 
 Any reason why it can't just be used instead?
 
 https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
 
 This code is in a functional state right now, using kombu (for the moment, 
 until oslo.messaging becomes py3 compliant).
 
 The concept of a engine which puts messages on a queue for a remote executor 
 is in-fact exactly the case taskflow is doing (the remote exeuctor/worker 
 will then respond when it is done and the engine will then initiate the next 
 piece of work to do) in the above listed etherpad (and which is implemented).
 
 Is it the case that in mistral the engine will be maintaining the 
 'orchestration' of the workflow during the lifetime of that workflow? In the 
 case of mistral what is an engine server? Is this a server that has engines 
 in it (where each engine is 'orchestrating' the remote/local workflows and 
 monitoring and recording the state transitions and data flow that is 
 occurring)? The details @ 
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
  seems to be already what taskflow provides via its engine object, creating a 
 application which runs engines and those engines initiate workflows is made 
 to be dead simple.
 
 From previous discussions with the mistral folks it seems like the overlap 
 here is getting more and more, which seems to be bad (and means something is 
 broken/wrong). In fact most of the concepts that u have blueprints for have 
 already been completed in taskflow (data-flow, engine being disconnected from 
 the rest api…) and ones u don't have listed (resumption, reversion…). 
 
 What can we do to fix this situation?
 
 -Josh
 
 From: Nikolay Makhotkin nmakhot...@mirantis.com
 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Tuesday, February 25, 2014 at 11:30 PM
 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
 oslo.messaging
 
 Looks good. Thanks, Winson! 
 
 Renat, What do you think?
 
 
 On Wed, Feb 26, 2014 at 10:00 AM, W Chan m4d.co...@gmail.com wrote:
 The following link is the google doc of the proposed engine/executor 
 message flow architecture.  
 https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing
   
 
 The diagram on the right is the scalable engine where one or more engine 
 sends requests over a transport to one or more executors.  The executor 
 client, transport, and executor server follows the RPC client/server design 
 pattern in oslo.messaging.
 
 The diagram represents the local engine.  In reality, it's following the 
 same RPC client/server design pattern.  The only difference is that it'll 
 be configured to use a fake RPC backend driver.  The fake driver uses in 
 process queues shared between a pair of engine and executor.
 
 The following are the stepwise changes I will make.
 1) Keep the local and scalable engine structure intact.  Create the 
 Executor Client at ./mistral/engine/scalable/executor/client.py.  Create 
 the Executor Server at ./mistral/engine/scalable/executor/service.py and 
 implement the task operations under 
 ./mistral/engine/scalable/executor/executor.py.  Delete 
 ./mistral/engine/scalable/executor/executor.py.  Modify the launcher 
 ./mistral/cmd/task_executor.py.  Modify ./mistral/engine/scalable/engine.py 
 to use the Executor Client instead of sending the message directly to 
 rabbit via pika.  The sum of this is the atomic change that keeps existing 
 structure and without breaking the code.
 2) Remove the local engine. 
 https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor
 3) Implement versioning for the engine.  
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning
 4) Port abstract engine to use oslo.messaging and implement the engine 
 client, engine server, and modify the API layer to consume the engine 
 client. 
 https://blueprints.launchpad.net

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-28 Thread Joshua Harlow
Sounds good,

Lets connect, the value of central oslo connected projects is that shared 
libraries == share the pain. Duplicating features and functionality is always 
more pain. In the end we are a community, not silos, so it seems like before 
mistral goes down the path of duplicating more and more features (I understand 
the desire to POC mistral and learn what mistral wants to become, and all that) 
that we should start the path to working together. I personally am worried that 
mistral will start to apply for incubation and then the question will come up 
as to this (mistral was doing POC, kept on doing POC, never came back to using 
common libraries, and then gets asked why this happened).

I'd like to make us all successful, and as a old saying goes,

“A single twig breaks, but the bundle of twigs is strong”, openstack needs to 
be a cohesive bundle and not a single twig ;)

From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, February 28, 2014 at 6:31 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

Hi Joshua,

Sorry, I’ve been very busy for the last couple of days and didn’t respond 
quickly enough.

Well, first of all, it’s my bad that I’ve not been following TaskFlow progress 
for a while and, honestly, I just need to get more info on the current TaskFlow 
status. So I’ll do that and get back to you soon. As you know, there were 
reasons why we decided to go this path (without using TaskFlow) but I’ve always 
thought we will be able to align our efforts as we move forward once 
requirements and design of Mistral become more clear. I really want to use 
TaskFlow for Mistral implementation. We just need to make sure that it will 
bring more value than pain (sorry if it sounds harsh).

Thanks for your feedback and this info. We’ll get in touch with you soon.

Renat Akhmerov
@ Mirantis Inc.



On 27 Feb 2014, at 03:22, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:

So this design is starting to look pretty familiar to a what we have in 
taskflow.

Any reason why it can't just be used instead?

https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

This code is in a functional state right now, using kombu (for the moment, 
until oslo.messaging becomes py3 compliant).

The concept of a engine which puts messages on a queue for a remote executor is 
in-fact exactly the case taskflow is doing (the remote exeuctor/worker will 
then respond when it is done and the engine will then initiate the next piece 
of work to do) in the above listed etherpad (and which is implemented).

Is it the case that in mistral the engine will be maintaining the 
'orchestration' of the workflow during the lifetime of that workflow? In the 
case of mistral what is an engine server? Is this a server that has engines in 
it (where each engine is 'orchestrating' the remote/local workflows and 
monitoring and recording the state transitions and data flow that is 
occurring)? The details @ 
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
 seems to be already what taskflow provides via its engine object, creating a 
application which runs engines and those engines initiate workflows is made to 
be dead simple.

From previous discussions with the mistral folks it seems like the overlap 
here is getting more and more, which seems to be bad (and means something is 
broken/wrong). In fact most of the concepts that u have blueprints for have 
already been completed in taskflow (data-flow, engine being disconnected from 
the rest api…) and ones u don't have listed (resumption, reversion…).

What can we do to fix this situation?

-Josh

From: Nikolay Makhotkin 
nmakhot...@mirantis.commailto:nmakhot...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, February 25, 2014 at 11:30 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

Looks good. Thanks, Winson!

Renat, What do you think?


On Wed, Feb 26, 2014 at 10:00 AM, W Chan 
m4d.co...@gmail.commailto:m4d.co...@gmail.com wrote:
The following link is the google doc of the proposed engine/executor message 
flow architecture.  
https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing

The diagram on the right is the scalable engine where one or more engine sends 
requests over a transport to one or more executors.  The executor client, 
transport

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-28 Thread W Chan
All,
This is a great start.  I think the sooner we have this discussion the
better.  Any uncertainty in the direction/architecture here is going to
stall progress.  How about Convection?  What's the status of the Convection
project and where it's heading?  Should we have similar discussion with the
contributors of that project?

Joshua,
I have a few questions about TaskFlow.
1) How does it handle conditional loop and expression evaluation for
decision branching?  I've looked at the Taskflow wiki/code briefly and it's
not obvious.  I assume it would be logic that user will embed within a task?
2) How about predefined catalog of standard tasks (i.e. REST call, SOAP
call, Email task, etc.)?  Is that within the scope of Taskflow or up to
TaskFlow consumers like Mistral?
3) Does TaskFlow have its own DSL?  The examples provided are mostly code
based.

Thanks.
Winson




On Fri, Feb 28, 2014 at 10:54 AM, Joshua Harlow harlo...@yahoo-inc.comwrote:

  Sounds good,

  Lets connect, the value of central oslo connected projects is that
 shared libraries == share the pain. Duplicating features and functionality
 is always more pain. In the end we are a community, not silos, so it seems
 like before mistral goes down the path of duplicating more and more
 features (I understand the desire to POC mistral and learn what mistral
 wants to become, and all that) that we should start the path to working
 together. I personally am worried that mistral will start to apply for
 incubation and then the question will come up as to this (mistral was doing
 POC, kept on doing POC, never came back to using common libraries, and then
 gets asked why this happened).

  I'd like to make us all successful, and as a old saying goes,

  A single twig breaks, but the bundle of twigs is strong, openstack
 needs to be a cohesive bundle and not a single twig ;)

   From: Renat Akhmerov rakhme...@mirantis.com

 Reply-To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Date: Friday, February 28, 2014 at 6:31 AM

 To: OpenStack Development Mailing List (not for usage questions) 
 openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to
 oslo.messaging

   Hi Joshua,

  Sorry, I've been very busy for the last couple of days and didn't
 respond quickly enough.

  Well, first of all, it's my bad that I've not been following TaskFlow
 progress for a while and, honestly, I just need to get more info on the
 current TaskFlow status. So I'll do that and get back to you soon. As you
 know, there were reasons why we decided to go this path (without using
 TaskFlow) but I've always thought we will be able to align our efforts as
 we move forward once requirements and design of Mistral become more clear.
 I really *want* to use TaskFlow for Mistral implementation. We just need
 to make sure that it will bring more value than pain (sorry if it sounds
 harsh).

  Thanks for your feedback and this info. We'll get in touch with you soon.

  Renat Akhmerov
 @ Mirantis Inc.



  On 27 Feb 2014, at 03:22, Joshua Harlow harlo...@yahoo-inc.com wrote:

  So this design is starting to look pretty familiar to a what we have in
 taskflow.

  Any reason why it can't just be used instead?

  https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

  This code is in a functional state right now, using kombu (for the
 moment, until oslo.messaging becomes py3 compliant).

  The concept of a engine which puts messages on a queue for a remote
 executor is in-fact exactly the case taskflow is doing (the remote
 exeuctor/worker will then respond when it is done and the engine will then
 initiate the next piece of work to do) in the above listed etherpad (and
 which is implemented).

  Is it the case that in mistral the engine will be maintaining the
 'orchestration' of the workflow during the lifetime of that workflow? In
 the case of mistral what is an engine server? Is this a server that has
 engines in it (where each engine is 'orchestrating' the remote/local
 workflows and monitoring and recording the state transitions and data flow
 that is occurring)? The details @
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
  seems
 to be already what taskflow provides via its engine object, creating a
 application which runs engines and those engines initiate workflows is made
 to be dead simple.

  From previous discussions with the mistral folks it seems like the
 overlap here is getting more and more, which seems to be bad (and means
 something is broken/wrong). In fact most of the concepts that u have
 blueprints for have already been completed in taskflow (data-flow, engine
 being disconnected from the rest api...) and ones u don't have listed
 (resumption, reversion...).

  What can we do to fix this situation?

  -Josh

   From: Nikolay Makhotkin nmakhot...@mirantis.com
 Reply-To: OpenStack Development Mailing List (not for usage questions

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-28 Thread Joshua Harlow
Convection? Afaik u guys are building convection (convection was just an idea, 
I see mistral as the POC/impl) ;)

https://wiki.openstack.org/wiki/Convection#NOTICE:_Similar_project_-.3E_Mistral

So questions around taskflow:

  1.  Correct u put it in your task, there was previous ideas/work done by the 
team @ https://etherpad.openstack.org/p/BrainstormFlowConditions but from 
previous people that have build said systems it was determined that actually 
there wasn't much use for conditionals being useful (yet). But expression 
evaluation, not sure what that means, being a library, any type of expression 
evaluation is just whatever u can imagine in python. Conditional tasks (and 
such) being managed by taskflows engines we can reconsider  might even be 
possible but this is imho dangerous territory that is being approached, 
expression evaluation and conditional branching and loops is basically a 
language specification ;)
  2.  I don't see taskflow managing a catalog (currently), that seems out of 
scope of a library that provides the execution, resumption parts (any consumer 
of taskflow should be free to define and organize there catalog as they choose).
  3.  Negative, taskflow is a execution and state-management library (not a 
full framework imho) that helps build the upper layers that services like 
mistral can use (or nova, or glance or…). I don't feel its the right place to 
have taskflow force a DSL onto people, since the underlying primitives that can 
form a upper level DSL are more service/app level choices (heat has there DSL, 
mistral has theres, both are fine, and both likely can take advantage of the 
same taskflow execution and state-management primitives to use in there 
service).

Hope that helps :)

-Josh

From: W Chan m4d.co...@gmail.commailto:m4d.co...@gmail.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, February 28, 2014 at 12:02 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

All,
This is a great start.  I think the sooner we have this discussion the better.  
Any uncertainty in the direction/architecture here is going to stall progress.  
How about Convection?  What's the status of the Convection project and where 
it's heading?  Should we have similar discussion with the contributors of that 
project?

Joshua,
I have a few questions about TaskFlow.
1) How does it handle conditional loop and expression evaluation for decision 
branching?  I've looked at the Taskflow wiki/code briefly and it's not obvious. 
 I assume it would be logic that user will embed within a task?
2) How about predefined catalog of standard tasks (i.e. REST call, SOAP call, 
Email task, etc.)?  Is that within the scope of Taskflow or up to TaskFlow 
consumers like Mistral?
3) Does TaskFlow have its own DSL?  The examples provided are mostly code based.

Thanks.
Winson




On Fri, Feb 28, 2014 at 10:54 AM, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:
Sounds good,

Lets connect, the value of central oslo connected projects is that shared 
libraries == share the pain. Duplicating features and functionality is always 
more pain. In the end we are a community, not silos, so it seems like before 
mistral goes down the path of duplicating more and more features (I understand 
the desire to POC mistral and learn what mistral wants to become, and all that) 
that we should start the path to working together. I personally am worried that 
mistral will start to apply for incubation and then the question will come up 
as to this (mistral was doing POC, kept on doing POC, never came back to using 
common libraries, and then gets asked why this happened).

I'd like to make us all successful, and as a old saying goes,

“A single twig breaks, but the bundle of twigs is strong”, openstack needs to 
be a cohesive bundle and not a single twig ;)

From: Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com

Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, February 28, 2014 at 6:31 AM

To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

Hi Joshua,

Sorry, I’ve been very busy for the last couple of days and didn’t respond 
quickly enough.

Well, first of all, it’s my bad that I’ve not been following TaskFlow progress 
for a while and, honestly, I just need to get more info on the current TaskFlow 
status. So I’ll do that and get back to you soon. As you know, there were 
reasons why we decided to go this path

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-28 Thread Keith Bray
Hey, yeah, so I started the Convection wiki and discussed that idea at the 
Portland summit in 2013.  Mirantis picked up the Convection proposal and 
decided to run with it in a project named Mistral from what I understand...  
Their was apparently some concerns over name infringement, which is why the 
name Convection was not kept when they began their Mistral proof-of-concept.  
Convection was an idea proposed.  There is no code for Convection, and no 
active contributors to Convection.  In the spirit of collaboration, if folks 
are interested in contributing to a Workflow system, I would encourage you to 
get involved in Mistral and see if it fits your needs... If not, I'm sure the 
Mistral (and other openstack folks) would love to have a discussion about that.

Kind regards,
-Keith

From: Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, February 28, 2014 2:46 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, 
W Chan m4d.co...@gmail.commailto:m4d.co...@gmail.com
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

Convection? Afaik u guys are building convection (convection was just an idea, 
I see mistral as the POC/impl) ;)

https://wiki.openstack.org/wiki/Convection#NOTICE:_Similar_project_-.3E_Mistral

So questions around taskflow:

  1.  Correct u put it in your task, there was previous ideas/work done by the 
team @ https://etherpad.openstack.org/p/BrainstormFlowConditions but from 
previous people that have build said systems it was determined that actually 
there wasn't much use for conditionals being useful (yet). But expression 
evaluation, not sure what that means, being a library, any type of expression 
evaluation is just whatever u can imagine in python. Conditional tasks (and 
such) being managed by taskflows engines we can reconsider  might even be 
possible but this is imho dangerous territory that is being approached, 
expression evaluation and conditional branching and loops is basically a 
language specification ;)
  2.  I don't see taskflow managing a catalog (currently), that seems out of 
scope of a library that provides the execution, resumption parts (any consumer 
of taskflow should be free to define and organize there catalog as they choose).
  3.  Negative, taskflow is a execution and state-management library (not a 
full framework imho) that helps build the upper layers that services like 
mistral can use (or nova, or glance or…). I don't feel its the right place to 
have taskflow force a DSL onto people, since the underlying primitives that can 
form a upper level DSL are more service/app level choices (heat has there DSL, 
mistral has theres, both are fine, and both likely can take advantage of the 
same taskflow execution and state-management primitives to use in there 
service).

Hope that helps :)

-Josh

From: W Chan m4d.co...@gmail.commailto:m4d.co...@gmail.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Friday, February 28, 2014 at 12:02 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

All,
This is a great start.  I think the sooner we have this discussion the better.  
Any uncertainty in the direction/architecture here is going to stall progress.  
How about Convection?  What's the status of the Convection project and where 
it's heading?  Should we have similar discussion with the contributors of that 
project?

Joshua,
I have a few questions about TaskFlow.
1) How does it handle conditional loop and expression evaluation for decision 
branching?  I've looked at the Taskflow wiki/code briefly and it's not obvious. 
 I assume it would be logic that user will embed within a task?
2) How about predefined catalog of standard tasks (i.e. REST call, SOAP call, 
Email task, etc.)?  Is that within the scope of Taskflow or up to TaskFlow 
consumers like Mistral?
3) Does TaskFlow have its own DSL?  The examples provided are mostly code based.

Thanks.
Winson




On Fri, Feb 28, 2014 at 10:54 AM, Joshua Harlow 
harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote:
Sounds good,

Lets connect, the value of central oslo connected projects is that shared 
libraries == share the pain. Duplicating features and functionality is always 
more pain. In the end we are a community, not silos, so it seems like before 
mistral goes down the path of duplicating more and more features (I understand 
the desire to POC mistral and learn what mistral wants to become, and all that) 
that we should start

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-26 Thread Renat Akhmerov
Winson, nice job!

Now it totally makes sense to me. You’re good to go with this unless others 
have objections.

Just one technical dummy question (sorry, I’m not yet familiar with 
oslo.messaging): at your picture you have “Transport”, so what can be 
specifically except RabbitMQ? 

Renat Akhmerov
@ Mirantis Inc.



On 26 Feb 2014, at 14:30, Nikolay Makhotkin nmakhot...@mirantis.com wrote:

 Looks good. Thanks, Winson! 
 
 Renat, What do you think?
 
 
 On Wed, Feb 26, 2014 at 10:00 AM, W Chan m4d.co...@gmail.com wrote:
 The following link is the google doc of the proposed engine/executor message 
 flow architecture.  
 https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing 
  
 
 The diagram on the right is the scalable engine where one or more engine 
 sends requests over a transport to one or more executors.  The executor 
 client, transport, and executor server follows the RPC client/server design 
 pattern in oslo.messaging.
 
 The diagram represents the local engine.  In reality, it's following the same 
 RPC client/server design pattern.  The only difference is that it'll be 
 configured to use a fake RPC backend driver.  The fake driver uses in process 
 queues shared between a pair of engine and executor.
 
 The following are the stepwise changes I will make.
 1) Keep the local and scalable engine structure intact.  Create the Executor 
 Client at ./mistral/engine/scalable/executor/client.py.  Create the Executor 
 Server at ./mistral/engine/scalable/executor/service.py and implement the 
 task operations under ./mistral/engine/scalable/executor/executor.py.  Delete 
 ./mistral/engine/scalable/executor/executor.py.  Modify the launcher 
 ./mistral/cmd/task_executor.py.  Modify ./mistral/engine/scalable/engine.py 
 to use the Executor Client instead of sending the message directly to rabbit 
 via pika.  The sum of this is the atomic change that keeps existing structure 
 and without breaking the code.
 2) Remove the local engine. 
 https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor
 3) Implement versioning for the engine.  
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning
 4) Port abstract engine to use oslo.messaging and implement the engine 
 client, engine server, and modify the API layer to consume the engine client. 
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process.
 
 Winson
 
 
 On Mon, Feb 24, 2014 at 8:07 PM, Renat Akhmerov rakhme...@mirantis.com 
 wrote:
 
 On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote:
 
 Renat,
 
 Regarding your comments on change https://review.openstack.org/#/c/75609/, I 
 don't think the port to oslo.messaging is just a swap from pika to 
 oslo.messaging.  OpenStack services as I understand is usually implemented 
 as an RPC client/server over a messaging transport.  Sync vs async calls are 
 done via the RPC client call and cast respectively.  The messaging transport 
 is abstracted and concrete implementation is done via drivers/plugins.  So 
 the architecture of the executor if ported to oslo.messaging needs to 
 include a client, a server, and a transport.  The consumer (in this case the 
 mistral engine) instantiates an instance of the client for the executor, 
 makes the method call to handle task, the client then sends the request over 
 the transport to the server.  The server picks up the request from the 
 exchange and processes the request.  If cast (async), the client side 
 returns immediately.  If call (sync), the client side waits for a response 
 from the server over a reply_q (a unique queue for the session in the 
 transport).  Also, oslo.messaging allows versioning in the message. Major 
 version change indicates API contract changes.  Minor version indicates 
 backend changes but with API compatibility.  
 
 My main concern about this patch is not related with messaging 
 infrastructure. I believe you know better than me how it should look like. 
 I’m mostly concerned with the way of making changes you chose. From my 
 perspective, it’s much better to make atomic changes where every changes 
 doesn’t affect too much in existing architecture. So the first step could be 
 to change pika to oslo.messaging with minimal structural changes without 
 introducing versioning (could be just TODO comment saying that the framework 
 allows it and we may want to use it in the future, to be decide), without 
 getting rid of the current engine structure (local, scalable). Some of the 
 things in the file structure and architecture came from the decisions made by 
 many people and we need to be careful about changing them.
 
 
 So, where I'm headed with this change...  I'm implementing the basic 
 structure/scaffolding for the new executor service using oslo.messaging 
 (default transport with rabbit).  Since the whole change will take a few 
 rounds, I don't want to disrupt any changes that the team is making at the 
 moment and so I'm building 

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-26 Thread W Chan
Thanks.  I'll start making the changes.  The other transport currently
implemented at oslo.messaging is located at
https://github.com/openstack/oslo.messaging/tree/master/oslo/messaging/_drivers,
prefixed with impl.  There are quid and zmq.


On Wed, Feb 26, 2014 at 12:03 AM, Renat Akhmerov rakhme...@mirantis.comwrote:

 Winson, nice job!

 Now it totally makes sense to me. You're good to go with this unless
 others have objections.

 Just one technical dummy question (sorry, I'm not yet familiar with
 oslo.messaging): at your picture you have Transport, so what can be
 specifically except RabbitMQ?

 Renat Akhmerov
 @ Mirantis Inc.



 On 26 Feb 2014, at 14:30, Nikolay Makhotkin nmakhot...@mirantis.com
 wrote:

 Looks good. Thanks, Winson!

 Renat, What do you think?


 On Wed, Feb 26, 2014 at 10:00 AM, W Chan m4d.co...@gmail.com wrote:

 The following link is the google doc of the proposed engine/executor
 message flow architecture.
 https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing

 The diagram on the right is the scalable engine where one or more engine
 sends requests over a transport to one or more executors.  The executor
 client, transport, and executor server follows the RPC client/server
 design 
 patternhttps://github.com/openstack/oslo.messaging/tree/master/oslo/messaging/rpcin
  oslo.messaging.

 The diagram represents the local engine.  In reality, it's following the
 same RPC client/server design pattern.  The only difference is that it'll
 be configured to use a 
 fakehttps://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_fake.pyRPC
  backend driver.  The fake driver uses in process
 queues http://docs.python.org/2/library/queue.html#module-Queue shared
 between a pair of engine and executor.

 The following are the stepwise changes I will make.
 1) Keep the local and scalable engine structure intact.  Create the
 Executor Client at ./mistral/engine/scalable/executor/client.py.  Create
 the Executor Server at ./mistral/engine/scalable/executor/service.py and
 implement the task operations under
 ./mistral/engine/scalable/executor/executor.py.  Delete
 ./mistral/engine/scalable/executor/executor.py.  Modify the launcher
 ./mistral/cmd/task_executor.py.  Modify ./mistral/engine/scalable/engine.py
 to use the Executor Client instead of sending the message directly to
 rabbit via pika.  The sum of this is the atomic change that keeps existing
 structure and without breaking the code.
 2) Remove the local engine.
 https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor
 3) Implement versioning for the engine.
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning
 4) Port abstract engine to use oslo.messaging and implement the engine
 client, engine server, and modify the API layer to consume the engine
 client.
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
 .

 Winson


 On Mon, Feb 24, 2014 at 8:07 PM, Renat Akhmerov 
 rakhme...@mirantis.comwrote:


 On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote:

 Renat,

 Regarding your comments on change
 https://review.openstack.org/#/c/75609/, I don't think the port to
 oslo.messaging is just a swap from pika to oslo.messaging.  OpenStack
 services as I understand is usually implemented as an RPC client/server
 over a messaging transport.  Sync vs async calls are done via the RPC
 client call and cast respectively.  The messaging transport is abstracted
 and concrete implementation is done via drivers/plugins.  So the
 architecture of the executor if ported to oslo.messaging needs to include a
 client, a server, and a transport.  The consumer (in this case the mistral
 engine) instantiates an instance of the client for the executor, makes the
 method call to handle task, the client then sends the request over the
 transport to the server.  The server picks up the request from the exchange
 and processes the request.  If cast (async), the client side returns
 immediately.  If call (sync), the client side waits for a response from the
 server over a reply_q (a unique queue for the session in the transport).
  Also, oslo.messaging allows versioning in the message. Major version
 change indicates API contract changes.  Minor version indicates backend
 changes but with API compatibility.


 My main concern about this patch is not related with messaging
 infrastructure. I believe you know better than me how it should look like.
 I'm mostly concerned with the way of making changes you chose. From my
 perspective, it's much better to make atomic changes where every changes
 doesn't affect too much in existing architecture. So the first step could
 be to change pika to oslo.messaging with minimal structural changes without
 introducing versioning (could be just TODO comment saying that the
 framework allows it and we may want to use it in the future, to be decide),
 without getting rid of the current engine structure 

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-26 Thread Joshua Harlow
So this design is starting to look pretty familiar to a what we have in 
taskflow.

Any reason why it can't just be used instead?

https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

This code is in a functional state right now, using kombu (for the moment, 
until oslo.messaging becomes py3 compliant).

The concept of a engine which puts messages on a queue for a remote executor is 
in-fact exactly the case taskflow is doing (the remote exeuctor/worker will 
then respond when it is done and the engine will then initiate the next piece 
of work to do) in the above listed etherpad (and which is implemented).

Is it the case that in mistral the engine will be maintaining the 
'orchestration' of the workflow during the lifetime of that workflow? In the 
case of mistral what is an engine server? Is this a server that has engines in 
it (where each engine is 'orchestrating' the remote/local workflows and 
monitoring and recording the state transitions and data flow that is 
occurring)? The details @ 
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
 seems to be already what taskflow provides via its engine object, creating a 
application which runs engines and those engines initiate workflows is made to 
be dead simple.

From previous discussions with the mistral folks it seems like the overlap 
here is getting more and more, which seems to be bad (and means something is 
broken/wrong). In fact most of the concepts that u have blueprints for have 
already been completed in taskflow (data-flow, engine being disconnected from 
the rest api…) and ones u don't have listed (resumption, reversion…).

What can we do to fix this situation?

-Josh

From: Nikolay Makhotkin 
nmakhot...@mirantis.commailto:nmakhot...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, February 25, 2014 at 11:30 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to 
oslo.messaging

Looks good. Thanks, Winson!

Renat, What do you think?


On Wed, Feb 26, 2014 at 10:00 AM, W Chan 
m4d.co...@gmail.commailto:m4d.co...@gmail.com wrote:
The following link is the google doc of the proposed engine/executor message 
flow architecture.  
https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing

The diagram on the right is the scalable engine where one or more engine sends 
requests over a transport to one or more executors.  The executor client, 
transport, and executor server follows the RPC client/server design 
patternhttps://github.com/openstack/oslo.messaging/tree/master/oslo/messaging/rpc
 in oslo.messaging.

The diagram represents the local engine.  In reality, it's following the same 
RPC client/server design pattern.  The only difference is that it'll be 
configured to use a 
fakehttps://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_fake.py
 RPC backend driver.  The fake driver uses in process 
queueshttp://docs.python.org/2/library/queue.html#module-Queue shared between 
a pair of engine and executor.

The following are the stepwise changes I will make.
1) Keep the local and scalable engine structure intact.  Create the Executor 
Client at ./mistral/engine/scalable/executor/client.py.  Create the Executor 
Server at ./mistral/engine/scalable/executor/service.py and implement the task 
operations under ./mistral/engine/scalable/executor/executor.py.  Delete 
./mistral/engine/scalable/executor/executor.py.  Modify the launcher 
./mistral/cmd/task_executor.py.  Modify ./mistral/engine/scalable/engine.py to 
use the Executor Client instead of sending the message directly to rabbit via 
pika.  The sum of this is the atomic change that keeps existing structure and 
without breaking the code.
2) Remove the local engine. 
https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor
3) Implement versioning for the engine.  
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning
4) Port abstract engine to use oslo.messaging and implement the engine client, 
engine server, and modify the API layer to consume the engine client. 
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process.

Winson


On Mon, Feb 24, 2014 at 8:07 PM, Renat Akhmerov 
rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote:

On 25 Feb 2014, at 02:21, W Chan 
m4d.co...@gmail.commailto:m4d.co...@gmail.com wrote:

Renat,

Regarding your comments on change https://review.openstack.org/#/c/75609/, I 
don't think the port to oslo.messaging is just a swap from pika to 
oslo.messaging.  OpenStack services as I understand is usually implemented as 
an RPC client/server over a messaging transport.  Sync vs async calls are done 
via the RPC client call and cast

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-25 Thread W Chan
Sure.  Let me give this some thoughts and work with you separately.  Before
we speak up, we should have a proposal for discussion.


On Mon, Feb 24, 2014 at 9:53 PM, Dmitri Zimine d...@stackstorm.com wrote:

 Winson,

 While you're looking into this and working on the design, may be also
 think through other executor/engine communications.

 We talked about executor communicating to engine over 3 channels (DB,
 REST, RabbitMQ) which I wasn't happy about ;) and put it off for some time.
 May be it can be rationalized as part of your design.

 DZ.

 On Feb 24, 2014, at 11:21 AM, W Chan m4d.co...@gmail.com wrote:

 Renat,

 Regarding your comments on change https://review.openstack.org/#/c/75609/,
 I don't think the port to oslo.messaging is just a swap from pika to
 oslo.messaging.  OpenStack services as I understand is usually implemented
 as an RPC client/server over a messaging transport.  Sync vs async calls
 are done via the RPC client call and cast respectively.  The messaging
 transport is abstracted and concrete implementation is done via
 drivers/plugins.  So the architecture of the executor if ported to
 oslo.messaging needs to include a client, a server, and a transport.  The
 consumer (in this case the mistral engine) instantiates an instance of the
 client for the executor, makes the method call to handle task, the client
 then sends the request over the transport to the server.  The server picks
 up the request from the exchange and processes the request.  If cast
 (async), the client side returns immediately.  If call (sync), the client
 side waits for a response from the server over a reply_q (a unique queue
 for the session in the transport).  Also, oslo.messaging allows versioning
 in the message. Major version change indicates API contract changes.  Minor
 version indicates backend changes but with API compatibility.

 So, where I'm headed with this change...  I'm implementing the basic
 structure/scaffolding for the new executor service using oslo.messaging
 (default transport with rabbit).  Since the whole change will take a few
 rounds, I don't want to disrupt any changes that the team is making at the
 moment and so I'm building the structure separately.  I'm also adding
 versioning (v1) in the module structure to anticipate any versioning
 changes in the future.   I expect the change request will lead to some
 discussion as we are doing here.  I will migrate the core operations of the
 executor (handle_task, handle_task_error, do_task_action) to the server
 component when we agree on the architecture and switch the consumer
 (engine) to use the new RPC client for the executor instead of sending the
 message to the queue over pika.  Also, the launcher for
 ./mistral/cmd/task_executor.py will change as well in subsequent round.  An
 example launcher is here
 https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine.
  The interceptor project here is what I use to research how oslo.messaging
 works.  I hope this is clear. The blueprint only changes how the request
 and response are being transported.  It shouldn't change how the executor
 currently works.

 Finally, can you clarify the difference between local vs scalable engine?
  I personally do not prefer to explicitly name the engine scalable because
 this requirement should be in the engine by default and we do not need to
 explicitly state/separate that.  But if this is a roadblock for the change,
 I can put the scalable structure back in the change to move this forward.

 Thanks.
 Winson

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-25 Thread Renat Akhmerov
Yes, right. Thanks Winson.

Renat Akhmerov
@ Mirantis Inc.



On 26 Feb 2014, at 01:39, W Chan m4d.co...@gmail.com wrote:

 Sure.  Let me give this some thoughts and work with you separately.  Before 
 we speak up, we should have a proposal for discussion.
 
 
 On Mon, Feb 24, 2014 at 9:53 PM, Dmitri Zimine d...@stackstorm.com wrote:
 Winson, 
 
 While you're looking into this and working on the design, may be also think 
 through other executor/engine communications.
 
 We talked about executor communicating to engine over 3 channels (DB, REST, 
 RabbitMQ) which I wasn't happy about ;) and put it off for some time. May be 
 it can be rationalized as part of your design. 
 
 DZ. 
 
 On Feb 24, 2014, at 11:21 AM, W Chan m4d.co...@gmail.com wrote:
 
 Renat,
 
 Regarding your comments on change https://review.openstack.org/#/c/75609/, I 
 don't think the port to oslo.messaging is just a swap from pika to 
 oslo.messaging.  OpenStack services as I understand is usually implemented 
 as an RPC client/server over a messaging transport.  Sync vs async calls are 
 done via the RPC client call and cast respectively.  The messaging transport 
 is abstracted and concrete implementation is done via drivers/plugins.  So 
 the architecture of the executor if ported to oslo.messaging needs to 
 include a client, a server, and a transport.  The consumer (in this case the 
 mistral engine) instantiates an instance of the client for the executor, 
 makes the method call to handle task, the client then sends the request over 
 the transport to the server.  The server picks up the request from the 
 exchange and processes the request.  If cast (async), the client side 
 returns immediately.  If call (sync), the client side waits for a response 
 from the server over a reply_q (a unique queue for the session in the 
 transport).  Also, oslo.messaging allows versioning in the message. Major 
 version change indicates API contract changes.  Minor version indicates 
 backend changes but with API compatibility.  
 
 So, where I'm headed with this change...  I'm implementing the basic 
 structure/scaffolding for the new executor service using oslo.messaging 
 (default transport with rabbit).  Since the whole change will take a few 
 rounds, I don't want to disrupt any changes that the team is making at the 
 moment and so I'm building the structure separately.  I'm also adding 
 versioning (v1) in the module structure to anticipate any versioning changes 
 in the future.   I expect the change request will lead to some discussion as 
 we are doing here.  I will migrate the core operations of the executor 
 (handle_task, handle_task_error, do_task_action) to the server component 
 when we agree on the architecture and switch the consumer (engine) to use 
 the new RPC client for the executor instead of sending the message to the 
 queue over pika.  Also, the launcher for ./mistral/cmd/task_executor.py will 
 change as well in subsequent round.  An example launcher is here 
 https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine.
   The interceptor project here is what I use to research how oslo.messaging 
 works.  I hope this is clear. The blueprint only changes how the request and 
 response are being transported.  It shouldn't change how the executor 
 currently works.
 
 Finally, can you clarify the difference between local vs scalable engine?  I 
 personally do not prefer to explicitly name the engine scalable because this 
 requirement should be in the engine by default and we do not need to 
 explicitly state/separate that.  But if this is a roadblock for the change, 
 I can put the scalable structure back in the change to move this forward.
 
 Thanks.
 Winson
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-25 Thread W Chan
The following link is the google doc of the proposed engine/executor
message flow architecture.
https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing

The diagram on the right is the scalable engine where one or more engine
sends requests over a transport to one or more executors.  The executor
client, transport, and executor server follows the RPC client/server design
patternhttps://github.com/openstack/oslo.messaging/tree/master/oslo/messaging/rpcin
oslo.messaging.

The diagram represents the local engine.  In reality, it's following the
same RPC client/server design pattern.  The only difference is that it'll
be configured to use a
fakehttps://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_fake.pyRPC
backend driver.  The fake driver uses in process
queues http://docs.python.org/2/library/queue.html#module-Queue shared
between a pair of engine and executor.

The following are the stepwise changes I will make.
1) Keep the local and scalable engine structure intact.  Create the
Executor Client at ./mistral/engine/scalable/executor/client.py.  Create
the Executor Server at ./mistral/engine/scalable/executor/service.py and
implement the task operations under
./mistral/engine/scalable/executor/executor.py.  Delete
./mistral/engine/scalable/executor/executor.py.  Modify the launcher
./mistral/cmd/task_executor.py.  Modify ./mistral/engine/scalable/engine.py
to use the Executor Client instead of sending the message directly to
rabbit via pika.  The sum of this is the atomic change that keeps existing
structure and without breaking the code.
2) Remove the local engine.
https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor
3) Implement versioning for the engine.
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning
4) Port abstract engine to use oslo.messaging and implement the engine
client, engine server, and modify the API layer to consume the engine
client.
https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
.

Winson


On Mon, Feb 24, 2014 at 8:07 PM, Renat Akhmerov rakhme...@mirantis.comwrote:


 On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote:

 Renat,

 Regarding your comments on change https://review.openstack.org/#/c/75609/,
 I don't think the port to oslo.messaging is just a swap from pika to
 oslo.messaging.  OpenStack services as I understand is usually implemented
 as an RPC client/server over a messaging transport.  Sync vs async calls
 are done via the RPC client call and cast respectively.  The messaging
 transport is abstracted and concrete implementation is done via
 drivers/plugins.  So the architecture of the executor if ported to
 oslo.messaging needs to include a client, a server, and a transport.  The
 consumer (in this case the mistral engine) instantiates an instance of the
 client for the executor, makes the method call to handle task, the client
 then sends the request over the transport to the server.  The server picks
 up the request from the exchange and processes the request.  If cast
 (async), the client side returns immediately.  If call (sync), the client
 side waits for a response from the server over a reply_q (a unique queue
 for the session in the transport).  Also, oslo.messaging allows versioning
 in the message. Major version change indicates API contract changes.  Minor
 version indicates backend changes but with API compatibility.


 My main concern about this patch is not related with messaging
 infrastructure. I believe you know better than me how it should look like.
 I'm mostly concerned with the way of making changes you chose. From my
 perspective, it's much better to make atomic changes where every changes
 doesn't affect too much in existing architecture. So the first step could
 be to change pika to oslo.messaging with minimal structural changes without
 introducing versioning (could be just TODO comment saying that the
 framework allows it and we may want to use it in the future, to be decide),
 without getting rid of the current engine structure (local, scalable). Some
 of the things in the file structure and architecture came from the
 decisions made by many people and we need to be careful about changing them.


 So, where I'm headed with this change...  I'm implementing the basic
 structure/scaffolding for the new executor service using oslo.messaging
 (default transport with rabbit).  Since the whole change will take a few
 rounds, I don't want to disrupt any changes that the team is making at the
 moment and so I'm building the structure separately.  I'm also adding
 versioning (v1) in the module structure to anticipate any versioning
 changes in the future.   I expect the change request will lead to some
 discussion as we are doing here.  I will migrate the core operations of the
 executor (handle_task, handle_task_error, do_task_action) to the server
 component when we agree on the architecture and switch the consumer
 

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-25 Thread Nikolay Makhotkin
Looks good. Thanks, Winson!

Renat, What do you think?


On Wed, Feb 26, 2014 at 10:00 AM, W Chan m4d.co...@gmail.com wrote:

 The following link is the google doc of the proposed engine/executor
 message flow architecture.
 https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing

 The diagram on the right is the scalable engine where one or more engine
 sends requests over a transport to one or more executors.  The executor
 client, transport, and executor server follows the RPC client/server
 design 
 patternhttps://github.com/openstack/oslo.messaging/tree/master/oslo/messaging/rpcin
  oslo.messaging.

 The diagram represents the local engine.  In reality, it's following the
 same RPC client/server design pattern.  The only difference is that it'll
 be configured to use a 
 fakehttps://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_fake.pyRPC
  backend driver.  The fake driver uses in process
 queues http://docs.python.org/2/library/queue.html#module-Queue shared
 between a pair of engine and executor.

 The following are the stepwise changes I will make.
 1) Keep the local and scalable engine structure intact.  Create the
 Executor Client at ./mistral/engine/scalable/executor/client.py.  Create
 the Executor Server at ./mistral/engine/scalable/executor/service.py and
 implement the task operations under
 ./mistral/engine/scalable/executor/executor.py.  Delete
 ./mistral/engine/scalable/executor/executor.py.  Modify the launcher
 ./mistral/cmd/task_executor.py.  Modify ./mistral/engine/scalable/engine.py
 to use the Executor Client instead of sending the message directly to
 rabbit via pika.  The sum of this is the atomic change that keeps existing
 structure and without breaking the code.
 2) Remove the local engine.
 https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor
 3) Implement versioning for the engine.
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning
 4) Port abstract engine to use oslo.messaging and implement the engine
 client, engine server, and modify the API layer to consume the engine
 client.
 https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process
 .

 Winson


 On Mon, Feb 24, 2014 at 8:07 PM, Renat Akhmerov rakhme...@mirantis.comwrote:


 On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote:

 Renat,

 Regarding your comments on change https://review.openstack.org/#/c/75609/,
 I don't think the port to oslo.messaging is just a swap from pika to
 oslo.messaging.  OpenStack services as I understand is usually implemented
 as an RPC client/server over a messaging transport.  Sync vs async calls
 are done via the RPC client call and cast respectively.  The messaging
 transport is abstracted and concrete implementation is done via
 drivers/plugins.  So the architecture of the executor if ported to
 oslo.messaging needs to include a client, a server, and a transport.  The
 consumer (in this case the mistral engine) instantiates an instance of the
 client for the executor, makes the method call to handle task, the client
 then sends the request over the transport to the server.  The server picks
 up the request from the exchange and processes the request.  If cast
 (async), the client side returns immediately.  If call (sync), the client
 side waits for a response from the server over a reply_q (a unique queue
 for the session in the transport).  Also, oslo.messaging allows versioning
 in the message. Major version change indicates API contract changes.  Minor
 version indicates backend changes but with API compatibility.


 My main concern about this patch is not related with messaging
 infrastructure. I believe you know better than me how it should look like.
 I'm mostly concerned with the way of making changes you chose. From my
 perspective, it's much better to make atomic changes where every changes
 doesn't affect too much in existing architecture. So the first step could
 be to change pika to oslo.messaging with minimal structural changes without
 introducing versioning (could be just TODO comment saying that the
 framework allows it and we may want to use it in the future, to be decide),
 without getting rid of the current engine structure (local, scalable). Some
 of the things in the file structure and architecture came from the
 decisions made by many people and we need to be careful about changing them.


 So, where I'm headed with this change...  I'm implementing the basic
 structure/scaffolding for the new executor service using oslo.messaging
 (default transport with rabbit).  Since the whole change will take a few
 rounds, I don't want to disrupt any changes that the team is making at the
 moment and so I'm building the structure separately.  I'm also adding
 versioning (v1) in the module structure to anticipate any versioning
 changes in the future.   I expect the change request will lead to some
 discussion as we are doing here.  I will migrate the 

Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-24 Thread Renat Akhmerov

On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote:

 Renat,
 
 Regarding your comments on change https://review.openstack.org/#/c/75609/, I 
 don't think the port to oslo.messaging is just a swap from pika to 
 oslo.messaging.  OpenStack services as I understand is usually implemented as 
 an RPC client/server over a messaging transport.  Sync vs async calls are 
 done via the RPC client call and cast respectively.  The messaging transport 
 is abstracted and concrete implementation is done via drivers/plugins.  So 
 the architecture of the executor if ported to oslo.messaging needs to include 
 a client, a server, and a transport.  The consumer (in this case the mistral 
 engine) instantiates an instance of the client for the executor, makes the 
 method call to handle task, the client then sends the request over the 
 transport to the server.  The server picks up the request from the exchange 
 and processes the request.  If cast (async), the client side returns 
 immediately.  If call (sync), the client side waits for a response from the 
 server over a reply_q (a unique queue for the session in the transport).  
 Also, oslo.messaging allows versioning in the message. Major version change 
 indicates API contract changes.  Minor version indicates backend changes but 
 with API compatibility.  

My main concern about this patch is not related with messaging infrastructure. 
I believe you know better than me how it should look like. I’m mostly concerned 
with the way of making changes you chose. From my perspective, it’s much better 
to make atomic changes where every changes doesn’t affect too much in existing 
architecture. So the first step could be to change pika to oslo.messaging with 
minimal structural changes without introducing versioning (could be just TODO 
comment saying that the framework allows it and we may want to use it in the 
future, to be decide), without getting rid of the current engine structure 
(local, scalable). Some of the things in the file structure and architecture 
came from the decisions made by many people and we need to be careful about 
changing them.


 So, where I'm headed with this change...  I'm implementing the basic 
 structure/scaffolding for the new executor service using oslo.messaging 
 (default transport with rabbit).  Since the whole change will take a few 
 rounds, I don't want to disrupt any changes that the team is making at the 
 moment and so I'm building the structure separately.  I'm also adding 
 versioning (v1) in the module structure to anticipate any versioning changes 
 in the future.   I expect the change request will lead to some discussion as 
 we are doing here.  I will migrate the core operations of the executor 
 (handle_task, handle_task_error, do_task_action) to the server component when 
 we agree on the architecture and switch the consumer (engine) to use the new 
 RPC client for the executor instead of sending the message to the queue over 
 pika.  Also, the launcher for ./mistral/cmd/task_executor.py will change as 
 well in subsequent round.  An example launcher is here 
 https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine. 
  The interceptor project here is what I use to research how oslo.messaging 
 works.  I hope this is clear. The blueprint only changes how the request and 
 response are being transported.  It shouldn't change how the executor 
 currently works.

Please create a document describing the approach you’re pursuing here. I would 
expect to see the main goals you want to achieve upon completion.

 Finally, can you clarify the difference between local vs scalable engine?  I 
 personally do not prefer to explicitly name the engine scalable because this 
 requirement should be in the engine by default and we do not need to 
 explicitly state/separate that.  But if this is a roadblock for the change, I 
 can put the scalable structure back in the change to move this forward.

Separation for local and scalable implementations appeared for historical 
reasons because from the beginning we didn’t see how it all would look like and 
hence we tried different approaches to implement the engine. At some point we 
got 2 working versions: the one that didn’t distribute anything (local) and 
another one that could distribute tasks over task executors via asynchronous HA 
transport (scalable). Later on we decided to leave them both since scalable is 
needed by the requirements and local might be useful for demonstration purposes 
and testing since it doesn’t require RabbitMQ to be installed. So we decided to 
refactor both and make them work similarly except the way they run tasks.

Thanks.

Renat Akhmerov
@Mirantis Inc.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging

2014-02-24 Thread Dmitri Zimine
Winson, 

While you're looking into this and working on the design, may be also think 
through other executor/engine communications.

We talked about executor communicating to engine over 3 channels (DB, REST, 
RabbitMQ) which I wasn't happy about ;) and put it off for some time. May be it 
can be rationalized as part of your design. 

DZ. 

On Feb 24, 2014, at 11:21 AM, W Chan m4d.co...@gmail.com wrote:

 Renat,
 
 Regarding your comments on change https://review.openstack.org/#/c/75609/, I 
 don't think the port to oslo.messaging is just a swap from pika to 
 oslo.messaging.  OpenStack services as I understand is usually implemented as 
 an RPC client/server over a messaging transport.  Sync vs async calls are 
 done via the RPC client call and cast respectively.  The messaging transport 
 is abstracted and concrete implementation is done via drivers/plugins.  So 
 the architecture of the executor if ported to oslo.messaging needs to include 
 a client, a server, and a transport.  The consumer (in this case the mistral 
 engine) instantiates an instance of the client for the executor, makes the 
 method call to handle task, the client then sends the request over the 
 transport to the server.  The server picks up the request from the exchange 
 and processes the request.  If cast (async), the client side returns 
 immediately.  If call (sync), the client side waits for a response from the 
 server over a reply_q (a unique queue for the session in the transport).  
 Also, oslo.messaging allows versioning in the message. Major version change 
 indicates API contract changes.  Minor version indicates backend changes but 
 with API compatibility.  
 
 So, where I'm headed with this change...  I'm implementing the basic 
 structure/scaffolding for the new executor service using oslo.messaging 
 (default transport with rabbit).  Since the whole change will take a few 
 rounds, I don't want to disrupt any changes that the team is making at the 
 moment and so I'm building the structure separately.  I'm also adding 
 versioning (v1) in the module structure to anticipate any versioning changes 
 in the future.   I expect the change request will lead to some discussion as 
 we are doing here.  I will migrate the core operations of the executor 
 (handle_task, handle_task_error, do_task_action) to the server component when 
 we agree on the architecture and switch the consumer (engine) to use the new 
 RPC client for the executor instead of sending the message to the queue over 
 pika.  Also, the launcher for ./mistral/cmd/task_executor.py will change as 
 well in subsequent round.  An example launcher is here 
 https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine. 
  The interceptor project here is what I use to research how oslo.messaging 
 works.  I hope this is clear. The blueprint only changes how the request and 
 response are being transported.  It shouldn't change how the executor 
 currently works.
 
 Finally, can you clarify the difference between local vs scalable engine?  I 
 personally do not prefer to explicitly name the engine scalable because this 
 requirement should be in the engine by default and we do not need to 
 explicitly state/separate that.  But if this is a roadblock for the change, I 
 can put the scalable structure back in the change to move this forward.
 
 Thanks.
 Winson
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev