Re: [openstack-dev] [Mistral] Mistral milestone 0.1 blueprint review
Thanks, Dmitri, I left my comments in the document you shared. Please take a loot at them. We also need to define a reasonable date for 0.1 release which is now set to Sept 11 according to my very raw estimates. We may want to move it a little bit. Renat Akhmerov @ Mirantis Inc. On 14 Aug 2014, at 08:30, Dmitri Zimine dzim...@stackstorm.com wrote: Renat, team: The current plan of record on 0.1 here, https://launchpad.net/mistral/+milestone/0.1 Some suggested adjustments: https://docs.google.com/a/stackstorm.com/document/d/1Ukd6SDk9tMpn8kpti-kXaOzVhl8Jm1YWTh3Kw1A6lgQ/edit# Pls use the doc to comment/disccuss and let’s decide by next mistral iRC meeting on Mon. Regards, Dmitri. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting reminder - 08/18/2014
Hi mistral folks, I’d like to remind that we’ll have a team meeting today as usually at 16.00 UTC at #openstack-meeting. Agenda: * Review action items * Current status (progress, issues, roadblocks) * Further plans * Release 0.1 scope (BPs and bugs) * Open discussion Please also look at https://wiki.openstack.org/wiki/Meetings/MistralAgenda if you need to find meeting archive. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting minutes/log - 08/18/2014
Folks, Thanks for joining the meeting today. As usually, Meeting minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-08-18-16.00.html Meeting log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-08-18-16.00.log.html Meeting agenda/archive: https://wiki.openstack.org/wiki/Meetings/MistralAgenda Next Monday on Aug 25 we’ll meet again. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] Mistral API v2 - discussion
Dmitri, Thanks for sharing this. We discussed the details of what captured in the etherpad with the team today and looks like folks support most of the ideas we came up with. Anyways, let’s take a couple of days till the next team meeting on Monday to think it over, discuss it all together and make corrections if needed. Renat Akhmerov @ Mirantis Inc. On 22 Aug 2014, at 11:12, Dmitri Zimine dzim...@stackstorm.com wrote: Hi Stackers, we are discussing the API, v2. The core team captured the initial thoughts here (most recent): https://etherpad.openstack.org/p/mistral-API-v2-discussion and here: https://docs.google.com/a/stackstorm.com/document/d/12j66DZiyJahdnV8zXM_fCRJb0qxpsVS_eyhZvdHlIVU/edit Have a look, leave your comments, questions, suggestions. On Monday’s IRC meeting we’ll talk more and finalize the draft. Thanks, DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting reminder - 08/25/2014
Hi Mistral folks, Don’t forget that we’ll have a team meeting today at 16.00 UTC at #openstack-meeting. Agenda: * Review action items * Current status (progress, issues, roadblocks, further plans) * API V2 Discussion (integration with Glance, versioned client etc.) * New action design discussion (storing actions in DB etc.) * Open discussion (See http://wiki.openstack.org/wiki/Meetings/MistralAgenda for meeting archive) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting minutes/log - 08/25/2014
Thanks for joining our community meeting today! Meeting minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-08-25-16.00.html Meeting log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-08-25-16.00.log.html The next meeting will be on Sept 1st at the same time. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Access filtering BP
Team, Please take a look at the new BP https://blueprints.launchpad.net/mistral/+spec/mistral-access-filtering. Interested in your feedback and shared experience. Although it’s a purely internal design thing I find pretty important to be accurate about things like that. Thanks Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Workflow on-finish
There are two blueprints that I supposed to use for this purpose: https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-amqp So my opinion: This functionality should be orthogonal to what we configure in DSL. The mechanism of listeners would is more generic and would your requirement as a special case. At this point, I see that we may want to implement a generic transport-agnostic listener mechanism internally (not that hard task) and then implement required transport specific plugins to it. Inviting everyone to discussion. Thanks Renat Akhmerov @ Mirantis Inc. On 28 Aug 2014, at 06:17, W Chan m4d.co...@gmail.com wrote: Renat, It will be helpful to perform a callback on completion of the async workflow. Can we add on-finish to the workflow spec and when workflow completes, runs task(s) defined in the on-finish section of the spec? This will allow the workflow author to define how the callback is to be done. Here's the bp link. https://blueprints.launchpad.net/mistral/+spec/mistral-workflow-on-finish Thanks. Winson ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Workflow on-finish
Right now, you can just include a special task into a workflow that, for example, sends an HTTP request to whatever you need to notify about workflow completion. Although, I see it rather as a hack (not so horrible though). Renat Akhmerov @ Mirantis Inc. On 28 Aug 2014, at 12:01, Renat Akhmerov rakhme...@mirantis.com wrote: There are two blueprints that I supposed to use for this purpose: https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-amqp So my opinion: This functionality should be orthogonal to what we configure in DSL. The mechanism of listeners would is more generic and would your requirement as a special case. At this point, I see that we may want to implement a generic transport-agnostic listener mechanism internally (not that hard task) and then implement required transport specific plugins to it. Inviting everyone to discussion. Thanks Renat Akhmerov @ Mirantis Inc. On 28 Aug 2014, at 06:17, W Chan m4d.co...@gmail.com wrote: Renat, It will be helpful to perform a callback on completion of the async workflow. Can we add on-finish to the workflow spec and when workflow completes, runs task(s) defined in the on-finish section of the spec? This will allow the workflow author to define how the callback is to be done. Here's the bp link. https://blueprints.launchpad.net/mistral/+spec/mistral-workflow-on-finish Thanks. Winson ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Workflow on-finish
Yes, it’s just a regular task that sends a request. Something like: notify_about_completion: action: std.http parameters: url: whatever_we_need.org method: GET You can also take a look at webhooks examples in mistral-extra. Renat Akhmerov @ Mirantis Inc. On 29 Aug 2014, at 01:22, W Chan m4d.co...@gmail.com wrote: Is there an example somewhere that I can reference on how to define this special task? Thanks! On Wed, Aug 27, 2014 at 10:02 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Right now, you can just include a special task into a workflow that, for example, sends an HTTP request to whatever you need to notify about workflow completion. Although, I see it rather as a hack (not so horrible though). Renat Akhmerov @ Mirantis Inc. On 28 Aug 2014, at 12:01, Renat Akhmerov rakhme...@mirantis.com wrote: There are two blueprints that I supposed to use for this purpose: https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-amqp So my opinion: This functionality should be orthogonal to what we configure in DSL. The mechanism of listeners would is more generic and would your requirement as a special case. At this point, I see that we may want to implement a generic transport-agnostic listener mechanism internally (not that hard task) and then implement required transport specific plugins to it. Inviting everyone to discussion. Thanks Renat Akhmerov @ Mirantis Inc. On 28 Aug 2014, at 06:17, W Chan m4d.co...@gmail.com wrote: Renat, It will be helpful to perform a callback on completion of the async workflow. Can we add on-finish to the workflow spec and when workflow completes, runs task(s) defined in the on-finish section of the spec? This will allow the workflow author to define how the callback is to be done. Here's the bp link. https://blueprints.launchpad.net/mistral/+spec/mistral-workflow-on-finish Thanks. Winson ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting reminder - 09/01/2014
Hi, As usually, this is a reminder about team meeting today at 16.00 UTC #openstack-meeting. Agenda: Review action items Current status (progress, issues, roadblocks, further plans) Release 0.1 progress Open discussion (See it also at https://wiki.openstack.org/wiki/Meetings/MistralAgenda as well as the meeting archive) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting minutes/log - 09/01/2014
Thanks for joining our team meeting today! Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-09-01-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-09-01-16.00.log.html Agenda/archive: https://wiki.openstack.org/wiki/Meetings/MistralAgenda The next meeting will be on Sep 8th. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Team meeting reminder - 09/08/2014
Hi, Please keep in mind that we’ll have a team meeting today at 16.00 UTC at #openstack-meeting. Agenda: Review action items Current status (progress, issues, roadblocks, further plans) Release 0.1 progress (go through the list of what's left) Metrics collector BPs Open discussion (see also at https://wiki.openstack.org/wiki/Meetings/MistralAgenda#Agenda as well as meeting archive) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Team meeting minutes/log - 08/09/2014
Thanks for joining us today at our team meeting! Meeting minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-09-08-16.00.html Meeting log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-09-08-16.00.log.html Agenda/Meeting archive: https://wiki.openstack.org/wiki/Meetings/MistralAgenda The next meeting will be at the same time/place on Sep 15. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] callback on workflow completion
Ok, here is what I think...
I totally support the first option for its easiness in terms of understanding
how it all should work (no need to figure out if some additional objects must
be deleted if a workflow has been removed etc. etc.). We actually have two BPS
[0] and [1] where the idea was similar to your option #2. But I admit that
they’ve been around for a while and I think are obsolete (even though having
eventually the same goal of notifying the outside world about executions/tasks
events).
The only thing I would like to suggest is how we define a callback (keeping in
mind it should be a valid JSON in reality):
POST /executions
workflow_name=flow
callbacks=[{
events: [[on-task-complete, on-execution-complete]
action: std.http url=‘http://foo.bar.com' method=POST headers=‘{}' ##
},
{# another callback}
]
and/or
POST /executions
workflow_name=flow
callbacks=[{
events: [[on-task-complete, on-execution-complete]
action: std.http
parameters: {
url: http://foo.bar.com,
method: POST
headers: {
# Whatever headers we need.
}
}
},
{# another callback}
]
In other word we can trivially generalise this so that:
we can use not only webhooks but any action accessible in Mistral (e.g. it may
be other transport)
it is consistent with our DSL
We might even want to allow “workflow” as well as “action” but not sure if we
need to get that far for now.
Thoughts?
[0] https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-amqp
[1] https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http
Renat Akhmerov
@ Mirantis Inc.
On 17 Sep 2014, at 10:36, Dmitri Zimine dzim...@stackstorm.com wrote:
Use case:
The client software fires the workflow execution and needs to be know when
the workflow is complete. There is no good pool strategy as workflow can take
arbitrary time from ms to days. Callback notification is needed.
Solution is a webhook
Option 1: pass callback URL as part of starting workflow execution:
POST /executions
workflow_name=flow
callback= {
events: [[on-task-complete, on-execution-complete]
url: http://bla.com
method:POST
headers: {}
… other stuff to form proper HTTP call, like API tokens, etc ...
}
…..
Option 2: webhook endpoint
Mistral exposes /webhook controller
Client creates a webhook and receives events for all executions for selected
workflows.
{
name: web,
active: true,
events: [ ]
“workflows”: [wf1, wf2]
url: http://example.com/webhook;,
}
Opinions:
DZ: my opinion:
Option 1 for it is simple and convenient for a client.
It seems like an optimal solution for “tracking executions and tasks” use
case.
Option 2 is an overkill: makes it harder for a client (post workflow, post
webhook, post execution, delete workflow, delete webhook) introduces
lifecycle management problems (e.g., workflow deleted - webhook orphaned).
I vaguely recall someone from Heat compared these options and regretted one
of them for security reasons, but can’t remember details.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] callback on workflow completion
Well, event types are supposed to be different depending on what we want.. I
just like the idea of using actions because corresponds to spirit of Mistral :)
Btw, zaqar can be easily supported in a form of action (again, to be consistent
with what we do in the system).
Renat Akhmerov
@ Mirantis Inc.
On 17 Sep 2014, at 17:07, Angus Salkeld asalk...@mirantis.com wrote:
On Thu, Sep 18, 2014 at 4:25 AM, Renat Akhmerov rakhme...@mirantis.com
wrote:
Ok, here is what I think...
I totally support the first option for its easiness in terms of understanding
how it all should work (no need to figure out if some additional objects must
be deleted if a workflow has been removed etc. etc.). We actually have two
BPS [0] and [1] where the idea was similar to your option #2. But I admit
that they’ve been around for a while and I think are obsolete (even though
having eventually the same goal of notifying the outside world about
executions/tasks events).
The only thing I would like to suggest is how we define a callback (keeping
in mind it should be a valid JSON in reality):
POST /executions
workflow_name=flow
callbacks=[{
events: [[on-task-complete, on-execution-complete]
action: std.http url=‘http://foo.bar.com' method=POST headers=‘{}' ##
},
{# another callback}
]
and/or
POST /executions
workflow_name=flow
callbacks=[{
events: [[on-task-complete, on-execution-complete]
action: std.http
parameters: {
url: http://foo.bar.com,
method: POST
headers: {
# Whatever headers we need.
}
}
},
{# another callback}
]
In other word we can trivially generalise this so that:
we can use not only webhooks but any action accessible in Mistral (e.g. it
may be other transport)
it is consistent with our DSL
We might even want to allow “workflow” as well as “action” but not sure if we
need to get that far for now.
Thoughts?
[0]
https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-amqp
[1]
https://blueprints.launchpad.net/mistral/+spec/mistral-event-listeners-http
Renat Akhmerov
@ Mirantis Inc.
On 17 Sep 2014, at 10:36, Dmitri Zimine dzim...@stackstorm.com wrote:
Use case:
The client software fires the workflow execution and needs to be know when
the workflow is complete. There is no good pool strategy as workflow can
take arbitrary time from ms to days. Callback notification is needed.
Solution is a webhook
I think another good solution is a zaqar queue/inbox. To me this is even
better as you can send updates on
each task that runs not just the whole workbook. This provides much better
feedback to the user.
(we are looking at something like this in Heat).
in this case you just need the name of the queue/inbox.
-Angus
Option 1: pass callback URL as part of starting workflow execution:
POST /executions
workflow_name=flow
callback= {
events: [[on-task-complete, on-execution-complete]
url: http://bla.com
method:POST
headers: {}
… other stuff to form proper HTTP call, like API tokens, etc ...
}
…..
Option 2: webhook endpoint
Mistral exposes /webhook controller
Client creates a webhook and receives events for all executions for selected
workflows.
{
name: web,
active: true,
events: [ ]
“workflows”: [wf1, wf2]
url: http://example.com/webhook;,
}
Opinions:
DZ: my opinion:
Option 1 for it is simple and convenient for a client.
It seems like an optimal solution for “tracking executions and tasks” use
case.
Option 2 is an overkill: makes it harder for a client (post workflow, post
webhook, post execution, delete workflow, delete webhook) introduces
lifecycle management problems (e.g., workflow deleted - webhook orphaned).
I vaguely recall someone from Heat compared these options and regretted one
of them for security reasons, but can’t remember details.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes and logs - 12/09/2013
Hi, Thanks for joining today’s Mistral community meeting. Here are the links to meeting minutes and log: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2013/mistral.2013-12-09-16.01.html Log: http://eavesdrop.openstack.org/meetings/mistral/2013/mistral.2013-12-09-16.01.log.html Join us next time to discuss Mistral PoC and Mistral Engine architecture. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting agenda - 12/16/2013
Hi! This is a reminder that we will have another community meeting today in IRC (#openstack-meeting) at 16.00 UTC. Here’s the agenda: https://wiki.openstack.org/wiki/Meetings/MistralAgenda As usually, you’re welcome to join! Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes - 12/16/2013
Hello, Thanks for joining us today in IRC, here are the links to meeting minutes and logs: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2013/mistral.2013-12-16-16.00.html Logs: http://eavesdrop.openstack.org/meetings/mistral/2013/mistral.2013-12-16-16.00.log.html Join us next time. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes - 12/23/2013
Hi, Here the links to minutes and logs for the IRC meeting that we had today: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2013/mistral.2013-12-23-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2013/mistral.2013-12-23-16.00.log.html Feel free to join us next time. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Evolving Mistral DSL: Data Flow
Hi all,
We’ve created another etherpad https://etherpad.openstack.org/p/mistral-poc
where we proposed additional Mistral features that are going to be reflected in
DSL. One of the most important things is Data Flow, it’s main idea, how it
matches to the initial workflow model and how it affects DSL. For convenience
I’ll provide the snippet from this etherpad about Data Flow in this email.
Data Flow
When we start a workflow we optionally set initial state of workflow execution
context (basically, just object model representable in JSON format).
Expose workflow execution context object accessible as $. in YAQL notation in
DSL.
A task optionally has a YAQL expression to select needed data from workflow
execution context. Selected data is considered task input and it gets
translated to parameters of corresponding REST action (http request body in
case of POST), AMQP action (as a JSON object) etc.
A task produces a result, the result gets merged into the context and the
context copy gets passed to the next task.
Context object is immutable and each task gets a copy of it to avoid needs to
use locking techniques. It gets passed to the next task with the task itself
via a message queue.
DSL snippet
Workflow:
tasks:
task1:
input: $.people.[$.age 30].address # Input selector expression
in YAQL notation that is used to select data needed for task1 from workflow
execution context
action: MyRest:action1
Services:
MyRest:
type: REST_API
parameters:
baseUrl: http://localhost:8988/my_service
actions:
action1:
parameters:
url: /action1
method: POST
Let's say the initial workflow execution context is as follows:
{
people: [
{
first_name: John,
last_name: Doe,
age: 32,
address: {
street: 25 Broadway Avenue,
city: Woodstock
}
},
{
first_name: Jane,
last_name: Doe,
age: 28,
address: {
street: 101 Jackson Street,
city: Gamilton
}
}
]}
Then input selector $.people.[$.age 30].address.city will evaluate to:
{
street: 25 Broadway Avenue,
city: Woodstock
}
And corresponding HTTP request will be:
http POST http://localhost:8988/my_service/action1
{
street: 25 Broadway Avenue,
city: Woodstock
}
In case of HTTP GET it will look like:
http GET
http://localhost:8988/my_service/action1?street=25+Broadway+Avenuecity=Woodstock
“
If you have any concerns or new ideas on how to improve what we’re proposing
please feel free to share with us.
Thanks.
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Evolving Mistral DSL: Conditional transitions
Hi,
This is an additional email to continue discussion about new Mistral DSL
capabilities. This time about conditional transitions and task dependencies.
Here’s the link to the corresponding etherpad:
https://etherpad.openstack.org/p/mistral-poc.
For convenience, I’m providing the snippet about conditional transitions right
in this message.
Conditional Transitions
Option #1 ('requires' with condition)
In this case we add additional optional condition in dependencies declaration.
task1 # task1 doesn't have dependencies
action: MyRest:action1
task2:
requires: task1 # task2 is allowed to run if task1 has
finished with success
action: MyRest:action2
task3:
requires: # task3 is allowed to run if task1
and task2 have finished and corresponding conditions are true
task1: $.value1 = 34
task2: $.value2 = 245
action: MyRest:action3
task4:
requires: [task2, task3] # task4 is allowed to run if task2 and
task3 has finished with success
action: MyRest:action4
Option #2 (direct transition)
In this case after task completion we decided what to do next depending on task
result.
task1:
action: MyRest:action1
SUCCESS:
- task2 # If task1 has finished with success then start task2
- task3: $.value1 == 123 # If after completion of task1 variable
value1 in the context equals to 123 then start task3
ERROR:
- task10 # If task1 has finished with error then start task10
Option #3 (hybrid of 'requires' and direct transitions)
In this case we can specify both requires and direct transitions. In the
example below task3 starts only when task1 and task2 have finished. Therefore
task2 is an implicit dependency of task3 since task1 initiated execution of
task2. It means that a task can start only if the entire sub-workflow initiated
by requires clause has finished.
task1:
action: MyRest:action1
SUCCESS
- task2
task2:
action: MyRest:action2
task3:
action: MyRest:action3
requires:
task1: $.value1 = 34
SUCCESS:
task4: '$.value1 == 123'
Note that we’re now considering 3 options to proceed with and our current
preference is option #3 (mix of ‘requires’ and direct transitions) since it
would give a lot of flexibility for designing real life workflows. In some
cases it would be more convenient to describe a business process using task
dependencies and in other cases use direct conditional transitions if
“straightforward” flow definition is preferred.
Regardless of the option described here a user would start new workflow
execution with specifying a particular task in a graph.
As always, we would like to get your feedback on this.
Thanks!
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Sorry for being a little bit verbose :) … Renat ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Greg,
This is really not a new discussion. Particularly, we’ve already discussed
EventScheduler thing before with heat folks
(http://lists.openstack.org/pipermail/openstack-dev/2013-November/019150.html).
We agreed that EventScheduler doesn’t make a lot of sense alone after Mistral
came into play. It’s because EventScheduler only describes calling webhooks on
schedule and nothing else. Mistral already has this functionality just because
it’s a part of its core idea. Timer events is just one of the event types
Mistral is going to support (there will also be others like Ceilometer alarms).
And one of the simple use case of Mistral would be exactly “calling a webhook
on schedule”. Since Mistral uses DSL as a main interaction means with a user
the team decided to implement ideas of EventScheduler as a part of Mistral DSL
as well. Here’s the simple Mistral DSL that describes basically the same as
described in EventScheduler:
Services:
MyRest:
type: REST_API
parameters:
baseUrl: http://localhost:8988/my_service
actions:
my-action:
parameters:
url: /my-action
method: GET
Workflow:
tasks:
my-task:
action: MyRest:my-action
events:
my-task:
type: periodic
tasks: my-task
parameters:
cron-pattern: */1 * * * *
I really hope it’s self-explaining enough (if not, let us know what is not
clear or what you don’t like, we’ll improve it). Basically, to make this simple
DSL snippet work we just need to upload it to Mistral via REST API. Just want
to repeat: this already works. To simplify this even more for use cases like
yours we also have an idea of providing some API sugar like scheduling webhook
calls using special REST API action like:
http POST /webhooks
{
“name”: “my every minute webhook”,
“webhook: “http://my.webhook.host/someurl”,
“schedule”: {
“cron”: “* * * * *
}
}
And under the hood Mistral would just generate a workflow definition similar to
provided above.
So my main point here that I was trying to make is that EventScheduler is just
a special case of Mistral capabilities. We can just provide a convenience for
scheduling webhook calls on the API (a couple of hours to implement).
Thanks! Happy New Year to everyone!
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [MIstral] Community meeting agenda - 01/13/2014
Hi, This is a reminder that we’ll have a community meeting at #openstack-meeting at 16.00 UTC tomorrow on Jan 13. The agenda is: Review action items Discuss new requirements (conditionals, direct transitions, data flow) Open discussion It can also be found at https://wiki.openstack.org/wiki/Meetings/MistralAgenda as well as logs and minutes for the previous meetings. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes - 01/13/2014
Hi, Thanks for joining us today in IRC at #openstack-meeting. Here are the links to minutes and log of the meeting: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-01-13-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-01-13-16.00.log.html Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] writing comments as complete sentences
I absolutely agree with that. Honestly, I don’t even understand why it can be different (not compliant with regular grammar rules). Comments are supposed to clarify things so they should be worded very clearly and read as normal English sentences. Additionally it’s one of the small bricks that the entire coding culture is built of. Renat Akhmerov @ Mirantis Inc. On 14 Jan 2014, at 06:00, Jay Pipes jaypi...@gmail.com wrote: On Tue, 2014-01-14 at 08:01 -0500, Doug Hellmann wrote: Ned Batchelder had an interesting post a few days ago about why code comments should be written as complete sentences, rather than fragments. The main point is that with an unterminated phrase or sentence fragment it isn't always clear that the thought has been completed (i.e., maybe the full comment isn't even present in the code). # Supported. -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [trove][mistral] scheduled tasks
Greg,
Looks like I missed your last email in an email storm after NY holidays but now
was able to see it accidentally. So answering your question about Mistral
incubation your concern is valid. We’re currently working on moving Mistral
into incubation and we’ll keep you (and others) posted on that.
Renat Akhmerov
@ Mirantis Inc.
On 02 Jan 2014, at 09:52, Greg Hill greg.h...@rackspace.com wrote:
Renat,
Thanks for the additional information. I've been trying to put the pieces of
history together and it seems I missed some of it. I think I now understand
the evolution of things. Mistral does seem like it would work for what we
need, so I'll definitely be paying attention to it.
I'm curious about the general OpenStack precedent of having an incubated
project depend on one that isn't yet incubated. Does that make it a
hindrance for graduating to an official project since it would require
operators to deploy something that isn't yet officially part of openstack, or
does that not matter?
Greg
On Jan 2, 2014, at 12:21 AM, Renat Akhmerov rakhme...@mirantis.com wrote:
Greg,
This is really not a new discussion. Particularly, we’ve already discussed
EventScheduler thing before with heat folks
(http://lists.openstack.org/pipermail/openstack-dev/2013-November/019150.html).
We agreed that EventScheduler doesn’t make a lot of sense alone after
Mistral came into play. It’s because EventScheduler only describes calling
webhooks on schedule and nothing else. Mistral already has this
functionality just because it’s a part of its core idea. Timer events is
just one of the event types Mistral is going to support (there will also be
others like Ceilometer alarms). And one of the simple use case of Mistral
would be exactly “calling a webhook on schedule”. Since Mistral uses DSL as
a main interaction means with a user the team decided to implement ideas of
EventScheduler as a part of Mistral DSL as well. Here’s the simple Mistral
DSL that describes basically the same as described in EventScheduler:
Services:
MyRest:
type: REST_API
parameters:
baseUrl: http://localhost:8988/my_service
actions:
my-action:
parameters:
url: /my-action
method: GET
Workflow:
tasks:
my-task:
action: MyRest:my-action
events:
my-task:
type: periodic
tasks: my-task
parameters:
cron-pattern: */1 * * * *
I really hope it’s self-explaining enough (if not, let us know what is not
clear or what you don’t like, we’ll improve it). Basically, to make this
simple DSL snippet work we just need to upload it to Mistral via REST API.
Just want to repeat: this already works. To simplify this even more for use
cases like yours we also have an idea of providing some API sugar like
scheduling webhook calls using special REST API action like:
http POST /webhooks
{
“name”: “my every minute webhook”,
“webhook: “http://my.webhook.host/someurl”,
“schedule”: {
“cron”: “* * * * *
}
}
And under the hood Mistral would just generate a workflow definition similar
to provided above.
So my main point here that I was trying to make is that EventScheduler is
just a special case of Mistral capabilities. We can just provide a
convenience for scheduling webhook calls on the API (a couple of hours to
implement).
Thanks! Happy New Year to everyone!
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Great idea, fully support it. We’re interested in that too. One specific thing that was mentioned is the ability to mock auth service seems to be very useful for some test scenarios, we came across that recently. Renat Akhmerov @ Mirantis Inc. On 15 Jan 2014, at 14:07, Sylvain Bauza sylvain.ba...@gmail.com wrote: Hi Doug, Count me in. Climate is currently working on delivering its first python-climateclient but it would be great if we could leverage any olso lib for this. -Sylvain 2014/1/15 Doug Hellmann doug.hellm...@dreamhost.com Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Renat Akhmerov On 16 Jan 2014, at 11:52, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 2:36 PM, Joe Gordon joe.gord...@gmail.com wrote: 2) major overhaul of client libraries so they are all based off a common base library. This would cover namespace changes, and possible a push to move CLI into python-openstackclient This seems like the biggest win to me. +1 Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 16 Jan 2014, at 12:36, Dean Troyer dtro...@gmail.com wrote: I've already written a POC for solum and some other things to demonstrate how to add additional projects simply by installing the python-*client package. https://github.com/dtroyer/python-oscplugin is a trivial example. Thanks, this link is helpful. In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. If it is not a library that is actually shared you will get back to the current situation over time. How is that different from any other oslo stuff? Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). When you have stable and accurate documents of this sort, let's talk. You may have noticed that few (any?) of the recent major API revs are documented in this manner… There’s still something to work on for us.. As far as what I’ve written, just a crazy idea that came to my mind. After all, everyone’s phone has lots of things that were considered crazy years ago. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. Renat Akhmerov___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Ok, I think most of the reasoning you’ve said makes sense. So for a non-incubated project we’re going to have separate clients and then get them integrated into this single client once the project itself gets incubated? Or it’s going to happen when the project gets integrated into core os projects? So if yes, it’s just going to be one more incubation/integration requirement, right? Renat On 16 Jan 2014, at 18:09, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 8:42 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. There’s also the discovery problem, it’s incredibly frustrating if, as I’m starting out to use an Openstack based cloud, everytime I want to touch some new segment of the service I need to go find out what the client lib is for that, possibly download the dependencies for it, possibly get it approved, etc. Splitting up services makes a lot of sense on the server side, but to the consumer a cloud often times isn’t a disjoint set of services that happen to be working in parallel, it is a single unified product where they may not know the boundary lines, or at the very least the boundaries can be fuzzy for them. Renat Akhmerov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 Jan 2014, at 10:04, Jonathan LaCour jonathan-li...@cleverdevil.org wrote: pip install openstack That would be awesome :) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes - 01/20/2014
Hi, Thank you for joining us today at #openstack-meeting. Here are the links to meeting minutes and logs: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-01-20-16.00.html Logs: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-01-20-16.00.log.html Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 Jan 2014, at 22:06, Robert Collins robe...@robertcollins.net wrote: On 17 January 2014 09:22, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Launchpadlib which builds on wadllib did *exactly* that. It worked fairly well with the one caveat that it fell into the ORM trap - just in time lookups for everything with crippling roundtrips. Thanks, I’ll have a look at it. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 18 Jan 2014, at 07:48, Sean Dague s...@dague.net wrote: I also think auto generated clients have a lot of challenges in the same way that full javascript pages in browsers have. If you screw up in a subtle way you can just completely disable your clients from connecting to your server entirely (or block them from using bits of the lesser used function path because a minor bit of schema fat fingered). So without a ton of additional verification on that path, I wouldn't want that anywhere near openstack. Especially with vendor extensions, which mean that enabling a network driver might break all your clients. Yes, this makes tons of sense. The more I think about it the more I come to a conclusion that it will never work well enough (more organizationally though than technically). Additionally, there can be not 1 to 1 match between API methods and python code. For example, we’ve been considering some improvements in Mistral client library so that we can make in some cases a series of calls without actually hitting the server therefore implementing a masking behavior (in some scenarios it seems to make a lot of sense). Typically I’m talking about building a complex object from small pieces. We haven’t made a decision though. We’ll be discussing it with the community. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 21 Jan 2014, at 09:07, Jesse Noller jesse.nol...@rackspace.com wrote: Do you use any other platform than Linux? Even donald - one of the python packaging leads and PyPI leads said this is a bad end-user experience for consumers of openstack clouds. That fact that someone (even very smart experience) said something doesn’t mean we should stop thinking forward. I am pretty new to Python though and apologize if I misunderstand things sometimes.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 21 Jan 2014, at 09:40, Sean Dague s...@dague.net wrote: On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out”. Honestly, I was very far but now I’m much closer :) Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. I see your point. Right now this is really a problem. I hope it’ll be solved someday globally so that python could have more than 1 version of a library. However, some organizational steps could be made to strictly control versions of libraries. Anyway, I admit you’re right here. So come spend a month working on requirements updates in OpenStack gate… Frankly, never had significant experience in that. I would love to :) well you are a braver man than I. :) Nope, it’s hardly true :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
+1 for opening new threads regarding specific questions. On 21 Jan 2014, at 11:54, Renat Akhmerov rakhme...@mirantis.com wrote: On 21 Jan 2014, at 09:40, Sean Dague s...@dague.net wrote: On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out”. Honestly, I was very far but now I’m much closer :) Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. I see your point. Right now this is really a problem. I hope it’ll be solved someday globally so that python could have more than 1 version of a library. However, some organizational steps could be made to strictly control versions of libraries. Anyway, I admit you’re right here. So come spend a month working on requirements updates in OpenStack gate… Frankly, never had significant experience in that. I would love to :) well you are a braver man than I. :) Nope, it’s hardly true :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Repositoris re-organization
On 21 Jan 2014, at 11:55, Alexander Tivelkov ativel...@mirantis.com wrote: murano - main services, common, agents docs, deployments scripts python-muranoclient - python bindings and CLI murano-dashboard - OS Dashboard plugin murano-apps - new repo for metadata, including core library and example apps. murano-tests - existing test-repo, not going to be transferred when incubated. I think this looks pretty nice as long as we don’t have significant problems with granularity. +1 that it will allow making more consistent changes and hence keeping the whole system robust. Renat Akhmerov___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] memoizer aka cache
Joining to providing our backgrounds.. I’d be happy to help here too since I
have pretty solid background in using and developing caching solutions, however
mostly in Java world (expertise in GemFire and Coherence, developing GridGain
distributed cache).
Renat Akhmerov
@ Mirantis Inc.
On 23 Jan 2014, at 18:38, Joshua Harlow harlo...@yahoo-inc.com wrote:
Same here; I've done pretty big memcache (and similar technologies) scale
caching invalidations at Y! before so here to help…
From: Morgan Fainberg m...@metacloud.com
Reply-To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Date: Thursday, January 23, 2014 at 4:17 PM
To: OpenStack Development Mailing List (not for usage questions)
openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [oslo] memoizer aka cache
Yes! There is a reason Keystone has a very small footprint of
caching/invalidation done so far. It really needs to be correct when it
comes to proper invalidation logic. I am happy to offer some help in
determining logic for caching/invalidation with Dogpile.cache in mind as we
get it into oslo and available for all to use.
--Morgan
On Thu, Jan 23, 2014 at 2:54 PM, Joshua Harlow harlo...@yahoo-inc.com wrote:
Sure, no cancelling cases of conscious usage, but we need to be careful
here and make sure its really appropriate. Caching and invalidation
techniques are right up there in terms of problems that appear easy and
simple to initially do/use, but doing it correctly is really really hard
(especially at any type of scale).
-Josh
On 1/23/14, 1:35 PM, Renat Akhmerov rakhme...@mirantis.com wrote:
On 23 Jan 2014, at 08:41, Joshua Harlow harlo...@yahoo-inc.com wrote:
So to me memoizing is typically a premature optimization in a lot of
cases. And doing it incorrectly leads to overfilling the python
processes memory (your global dict will have objects in it that can't be
garbage collected, and with enough keys+values being stored will act
just like a memory leak; basically it acts as a new GC root object in a
way) or more cache invalidation races/inconsistencies than just
recomputing the initial valueŠ
I agree with your concerns here. At the same time, I think this thinking
shouldn¹t cancel cases of conscious usage of caching technics. A decent
cache implementation would help to solve lots of performance problems
(which eventually becomes a concern for any project).
Overall though there are a few caching libraries I've seen being used,
any of which could be used for memoization.
-
https://github.com/openstack/oslo-incubator/tree/master/openstack/common/
cache
-
https://github.com/openstack/oslo-incubator/blob/master/openstack/common/
memorycache.py
I looked at the code. I have lots of question to the implementation (like
cache eviction policies, whether or not it works well with green threads,
but I think it¹s a subject for a separate discussion though). Could you
please share your experience of using it? Were there specific problems
that you could point to? May be they are already described somewhere?
- dogpile cache @ https://pypi.python.org/pypi/dogpile.cache
This one looks really interesting in terms of claimed feature set. It
seems to be compatible with Python 2.7, not sure about 2.6. As above, it
would be cool you told about your experience with it.
I am personally weary of using them for memoization, what expensive
method calls do u see the complexity of this being useful? I didn't
think that many method calls being done in openstack warranted the
complexity added by doing this (premature optimization is the root of
all evil...). Do u have data showing where it would be
applicable/beneficial?
I believe there¹s a great deal of use cases like caching db objects or
more generally caching any heavy objects involving interprocess
communication. For instance, API clients may be caching objects that are
known to be immutable on the server side.
Sent from my really tiny device...
On Jan 23, 2014, at 8:19 AM, Shawn Hartsock harts...@acm.org wrote:
I would like to have us adopt a memoizing caching library of some kind
for use with OpenStack projects. I have no strong preference at this
time and I would like suggestions on what to use.
I have seen a number of patches where people have begun to implement
their own caches in dictionaries. This typically confuses the code and
mixes issues of correctness and performance in code.
Here's an example:
We start with:
def my_thing_method(some_args):
# do expensive work
return value
... but a performance problem is detected... maybe the method is
called 15 times in 10 seconds but then not again for 5 minutes and the
return value can only logically change every minute or two... so we
end up with ...
_GLOBAL_THING_CACHE = {}
def my_thing_method(some_args):
key = key_from
Re: [openstack-dev] Mistral + taskflow mini-meetup
Josh, thanks for sharing this with the community. Just a couple of words as an addition to that.. The driver for this conversation is that TaskFlow library and Mistral service in many ways do similar things: task processing combined somehow (flow or workflow). However, there’s a number of differences in approaches that the two technologies follow. Initially, when Mistral’s development phase started about a couple of months ago the team was willing to use TaskFlow at implementation level. Basically, we can potentially represent Mistral tasks as TaskFlow tasks and use TaskFlow API to run them. One of the problems though is that TaskFlow tasks are basically python methods and hence run synchronously (once we get out of the method the task is considered finished) whereas Mistral is primarily designed to run asynchronous tasks (send a signal to an external system and start waiting for a result which may arrive minutes or hours later. Mistral is more like event-driven system versus traditional executor architecture. So now Mistral PoC is not using TaskFlow but moving forward we we’d like to try to marry these two technologies to be more aligned in terms of APIs and feature sets. Renat Akhmerov @ Mirantis Inc. On 27 Jan 2014, at 13:21, Joshua Harlow harlo...@yahoo-inc.com wrote: Hi all, In order to encourage further discussion off IRC and more in public I'd like to share a etherpad that was worked on during a 'meetup' with some of the mistral folks and me. https://etherpad.openstack.org/p/taskflow-mistral-jan-meetup It was more of a (mini) in-person meetup but I thought I'd be good to gather some feedback there and let the more general audience see this and ask any questions/feedback/other... Some of the key distinctions between taskflow/mistral we talked about and as well other various DSL aspects and some possible action items. Feel free to ask questions, -Josh ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Community meeting agenda - 02/03/2014
Hi, This is a reminder that we will have a weekly community meeting in IRC (#openstack-meeting) tomorrow at 16.00 UTC. Here’s the agenda (also published at [0] along with links to the previous meetings): Review action items Discuss capabilities and DSL again (since we now have new contributors) Discuss DSL example (https://etherpad.openstack.org/p/mistral-poc) Discuss current PoC status Review Blueprints (at least part) Open discussion (roadblocks, suggestions, etc.) As usually, everyone is welcome to join! [0] https://wiki.openstack.org/wiki/Meetings/MistralAgenda Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [mistral] Community meeting minutes and logs - 02/03/2014
Hi, Thanks for joining us today! Here are the links to minutes and log: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-02-03-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-02-03-16.00.log.html Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Comments on DSL
Dmitri, Sure, no problem. Good considerations. I’m now in the process of reviewing your notes.. Renat Akhmerov @ Mirantis Inc. On 04 Feb 2014, at 09:05, Dmitri Zimine d...@stackstorm.com wrote: Following up from yesterday's community meeting I am still catching up with the project, still miss a lot of context. Put my questions and comments on DSL definition: https://etherpad.openstack.org/p/mistral-dsl-discussion Let's review, and thanks for helping us get up to speed. DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] WSME 0.6 released
I guess it should be but just in case… Renat Akhmerov @ Mirantis Inc. On 06 Feb 2014, at 07:58, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Thu, Feb 6, 2014 at 10:11 AM, Sylvain Bauza sylvain.ba...@gmail.com wrote: Thanks Doug, 2014-02-06 15:54 GMT+01:00 Doug Hellmann doug.hellm...@dreamhost.com: cdf74daac2a204d5fe77f4b2bf5a956f65a73a6f Support dynamic types f191f32a722ef0c2eaad71dd33da4e7787ac2424 Add IntegerType and some classes for validation Doug Do you know when the docs will be updated ? [1] Some complex types can already be found on Ironic/Ceilometer/Climate and I would love to see if some have been backported to WSME as native types (like the UUID type or the String one) I'll look into why the doc build is broken. The gerrit merge should have triggered an update on http://wsme.readthedocs.org/en/latest/ Doug -Sylvain [1] : http://pythonhosted.org//WSME/types.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] WSME 0.6 released
Doug, is it backwards compatible with 0.5b6? Renat Akhmerov @ Mirantis Inc. On 06 Feb 2014, at 07:58, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Thu, Feb 6, 2014 at 10:11 AM, Sylvain Bauza sylvain.ba...@gmail.com wrote: Thanks Doug, 2014-02-06 15:54 GMT+01:00 Doug Hellmann doug.hellm...@dreamhost.com: cdf74daac2a204d5fe77f4b2bf5a956f65a73a6f Support dynamic types f191f32a722ef0c2eaad71dd33da4e7787ac2424 Add IntegerType and some classes for validation Doug Do you know when the docs will be updated ? [1] Some complex types can already be found on Ironic/Ceilometer/Climate and I would love to see if some have been backported to WSME as native types (like the UUID type or the String one) I'll look into why the doc build is broken. The gerrit merge should have triggered an update on http://wsme.readthedocs.org/en/latest/ Doug -Sylvain [1] : http://pythonhosted.org//WSME/types.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Cancelling community meeting today on Feb 10
Hi guys, I would suggest we skip today’s community meeting since some folks from the team (including myself) won’t be able to participate. My apologies for the late notice.. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral]
Dmitry, I think you are right here. I think for simple case we should be able to use in-place action definition without having to define the action separately. Like you said it’s only valuable if we need to reuse it. The only difference I see between std:send-email and something like REST_API is that a set of parameters for the latter is dynamic (versus std:send-email where it’s always “recipients”, “subject”, “body”). Even though it’s still the same protocol (HTTP) but a particular request representation may be different (i.e. query string, headers, the structure of body in case POST etc.). But I think that doesn’t cancel the idea of being able to define the action along with the task itself. So good point. As for the syntax itself, we need to think it over. In the snippet you provided “action: std:REST_API”, so we need to make sure not to have ambiguities in the ways how we can refer actions. A convention could be “if we don’t use a namespace we assume that there’s a separate action definition included into the same workbook, otherwise it should be considered in-place action definition and task property “action” refers to an action type rather than the action itself”. Does that make sense? Renat Akhmerov @ Mirantis Inc. On 11 Feb 2014, at 16:23, Dmitri Zimine d...@stackstorm.com wrote: Do we have (or think about) a shorthand to calling REST_API action, without defining a service? FULL DSL: Services: TimeService: type: REST_API parameters: baseUrl:http://api.timezonedb.com key:my_api_key actions: get-time: task-parameters: zone: Workflow: tasks: timeInToronto: action: TimeService:get-time parameters: zone: America/Toronto SHORTCUT - may look something like this: Workflow: tasks: timeInToronto: action:std:REST_API parameters: baseUrl: http://api.timezonedb.com; method: GET parameters: zone=/America/Torontokey=my_api_key Why asking: 1) analogy with std:send-email action. I wonder do we have to make user define Service for std:send-email? and I think that for standard tasks we shouldn't have to. If there is any thinking on REST_API, it may apply here. 2) For a one-off web service calls the complete syntax is may be overkill (but yes, it comes handy for reuse). See examples below. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Notes on action YAML declaration and naming
On 14 Feb 2014, at 15:02, Dmitri Zimine d...@stackstorm.com wrote: Current DSL snippet: actions: my-action parameters: foo: bar response: # just agreed to change to 'results’ Just a note: “response” indentation here is not correct, it’s not a parameter called “response” but rather a property of “my-action”. select: '$.server_id' store_as: v1 In the code, we refer to action.result_helper 1) Note that response is not exactly a parameter. It doesn't doesn't refer to data. It's (query, variable) pairs, that are used to parse the results and post data to global context [1]. The terms response, or result, do not reflect what is actually happening here. Suggestions? Save? Publish? Result Handler? For explicitness we can use something like “result-handler” and initially I thought about this option. But I personally love conciseness and I think name “result” would be ok for this section meaning it defines the structure of the action result. “handler” is not 100% precise too because we don’t actually handle a result here, we define the rules how to get this result. I would appreciate to see other suggestions though. 2) Whichever name we use for this output transformer, shall it be under parameters? No, what we have in this section is like a return value type for a regular method. Parameters define action input. 3) how do we call action/task parameters? Think 'model' (which reflects in yaml, code, docs, talk, etc.) input and output? (+1) in and out? (-1) request and response? (-1) good for WebServices but not generic enough parameters and results? (ok) Could you please clarify your questions here? Not sure I’m following... 4) Syntax simplification: can we drop 'parameters' keyword? Anything under action is action parameters, unless it's a reserved keyword, which the implementation can parse out. actions: my-action foo: bar task-parameters: # not a keyword, specific to REST_API flavor_id: image_id: publish: # keyword select: '$.server_id' store_as: v1 It will create problems like name ambiguity in case we need to have a parameter with the same names as keywords (“task-parameters” and “publish” in your example). My preference would be to leave it explicit. Renat ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Notes on action YAML declaration and naming
“output” looks nice! Renat Akhmerov @ Mirantis Inc. On 14 Feb 2014, at 20:26, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Current DSL snippet: actions: my-action parameters: foo: bar response: # just agreed to change to 'results' select: '$.server_id' store_as: v1 'result' sounds better than 'response' and, I think, more fit to action description. And I suggest for a new word - 'output'; actually, this block describe how the output will be taken and stored. However, I agree that this block should be at action-property level: actions: my-action result: select: '$.server_id' store_as: vm_id parameters: foo: bar On Fri, Feb 14, 2014 at 12:36 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 14 Feb 2014, at 15:02, Dmitri Zimine d...@stackstorm.com wrote: Current DSL snippet: actions: my-action parameters: foo: bar response: # just agreed to change to 'results’ Just a note: “response” indentation here is not correct, it’s not a parameter called “response” but rather a property of “my-action”. select: '$.server_id' store_as: v1 In the code, we refer to action.result_helper 1) Note that response is not exactly a parameter. It doesn't doesn't refer to data. It's (query, variable) pairs, that are used to parse the results and post data to global context [1]. The terms response, or result, do not reflect what is actually happening here. Suggestions? Save? Publish? Result Handler? For explicitness we can use something like “result-handler” and initially I thought about this option. But I personally love conciseness and I think name “result” would be ok for this section meaning it defines the structure of the action result. “handler” is not 100% precise too because we don’t actually handle a result here, we define the rules how to get this result. I would appreciate to see other suggestions though. 2) Whichever name we use for this output transformer, shall it be under parameters? No, what we have in this section is like a return value type for a regular method. Parameters define action input. 3) how do we call action/task parameters? Think 'model' (which reflects in yaml, code, docs, talk, etc.) input and output? (+1) in and out? (-1) request and response? (-1) good for WebServices but not generic enough parameters and results? (ok) Could you please clarify your questions here? Not sure I’m following... 4) Syntax simplification: can we drop 'parameters' keyword? Anything under action is action parameters, unless it's a reserved keyword, which the implementation can parse out. actions: my-action foo: bar task-parameters: # not a keyword, specific to REST_API flavor_id: image_id: publish: # keyword select: '$.server_id' store_as: v1 It will create problems like name ambiguity in case we need to have a parameter with the same names as keywords (“task-parameters” and “publish” in your example). My preference would be to leave it explicit. Renat ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Nikolay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Notes on action YAML declaration and naming
On 15 Feb 2014, at 04:01, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Dmitri, in our concerns under word 'input' we assume a block contains the info about how the input data will be taken for corresponding task from initial context. So, it will be a kind of expression (e.g. YAQL). Renat, am I right? Yes, basically we should keep in mind that we have 2 types of parameters for an action: Parameters that specify nature of the action itself (for example, “method: POST” for REST_API action type). Input parameters determined dynamically using a selector expression defined on task level (it’s what we have as “input: $.people.[$.age 30].address” in [0]). These define are a real action input interesting from a workflow logic perspective. [0] https://etherpad.openstack.org/p/mistral-poc Renat___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Notes on action YAML declaration and naming
Dmitri, Right now https://etherpad.openstack.org/p/mistral-poc is the only place where we described it. It shouldn’t be considered a specification, it was rather a playground where we tried to shape up our ideas. We’ll fix it using our latest ideas and changes captured in the code and create another etherpad for further long-term discussions. Renat Akhmerov @ Mirantis Inc. On 15 Feb 2014, at 06:26, Dmitri Zimine d...@stackstorm.com wrote: Ok, I see. Do we have a spec that describes this? Lets spell it out and describe the whole picture of input, output, parameters, and result. DZ On Feb 14, 2014, at 1:01 PM, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Dmitri, in our concerns under word 'input' we assume a block contains the info about how the input data will be taken for corresponding task from initial context. So, it will be a kind of expression (e.g. YAQL). Renat, am I right? On Fri, Feb 14, 2014 at 9:51 PM, Dmitri Zimine d...@stackstorm.com wrote: I like output, too. But it should go with 'input' In summary, there are two alternatives. Note that I moved task-parameters under parameters. Ok with this? actions: my-action input: foo: bar task-parameters: flavor_id: image_id: output: select: '$.server_id' store_as: v1 this maps to action(input, *output) actions: my-action parameters: foo: bar task-parameters: flavor_id: image_id: result: select: '$.server_id' store_as: v1 this maps to result=action(parameters) On Feb 14, 2014, at 8:40 AM, Renat Akhmerov rakhme...@mirantis.com wrote: “output” looks nice! Renat Akhmerov @ Mirantis Inc. On 14 Feb 2014, at 20:26, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Current DSL snippet: actions: my-action parameters: foo: bar response: # just agreed to change to 'results' select: '$.server_id' store_as: v1 'result' sounds better than 'response' and, I think, more fit to action description. And I suggest for a new word - 'output'; actually, this block describe how the output will be taken and stored. However, I agree that this block should be at action-property level: actions: my-action result: select: '$.server_id' store_as: vm_id parameters: foo: bar On Fri, Feb 14, 2014 at 12:36 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 14 Feb 2014, at 15:02, Dmitri Zimine d...@stackstorm.com wrote: Current DSL snippet: actions: my-action parameters: foo: bar response: # just agreed to change to 'results’ Just a note: “response” indentation here is not correct, it’s not a parameter called “response” but rather a property of “my-action”. select: '$.server_id' store_as: v1 In the code, we refer to action.result_helper 1) Note that response is not exactly a parameter. It doesn't doesn't refer to data. It's (query, variable) pairs, that are used to parse the results and post data to global context [1]. The terms response, or result, do not reflect what is actually happening here. Suggestions? Save? Publish? Result Handler? For explicitness we can use something like “result-handler” and initially I thought about this option. But I personally love conciseness and I think name “result” would be ok for this section meaning it defines the structure of the action result. “handler” is not 100% precise too because we don’t actually handle a result here, we define the rules how to get this result. I would appreciate to see other suggestions though. 2) Whichever name we use for this output transformer, shall it be under parameters? No, what we have in this section is like a return value type for a regular method. Parameters define action input. 3) how do we call action/task parameters? Think 'model' (which reflects in yaml, code, docs, talk, etc.) input and output? (+1) in and out? (-1) request and response? (-1) good for WebServices but not generic enough parameters and results? (ok) Could you please clarify your questions here? Not sure I’m following... 4) Syntax simplification: can we drop 'parameters' keyword? Anything under action is action parameters, unless it's a reserved keyword, which the implementation can parse out. actions: my-action foo: bar task-parameters: # not a keyword, specific to REST_API flavor_id: image_id: publish: # keyword select: '$.server_id' store_as: v1 It will create problems like name ambiguity in case we need to have
Re: [openstack-dev] [Murano] Need a new DSL for Murano
Clint, We're collaborating with Murano. We may need to do it in a way that others could see it though. There are several things here: Murano doesn’t really have a “workflow engine” similar to Mistral’s. People get confused with that but it’s just a legacy terminology, I think Murano folks were going to rename this component to be more precise about it. Mistral DSL doesn’t seem to be a good option for solving tasks that Murano is intended to solve. Specifically I mean things like complex object composition, description of data types, contracts and so on. Like Alex and Stan mentioned Murano DSL tends to grow into a full programming language. Most likely Mistral will be used in Murano for implementation, at least we see where it would be valuable. But Mistral is not so matured yet, we need to keep working hard and be patient :) Anyway, we keep thinking on how to make both languages look similar or at least the possibility to use them seamlessly, if needed (call Mistral workflows from Murano DSL or vice versa). Renat Akhmerov @ Mirantis Inc. On 16 Feb 2014, at 05:48, Clint Byrum cl...@fewbar.com wrote: Excerpts from Alexander Tivelkov's message of 2014-02-14 18:17:10 -0800: Hi folks, Murano matures, and we are getting more and more feedback from our early adopters. The overall reception is very positive, but at the same time there are some complaints as well. By now the most significant complaint is is hard to write workflows for application deployment and maintenance. Current version of workflow definition markup really have some design drawbacks which limit its potential adoption. They are caused by the fact that it was never intended for use for Application Catalog use-cases. Just curious, is there any reason you're not collaborating on Mistral for this rather than both having a workflow engine? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting reminder - 02/17/2014
Hi, This is a reminder that we’ll have an IRC community meeting today at #openstack-meeting at 16.00 UTC. Here’s the agenda: Review action items Discuss current status Continue DSL discussion Open discussion (roadblocks, suggestions, etc.) Please let us know if you have other topics to discuss. Thanks! Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes - 02/17/2014
Hi, Thanks for joining the community meeting at #openstack-meeting today. Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-02-17-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-02-17-16.00.log.html Looking forward to see you again! Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting reminder - 02/24/2014
Hi Mistral team, This is a reminder that we’ll have another IRC community meeting today (#openstack-meeting) at 16.00 UTC. Here’s the agenda: Review action items Discuss current status Continue DSL discussion Open discussion (roadblocks, suggestions, etc.) You can also find the agenda and the links to the previous meeting minutes and logs at https://wiki.openstack.org/wiki/Meetings/MistralAgenda. Please follow up on this email if you have additional items to discuss. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes - 02/24/2014
Folks, Thanks for joining us at #openstack-meeting. Here are the links to the meeting minutes and log: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-02-24-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-02-24-16.00.log.html Next meeting will be held on March 3. Looking forward to chat with you again. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging
On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote: Renat, Regarding your comments on change https://review.openstack.org/#/c/75609/, I don't think the port to oslo.messaging is just a swap from pika to oslo.messaging. OpenStack services as I understand is usually implemented as an RPC client/server over a messaging transport. Sync vs async calls are done via the RPC client call and cast respectively. The messaging transport is abstracted and concrete implementation is done via drivers/plugins. So the architecture of the executor if ported to oslo.messaging needs to include a client, a server, and a transport. The consumer (in this case the mistral engine) instantiates an instance of the client for the executor, makes the method call to handle task, the client then sends the request over the transport to the server. The server picks up the request from the exchange and processes the request. If cast (async), the client side returns immediately. If call (sync), the client side waits for a response from the server over a reply_q (a unique queue for the session in the transport). Also, oslo.messaging allows versioning in the message. Major version change indicates API contract changes. Minor version indicates backend changes but with API compatibility. My main concern about this patch is not related with messaging infrastructure. I believe you know better than me how it should look like. I’m mostly concerned with the way of making changes you chose. From my perspective, it’s much better to make atomic changes where every changes doesn’t affect too much in existing architecture. So the first step could be to change pika to oslo.messaging with minimal structural changes without introducing versioning (could be just TODO comment saying that the framework allows it and we may want to use it in the future, to be decide), without getting rid of the current engine structure (local, scalable). Some of the things in the file structure and architecture came from the decisions made by many people and we need to be careful about changing them. So, where I'm headed with this change... I'm implementing the basic structure/scaffolding for the new executor service using oslo.messaging (default transport with rabbit). Since the whole change will take a few rounds, I don't want to disrupt any changes that the team is making at the moment and so I'm building the structure separately. I'm also adding versioning (v1) in the module structure to anticipate any versioning changes in the future. I expect the change request will lead to some discussion as we are doing here. I will migrate the core operations of the executor (handle_task, handle_task_error, do_task_action) to the server component when we agree on the architecture and switch the consumer (engine) to use the new RPC client for the executor instead of sending the message to the queue over pika. Also, the launcher for ./mistral/cmd/task_executor.py will change as well in subsequent round. An example launcher is here https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine. The interceptor project here is what I use to research how oslo.messaging works. I hope this is clear. The blueprint only changes how the request and response are being transported. It shouldn't change how the executor currently works. Please create a document describing the approach you’re pursuing here. I would expect to see the main goals you want to achieve upon completion. Finally, can you clarify the difference between local vs scalable engine? I personally do not prefer to explicitly name the engine scalable because this requirement should be in the engine by default and we do not need to explicitly state/separate that. But if this is a roadblock for the change, I can put the scalable structure back in the change to move this forward. Separation for local and scalable implementations appeared for historical reasons because from the beginning we didn’t see how it all would look like and hence we tried different approaches to implement the engine. At some point we got 2 working versions: the one that didn’t distribute anything (local) and another one that could distribute tasks over task executors via asynchronous HA transport (scalable). Later on we decided to leave them both since scalable is needed by the requirements and local might be useful for demonstration purposes and testing since it doesn’t require RabbitMQ to be installed. So we decided to refactor both and make them work similarly except the way they run tasks. Thanks. Renat Akhmerov @Mirantis Inc.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Local vs. Scalable Engine
On 25 Feb 2014, at 07:12, W Chan m4d.co...@gmail.com wrote: As I understand, the local engine runs the task immediately whereas the scalable engine sends it over the message queue to one or more executors. Correct. In what circumstances would we see a Mistral user using a local engine (other than testing) instead of the scalable engine? Yes, mostly testing we it could be used for demonstration purposes also or in the environments where installing RabbitMQ is not desirable. If we are keeping the local engine, can we move the abstraction to the executor instead, having drivers for a local executor and remote executor? The message flow from the engine to the executor would be consistent, it's just where the request will be processed. I think I get the idea and it sounds good to me. We could really have executor in both cases but the transport from engine to executor can be different. Is that what you’re suggesting? And what do you call driver here? And since we are porting to oslo.messaging, there's already a fake driver that allows for an in process Queue for local execution. The local executor can be a derivative of that fake driver for non-testing purposes. And if we don't want to use an in process queue here to avoid the complexity, we can have the client side module of the executor determine whether to dispatch to a local executor vs. RPC call to a remote executor. Yes, that sounds interesting. Could you please write up some etherpad with details explaining your idea? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Local vs. Scalable Engine
“In process” is fine to me. Winson, please register a blueprint for this change and put the link in here so that everyone can see what it all means exactly. My feeling is that we can approve and get it done pretty soon. Renat Akhmerov @ Mirantis Inc. On 25 Feb 2014, at 12:40, Dmitri Zimine d...@stackstorm.com wrote: I agree with Winson's points. Inline. On Feb 24, 2014, at 8:31 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 25 Feb 2014, at 07:12, W Chan m4d.co...@gmail.com wrote: As I understand, the local engine runs the task immediately whereas the scalable engine sends it over the message queue to one or more executors. Correct. Note: that local is confusing here, in process will reflect what it is doing better. In what circumstances would we see a Mistral user using a local engine (other than testing) instead of the scalable engine? Yes, mostly testing we it could be used for demonstration purposes also or in the environments where installing RabbitMQ is not desirable. If we are keeping the local engine, can we move the abstraction to the executor instead, having drivers for a local executor and remote executor? The message flow from the engine to the executor would be consistent, it's just where the request will be processed. I think I get the idea and it sounds good to me. We could really have executor in both cases but the transport from engine to executor can be different. Is that what you’re suggesting? And what do you call driver here? +1 to abstraction to the executor, indeed the local and remote engines today differ only by how they invoke executor, e.g. transport / driver. And since we are porting to oslo.messaging, there's already a fake driver that allows for an in process Queue for local execution. The local executor can be a derivative of that fake driver for non-testing purposes. And if we don't want to use an in process queue here to avoid the complexity, we can have the client side module of the executor determine whether to dispatch to a local executor vs. RPC call to a remote executor. Yes, that sounds interesting. Could you please write up some etherpad with details explaining your idea? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging
Yes, right. Thanks Winson. Renat Akhmerov @ Mirantis Inc. On 26 Feb 2014, at 01:39, W Chan m4d.co...@gmail.com wrote: Sure. Let me give this some thoughts and work with you separately. Before we speak up, we should have a proposal for discussion. On Mon, Feb 24, 2014 at 9:53 PM, Dmitri Zimine d...@stackstorm.com wrote: Winson, While you're looking into this and working on the design, may be also think through other executor/engine communications. We talked about executor communicating to engine over 3 channels (DB, REST, RabbitMQ) which I wasn't happy about ;) and put it off for some time. May be it can be rationalized as part of your design. DZ. On Feb 24, 2014, at 11:21 AM, W Chan m4d.co...@gmail.com wrote: Renat, Regarding your comments on change https://review.openstack.org/#/c/75609/, I don't think the port to oslo.messaging is just a swap from pika to oslo.messaging. OpenStack services as I understand is usually implemented as an RPC client/server over a messaging transport. Sync vs async calls are done via the RPC client call and cast respectively. The messaging transport is abstracted and concrete implementation is done via drivers/plugins. So the architecture of the executor if ported to oslo.messaging needs to include a client, a server, and a transport. The consumer (in this case the mistral engine) instantiates an instance of the client for the executor, makes the method call to handle task, the client then sends the request over the transport to the server. The server picks up the request from the exchange and processes the request. If cast (async), the client side returns immediately. If call (sync), the client side waits for a response from the server over a reply_q (a unique queue for the session in the transport). Also, oslo.messaging allows versioning in the message. Major version change indicates API contract changes. Minor version indicates backend changes but with API compatibility. So, where I'm headed with this change... I'm implementing the basic structure/scaffolding for the new executor service using oslo.messaging (default transport with rabbit). Since the whole change will take a few rounds, I don't want to disrupt any changes that the team is making at the moment and so I'm building the structure separately. I'm also adding versioning (v1) in the module structure to anticipate any versioning changes in the future. I expect the change request will lead to some discussion as we are doing here. I will migrate the core operations of the executor (handle_task, handle_task_error, do_task_action) to the server component when we agree on the architecture and switch the consumer (engine) to use the new RPC client for the executor instead of sending the message to the queue over pika. Also, the launcher for ./mistral/cmd/task_executor.py will change as well in subsequent round. An example launcher is here https://github.com/uhobawuhot/interceptor/blob/master/bin/interceptor-engine. The interceptor project here is what I use to research how oslo.messaging works. I hope this is clear. The blueprint only changes how the request and response are being transported. It shouldn't change how the executor currently works. Finally, can you clarify the difference between local vs scalable engine? I personally do not prefer to explicitly name the engine scalable because this requirement should be in the engine by default and we do not need to explicitly state/separate that. But if this is a roadblock for the change, I can put the scalable structure back in the change to move this forward. Thanks. Winson ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Understanding parameters for tasks and actions
Hi team,
I’m currently working on the first version of Data Flow and I would like to
make sure we all clearly understand how to interpret “parameters for tasks and
actions when we declare them in Mistral DSL. I feel like I’m getting lost here
a little bit. The problem is that we still don’t have a solid DSL spec since we
keep changing our vision (especially after new members joined the team). But
that may be fine, it’s life.
I also have a couple of suggestions that I’d like to discuss with you. Sorry if
that seems verbose, I’ll try to be as concise as possible.
I took a couple of snippets from [1] and put them in here.
# Snippet 1.
Services:
Nova:
type: REST_API
parameters:
baseUrl: $.novaURL
actions:
createVM:
parameters:
url: /servers/{$.vm_id}
method: POST
output:
select: $.server.id
store-as: vm_id
# Snippet 2.
Workflow:
tasks:
createVM:
action: Nova:createVM
on-success: waitForIP
on-error: sendCreateVMError
“$.” - handle to workflow execution storage (what we call ‘context’ now) where
we keep workflow variables.
Let’s say our workflow input is JSON like this:
{
“novaURL”: “http://localhost:123”,
“image_id”: “123
}
Questions
So the things that I don’t like or am not sure about:
1. Task “createVM” needs to use “image_id” but it doesn’t have any information
about it its declaration.
According to the current vision it should be like
createVM:
action: Nova:createVM
parameters:
image_id: $.image_id
And at runtime “image_id should be resolved to “123” get passed to action and,
in fact, be kind of the third parameter along with “url” and “method”. This is
specifically interesting because on one hand we have different types of
parameters: “url” and “method” for REST_API action define the nature of the
action itself. But “image_id” on the other hand is a dynamic data coming
eventually from user input.
So the question is: do we need to differentiate between these types of
parameters explicitly and make a part of the specification?
We also had a notion of “task-parameters” for action declarations which is
supposed to be used to declare this second type of parameters (dynamic) but do
we really need it? I guess if we clearly declare input and output at task level
then actions should be able to use them according to their nature.
2. Action declaration “createVM” has section “response” which may not be ok in
terms of level of abstraction.
My current vision is that actions should not declare how we store the result
(“output”) in execution. Ideally looking at tasks only should give us
comprehensive understanding of how workflow data flows. So I would move
“store-as” to task level.
Suggestions
1. Define “input” and “output” at task level like this:
createVM:
input:
image_id: $.image_id
output: vm_id
Where “output: vm_id” is basically a replacement for “store-as: vm_id” at
action level, i.e. it’s a hint to Mistral to store the output of this task
under “vm_id” key in execution context. Again, the idea is to define task and
action responsibilities more strictly:
Task is a high-level workflow building block which defines workflow logical
step and how it modifies workflow data. Task doesn’t contain technical details
on how it’s implemented.
Action is an implementor of the workflow logical step defined by a task. Action
defines specific algorithm of how task is implemented.
2. User “parameters” only for actions to specify their additional properties
influencing their nature (like method for HTTP actions).
Please let me know your thoughts. We can make required adjustments right now.
[1] https://etherpad.openstack.org/p/mistral-poc
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging
Winson, nice job! Now it totally makes sense to me. You’re good to go with this unless others have objections. Just one technical dummy question (sorry, I’m not yet familiar with oslo.messaging): at your picture you have “Transport”, so what can be specifically except RabbitMQ? Renat Akhmerov @ Mirantis Inc. On 26 Feb 2014, at 14:30, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Looks good. Thanks, Winson! Renat, What do you think? On Wed, Feb 26, 2014 at 10:00 AM, W Chan m4d.co...@gmail.com wrote: The following link is the google doc of the proposed engine/executor message flow architecture. https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing The diagram on the right is the scalable engine where one or more engine sends requests over a transport to one or more executors. The executor client, transport, and executor server follows the RPC client/server design pattern in oslo.messaging. The diagram represents the local engine. In reality, it's following the same RPC client/server design pattern. The only difference is that it'll be configured to use a fake RPC backend driver. The fake driver uses in process queues shared between a pair of engine and executor. The following are the stepwise changes I will make. 1) Keep the local and scalable engine structure intact. Create the Executor Client at ./mistral/engine/scalable/executor/client.py. Create the Executor Server at ./mistral/engine/scalable/executor/service.py and implement the task operations under ./mistral/engine/scalable/executor/executor.py. Delete ./mistral/engine/scalable/executor/executor.py. Modify the launcher ./mistral/cmd/task_executor.py. Modify ./mistral/engine/scalable/engine.py to use the Executor Client instead of sending the message directly to rabbit via pika. The sum of this is the atomic change that keeps existing structure and without breaking the code. 2) Remove the local engine. https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor 3) Implement versioning for the engine. https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning 4) Port abstract engine to use oslo.messaging and implement the engine client, engine server, and modify the API layer to consume the engine client. https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process. Winson On Mon, Feb 24, 2014 at 8:07 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 25 Feb 2014, at 02:21, W Chan m4d.co...@gmail.com wrote: Renat, Regarding your comments on change https://review.openstack.org/#/c/75609/, I don't think the port to oslo.messaging is just a swap from pika to oslo.messaging. OpenStack services as I understand is usually implemented as an RPC client/server over a messaging transport. Sync vs async calls are done via the RPC client call and cast respectively. The messaging transport is abstracted and concrete implementation is done via drivers/plugins. So the architecture of the executor if ported to oslo.messaging needs to include a client, a server, and a transport. The consumer (in this case the mistral engine) instantiates an instance of the client for the executor, makes the method call to handle task, the client then sends the request over the transport to the server. The server picks up the request from the exchange and processes the request. If cast (async), the client side returns immediately. If call (sync), the client side waits for a response from the server over a reply_q (a unique queue for the session in the transport). Also, oslo.messaging allows versioning in the message. Major version change indicates API contract changes. Minor version indicates backend changes but with API compatibility. My main concern about this patch is not related with messaging infrastructure. I believe you know better than me how it should look like. I’m mostly concerned with the way of making changes you chose. From my perspective, it’s much better to make atomic changes where every changes doesn’t affect too much in existing architecture. So the first step could be to change pika to oslo.messaging with minimal structural changes without introducing versioning (could be just TODO comment saying that the framework allows it and we may want to use it in the future, to be decide), without getting rid of the current engine structure (local, scalable). Some of the things in the file structure and architecture came from the decisions made by many people and we need to be careful about changing them. So, where I'm headed with this change... I'm implementing the basic structure/scaffolding for the new executor service using oslo.messaging (default transport with rabbit). Since the whole change will take a few rounds, I don't want to disrupt any changes that the team is making at the moment and so I'm building
Re: [openstack-dev] [Mistral] Renaming action types
Hm.. I see your point. Generally, I like short names but expressive enough to
understand what it is. VERB_NOUN would be good but nothing decent comes to my
mind regarding HTTP :)
If you guys have any suggestions you’re welcome.
Renat Akhmerov
@ Mirantis Inc.
On 26 Feb 2014, at 16:33, Dmitri Zimine d...@stackstorm.com wrote:
+1
Should we use VERB_NOUN pattern? Or relax it for some obvious? I can't figure
a good VERB_NOUN for HTTP. REQUEST_HTTP is dull.
BTW it's interesting that we assume that a service can contain only actions
of the same type.
DZ
On Feb 26, 2014, at 1:10 AM, Nikolay Makhotkin nmakhot...@mirantis.com
wrote:
Agree, I don't see any thing that makes sense with words 'REST' and 'API'
too.
On Wed, Feb 26, 2014 at 11:38 AM, Renat Akhmerov rakhme...@mirantis.com
wrote:
Folks,
I’m proposing to rename these two action types REST_API and MISTRAL_REST_API
to HTTP and MISTRAL_HTTP. Words “REST” and “API” don’t look correct to me,
if you look at
Services:
Nova:
type: REST_API
parameters:
baseUrl: {$.novaURL}
actions:
createVM:
parameters:
url: /servers/{$.vm_id}
method: POST
There’s no information about “REST” or “API” here. It’s just a spec how to
form an HTTP request.
Thoughts?
Renat Akhmerov
@ Mirantis Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Best Regards,
Nikolay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Understanding parameters for tasks and actions
On 26 Feb 2014, at 15:18, Timur Nurlygayanov tnurlygaya...@mirantis.com wrote: for me just unclear the following syntacsis: $.image_id what is $ in this case? It will be more clear if we can replace $ to something - any instance with readable name, like global.image_id or context.image_id. looks like $ can be the different in different name spaces. it will be unclear for new users and of course it will be easy when we have experience with the DSL. Yes, “$” actually comes from YAQL which is simple expression language with pythonic-like syntax. We were planning to use it here. However, we got an idea to make it possible to use other languages by having a simple abstraction ExpressionEvaluator in the system. It already exists.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Renaming action types
Ooh, I was wrong. Sorry. We use dash naming. We have “on-success”, “on-error”
and so forth.
Please let us know if you see other inconsistencies.
Thanks
Renat Akhmerov
@ Mirantis Inc.
On 26 Feb 2014, at 21:00, Renat Akhmerov rakhme...@mirantis.com wrote:
Thanks Jay.
Regarding underscore naming. If you meant using underscore naming for
“createVM” and “novaURL” then yes, “createVM” is just a task name and it’s a
user preference. The same about “novaURL” which will be defined by users. As
for keywords, seemingly we follow underscore naming.
Renat Akhmerov
@ Mirantis Inc.
On 26 Feb 2014, at 17:58, Jay Pipes jaypi...@gmail.com wrote:
On Wed, 2014-02-26 at 14:38 +0700, Renat Akhmerov wrote:
Folks,
I’m proposing to rename these two action types REST_API and
MISTRAL_REST_API to HTTP and MISTRAL_HTTP. Words “REST” and “API”
don’t look correct to me, if you look at
Services:
Nova:
type: REST_API
parameters:
baseUrl: {$.novaURL}
actions:
createVM:
parameters:
url: /servers/{$.vm_id}
method: POST
There’s no information about “REST” or “API” here. It’s just a spec
how to form an HTTP request.
+1 on HTTP and MISTRAL_HTTP.
On an unrelated note, would it be possible to use under_score_naming
instead of camelCase naming?
Best,
-jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Defining term DSL
I don’t see any issues with term DSL (Domain Specific Language). This is really a language which 'workbook definitions’ are written in. Dmitri, could you please provide more details on why you question it? Thanks Renat Akhmerov @ Mirantis Inc. On 26 Feb 2014, at 20:12, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Due to the comment to https://review.openstack.org/#/c/75888/1 there is a quiestion: Do we use term DSL or something else? I think the word 'DSL' is more fit thing that we call 'workbook definition', some text describing workflows, services, tasks and actions. And processing module for this also has name 'dsl'. Thoughts? Dmitri? Nikolay, Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Renaming events to triggers and move them out of Workflow
Hi team, When I tell peopleI about Mistral I always have a hard time explaining why we use term “event” for declaring ways to start workflows. For example, take a look at the following snippet: Workflow: ... events: execute_backup: type: periodic tasks: execute_backup parameters: cron-pattern: */1 * * * * Here we just tell Mistral “we should run backup workflow every minute”. My suggestion is to rename “events” to “triggers” because actually events are going to be sent by Mistral when we implement notification mechanism (sending events about over HTTP or AMQP about changes in workflows' and tasks’ state). I would also suggest we move events out of “Workflow” section since it’s not actually a part of workflow. Thoughts? If you agree I’ll create a blueprint for this. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Defining term DSL
On 26 Feb 2014, at 22:54, Dmitri Zimine d...@stackstorm.com wrote: Based on the terminology from [1], it's not part of the model, but the language that describes the model in the file. Sorry, I’m having a hard time trying to understand this phrase :) What do you mean by “model” here? And why should DSL be a part of the model? And theoretically this may be not the only language to express the workflow. Sure, from that perspective, for example, JVM has many “DSLs”: Java, Scala, Groovy etc. Once the file is parsed, we operate on model, not on the language. How does it influence using term DSL? DSL is, in fact, a user interface. Model is something we build inside a system to process DSL in a more convenient way. I am afraid we are breaking an abstraction when begin to call things DSLWorkbook or DSLWorkflow. What is the difference between Workbook and DSLWorkbook, and how DSL is relevant here? Prefix “DSL” tells that this exactly matches the structure of an object declared with using DSL. But, for example, a workbook in a database may have (and it has) a different structure better suitable for storing it in a relational model. So I’m not sure what you mean by saying “we are breaking an abstraction” here. What abstraction? [1] https://wiki.openstack.org/wiki/Mistral, ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Understanding parameters for tasks and actions
. The point here is we don't refer the context variables in Service definitions. Only transformation of input, with some static text. The service actions now look symmetric on input/output. Yes, makes sense. As you said “we don’t refer the context variables” and that’s why I’m against of having “store-as” in action declaration, it indirectly refers to the context (the output of action will be stored under this name in the context). Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] What is the model?
On 27 Feb 2014, at 08:11, Dmitri Zimine d...@stackstorm.com wrote: As a learning exercise, I was trying to extract the intend behind the DSL and current code. Giving that many things are still moving, I filled up the blanks with imagination. Here is my understanding on where you are going: https://docs.google.com/a/stackstorm.com/drawings/d/1qBxrmQ8F7YFlz930zEbHKUxgzc3ty7JrlmoTR5TxQlo/edit Is it where we are going? Can we use it to bring us the new team members to your understanding? Please comment here or right on the document. Yes, Dmitri. This is correct. The picture you created seems to express the intention. The reasons why we decided to do this are: We don’t have any validation now when parsing DSL. It means that incorrect part of DSL may only get revealed, for example, hours later after workflow started when Mistral process a particular action and sees that something is missing. So far the whole work with dsl has been based on having a dictionary consisting of other nested dictionaries that reflect exactly what we have in YAML and hence we’ve had to do something like wb_dsl[‘Workflow’][‘tasks’][task_name][‘action’] to access something. It just looks ugly to use string literals all over the code: it’s fragile, we need to remember everything about DSL structure and so on. It seems to be a good idea to have classes in the code that reflect DSL structure so that we can always see how it looks like and use it in development process instead of keeping everything in the head. In other words, from code perspective these classes play the role of DSL specification (like DTOs reflect interprocess communication protocols or rest resources that we define explicitly in API layer). It will be much easier to do refactoring having these classes. Although Python is a dynamic language and refactoring support is very limited due to fundamental reasons IDEs can still help with lots of things. Nikolay's recent refactoring seems to be going in this direction : https://review.openstack.org/#/c/75888/ Yes, we have blueprints https://blueprints.launchpad.net/mistral/+spec/mistral-dsl-model https://blueprints.launchpad.net/mistral/+spec/mistral-dsl-validation which were registered to address the problems I mentioned above. I guess we should put more information in the first one though. We are missing a separation of ActionSpec and ActionData. They have distinct roles. First - action spec, from Service/actions section - defines the new action declaratively from existing action. Second - action data from Workflow/tasks/task - defines parameters for a particular action instance. There’s still a lot of work left on actions including the ability to write custom actions and plug them in to Mistral. What you call ActionSpec, in my understanding, just an implementation of BaseAction class that we have now (I would rename it just to Action) that has a constructor with all parameters required by this action and its implementation logic. ActionData is what we declare in DSL. Renat Akhmerov @Mistral Inc___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] std:repeat action
Thanks Manas! This is one of the important things we need to get done within the next couple of weeks. Since it’s going to affect engine I think we need to wait for a couple of days with the implementation till we merge the changes that are being worked on and that also affect engine significantly. Team, please research carefully this etherpad and leave your comments. It’s a pretty tricky thing and we need to figure out the best strategy how to approach this kind of things. We’re going to have more problems similar to this one. Renat Akhmerov @ Mirantis Inc. On 25 Feb 2014, at 10:07, manas kelshikar mana...@gmail.com wrote: Hi everyone, I have put down my thoughts about the standard repeat action blueprint. https://blueprints.launchpad.net/mistral/+spec/mistral-std-repeat-action I have added link to an etherpad document which explore a few alternatives of the approach. I have explored details of how the std:repeat action should behave as defined in the blueprint. Further there are some thoughts on how it could be designed to remove ambiguity in the chaining. Please take a look. Thanks, Manas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] a question about Mistral
You’re welcome! Please let us know if you need more info. Renat Akhmerov @ Mirantis Inc. On 28 Feb 2014, at 09:48, Liuji (Jeremy) jeremy@huawei.com wrote: Hi, Yes. I mean the feature like VMware Distributed Resource Scheduler. Now I am totally clear about the question. Thanks for your explanations. Thanks, Jeremy Liu -Original Message- From: Renat Akhmerov [mailto:rakhme...@mirantis.com] Sent: Thursday, February 27, 2014 5:03 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Mistral] a question about Mistral Hey, Can you please provide more details on what you're interested in? What do you mean by DRS? If you mean VMware Distributed Resource Scheduler then yes and no. It's not the major goal of Mistral but Mistral is a more generic tool that could be used to build something like this. The primary goal of Mistral is to provide a workflow engine and easy way to integrate Mistral with other systems so that we can trigger workflow execution upon external events like Ceilometer alarms, timer or anything else. Feel free to ask any questions, thanks! Renat Akhmerov @ Mirantis Inc. On 27 Feb 2014, at 15:08, Liuji (Jeremy) jeremy@huawei.com wrote: Hi, Mistral members I am very interesting in the project Mistral. About the wiki of the Mistral, I have a question about the use case description as the follow. Live migration A user specifies tasks for VM live migration triggered upon an event from Ceilometer (CPU consumption 100%). Is this mean Mistral has the plan to provide the feature like DRS? I am a newbie for Mistral and I apologize if I am missing something very obvious. Thanks, Jeremy Liu ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Renaming events to triggers and move them out of Workflow
Ok, thanks guys! https://blueprints.launchpad.net/mistral/+spec/mistral-rename-event-to-trigger Renat Akhmerov @ Mirantis Inc. On 28 Feb 2014, at 07:41, Dmitri Zimine d...@stackstorm.com wrote: Agree on both. On Feb 26, 2014, at 8:51 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Hi team, When I tell peopleI about Mistral I always have a hard time explaining why we use term “event” for declaring ways to start workflows. For example, take a look at the following snippet: Workflow: ... events: execute_backup: type: periodic tasks: execute_backup parameters: cron-pattern: */1 * * * * Here we just tell Mistral “we should run backup workflow every minute”. My suggestion is to rename “events” to “triggers” because actually events are going to be sent by Mistral when we implement notification mechanism (sending events about over HTTP or AMQP about changes in workflows' and tasks’ state). I would also suggest we move events out of “Workflow” section since it’s not actually a part of workflow. Thoughts? If you agree I’ll create a blueprint for this. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Renaming action types
Haah :) Honestly, I don’t like it. “invoke” doesn’t seem to be carrying any
useful information here. And “invoke_mistral” looks completely confusing since
it’s not clear it’s related with HTTP.
Renat Akhmerov
@ Mirantis Inc.
On 27 Feb 2014, at 23:42, Manas Kelshikar ma...@stackstorm.com wrote:
How about ...
invoke_http invoke_mistral to fit the verb_noun pattern.
On Wed, Feb 26, 2014 at 6:04 AM, Renat Akhmerov rakhme...@mirantis.com
wrote:
Ooh, I was wrong. Sorry. We use dash naming. We have “on-success”, “on-error”
and so forth.
Please let us know if you see other inconsistencies.
Thanks
Renat Akhmerov
@ Mirantis Inc.
On 26 Feb 2014, at 21:00, Renat Akhmerov rakhme...@mirantis.com wrote:
Thanks Jay.
Regarding underscore naming. If you meant using underscore naming for
“createVM” and “novaURL” then yes, “createVM” is just a task name and it’s
a user preference. The same about “novaURL” which will be defined by users.
As for keywords, seemingly we follow underscore naming.
Renat Akhmerov
@ Mirantis Inc.
On 26 Feb 2014, at 17:58, Jay Pipes jaypi...@gmail.com wrote:
On Wed, 2014-02-26 at 14:38 +0700, Renat Akhmerov wrote:
Folks,
I’m proposing to rename these two action types REST_API and
MISTRAL_REST_API to HTTP and MISTRAL_HTTP. Words “REST” and “API”
don’t look correct to me, if you look at
Services:
Nova:
type: REST_API
parameters:
baseUrl: {$.novaURL}
actions:
createVM:
parameters:
url: /servers/{$.vm_id}
method: POST
There’s no information about “REST” or “API” here. It’s just a spec
how to form an HTTP request.
+1 on HTTP and MISTRAL_HTTP.
On an unrelated note, would it be possible to use under_score_naming
instead of camelCase naming?
Best,
-jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Defining term DSL
Yes. Guys, thanks for your feedback. I had a conversation with Dmitri today and realized you guys are right here. We should think about building basically a “domain model” which the system operates with and once we built it we should forget that we have some DSL or whatever we used to describe this model (could be other language, for example). Our initial intention actually was different but anyway what you’re saying is valid. Looks like Nikolay agrees with me too and he’s now reworking this commit. Coming up soon. Renat Akhmerov @ Mirantis Inc. On 27 Feb 2014, at 23:36, Manas Kelshikar ma...@stackstorm.com wrote: I looked at the review prior to looking at the discussion and even I was confused by names like DSL*. The way I see it DSL is largely syntatic sugar and therefore it will be good to have a clear separation between DSL and model. The fact that something is defined in a DSL is irrelevant once it crosses mistral API border in effect within mistral itself DSLTask, DSLAction etc are simply description objects and how they were defined does not matter to mistral implementation. Each description object being a recipe to eventually execute a task. We therefore already see these two manifestations in current code i.e. DSLTask(per Nikolay's change) and Task (https://github.com/stackforge/mistral/blob/master/mistral/api/controllers/v1/task.py#L30). To me it seems like we only need to agree upon names. Here are my suggestions - i) DSLTask - Task Task - TaskInstance (Similarly for workflow, action etc.) OR ii) DSLTask - TaskSpec Task - Task (Similarly for workflow, action etc.) On Wed, Feb 26, 2014 at 9:31 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 26 Feb 2014, at 22:54, Dmitri Zimine d...@stackstorm.com wrote: Based on the terminology from [1], it's not part of the model, but the language that describes the model in the file. Sorry, I’m having a hard time trying to understand this phrase :) What do you mean by “model” here? And why should DSL be a part of the model? And theoretically this may be not the only language to express the workflow. Sure, from that perspective, for example, JVM has many “DSLs”: Java, Scala, Groovy etc. Once the file is parsed, we operate on model, not on the language. How does it influence using term DSL? DSL is, in fact, a user interface. Model is something we build inside a system to process DSL in a more convenient way. I am afraid we are breaking an abstraction when begin to call things DSLWorkbook or DSLWorkflow. What is the difference between Workbook and DSLWorkbook, and how DSL is relevant here? Prefix “DSL” tells that this exactly matches the structure of an object declared with using DSL. But, for example, a workbook in a database may have (and it has) a different structure better suitable for storing it in a relational model. So I’m not sure what you mean by saying “we are breaking an abstraction” here. What abstraction? [1] https://wiki.openstack.org/wiki/Mistral, ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging
Hi Joshua, Sorry, I’ve been very busy for the last couple of days and didn’t respond quickly enough. Well, first of all, it’s my bad that I’ve not been following TaskFlow progress for a while and, honestly, I just need to get more info on the current TaskFlow status. So I’ll do that and get back to you soon. As you know, there were reasons why we decided to go this path (without using TaskFlow) but I’ve always thought we will be able to align our efforts as we move forward once requirements and design of Mistral become more clear. I really want to use TaskFlow for Mistral implementation. We just need to make sure that it will bring more value than pain (sorry if it sounds harsh). Thanks for your feedback and this info. We’ll get in touch with you soon. Renat Akhmerov @ Mirantis Inc. On 27 Feb 2014, at 03:22, Joshua Harlow harlo...@yahoo-inc.com wrote: So this design is starting to look pretty familiar to a what we have in taskflow. Any reason why it can't just be used instead? https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine This code is in a functional state right now, using kombu (for the moment, until oslo.messaging becomes py3 compliant). The concept of a engine which puts messages on a queue for a remote executor is in-fact exactly the case taskflow is doing (the remote exeuctor/worker will then respond when it is done and the engine will then initiate the next piece of work to do) in the above listed etherpad (and which is implemented). Is it the case that in mistral the engine will be maintaining the 'orchestration' of the workflow during the lifetime of that workflow? In the case of mistral what is an engine server? Is this a server that has engines in it (where each engine is 'orchestrating' the remote/local workflows and monitoring and recording the state transitions and data flow that is occurring)? The details @ https://blueprints.launchpad.net/mistral/+spec/mistral-engine-standalone-process seems to be already what taskflow provides via its engine object, creating a application which runs engines and those engines initiate workflows is made to be dead simple. From previous discussions with the mistral folks it seems like the overlap here is getting more and more, which seems to be bad (and means something is broken/wrong). In fact most of the concepts that u have blueprints for have already been completed in taskflow (data-flow, engine being disconnected from the rest api…) and ones u don't have listed (resumption, reversion…). What can we do to fix this situation? -Josh From: Nikolay Makhotkin nmakhot...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, February 25, 2014 at 11:30 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Porting executor and engine to oslo.messaging Looks good. Thanks, Winson! Renat, What do you think? On Wed, Feb 26, 2014 at 10:00 AM, W Chan m4d.co...@gmail.com wrote: The following link is the google doc of the proposed engine/executor message flow architecture. https://drive.google.com/file/d/0B4TqA9lkW12PZ2dJVFRsS0pGdEU/edit?usp=sharing The diagram on the right is the scalable engine where one or more engine sends requests over a transport to one or more executors. The executor client, transport, and executor server follows the RPC client/server design pattern in oslo.messaging. The diagram represents the local engine. In reality, it's following the same RPC client/server design pattern. The only difference is that it'll be configured to use a fake RPC backend driver. The fake driver uses in process queues shared between a pair of engine and executor. The following are the stepwise changes I will make. 1) Keep the local and scalable engine structure intact. Create the Executor Client at ./mistral/engine/scalable/executor/client.py. Create the Executor Server at ./mistral/engine/scalable/executor/service.py and implement the task operations under ./mistral/engine/scalable/executor/executor.py. Delete ./mistral/engine/scalable/executor/executor.py. Modify the launcher ./mistral/cmd/task_executor.py. Modify ./mistral/engine/scalable/engine.py to use the Executor Client instead of sending the message directly to rabbit via pika. The sum of this is the atomic change that keeps existing structure and without breaking the code. 2) Remove the local engine. https://blueprints.launchpad.net/mistral/+spec/mistral-inproc-executor 3) Implement versioning for the engine. https://blueprints.launchpad.net/mistral/+spec/mistral-engine-versioning 4) Port abstract engine to use oslo.messaging and implement the engine client, engine server, and modify the API layer to consume the engine client. https://blueprints.launchpad.net
[openstack-dev] [Mistral] Community meeting minutes - 03/03/2014
Thanks for joining us today #openstack-meeting for our weekly community meeting. As always, meeting minutes and log: Minutes: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-03-03-16.00.html Log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-03-03-16.00.log.html I’ll see you next time! Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] DSL model vs. DB model, renaming
I think we forgot to point to the commit itself. Here it is: https://review.openstack.org/#/c/77126/ Manas, can you please provide more details on your suggestion? For now let me just describe the background of Nikolay’s question. Basically, we are talking about how we are working with data inside Mistral. So far, for example, if a user sent a request to Mistral “start workflow” then Mistral would do the following: Get workbook DSL (YAML) from the DB (given that it’s been already persisted earlier). Load it into a dictionary-like structure using standard ‘yaml’ library. Based on this dictionary-like structure create all necessary DB objects to track the state of workflow execution objects and individual tasks. Perform all the necessary logic in engine and so on. The important thing here is that DB objects contain corresponding DSL snippets as they are described in DSL (e.g. tasks have property “task_dsl”) to reduce the complexity of relational model that we have in DB. Otherwise it would be really complicated and most of the queries would contain lots of joins. The example of non-trivial relation in DSL is “task”-”action name”-”service”-”service actions”-”action”, as you can see it would be hard to navigate to action in the DB from a task if our relational model matches to what we have in DSL. this approach leads to the problem of addressing any dsl properties using hardcoded strings which are spread across the code and that brings lots of pain when doing refactoring, when trying to understand the structure of the model we describe in DSL, it doesn’t allow to do validation easily and so on. So what we have in DB we’ve called “model” so far and we’ve called just “dsl” the dictionary structure coming from DSL. So if we got a part of the structure related to a task we would call it “dsl_task”. So what Nikolay is doing now is he’s reworking the approach how we work with DSL. Now we assume that after we parsed a workbook DSL we get some “model”. So that we don’t use “dsl” in the code anywhere this “model” describes basically the structure of what we have in DSL and that would allow to address the problems I mentioned above (hardcoded strings are replaced with access methods, we clearly see the structure of what we’re working with, we can easily validate it and so on). So when we need to access some DSL properties we would need to get workbook DSL from DB, build this model out of it and continue to work with it. Long story short, this model parsed from DSL is not the model we store in DB but they’re both called “model” which may be confusing. For me this non-DB model more looks like “domain model” or something like this. So the question I would ask ourselves here: Is the approach itself reasonable? Do we have better ideas on how to work with DSL? A good mental exercise here would be to imagine that we have more than one DSL, not only YAML but say XML. How would it change the picture? How can we clearly distinguish between these two models so that it wouldn’t be confusing? Do we have a better naming in mind? Thanks. Renat Akhmerov @ Mirantis Inc. On 05 Mar 2014, at 08:56, Manas Kelshikar ma...@stackstorm.com wrote: Since the renaming is for types in mistral.model.*. I am thinking we suffix with Spec e.g. TaskObject - TaskSpec ActionObject - ActionSpec and so on. The Spec suggest that it is a specification of the final object that ends up in the DB and not the actual object. Multiple actual objects can be derived from these Spec objects which fits well with the current paradigm. Thoughts? On Mon, Mar 3, 2014 at 9:43 PM, Manas Kelshikar ma...@stackstorm.com wrote: Hi Nikolay - Is your concern that mistral.db.sqlalchemy.models.* and mistral.model.* will lead to confusion or something else? IMHO as per your change model seems like the appropriate usage while what is stored in the DB is also a model. If we pick appropriate names to distinguish between nature of the objects we should be able to avoid any confusion and whether or not model appears in the module name should not matter much. Thanks, Manas On Mon, Mar 3, 2014 at 8:43 AM, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Hi, team! Please look at the commit . Module 'mistral/model' now is responsible for object model representation which is used for accessing properties of actions, tasks etc. We have a name problem - looks like we should rename module 'mistral/model' since we have DB models and they are absolutely different. Thoughts? Best Regards, Nikolay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] DSL model vs. DB model, renaming
Alright, good input Manas, appreciate. My comments are below... On 06 Mar 2014, at 10:47, Manas Kelshikar ma...@stackstorm.com wrote: Do we have better ideas on how to work with DSL? A good mental exercise here would be to imagine that we have more than one DSL, not only YAML but say XML. How would it change the picture? [Manas] As long as we form an abstraction between the DSL format i.e. YAML/XML and it consumption we will be able to move between various formats. (wishful) My personal preference is to not even have DSL show up anywhere in Mistral code apart from take it as input and transforming into this first level specification model - I know this is not the current state. Totally agree with your point. That’s what we’re trying to achieve. How can we clearly distinguish between these two models so that it wouldn’t be confusing? Do we have a better naming in mind? [Manas] I think we all would agree that the best approach is to have precise naming. I see your point of de-normalizing the dsl data into respective db model objects. In a previous email I suggested using *Spec. I will try to build on this - 1. Everything specified via the YAML input is a specification or definition or template. Therefore I suggest we suffix all these types with Spec/Definition/Template. So TaskSpec/TaskDefinition/TaskTemplate etc.. As per the latest change these are TaskData ... ActionData. After all the time I spent thinking about it I would choose Spec suffix since it’s short and expresses the intention well enough. In conjunction with “workbook” package name it would look very nice (basically we get specification of workbook which is what we’re talking about, right?) So if you agree then let’s change to TaskSpec, ActionSpec etc. Nikolay, sorry for making you change this patch again and again :) But it’s really important and going to have a long-term effect at the entire system. 2. As per current impl the YAML is stored as a key-value in the DB. This is fine since that is front-ended by objects that Nikolay has introduced. e.g. TaskData, ActionData etc. Yep, right. The only thing I would suggest is to avoid DB fields like “task_dsl” like we have now. The alternative could be “task_spec”. 3. As per my thinking a model object that ends up in the DB or a model object that is in memory all can reside in the same module. I view persistence as an orthogonal concern so no real reason to distinguish the module names of the two set of models. If we do choose to distinguish as per latest change i.e. mistral/workbook that works too. Sorry, I believe I wasn’t clear enough on this thing. I think we shouldn’t have these two models in the same package since what I meant by “DB model” is actually “execution” and “task” that carry workflow runtime information and refer to a particular execution (we could also call it “session”). So my point is that these are fundamentally different types of models. The best analogy that comes to my mind is the relationship “class - instance” where in our case “class = Specification (TaskSpec etc.) and “instance = Execution/Task”. Does it make any sense? @Nikolay - I am generally ok with the approach. I hope that this helps clarify my thinking and perception. Please ask more questions. Overall I like the approach of formalizing the 2 models. I am ok with current state of the review and have laid out my preferences. I like the current state of this patch. The only thing I would do is renaming “Data” to “Spec”. Thank you. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] DSL model vs. DB model, renaming
Ok, good! Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 15:25, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Manas, Renat, no problem :) The commit is sent already - https://review.openstack.org/#/c/75888/ On Thu, Mar 6, 2014 at 12:14 PM, Manas Kelshikar ma...@stackstorm.com wrote: I agree, let's rename data to spec and unblock the check-in. Nikolay - Sorry for the trouble :) On Mar 5, 2014 10:17 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Alright, good input Manas, appreciate. My comments are below... On 06 Mar 2014, at 10:47, Manas Kelshikar ma...@stackstorm.com wrote: Do we have better ideas on how to work with DSL? A good mental exercise here would be to imagine that we have more than one DSL, not only YAML but say XML. How would it change the picture? [Manas] As long as we form an abstraction between the DSL format i.e. YAML/XML and it consumption we will be able to move between various formats. (wishful) My personal preference is to not even have DSL show up anywhere in Mistral code apart from take it as input and transforming into this first level specification model - I know this is not the current state. Totally agree with your point. That’s what we’re trying to achieve. How can we clearly distinguish between these two models so that it wouldn’t be confusing? Do we have a better naming in mind? [Manas] I think we all would agree that the best approach is to have precise naming. I see your point of de-normalizing the dsl data into respective db model objects. In a previous email I suggested using *Spec. I will try to build on this - 1. Everything specified via the YAML input is a specification or definition or template. Therefore I suggest we suffix all these types with Spec/Definition/Template. So TaskSpec/TaskDefinition/TaskTemplate etc.. As per the latest change these are TaskData ... ActionData. After all the time I spent thinking about it I would choose Spec suffix since it’s short and expresses the intention well enough. In conjunction with “workbook” package name it would look very nice (basically we get specification of workbook which is what we’re talking about, right?) So if you agree then let’s change to TaskSpec, ActionSpec etc. Nikolay, sorry for making you change this patch again and again :) But it’s really important and going to have a long-term effect at the entire system. 2. As per current impl the YAML is stored as a key-value in the DB. This is fine since that is front-ended by objects that Nikolay has introduced. e.g. TaskData, ActionData etc. Yep, right. The only thing I would suggest is to avoid DB fields like “task_dsl” like we have now. The alternative could be “task_spec”. 3. As per my thinking a model object that ends up in the DB or a model object that is in memory all can reside in the same module. I view persistence as an orthogonal concern so no real reason to distinguish the module names of the two set of models. If we do choose to distinguish as per latest change i.e. mistral/workbook that works too. Sorry, I believe I wasn’t clear enough on this thing. I think we shouldn’t have these two models in the same package since what I meant by “DB model” is actually “execution” and “task” that carry workflow runtime information and refer to a particular execution (we could also call it “session”). So my point is that these are fundamentally different types of models. The best analogy that comes to my mind is the relationship “class - instance” where in our case “class = Specification (TaskSpec etc.) and “instance = Execution/Task”. Does it make any sense? @Nikolay - I am generally ok with the approach. I hope that this helps clarify my thinking and perception. Please ask more questions. Overall I like the approach of formalizing the 2 models. I am ok with current state of the review and have laid out my preferences. I like the current state of this patch. The only thing I would do is renaming “Data” to “Spec”. Thank you. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Nikolay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.com wrote: That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to see a killer object model for defining these workflows as a first step. What would a python-based equivalent of that real-world workflow look like? Then we can ask ourselves, does the DSL make this better or worse? Would we need to expose things like email handlers, or leave that to the general python libraries? $0.02 -S On 03/05/2014 10:50 PM, Dmitri Zimine wrote: Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting reminder - 03/10/2014
Hi team, This is a reminder about the community meeting today at #openstack-meeting (16.00 UTC). Here’s the agenda: Review action items Discuss current status Repeater What's left on Pilot Open discussion (roadblocks, suggestions, etc.) If you have anything else to discuss please let us know, Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Community meeting minutes/log - 03/10/2014
Hi, Thanks to everyone who joined us today at #openstack-meeting. For those who didn’t make it to attend the meeting here are the links to minutes and log: http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-03-10-16.00.html http://eavesdrop.openstack.org/meetings/mistral/2014/mistral.2014-03-10-16.00.log.html The next meeting will be on March 17 at the same time (16.00 UTC) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] MuranoPL questions?
Although being a little bit verbose it makes a lot of sense to me. @Joshua, Even assuming Python could be sandboxed and whatever else that’s needed to be able to use it as DSL (for something like Mistral, Murano or Heat) is done why do you think Python would be a better alternative for people who don’t know neither these new DSLs nor Python itself. Especially, given the fact that Python has A LOT of things that they’d never use. I know many people who have been programming in Python for a while and they admit they don’t know all the nuances of Python and actually use 30-40% of all of its capabilities. Even not in domain specific development. So narrowing a feature set that a language provides and limiting it to a certain domain vocabulary is what helps people solve tasks of that specific domain much easier and in the most expressive natural way. Without having to learn tons and tons of details that a general purpose language (GPL, hah :) ) provides (btw, the reason to write thick books). I agree with Stan, if you begin to use a technology you’ll have to learn something anyway, be it TaskFlow API and principles or DSL. Well-designed DSL just encapsulates essential principles of a system it is used for. By learning DSL you’re leaning the system itself, as simple as that. Renat Akhmerov @ Mirantis Inc. On 10 Mar 2014, at 05:35, Stan Lagun sla...@mirantis.com wrote: I'd be very interested in knowing the resource controls u plan to add. Memory, CPU... We haven't discussed it yet. Any suggestions are welcomed I'm still trying to figure out where something like https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.demoApp.DemoInstance/manifest.yaml would be beneficial, why not just spend effort sand boxing lua, python... Instead of spending effort on creating a new language and then having to sandbox it as well... Especially if u picked languages that are made to be sandboxed from the start (not python)... 1. See my detailed answer in Mistral thread why haven't we used any of those languages. There are many reasons besides sandboxing. 2. You don't need to sandbox MuranoPL. Sandboxing is restricting some operations. In MuranoPL ALL operations (including operators in expressions, functions, methods etc.) are just those that you explicitly provided. So there is nothing to restrict. There are no builtins that throw AccessViolationError 3. Most of the value of MuranoPL comes not form the workflow code but from class declarations. In all OOP languages classes are just a convenient to organize your code. There are classes that represent real-life objects and classes that are nothing more than data-structures, DTOs etc. In Murano classes in MuranoPL are deployable entities like Heat resources application components, services etc. In dashboard UI user works with those entities. He (in UI!) creates instances of those classes, fills their property values, binds objects together (assigns on object to a property of another). And this is done without even a single MuranoPL line being executed! That is possible because everything in MuranoPL is a subject to declaration and because it is just plain YAML anyone can easily extract those declarations from MuranoPL classes. Now suppose it was Python instead of MuranoPL. Then you would have to parse *.py files to get list of declared classes (without executing anything). Suppose that you managed to solve this somehow. Probably you wrote regexp that finds all class declarations it text files. Are you done? No! There are no properties (attributes) declarations in Python. You cannot infer all possible class attributes just from class declaration. You can try to parse __init__ method to find attribute initialization code but then you find that you cannot infer property types. You would not know what values does class expect in those attributes, what are constraints etc. Besides there are plenty of ways to fool such naive algorithms. Classes can be created using type() built-in function without declarations at all. Attributes may be initialized anywhere. Everything can be made in runtime. As you can see Python is not good for the task. Ans all this true for all of the languages you mentioned. The reason is fundamental - all of those languages are dynamic. So maybe you can take some statically-typed language like C++ or Java to do the job? Again, no! There are other problems with those type of languages. Have you ever seen statically-typed embeddable language? Those language require compilation and linker, binary distribution of classes, handling of versioning problems, dependencies on 3-rd party libraries etc. This is the hell you don't want to step in. But even in statically typed languages there are no contracts. You know the list of properties and their types. But knowing that class foo has property bar of type int doesn't give
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
On 10 Mar 2014, at 16:53, Stan Lagun sla...@mirantis.com wrote: On Mon, Mar 10, 2014 at 12:26 PM, Renat Akhmerov rakhme...@mirantis.com wrote: In case of Amazon SWF it works in the opposite way. First of all it’s a language agnostic web service, then they have language specific frameworks working on top of the service. The big question here is would SWF be popular (or at least usable in real-world scenarios) without language-specific framework on top? Legitimate question. I would say “no”. The thing is that SWF web service was not designed for direct usage in the first place. We believe it’s possible to do for a number of use cases although doesn’t cancel the idea of having language bindings. Life will tell. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Error on running tox
Hm.. Interesting. CI wasn’t able to reveal this for some reason. My first guess is that there’s a race condition somewhere. Did you try to debug it? And is this error 100% repeatable? Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 11:18, Manas Kelshikar ma...@stackstorm.com wrote: I see this error when I run tox. I pulled down a latest copy of master and tried to setup the environment. Any ideas? See http://paste.openstack.org/show/73213/ for details. Any help is appreciated. Thanks, Manas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Local vs. Scalable Engine
On 12 Mar 2014, at 06:37, W Chan m4d.co...@gmail.com wrote: Here're the proposed changes. 1) Rewrite the launch script to be more generic which contains option to launch all components (i.e. API, engine, executor) on the same process but over separate threads or launch each individually. You mentioned test_executor.py so I think it would make sense first to refactor the code in there related with acquiring transport and launching executor. My suggestions are: In test base class (mistral.tests.base.BaseTest) create the new method start_local_executor() that would deal with getting a fake driver inside and all that stuff. This would be enough for tests where we need to run engine and check something. start_local_executor() can be just a part of setUp() method for such tests. As for the launch script I have the following thoughts: Long-term launch scripts should be different for all API, engine and executor. Now API and engine start within the same process but it’s just a temporary solution. Launch script for engine (which is the same as API’s for now) should have an option --use-local-executor to be able to run an executor along with engine itself within the same process. 2) Move transport to a global variables, similar to global _engine and then shared by the different component. Not sure why we need it. Can you please explain more detailed here? The better way would be to initialize engine and executor with transport when we create them. If our current structure doesn’t allow this easily we should discuss it and change it. In mistral.engine.engine.py we now have: def load_engine(): global _engine module_name = cfg.CONF.engine.engine module = importutils.import_module(module_name) _engine = module.get_engine() As an option we could have the code that loads engine in engine launch script (once we decouple it from API process) so that when we call get_engine() we could pass in all needed configuration parameters like transport. 3) Modified the engine and the executor to use a factory method to get the global transport If we made a decision on #2 we won’t need it. A side note: when we discuss things like that I really miss DI container :) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Error on running tox
Ok. I might be related with oslo.messaging change that we merged in yesterday but I don’t see at this point how exactly. Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 12:38, Manas Kelshikar ma...@stackstorm.com wrote: Yes it is 100% reproducible. Was hoping it was environmental i.e. missing some dependency etc. but since that does not seem to be the case I shall debug locally and report back. Thanks! On Tue, Mar 11, 2014 at 9:54 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Hm.. Interesting. CI wasn’t able to reveal this for some reason. My first guess is that there’s a race condition somewhere. Did you try to debug it? And is this error 100% repeatable? Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 11:18, Manas Kelshikar ma...@stackstorm.com wrote: I see this error when I run tox. I pulled down a latest copy of master and tried to setup the environment. Any ideas? See http://paste.openstack.org/show/73213/ for details. Any help is appreciated. Thanks, Manas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Mistral] Actions design BP
Team, I started summarizing all the thoughts and ideas that we’ve been discussing for a while regarding actions. The main driver for this work is that the system keeps evolving and we still don’t have a comprehensive understanding of that part. Additionally, we keep getting a lot of requests and questions from our potential users which are related to actions (‘will they be extensible?’, ‘will they have dry-run feature?’, ‘what are the ways to configure and group them?’ and so on and so forth). So although we’re still in a Pilot phase we need to start this work in parallel. Even now lack of solid understanding of it creates a lot of problems in pilot development. I created a BP at launchpad [0] which has a reference to detailed specification [1]. It’s still in progress but you could already leave your early feedback so that I don’t go in a wrong direction too far. The highest priority now is still finishing the pilot so we shouldn’t start implementing everything described in BP right now. However, some of the things have to be adjusted asap (like Action interface and the main implementation principles). [0]: https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design [1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Error on running tox
I would just try to recreate virtual environments. We haven’t been able to reproduce this problem so far. Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 16:32, Nikolay Makhotkin nmakhot...@mirantis.com wrote: maybe something wrong with python2.6? .tox/py26/lib/python2.6/site-packages/mock.py, line 1201, in patched what if you try it on py27? On Wed, Mar 12, 2014 at 10:08 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Ok. I might be related with oslo.messaging change that we merged in yesterday but I don’t see at this point how exactly. Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 12:38, Manas Kelshikar ma...@stackstorm.com wrote: Yes it is 100% reproducible. Was hoping it was environmental i.e. missing some dependency etc. but since that does not seem to be the case I shall debug locally and report back. Thanks! On Tue, Mar 11, 2014 at 9:54 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Hm.. Interesting. CI wasn’t able to reveal this for some reason. My first guess is that there’s a race condition somewhere. Did you try to debug it? And is this error 100% repeatable? Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 11:18, Manas Kelshikar ma...@stackstorm.com wrote: I see this error when I run tox. I pulled down a latest copy of master and tried to setup the environment. Any ideas? See http://paste.openstack.org/show/73213/ for details. Any help is appreciated. Thanks, Manas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Nikolay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Error on running tox
Ok, awesome. Btw, when I was writing tests for Data Flow I didn’t make assumptions about order of tasks. You can take a look at how it’s achieved. Renat Akhmerov @ Mirantis Inc. On 13 Mar 2014, at 02:04, Manas Kelshikar ma...@stackstorm.com wrote: Works ok if I directly run nosetests from the virtual environment or even in the IDE. I see this error only when I run tox. In anycase after looking at the specific testcases I can tell that the test makes some ordering assumptions which may not be required. I will send out a patch soon and we can move the conversation to the review. Thanks! On Wed, Mar 12, 2014 at 11:00 AM, Manas Kelshikar ma...@stackstorm.com wrote: I pasted only for python 2.6 but exact same errors with 2.7. Also, I posted this question after I nuked my entire dev folder so this was being run on a new environment. /Manas On Wed, Mar 12, 2014 at 4:44 AM, Renat Akhmerov rakhme...@mirantis.com wrote: I would just try to recreate virtual environments. We haven’t been able to reproduce this problem so far. Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 16:32, Nikolay Makhotkin nmakhot...@mirantis.com wrote: maybe something wrong with python2.6? .tox/py26/lib/python2.6/site-packages/mock.py, line 1201, in patched what if you try it on py27? On Wed, Mar 12, 2014 at 10:08 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Ok. I might be related with oslo.messaging change that we merged in yesterday but I don’t see at this point how exactly. Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 12:38, Manas Kelshikar ma...@stackstorm.com wrote: Yes it is 100% reproducible. Was hoping it was environmental i.e. missing some dependency etc. but since that does not seem to be the case I shall debug locally and report back. Thanks! On Tue, Mar 11, 2014 at 9:54 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Hm.. Interesting. CI wasn’t able to reveal this for some reason. My first guess is that there’s a race condition somewhere. Did you try to debug it? And is this error 100% repeatable? Renat Akhmerov @ Mirantis Inc. On 12 Mar 2014, at 11:18, Manas Kelshikar ma...@stackstorm.com wrote: I see this error when I run tox. I pulled down a latest copy of master and tried to setup the environment. Any ideas? See http://paste.openstack.org/show/73213/ for details. Any help is appreciated. Thanks, Manas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best Regards, Nikolay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Local vs. Scalable Engine
On 13 Mar 2014, at 10:40, W Chan m4d.co...@gmail.com wrote: I can write a method in base test to start local executor. I will do that as a separate bp. Ok. After the engine is made standalone, the API will communicate to the engine and the engine to the executor via the oslo.messaging transport. This means that for the local option, we need to start all three components (API, engine, and executor) on the same process. If the long term goal as you stated above is to use separate launchers for these components, this means that the API launcher needs to duplicate all the logic to launch the engine and the executor. Hence, my proposal here is to move the logic to launch the components into a common module and either have a single generic launch script that launch specific components based on the CLI options or have separate launch scripts that reference the appropriate launch function from the common module. Ok, I see your point. Then I would suggest we have one script which we could use to run all the components (any subset of of them). So for those components we specified when launching the script we use this local transport. Btw, scheduler eventually should become a standalone component too, so we have 4 components. The RPC client/server in oslo.messaging do not determine the transport. The transport is determine via oslo.config and then given explicitly to the RPC client/server. https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/engine.py#L31 and https://github.com/stackforge/mistral/blob/master/mistral/cmd/task_executor.py#L63 are examples for the client and server respectively. The in process Queue is instantiated within this transport object from the fake driver. For the local option, all three components need to share the same transport in order to have the Queue in scope. Thus, we will need some method to have this transport object visible to all three components and hence my proposal to use a global variable and a factory method. I’m still not sure I follow your point here.. Looking at the links you provided I see this: transport = messaging.get_transport(cfg.CONF) So my point here is we can make this call once in the launching script and pass it to engine/executor (and now API too if we want it to be launched by the same script). Of course, we’ll have to change the way how we initialize these components, but I believe we can do it. So it’s just a dependency injection. And in this case we wouldn’t need to use a global variable. Am I still missing something? Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
So no need to ask the same…” Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Actions design BP
Joshua, Thanks for your interest and feedback. I believe you were able to deliver your message already, we definitely hear you. So no need to the same stuff again and again ;) As I promised before, we are now evaluating what’s been going on with TaskFlow for the last couple of months and preparing our questions/concerns/suggestions on using it within Mistral. But that’s not very easy and quick thing to do since there’s a bunch of details to take into account, especially given the fact that Mistral codebase has become much more solid and Mistral itself now has lots of requirements dictated by its use cases and roadmap vision. So patience would be really appreciated here. If you don’t mind I would prefer to discuss things like that in separate threads, not in threads devoted to daily project activities. So that we can split our conceptual discussions and current work that’s going on according to our plans. Otherwise we have a risk to make spaghetti out of our ML threads. Thanks Renat Akhmerov @ Mirantis Inc. On 13 Mar 2014, at 08:54, Joshua Harlow harlo...@yahoo-inc.com wrote: So taskflow has tasks, which seems comparable to actions? I guess I should get tired of asking but why recreate the same stuff ;) The questions listed: - Does action need to have revert() method along with run() method? - How does action expose errors occurring during it's work? - In what form does action return a result? And more @ https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign And quite a few others that haven't been mentioned (how does a action retry? How does a action report partial progress? What's the intertask/state persistence mechanism?) have been worked on by the taskflow team for a while now... https://github.com/openstack/taskflow/blob/master/taskflow/task.py#L31 (and others...) Anyways, I know mistral is still POC/pilot/prototype... but seems like more duplicate worked that could just be avoided ;) -Josh -Original Message- From: Renat Akhmerov rakhme...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 11, 2014 at 11:32 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: [openstack-dev] [Mistral] Actions design BP Team, I started summarizing all the thoughts and ideas that we¹ve been discussing for a while regarding actions. The main driver for this work is that the system keeps evolving and we still don¹t have a comprehensive understanding of that part. Additionally, we keep getting a lot of requests and questions from our potential users which are related to actions (Œwill they be extensible?¹, Œwill they have dry-run feature?¹, Œwhat are the ways to configure and group them?¹ and so on and so forth). So although we¹re still in a Pilot phase we need to start this work in parallel. Even now lack of solid understanding of it creates a lot of problems in pilot development. I created a BP at launchpad [0] which has a reference to detailed specification [1]. It¹s still in progress but you could already leave your early feedback so that I don¹t go in a wrong direction too far. The highest priority now is still finishing the pilot so we shouldn¹t start implementing everything described in BP right now. However, some of the things have to be adjusted asap (like Action interface and the main implementation principles). [0]: https://blueprints.launchpad.net/mistral/+spec/mistral-actions-design [1]: https://wiki.openstack.org/wiki/Mistral/Blueprints/ActionsDesign Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][Taskflow][all] Mistral + taskflow
Folks, Mistral and TaskFlow are significantly different technologies. With different set of capabilities, with different target audience. We may not be doing enough to clarify all the differences, I admit that. The challenge here is that people tend to judge having minimal amount of information about both things. As always, devil in the details. Stan is 100% right, “seems” is not an appropriate word here. Java seems to be similar to C++ at the first glance for those who have little or no knowledge about them. To be more consistent I won’t be providing all the general considerations that I’ve been using so far (in etherpads, MLs, in personal discussions), it doesn’t seem to be working well, at least not with everyone. So to make it better, like I said in that different thread: we’re evaluating TaskFlow now and will share the results. Basically, it’s what Boris said about what could and could not be implemented in TaskFlow. But since the very beginning of the project I never abandoned the idea of using TaskFlow some day when it’s possible. So, again: Joshua, we hear you, we’re working in that direction. I'm reminded of http://www.slideshare.net/RenatAkhmerov/mistral-hong-kong-unconference-trac k/2 where it seemed like we were doing much better collaboration, what has happened to break this continuity? Not sure why you think something is broken. We just want to finish the pilot with all the ‘must’ things working in it. This is a plan. Then we can revisit and change absolutely everything. Remember, to the great extent this is research. Joshua, this is what we talked about and agreed on many times. I know you might be anxious about that given the fact it’s taking more time than planned but our vision of the project has drastically evolved and gone far far beyond the initial Convection proposal. So the initial idea of POC is no longer relevant. Even though we finished the first version in December, we realized it wasn’t something that should have been shared with the community since it lacked some essential things. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Local vs. Scalable Engine
Take a look at method get_pecan_config() in mistral/api/app.py. It’s where you can pass any parameters into pecan app (see a dictionary ‘cfg_dict’ initialization). They can be then accessed via pecan.conf as described here: http://pecan.readthedocs.org/en/latest/configuration.html#application-configuration. If I understood the problem correctly this should be helpful. Renat Akhmerov @ Mirantis Inc. On 14 Mar 2014, at 05:14, Dmitri Zimine d...@stackstorm.com wrote: We have access to all configuration parameters in the context of api.py. May be you don't pass it but just instantiate it where you need it? Or I may misunderstand what you're trying to do... DZ PS: can you generate and update mistral.config.example to include new oslo messaging options? I forgot to mention it on review on time. On Mar 13, 2014, at 11:15 AM, W Chan m4d.co...@gmail.com wrote: On the transport variable, the problem I see isn't with passing the variable to the engine and executor. It's passing the transport into the API layer. The API layer is a pecan app and I currently don't see a way where the transport variable can be passed to it directly. I'm looking at https://github.com/stackforge/mistral/blob/master/mistral/cmd/api.py#L50 and https://github.com/stackforge/mistral/blob/master/mistral/api/app.py#L44. Do you have any suggestion? Thanks. On Thu, Mar 13, 2014 at 1:30 AM, Renat Akhmerov rakhme...@mirantis.com wrote: On 13 Mar 2014, at 10:40, W Chan m4d.co...@gmail.com wrote: I can write a method in base test to start local executor. I will do that as a separate bp. Ok. After the engine is made standalone, the API will communicate to the engine and the engine to the executor via the oslo.messaging transport. This means that for the local option, we need to start all three components (API, engine, and executor) on the same process. If the long term goal as you stated above is to use separate launchers for these components, this means that the API launcher needs to duplicate all the logic to launch the engine and the executor. Hence, my proposal here is to move the logic to launch the components into a common module and either have a single generic launch script that launch specific components based on the CLI options or have separate launch scripts that reference the appropriate launch function from the common module. Ok, I see your point. Then I would suggest we have one script which we could use to run all the components (any subset of of them). So for those components we specified when launching the script we use this local transport. Btw, scheduler eventually should become a standalone component too, so we have 4 components. The RPC client/server in oslo.messaging do not determine the transport. The transport is determine via oslo.config and then given explicitly to the RPC client/server. https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/engine.py#L31 and https://github.com/stackforge/mistral/blob/master/mistral/cmd/task_executor.py#L63 are examples for the client and server respectively. The in process Queue is instantiated within this transport object from the fake driver. For the local option, all three components need to share the same transport in order to have the Queue in scope. Thus, we will need some method to have this transport object visible to all three components and hence my proposal to use a global variable and a factory method. I’m still not sure I follow your point here.. Looking at the links you provided I see this: transport = messaging.get_transport(cfg.CONF) So my point here is we can make this call once in the launching script and pass it to engine/executor (and now API too if we want it to be launched by the same script). Of course, we’ll have to change the way how we initialize these components, but I believe we can do it. So it’s just a dependency injection. And in this case we wouldn’t need to use a global variable. Am I still missing something? Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
