Re: [openstack-dev] [Mistral] Crack at a Real life workflow
On Mon, Mar 10, 2014 at 12:26 PM, Renat Akhmerov rakhme...@mirantis.comwrote: In case of Amazon SWF it works in the opposite way. First of all it's a language agnostic web service, then they have language specific frameworks working on top of the service. The big question here is would SWF be popular (or at least usable in real-world scenarios) without language-specific framework on top? -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
On 10 Mar 2014, at 16:53, Stan Lagun sla...@mirantis.com wrote: On Mon, Mar 10, 2014 at 12:26 PM, Renat Akhmerov rakhme...@mirantis.com wrote: In case of Amazon SWF it works in the opposite way. First of all it’s a language agnostic web service, then they have language specific frameworks working on top of the service. The big question here is would SWF be popular (or at least usable in real-world scenarios) without language-specific framework on top? Legitimate question. I would say “no”. The thing is that SWF web service was not designed for direct usage in the first place. We believe it’s possible to do for a number of use cases although doesn’t cancel the idea of having language bindings. Life will tell. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
deserialized version of it. Thus you can control where those YAMLs are located. You can store them in Glance, database, file-system or generate them on the fly without DSL even notice. 9. With YAML you has less to explain (people already know it) and more of a tooling and IDE support Q: Isn't DSL of that power would be a subject to DoS attacks, resource starvation attacks etc.? A: This is a really good question cause nobody asked it yet :) And the answer is NO. Here is why: With MuranoPL you have complete control over everything. You can have time quota for DSL code (remember 30 second quota for PHP scrips in typical web setup?). Or you can limit DSL scripts to some reasonable number of executed instructions. Or count raw CPU time for the script. You can make all loop constructs to have upper limit on iterations. You can do the same with YAQL functions. You can enforce timeouts on all I/O operations. MuranoPL uses green threads and you can have a lot of them (unlike regular threads) and it is possible to limit total number of green threads used for DSL execution. It is possible to even limit memory usage. DSL code cannot allocate operating-system-level resources like file handles of TCP sockets. And there is garbage collector as DSL in interpreted on top of Python. How easy it would be to do the same with JavaScript/Lua/whatever? On Sat, Mar 8, 2014 at 7:05 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: What are the benefits of MuranoPL over a already established language? I noted the following and don't quite understand why they are needed (reinventing a language): - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Extends - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Block_constructs Q: where are those looping constructs executed? How hard is it to DOS murano servers (submitting jobs that loop forever). What execution limits are imposed? I noted that the parallel construct actually exposes the number of green threads (isn't this an avenue for resource starvation?). - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Object_model IMHO, something just doesn't seem right when the above is created, fitting a language into YAML seems about as awkward as creating a language in XML (xquery[1] for example) . Why was this really preferred over just python or something simpler for example, [lua, javascript...], that already has language/object constructs... built-in and have runtimes that u can control the security domain of (python is not a good choice to run arbitrary code-in, find out how much energy google put into python + app-engine and u'll see what it takes). http://en.wikipedia.org/wiki/XQuery#Examples From: Stan Lagun sla...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Friday, March 7, 2014 at 9:36 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Crack at a Real life workflow Hello everyone! Actually it is possible to construct YAML-based DSL that has all the constructs of regular OOP language like Python and at the same time be safe enough to be used for execution of untrusted code on shared server. Take a look at Murano DSL. For example the code above defines class Instance: https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.services.Instance/manifest.yaml The part that may be useful for Mistral is under Workflow key. Here is some doc on the language: https://wiki.openstack.org/wiki/Murano/DSL/Blueprint Technically you can code any workflow that you can in Python using such language (just don't look at all OOP-related stuff) and it will look very similar to Python but be safe as you can only call APIs that are explicitly provided for DSL Hope this might be helpful for Mistral On Fri, Mar 7, 2014 at 10:38 AM, Dmitri Zimine d...@stackstorm.com wrote: I just moved the sample to Git; let's leverage git review for specific comments on the syntax. https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.com wrote: Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Friday, March 7, 2014 at 9:36 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Crack at a Real life workflow Hello everyone! Actually it is possible to construct YAML-based DSL that has all the constructs of regular OOP language like Python and at the same time be safe enough to be used for execution of untrusted code on shared server. Take a look at Murano DSL. For example the code above defines class Instance: https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.services.Instance/manifest.yaml The part that may be useful for Mistral is under Workflow key. Here is some doc on the language: https://wiki.openstack.org/wiki/Murano/DSL/Blueprint Technically you can code any workflow that you can in Python using such language (just don't look at all OOP-related stuff) and it will look very similar to Python but be safe as you can only call APIs that are explicitly provided for DSL Hope this might be helpful for Mistral On Fri, Mar 7, 2014 at 10:38 AM, Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com wrote: I just moved the sample to Git; let's leverage git review for specific comments on the syntax. https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com wrote: Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. There must be some misunderstanding. Mistral _does follow AWS / BPEL engines approach, it is both doing I'm done this task, what should I do next? (executor) and callback service (engine that coordinates the flow and keeps the state). Like decider and activity workers in AWS Simple Workflow. Engine maintains the state. Executors run tasks. Object model describes workflow as a graph of tasks with transitions, conditions, etc. YAML is one way to define a workflow. Nothing controversial :) @all: Wether one writes Python code or uses yaml? Depends on the user. There are good arguments for YAML. But if it's crappy, it looses. We want to see how it feels to write it. To me, mixed feelings so far, but promising. What do you guys think? Comments welcome here: https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:41 AM, Sandy Walsh sandy.wa...@rackspace.commailto:sandy.wa...@rackspace.com wrote: On 03/06/2014 02:16 PM, Renat Akhmerov wrote: IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Yeah, the REST aspect is where the tricky part comes in :) Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. Likewise with things like sending emails from the backend. Minstral should just call a webhook and let the receiver deal with active states as they choose. Which is why modelling this stuff in code is usually always better and why I'd lean towards
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
of a tooling and IDE support Q: Isn't DSL of that power would be a subject to DoS attacks, resource starvation attacks etc.? A: This is a really good question cause nobody asked it yet :) And the answer is NO. Here is why: With MuranoPL you have complete control over everything. You can have time quota for DSL code (remember 30 second quota for PHP scrips in typical web setup?). Or you can limit DSL scripts to some reasonable number of executed instructions. Or count raw CPU time for the script. You can make all loop constructs to have upper limit on iterations. You can do the same with YAQL functions. You can enforce timeouts on all I/O operations. MuranoPL uses green threads and you can have a lot of them (unlike regular threads) and it is possible to limit total number of green threads used for DSL execution. It is possible to even limit memory usage. DSL code cannot allocate operating-system-level resources like file handles of TCP sockets. And there is garbage collector as DSL in interpreted on top of Python. How easy it would be to do the same with JavaScript/Lua/whatever? On Sat, Mar 8, 2014 at 7:05 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: What are the benefits of MuranoPL over a already established language? I noted the following and don't quite understand why they are needed (reinventing a language): - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Extends - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Block_constructs Q: where are those looping constructs executed? How hard is it to DOS murano servers (submitting jobs that loop forever). What execution limits are imposed? I noted that the parallel construct actually exposes the number of green threads (isn't this an avenue for resource starvation?). - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Object_model IMHO, something just doesn't seem right when the above is created, fitting a language into YAML seems about as awkward as creating a language in XML (xquery[1] for example) . Why was this really preferred over just python or something simpler for example, [lua, javascript...], that already has language/object constructs... built-in and have runtimes that u can control the security domain of (python is not a good choice to run arbitrary code-in, find out how much energy google put into python + app-engine and u'll see what it takes). http://en.wikipedia.org/wiki/XQuery#Examples From: Stan Lagun sla...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Friday, March 7, 2014 at 9:36 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Crack at a Real life workflow Hello everyone! Actually it is possible to construct YAML-based DSL that has all the constructs of regular OOP language like Python and at the same time be safe enough to be used for execution of untrusted code on shared server. Take a look at Murano DSL. For example the code above defines class Instance: https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.services.Instance/manifest.yaml The part that may be useful for Mistral is under Workflow key. Here is some doc on the language: https://wiki.openstack.org/wiki/Murano/DSL/Blueprint Technically you can code any workflow that you can in Python using such language (just don't look at all OOP-related stuff) and it will look very similar to Python but be safe as you can only call APIs that are explicitly provided for DSL Hope this might be helpful for Mistral On Fri, Mar 7, 2014 at 10:38 AM, Dmitri Zimine d...@stackstorm.com wrote: I just moved the sample to Git; let's leverage git review for specific comments on the syntax. https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.com wrote: Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
@ Dmitri ;) Sure lets catch up, most always I'm on IRC, feel free to ping me. Or jump into #openstack-state-management (for more good times). From: Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, March 6, 2014 at 10:36 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Crack at a Real life workflow Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. There must be some misunderstanding. Mistral _does follow AWS / BPEL engines approach, it is both doing I'm done this task, what should I do next? (executor) and callback service (engine that coordinates the flow and keeps the state). Like decider and activity workers in AWS Simple Workflow. Engine maintains the state. Executors run tasks. Object model describes workflow as a graph of tasks with transitions, conditions, etc. YAML is one way to define a workflow. Nothing controversial :) @all: Wether one writes Python code or uses yaml? Depends on the user. There are good arguments for YAML. But if it's crappy, it looses. We want to see how it feels to write it. To me, mixed feelings so far, but promising. What do you guys think? Comments welcome here: https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:41 AM, Sandy Walsh sandy.wa...@rackspace.commailto:sandy.wa...@rackspace.com wrote: On 03/06/2014 02:16 PM, Renat Akhmerov wrote: IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Yeah, the REST aspect is where the tricky part comes in :) Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. Likewise with things like sending emails from the backend. Minstral should just call a webhook and let the receiver deal with active states as they choose. Which is why modelling this stuff in code is usually always better and why I'd lean towards the TaskFlow approach to the problem. They're tackling this from a library perspective first and then (possibly) turning it into a service. Just seems like a better fit. It's also the approach taken by Amazon Simple Workflow and many BPEL engines. -S Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.commailto:harlo...@yahoo-inc.com wrote: That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.commailto:sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
What are the benefits of MuranoPL over a already established language? I noted the following and don't quite understand why they are needed (reinventing a language): - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Extends - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Block_constructs Q: where are those looping constructs executed? How hard is it to DOS murano servers (submitting jobs that loop forever). What execution limits are imposed? I noted that the parallel construct actually exposes the number of green threads (isn't this an avenue for resource starvation?). - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Object_model IMHO, something just doesn't seem right when the above is created, fitting a language into YAML seems about as awkward as creating a language in XML (xquery[1] for example) . Why was this really preferred over just python or something simpler for example, [lua, javascript…], that already has language/object constructs… built-in and have runtimes that u can control the security domain of (python is not a good choice to run arbitrary code-in, find out how much energy google put into python + app-engine and u'll see what it takes). http://en.wikipedia.org/wiki/XQuery#Examples From: Stan Lagun sla...@mirantis.commailto:sla...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Friday, March 7, 2014 at 9:36 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Crack at a Real life workflow Hello everyone! Actually it is possible to construct YAML-based DSL that has all the constructs of regular OOP language like Python and at the same time be safe enough to be used for execution of untrusted code on shared server. Take a look at Murano DSL. For example the code above defines class Instance: https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.services.Instance/manifest.yaml The part that may be useful for Mistral is under Workflow key. Here is some doc on the language: https://wiki.openstack.org/wiki/Murano/DSL/Blueprint Technically you can code any workflow that you can in Python using such language (just don't look at all OOP-related stuff) and it will look very similar to Python but be safe as you can only call APIs that are explicitly provided for DSL Hope this might be helpful for Mistral On Fri, Mar 7, 2014 at 10:38 AM, Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com wrote: I just moved the sample to Git; let's leverage git review for specific comments on the syntax. https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.commailto:d...@stackstorm.com wrote: Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. There must be some misunderstanding. Mistral _does follow AWS / BPEL engines approach, it is both doing I'm done this task, what should I do next? (executor) and callback service (engine that coordinates the flow and keeps the state). Like decider and activity workers in AWS Simple Workflow. Engine maintains the state. Executors run tasks. Object model describes workflow as a graph of tasks with transitions, conditions, etc. YAML is one way to define a workflow. Nothing controversial :) @all: Wether one writes Python code or uses yaml? Depends on the user. There are good arguments for YAML. But if it's crappy, it looses. We want to see how it feels to write it. To me, mixed feelings so far, but promising. What do you guys think? Comments welcome here: https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:41 AM, Sandy Walsh sandy.wa...@rackspace.commailto:sandy.wa...@rackspace.com
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to see a killer object model for defining these workflows as a first step. What would a python-based equivalent of that real-world workflow look like? Then we can ask ourselves, does the DSL make this better or worse? Would we need to expose things like email handlers, or leave that to the general python libraries? $0.02 -S On 03/05/2014 10:50 PM, Dmitri Zimine wrote: Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to see a killer object model for defining these workflows as a first step. What would a python-based equivalent of that real-world workflow look like? Then we can ask ourselves, does the DSL make this better or worse? Would we need to expose things like email handlers, or leave that to the general python libraries? $0.02 -S On 03/05/2014 10:50 PM, Dmitri Zimine wrote: Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.com wrote: That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to see a killer object model for defining these workflows as a first step. What would a python-based equivalent of that real-world workflow look like? Then we can ask ourselves, does the DSL make this better or worse? Would we need to expose things like email handlers, or leave that to the general python libraries? $0.02 -S On 03/05/2014 10:50 PM, Dmitri Zimine wrote: Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
On 03/06/2014 02:16 PM, Renat Akhmerov wrote: IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Yeah, the REST aspect is where the tricky part comes in :) Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. Likewise with things like sending emails from the backend. Minstral should just call a webhook and let the receiver deal with active states as they choose. Which is why modelling this stuff in code is usually always better and why I'd lean towards the TaskFlow approach to the problem. They're tackling this from a library perspective first and then (possibly) turning it into a service. Just seems like a better fit. It's also the approach taken by Amazon Simple Workflow and many BPEL engines. -S Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.com wrote: That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to see a killer object model for defining these workflows as a first step. What would a python-based equivalent of that real-world workflow look like? Then we can ask ourselves, does the DSL make this better or worse? Would we need to expose things like email handlers, or leave that to the general python libraries? $0.02 -S On 03/05/2014 10:50 PM, Dmitri Zimine wrote: Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. There must be some misunderstanding. Mistral _does follow AWS / BPEL engines approach, it is both doing I'm done this task, what should I do next? (executor) and callback service (engine that coordinates the flow and keeps the state). Like decider and activity workers in AWS Simple Workflow. Engine maintains the state. Executors run tasks. Object model describes workflow as a graph of tasks with transitions, conditions, etc. YAML is one way to define a workflow. Nothing controversial :) @all: Wether one writes Python code or uses yaml? Depends on the user. There are good arguments for YAML. But if it's crappy, it looses. We want to see how it feels to write it. To me, mixed feelings so far, but promising. What do you guys think? Comments welcome here: https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:41 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: On 03/06/2014 02:16 PM, Renat Akhmerov wrote: IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Yeah, the REST aspect is where the tricky part comes in :) Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. Likewise with things like sending emails from the backend. Minstral should just call a webhook and let the receiver deal with active states as they choose. Which is why modelling this stuff in code is usually always better and why I'd lean towards the TaskFlow approach to the problem. They're tackling this from a library perspective first and then (possibly) turning it into a service. Just seems like a better fit. It's also the approach taken by Amazon Simple Workflow and many BPEL engines. -S Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.com wrote: That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to see a killer object model for defining these workflows as a first step. What would a python-based equivalent of that real-world workflow look like? Then we can ask ourselves, does the DSL make this better or worse? Would we need to expose things like email handlers, or leave that to the general python libraries?
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
I just moved the sample to Git; let's leverage git review for specific comments on the syntax. https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.com wrote: Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. There must be some misunderstanding. Mistral _does follow AWS / BPEL engines approach, it is both doing I'm done this task, what should I do next? (executor) and callback service (engine that coordinates the flow and keeps the state). Like decider and activity workers in AWS Simple Workflow. Engine maintains the state. Executors run tasks. Object model describes workflow as a graph of tasks with transitions, conditions, etc. YAML is one way to define a workflow. Nothing controversial :) @all: Wether one writes Python code or uses yaml? Depends on the user. There are good arguments for YAML. But if it's crappy, it looses. We want to see how it feels to write it. To me, mixed feelings so far, but promising. What do you guys think? Comments welcome here: https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:41 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: On 03/06/2014 02:16 PM, Renat Akhmerov wrote: IMO, it looks not bad (sorry, I’m biased too) even now. Keep in mind this is not the final version, we keep making it more expressive and concise. As for killer object model it’s not 100% clear what you mean. As always, devil in the details. This is a web service with all the consequences. I assume what you call “object model” here is nothing else but a python binding for the web service which we’re also working on. Custom python logic you mentioned will also be possible to easily integrate. Like I said, it’s still a pilot stage of the project. Yeah, the REST aspect is where the tricky part comes in :) Basically, in order to make a grammar expressive enough to work across a web interface, we essentially end up writing a crappy language. Instead, we should focus on the callback hooks to something higher level to deal with these issues. Minstral should just say I'm done this task, what should I do next? and the callback service can make decisions on where in the graph to go next. Likewise with things like sending emails from the backend. Minstral should just call a webhook and let the receiver deal with active states as they choose. Which is why modelling this stuff in code is usually always better and why I'd lean towards the TaskFlow approach to the problem. They're tackling this from a library perspective first and then (possibly) turning it into a service. Just seems like a better fit. It's also the approach taken by Amazon Simple Workflow and many BPEL engines. -S Renat Akhmerov @ Mirantis Inc. On 06 Mar 2014, at 22:26, Joshua Harlow harlo...@yahoo-inc.com wrote: That sounds a little similar to what taskflow is trying to do (I am of course biased). I agree with letting the native language implement the basics (expressions, assignment...) and then building the domain ontop of that. Just seems more natural IMHO, and is similar to what linq (in c#) has done. My 3 cents. Sent from my really tiny device... On Mar 6, 2014, at 5:33 AM, Sandy Walsh sandy.wa...@rackspace.com wrote: DSL's are tricky beasts. On one hand I like giving a tool to non-developers so they can do their jobs, but I always cringe when the DSL reinvents the wheel for basic stuff (compound assignment expressions, conditionals, etc). YAML isn't really a DSL per se, in the sense that it has no language constructs. As compared to a Ruby-based DSL (for example) where you still have Ruby under the hood for the basic stuff and extensions to the language for the domain-specific stuff. Honestly, I'd like to
[openstack-dev] [Mistral] Crack at a Real life workflow
Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
On Mar 5, 2014 6:58 PM, Dmitri Zimine d...@stackstorm.com wrote: Folks, I took a crack at using our DSL to build a real-world workflow. Just to see how it feels to write it. And how it compares with alternative tools. This one automates a page from OpenStack operation guide: http://docs.openstack.org/trunk/openstack-ops/content/maintenance.html#planned_maintenance_compute_node Here it is https://gist.github.com/dzimine/9380941 or here http://paste.openstack.org/show/72741/ I have a bunch of comments, implicit assumptions, and questions which came to mind while writing it. Want your and other people's opinions on it. But gist and paste don't let annotate lines!!! :( May be we can put it on the review board, even with no intention to check in, to use for discussion? How is this different then chef/puppet/ansible ...? Forgive me if that has already been answered, I didn't see an answer to that under the FAQ at https://wiki.openstack.org/wiki/Mistral Any interest? DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
