Thanks Winson, Since we discussed all this already I just want to confirm that I fully support this model, it will significantly help us make much more concise, readable and maintainable workflows. I spent a lot of time thinking about it and don’t see any problems with it. Nice job!
However, all additional comments and questions are more than welcomed! Renat Akhmerov @ Mirantis Inc. > On 24 Dec 2014, at 04:32, W Chan <[email protected]> wrote: > > After some online discussions with Renat, the following is a revision of the > proposal to address the following related blueprints. > * > https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment > <https://blueprints.launchpad.net/mistral/+spec/mistral-execution-environment> > * https://blueprints.launchpad.net/mistral/+spec/mistral-global-context > <https://blueprints.launchpad.net/mistral/+spec/mistral-global-context> > * https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values > <https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values> > * https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context > <https://blueprints.launchpad.net/mistral/+spec/mistral-runtime-context> > > Please refer to the following threads for backgrounds. > * > http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html > <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052643.html> > * > http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html > <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052960.html> > * > http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html > <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html> > > > Workflow Context Scope > 1. context to workflow is passed to all its subflows and subtasks/actions > (aka children) only explicitly via inputs > 2. context are passed by value (copy.deepcopy) to children > 3. change to context is passed to parent only when it's explicitly published > at the end of the child execution > 4. change to context at the parent (after a publish from a child) is passed > to subsequent children > > Environment Variables > Solves the problem for quickly passing pre-defined inputs to a WF execution. > In the WF spec, environment variables are referenced as $.env.var1, > $.env.var2, etc. We should implement an API and DB model where users can > pre-defined different environments with their own set of variables. An > environment can be passed either by name from the DB or adhoc by dict in > start_workflow. On workflow execution, a copy of the environment is saved > with the execution object. Action inputs are still declared explicitly in > the WF spec. This does not solve the problem where common inputs are > specified over and over again. So if there are multiple SQL tasks in the WF, > the WF author still needs to supply the conn_str explicitly for each task. > In the example below, let's say we have a SQL Query Action that takes a > connection string and a query statement as inputs. The WF author can specify > that the conn_str input is supplied from the $.env.conn_str. > > Example: > > # Assume this SqlAction is registered as std.sql in Mistral's Action table. > class SqlAction(object): > def __init__(self, conn_str, query): > ... > > ... > > version: "2.0" > workflows: > demo: > type: direct > input: > - query > output: > - records > tasks: > query: > action: std.sql conn_str={$.env.conn_str} query={$.query} > publish: > records: $ > > ... > > my_adhoc_env = { > "conn_str": "mysql://admin:secrete > <mysql://admin:secrete@>@localhost/test" > } > > ... > > # adhoc by dict > start_workflow(wf_name, wf_inputs, env=my_adhoc_env) > > OR > > # lookup by name from DB model > start_workflow(wf_name, wf_inputs, env="my_lab_env") > > Define Default Action Inputs as Environment Variables > Solves the problem where we're specifying the same inputs to subflows and > subtasks/actions over and over again. On command execution, if action inputs > are not explicitly supplied, then defaults will be lookup from the > environment. > > Example: > Using the same example from above, the WF author can still supply both > conn_str and query inputs in the WF spec. However, the author also has the > option to supply that as default action inputs. An example environment > structure is below. "__actions" should be reserved and immutable. Users can > specific one or more default inputs for the sql action as nested dict under > "__actions". Recursive YAQL eval should be supported in the env variables. > > version: "2.0" > workflows: > demo: > type: direct > input: > - query > output: > - records > tasks: > query: > action: std.sql query={$.query} > publish: > records: $ > > ... > > my_adhoc_env = { > "sql_server": "localhost", > "__actions": { > "std.sql": { > "conn_str": "mysql://admin:secrete@ > <mysql://admin:secrete@>{$.env.sql_server}/test" > } > } > } > > Default Input Values Supplied Explicitly in WF Spec > Please refer to this blueprint > <https://blueprints.launchpad.net/mistral/+spec/mistral-default-input-values> > for background. This is a different use case. To support, we just need to > set the correct order of precedence in applying values. > 1. Input explicitly given to the sub flow/task in the WF spec > 2. Default input supplied from env > 3. Default input supplied at WF spec > > Putting this together... > At runtime, the WF context would be similar to the following example. This > will be use to recursively eval the inputs for subflow/tasks/actions. > > ctx = { > "var1": …, > "var2": …, > "my_server_ip": "10.1.23.250", > "env": { > "sql_server": "localhost", > "__actions": { > "std.sql": { > "conn": "mysql://admin:secrete@{$.env.sql_server}/test > <mysql://admin:secrete@{$.env.sql_server}/test>" > }, > "my.action": { > "endpoint": "http://{$.my_server_ip}/v1/foo"; > } > } > } > } > > Runtime Context > Please refer to this thread > <http://lists.openstack.org/pipermail/openstack-dev/2014-December/052824.html> > for the background and discussion. The only change here is that on > run_action, we will pass the runtime data as kwargs to all action invocation. > This means all Action classes should have at least **kwargs added to the > __init__ method. > > runtime = { > "execution_id": ..., > "task_id": ..., > ... > } > > ... > > action = SomeAction(..., runtime=runtime) > > > > _______________________________________________ > OpenStack-dev mailing list > [email protected] > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________ OpenStack-dev mailing list [email protected] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
