Re: [openstack-dev] [Oslo][TaskFlow] Proposal for new core reviewer (pranesh pandurangan)
On 01.07.2014 04:05, Joshua Harlow wrote: Greetings all stackers, I propose that we add Pranesh Pandurangan[1] to the taskflow-core team[2]. Pranesh has been actively contributing to taskflow for a while now, both in helping develop code and helping with the review load. He has provided quality reviews and is doing an awesome job with the various taskflow concepts and helping make taskflow the best library it can be! Overall I think he would make a great addition to the core review team. Please respond with +1/-1. +1 -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] How Mistral handling long running delegate tasks
I'm trying to catch up this rather long and interesting discussion, sorry for somewhat late reply. I can see aspects of 'lazy model' support in TaskFlow: - how tasks are executed and reverted - how flows are run - how engine works internally Let me address those aspects separately. == Executing and reverting tasks == I think that should be done via different interface then running a flow (or scheduling it to run), as it is completely different thing. In current TaskFlow this interface is called task executor: https://github.com/openstack/taskflow/blob/master/taskflow/engines/action_engine/executor.py#L57 That is actually how our WorkerBasedEngine was implemented: it's the same engine with special task executor that schedules tasks on worker instead of running task code locally. Task executors are not aware of flows by design, all they do is executing and reverting tasks. That means that task executors can be easily shared between engines if that's wanted. Current TaskExecutorBase interface uses futures (PEP 3148-like). When I proposed it, futures looked like good tool for the task at hand (see e.g. async task etherpad https://etherpad.openstack.org/p/async-taskflow-tasks) Now it may be time to reconsider that: having one future object per running task may become a scalability issue. It may be worth to use callbacks instead. It should not be too hard to refactor current engine for that. Also, as TaskExecutorBase is an internal API, there should not be any compatibility issues. Then, we can make task executor interface public and allow clients to provide their own task executors. It will be possible then for Mistral to implement its own task executor, or several, and share the executors between all the engine instances. You can call it a plan;) == Running flows == To run the flow TaskFlow client uses engine interface; also, there are few of helper functions provided for convenience: http://docs.openstack.org/developer/taskflow/engines.html#module-taskflow.engines.base http://docs.openstack.org/developer/taskflow/engines.html#creating-engines That is part of our public API, it is stable and good enough. Basically, I don't think this API needs any major change. Maybe it worth to add function or method to schedule running flow without actually waiting for flow completion (at least, it was on my top secret TODO list for quite a long time). == Engine internals == Each engine eats resources, like thread it runs on; using these resources to run one flow only is somewhat wasteful. Some work is already planned to address this situation (see e.g. https://blueprints.launchpad.net/taskflow/+spec/share-engine-thread). Also, it might be good idea to implement different 'type' of engine to support 'lazy' model, as Joshua suggests. But whatever should and will be done about it, I daresay all that work can be done without affecting API more then I described above. -- WBR, Ivan A. Melnikov ... tasks must flow ... On 02.04.2014 01:51, Dmitri Zimine wrote: Even more responses inline :) [...] ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [State-Management] Agenda for meeting (tommorow) at 2000 UTC
On 06.03.2014 04:00, Joshua Harlow wrote: Hi all, The [state-management] project team holds a weekly meeting in #openstack-meeting on thursdays, 2000 UTC. The next meeting is tommorow, 2014-03-06!!! [...] Any other topics are welcome :-) I dared to add item on documentation improvements. -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] Lots of gating failures because of testtools
Hi there, I see lots of unit tests jobs on gate fail with errors like 2014-01-29 10:48:44.933 | File /home/jenkins/workspace/gate-taskflow-python26/.tox/py26/lib/python2.6/site-packages/subunit/test_results.py, line 23, in module 2014-01-29 10:48:44.934 | from testtools.compat import all 2014-01-29 10:48:44.935 | ImportError: cannot import name all 2014-01-29 10:48:44.992 | ERROR: InvocationError: '/home/jenkins/workspace/gate-taskflow-python26/.tox/py26/bin/python setup.py testr --slowest --testr-args=' Looks like subunit is not compatible with just-released testtools 0.9.35. I guess we will need to pin testtools to 0.9.34 in test-requirements.txt. Or there are better solution? I filed a bug to subunit upstream: https://bugs.launchpad.net/subunit/+bug/1274056 I also filed a bug for taskflow, feel free to add your projects there if it's affected, too: https://bugs.launchpad.net/taskflow/+bug/1274050 -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [State-Management] Proposal to add Changbin Liu to taskflow-core
On Jan 17, 2014 2:27 AM, Joshua Harlow harlo...@yahoo-inc.com wrote: Greetings all stackers, I propose that we add Changbin Liu[1] to the taskflow-core team[2]. +1 -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [infra] Unit tests, gating, and real databases
Hi there, As far as I understand, slaves that run gate-*-python27 and python26 jobs have MySQL and Postgres servers installed and running so we can test migrations and do functional testing for database-related code. I wanted to use this to improve TaskFlow gating, but I failed to find docs about it and to derive how this database instances should be used from nova and oslo.db tests code. Can anyone give some hints or pointers on where should I get connection config and what can I do with those database servers in unit and functional tests? -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade
On 04.01.2014 01:29, Sean Dague wrote: On 01/03/2014 04:17 PM, Doug Hellmann wrote: [...] That's what made me think of the solution. But isn't setuptools in fact telling us that somehow the versions of things we expected to have installed are no longer installed and so something *is* broken (even if the versions of the installed libraries work together). It actually tells us that a human, somewhere, decided that their software did not work with some combination of other software, and that we are no longer able to correct their mistaken assumptions. [...] But sometimes humans are not wrong. For example, no released TaskFlow version really works with SQLAlchemy = 0.8 -- that was fixed only recently (https://review.openstack.org/#/c/62661/). I consider requirements to be part of the code, so if they are too strict (or too broad), that must be fixed, in a usual opensource way: via filing bug report, suggesting patch and so on. Requirements should reflect reality, especially for libraries that are intended to be useful outside of OpenStack, too. For example, current TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that, and release new version with that fix. -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Unicode strings in Python3
On 05.12.2013 20:42, Georgy Okrokvertskhov wrote: On Thu, Dec 5, 2013 at 8:32 AM, Christopher Armstrong chris.armstr...@rackspace.com wrote: On Thu, Dec 5, 2013 at 3:26 AM, Julien Danjou jul...@danjou.info wrote: On Wed, Dec 04 2013, Georgy Okrokvertskhov wrote: Quick summary: you can't use unicode() function and u' ' strings in Pyhton3. Not that it's advised, but you can use u' ' back again with Python 3.3. And this is a very useful feature for projects that want to have a single codebase that runs on both python 2 and python 3, so it's worth taking advantage of. You are right. PEP 414 introduces u literals in Python3.3. Unicode() function still does not work and should be avoided in the code, though. Also let me remind that there is always six.text_type for those who really need it. -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Cinder] TaskFlow 0.1 integration
On 19.11.2013 10:38, Kekane, Abhishek wrote: Hi All, Greetings!!! Hi there! And thanks for your interest in cinder and taskflow! We are in process of implementing the TaskFlow 0.1 in Cinder for copy volume to image and delete volume. I have added two blueprints for the same. 1. https://blueprints.launchpad.net/cinder/+spec/copy-volume-to-image-task-flow 2. https://blueprints.launchpad.net/cinder/+spec/delete-volume-task-flow I would like to know if any other developers/teams are working or planning to work on any cinder api apart from above two api's. Your help is appreciated. Anastasia Karpinska works on updating existing flows to use released TaskFlow 0.1.1 instead of internal copy: https://review.openstack.org/53922 It was blocked because taskflow was not in openstack/requirements, but now we're there, and Anastasia promised to finish the work and submit updated changeset for review in couple of days. There are also two changesets that convert cinder APIs to use TaskFlow: - https://review.openstack.org/53480 for create_backup by Victor Rodionov - https://review.openstack.org/55134 for create_snapshot by Stanislav Kudriashev As far as I know both Stanislav and Victor suspended their work unitil Anastasia's change lands. -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [State-Management] Proposal to add Anastasia Karpinska to taskflow-core
On 18.11.2013 22:05, Joshua Harlow wrote: Greetings all stackers, I propose that we add Anastasia Karpinska to the taskflow-core team [1]. [...] +1 -- WBR, Ivan A. Melnikov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
