Hi, Eswar, In case you're interested, there is another choice of 'workflow engine' you can refer to: http://docs.openstack.org/developer/mistral/
Hope you'll find it interesting :-) On Sat, Jan 9, 2016 at 10:33 AM, ESWAR RAO <eswar7...@gmail.com> wrote: > Thanks Joshua for the clear explanation. > > On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow <harlo...@fastmail.com> wrote: >> >> ESWAR RAO wrote: >>> >>> Hi Joshua, >>> >>> Thanks for the response. >>> >>> As you mentioned, I went through below which has taskflow as celery >>> front end: >>> >>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine >>> >>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery >> >> >> I think the idea way-back-when was celery would be a backend, not a >> front-end, although my memory might be fuzzy in this area (since it was a >> few years ago). >> >>> >>> >>> I guess if we have a job = task1 + task2 ; if we execute them through a >>> taskflow parallel pattern, they will be executed in parallel. >>> >>> Just curious to know that since this threading is built on >>> eventlets/green threads/python threads they will be affected by GIL and >>> we may not utilize multi-core capability of our systems. >> >> >> So that assumes u are using a taskflow engine which is powered by threads >> (or greenthreads), there are two types that do not use threads but actually >> run out of process: >> >> http://docs.openstack.org/developer/taskflow/engines.html#types >> >> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of >> process, across different machines, using kombu for triggering >> start/status... of tasks on other machines) >> >> 2. A parallel engine that uses a process pool executor >> (https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor) >> will then execute tasks using out of process child-processes. >> >> So both of these don't have the same GIL issue (although greenthreads by >> there very nature are not affected by the GIL either). >> >> The example (aptly named hello world) shows these different types >> (although it doesn't show the worker based one, since that requires more >> things to setup) @ >> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82 >> >> -Josh >> >>> >>> It seems in celery, we can have these tasks run as multiple processes so >>> that they are not affected by GIL. >> >> >> Right, the above starts to show why taskflow is really a super-set of >> celery (for some combination/interpretation of what taskflow or celery do). >> >>> >>> Please correct me if my understanding is wrong ?? >>> >>> Thanks >>> Eswar >>> >>> >>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <harlo...@fastmail.com >>> <mailto:harlo...@fastmail.com>> wrote: >>> >>> I also updated >>> https://wiki.openstack.org/wiki/DistributedTaskManagement to denote >>> that said wiki is no longer active (it was an attempt to back a >>> taskflow engine[1] with celery); although if u are interested in >>> continuing down this path feel free. >>> >>> Hopefully that clears up some 'confusion' around that wiki. >>> >>> [1] http://docs.openstack.org/developer/taskflow/engines.html >>> >>> >>> Joshua Harlow wrote: >>> >>> So actually they are quite different, (although similar at some >>> level), >>> >>> Given that celery isn't really a replacement for taskflow >>> although one >>> could say, from what I've heard from others, that taskflow is a >>> super-set of what celery is so taskflow likely can replace parts >>> of >>> celery (but not vice-versa). >>> >>> Feel free to jump on #openstack-state-management IRC channel if >>> u want >>> to chat in person more about why (it gets into details that >>> might just >>> be easier to explain in person). >>> >>> ESWAR RAO wrote: >>> >>> Hi All, >>> >>> Please let me know whether celery is replacement for >>> taskflow. >>> >>> As per my understanding, task-flow can break jobs into tasks >>> and execute >>> them. >>> >>> From celery wiki, it also does almost similar behaviour. >>> >>> I guess in most of openstack components taskflow is widely >>> used. >>> Any places where its being replaced with celery ?? >>> >>> Celery: https://wiki.openstack.org/wiki/Celery >>> Distributed: >>> https://wiki.openstack.org/wiki/DistributedTaskManagement >>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow >>> >>> Thanks >>> Eswar >>> >>> >>> __________________________________________________________________________ >>> >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> >>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> _______________________________________________ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openst...@lists.openstack.org >>> <mailto:openst...@lists.openstack.org> >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> >>> _______________________________________________ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openst...@lists.openstack.org >>> <mailto:openst...@lists.openstack.org> >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Regards! ----------------------------------- Lingxian Kong __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev