Thanks Joshua for the clear explanation. On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow <harlo...@fastmail.com> wrote:
> ESWAR RAO wrote: > >> Hi Joshua, >> >> Thanks for the response. >> >> As you mentioned, I went through below which has taskflow as celery >> front end: >> >> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine >> >> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery >> > > I think the idea way-back-when was celery would be a backend, not a > front-end, although my memory might be fuzzy in this area (since it was a > few years ago). > > >> >> I guess if we have a job = task1 + task2 ; if we execute them through a >> taskflow parallel pattern, they will be executed in parallel. >> >> Just curious to know that since this threading is built on >> eventlets/green threads/python threads they will be affected by GIL and >> we may not utilize multi-core capability of our systems. >> > > So that assumes u are using a taskflow engine which is powered by threads > (or greenthreads), there are two types that do not use threads but actually > run out of process: > > http://docs.openstack.org/developer/taskflow/engines.html#types > > 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of > process, across different machines, using kombu for triggering > start/status... of tasks on other machines) > > 2. A parallel engine that uses a process pool executor ( > https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor) > will then execute tasks using out of process child-processes. > > So both of these don't have the same GIL issue (although greenthreads by > there very nature are not affected by the GIL either). > > The example (aptly named hello world) shows these different types > (although it doesn't show the worker based one, since that requires more > things to setup) @ > https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82 > > -Josh > > >> It seems in celery, we can have these tasks run as multiple processes so >> that they are not affected by GIL. >> > > Right, the above starts to show why taskflow is really a super-set of > celery (for some combination/interpretation of what taskflow or celery do). > > >> Please correct me if my understanding is wrong ?? >> >> Thanks >> Eswar >> >> >> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow <harlo...@fastmail.com >> <mailto:harlo...@fastmail.com>> wrote: >> >> I also updated >> https://wiki.openstack.org/wiki/DistributedTaskManagement to denote >> that said wiki is no longer active (it was an attempt to back a >> taskflow engine[1] with celery); although if u are interested in >> continuing down this path feel free. >> >> Hopefully that clears up some 'confusion' around that wiki. >> >> [1] http://docs.openstack.org/developer/taskflow/engines.html >> >> >> Joshua Harlow wrote: >> >> So actually they are quite different, (although similar at some >> level), >> >> Given that celery isn't really a replacement for taskflow >> although one >> could say, from what I've heard from others, that taskflow is a >> super-set of what celery is so taskflow likely can replace parts >> of >> celery (but not vice-versa). >> >> Feel free to jump on #openstack-state-management IRC channel if >> u want >> to chat in person more about why (it gets into details that >> might just >> be easier to explain in person). >> >> ESWAR RAO wrote: >> >> Hi All, >> >> Please let me know whether celery is replacement for taskflow. >> >> As per my understanding, task-flow can break jobs into tasks >> and execute >> them. >> >> From celery wiki, it also does almost similar behaviour. >> >> I guess in most of openstack components taskflow is widely >> used. >> Any places where its being replaced with celery ?? >> >> Celery: https://wiki.openstack.org/wiki/Celery >> Distributed: >> https://wiki.openstack.org/wiki/DistributedTaskManagement >> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow >> >> Thanks >> Eswar >> >> >> __________________________________________________________________________ >> >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> _______________________________________________ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openst...@lists.openstack.org >> <mailto:openst...@lists.openstack.org> >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> >> _______________________________________________ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openst...@lists.openstack.org >> <mailto:openst...@lists.openstack.org> >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> >>
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev