Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

[Lingxian Kong]

Hi, Eswar,

In case you're interested, there is another choice of 'workflow
engine' you can refer to: http://docs.openstack.org/developer/mistral/

Hope you'll find it interesting :-)

On Sat, Jan 9, 2016 at 10:33 AM, ESWAR RAO  wrote:
> Thanks Joshua for the clear explanation.
>
> On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow  wrote:
>>
>> ESWAR RAO wrote:
>>>
>>> Hi Joshua,
>>>
>>> Thanks for the response.
>>>
>>> As you mentioned, I went through below which has taskflow as celery
>>> front end:
>>>
>>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>>>
>>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
>>
>>
>> I think the idea way-back-when was celery would be a backend, not a
>> front-end, although my memory might be fuzzy in this area (since it was a
>> few years ago).
>>
>>>
>>>
>>> I guess if we have a job = task1 + task2 ; if we execute them through a
>>> taskflow parallel pattern, they will be executed in parallel.
>>>
>>> Just curious to know that since this threading is built on
>>> eventlets/green threads/python threads they will be affected by GIL and
>>> we may not utilize multi-core capability of our systems.
>>
>>
>> So that assumes u are using a taskflow engine which is powered by threads
>> (or greenthreads), there are two types that do not use threads but actually
>> run out of process:
>>
>> http://docs.openstack.org/developer/taskflow/engines.html#types
>>
>> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of
>> process, across different machines, using kombu for triggering
>> start/status... of tasks on other machines)
>>
>> 2. A parallel engine that uses a process pool executor
>> (https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
>> will then execute tasks using out of process child-processes.
>>
>> So both of these don't have the same GIL issue (although greenthreads by
>> there very nature are not affected by the GIL either).
>>
>> The example (aptly named hello world) shows these different types
>> (although it doesn't show the worker based one, since that requires more
>> things to setup) @
>> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
>>
>> -Josh
>>
>>>
>>> It seems in celery, we can have these tasks run as multiple processes so
>>> that they are not affected by GIL.
>>
>>
>> Right, the above starts to show why taskflow is really a super-set of
>> celery (for some combination/interpretation of what taskflow or celery do).
>>
>>>
>>> Please correct me if my understanding is wrong ??
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow >> > wrote:
>>>
>>> I also updated
>>> https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>>> that said wiki is no longer active (it was an attempt to back a
>>> taskflow engine[1] with celery); although if u are interested in
>>> continuing down this path feel free.
>>>
>>> Hopefully that clears up some 'confusion' around that wiki.
>>>
>>> [1] http://docs.openstack.org/developer/taskflow/engines.html
>>>
>>>
>>> Joshua Harlow wrote:
>>>
>>> So actually they are quite different, (although similar at some
>>> level),
>>>
>>> Given that celery isn't really a replacement for taskflow
>>> although one
>>> could say, from what I've heard from others, that taskflow is a
>>> super-set of what celery is so taskflow likely can replace parts
>>> of
>>> celery (but not vice-versa).
>>>
>>> Feel free to jump on #openstack-state-management IRC channel if
>>> u want
>>> to chat in person more about why (it gets into details that
>>> might just
>>> be easier to explain in person).
>>>
>>> ESWAR RAO wrote:
>>>
>>> Hi All,
>>>
>>> Please let me know whether celery is replacement for
>>> taskflow.
>>>
>>> As per my understanding, task-flow can break jobs into tasks
>>> and execute
>>> them.
>>>
>>>  From celery wiki, it also does almost similar behaviour.
>>>
>>> I guess in most of openstack components taskflow is widely
>>> used.
>>> Any places where its being replaced with celery ??
>>>
>>> Celery: https://wiki.openstack.org/wiki/Celery
>>> Distributed:
>>> https://wiki.openstack.org/wiki/DistributedTaskManagement
>>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> 

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

[Joshua Harlow]

I also updated https://wiki.openstack.org/wiki/DistributedTaskManagement 
to denote that said wiki is no longer active (it was an attempt to back 
a taskflow engine[1] with celery); although if u are interested in 
continuing down this path feel free.


Hopefully that clears up some 'confusion' around that wiki.

[1] http://docs.openstack.org/developer/taskflow/engines.html

Joshua Harlow wrote:

So actually they are quite different, (although similar at some level),

Given that celery isn't really a replacement for taskflow although one
could say, from what I've heard from others, that taskflow is a
super-set of what celery is so taskflow likely can replace parts of
celery (but not vice-versa).

Feel free to jump on #openstack-state-management IRC channel if u want
to chat in person more about why (it gets into details that might just
be easier to explain in person).

ESWAR RAO wrote:

Hi All,

Please let me know whether celery is replacement for taskflow.

As per my understanding, task-flow can break jobs into tasks and execute
them.

From celery wiki, it also does almost similar behaviour.

I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??

Celery: https://wiki.openstack.org/wiki/Celery
Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow

Thanks
Eswar

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

[Joshua Harlow]

ESWAR RAO wrote:

Hi Joshua,

Thanks for the response.

As you mentioned, I went through below which has taskflow as celery
front end:

https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

https://blueprints.launchpad.net/taskflow/+spec/distributed-celery


I think the idea way-back-when was celery would be a backend, not a 
front-end, although my memory might be fuzzy in this area (since it was 
a few years ago).





I guess if we have a job = task1 + task2 ; if we execute them through a
taskflow parallel pattern, they will be executed in parallel.

Just curious to know that since this threading is built on
eventlets/green threads/python threads they will be affected by GIL and
we may not utilize multi-core capability of our systems.


So that assumes u are using a taskflow engine which is powered by 
threads (or greenthreads), there are two types that do not use threads 
but actually run out of process:


http://docs.openstack.org/developer/taskflow/engines.html#types

1. http://docs.openstack.org/developer/taskflow/workers.html (runs out 
of process, across different machines, using kombu for triggering 
start/status... of tasks on other machines)


2. A parallel engine that uses a process pool executor 
(https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor) 
will then execute tasks using out of process child-processes.


So both of these don't have the same GIL issue (although greenthreads by 
there very nature are not affected by the GIL either).


The example (aptly named hello world) shows these different types 
(although it doesn't show the worker based one, since that requires more 
things to setup) @ 
https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82


-Josh



It seems in celery, we can have these tasks run as multiple processes so
that they are not affected by GIL.


Right, the above starts to show why taskflow is really a super-set of 
celery (for some combination/interpretation of what taskflow or celery do).




Please correct me if my understanding is wrong ??

Thanks
Eswar


On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow > wrote:

I also updated
https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
that said wiki is no longer active (it was an attempt to back a
taskflow engine[1] with celery); although if u are interested in
continuing down this path feel free.

Hopefully that clears up some 'confusion' around that wiki.

[1] http://docs.openstack.org/developer/taskflow/engines.html


Joshua Harlow wrote:

So actually they are quite different, (although similar at some
level),

Given that celery isn't really a replacement for taskflow
although one
could say, from what I've heard from others, that taskflow is a
super-set of what celery is so taskflow likely can replace parts of
celery (but not vice-versa).

Feel free to jump on #openstack-state-management IRC channel if
u want
to chat in person more about why (it gets into details that
might just
be easier to explain in person).

ESWAR RAO wrote:

Hi All,

Please let me know whether celery is replacement for taskflow.

As per my understanding, task-flow can break jobs into tasks
and execute
them.

 From celery wiki, it also does almost similar behaviour.

I guess in most of openstack components taskflow is widely used.
Any places where its being replaced with celery ??

Celery: https://wiki.openstack.org/wiki/Celery
Distributed:
https://wiki.openstack.org/wiki/DistributedTaskManagement
TaskFlow: https://wiki.openstack.org/wiki/TaskFlow

Thanks
Eswar


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe 

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org

Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openst...@lists.openstack.org

Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

[ESWAR RAO]

Hi Joshua,

Thanks for the response.

As you mentioned, I went through below which has taskflow as celery front
end:

 https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine

 https://blueprints.launchpad.net/taskflow/+spec/distributed-celery

I guess if we have a job = task1 + task2 ; if we execute them through a
taskflow parallel pattern, they will be executed in parallel.

Just curious to know that since this threading is built on eventlets/green
threads/python threads they will be affected by GIL and we may not utilize
multi-core capability of our systems.

It seems in celery, we can have these tasks run as multiple processes so
that they are not affected by GIL.

Please correct me if my understanding is wrong ??

Thanks
Eswar


On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow  wrote:

> I also updated https://wiki.openstack.org/wiki/DistributedTaskManagement
> to denote that said wiki is no longer active (it was an attempt to back a
> taskflow engine[1] with celery); although if u are interested in continuing
> down this path feel free.
>
> Hopefully that clears up some 'confusion' around that wiki.
>
> [1] http://docs.openstack.org/developer/taskflow/engines.html
>
>
> Joshua Harlow wrote:
>
>> So actually they are quite different, (although similar at some level),
>>
>> Given that celery isn't really a replacement for taskflow although one
>> could say, from what I've heard from others, that taskflow is a
>> super-set of what celery is so taskflow likely can replace parts of
>> celery (but not vice-versa).
>>
>> Feel free to jump on #openstack-state-management IRC channel if u want
>> to chat in person more about why (it gets into details that might just
>> be easier to explain in person).
>>
>> ESWAR RAO wrote:
>>
>>> Hi All,
>>>
>>> Please let me know whether celery is replacement for taskflow.
>>>
>>> As per my understanding, task-flow can break jobs into tasks and execute
>>> them.
>>>
>>> From celery wiki, it also does almost similar behaviour.
>>>
>>> I guess in most of openstack components taskflow is widely used.
>>> Any places where its being replaced with celery ??
>>>
>>> Celery: https://wiki.openstack.org/wiki/Celery
>>> Distributed: https://wiki.openstack.org/wiki/DistributedTaskManagement
>>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> ___
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openst...@lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
> ___
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openst...@lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

[ESWAR RAO]

Thanks Joshua for the clear explanation.

On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow  wrote:

> ESWAR RAO wrote:
>
>> Hi Joshua,
>>
>> Thanks for the response.
>>
>> As you mentioned, I went through below which has taskflow as celery
>> front end:
>>
>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>>
>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
>>
>
> I think the idea way-back-when was celery would be a backend, not a
> front-end, although my memory might be fuzzy in this area (since it was a
> few years ago).
>
>
>>
>> I guess if we have a job = task1 + task2 ; if we execute them through a
>> taskflow parallel pattern, they will be executed in parallel.
>>
>> Just curious to know that since this threading is built on
>> eventlets/green threads/python threads they will be affected by GIL and
>> we may not utilize multi-core capability of our systems.
>>
>
> So that assumes u are using a taskflow engine which is powered by threads
> (or greenthreads), there are two types that do not use threads but actually
> run out of process:
>
> http://docs.openstack.org/developer/taskflow/engines.html#types
>
> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of
> process, across different machines, using kombu for triggering
> start/status... of tasks on other machines)
>
> 2. A parallel engine that uses a process pool executor (
> https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
> will then execute tasks using out of process child-processes.
>
> So both of these don't have the same GIL issue (although greenthreads by
> there very nature are not affected by the GIL either).
>
> The example (aptly named hello world) shows these different types
> (although it doesn't show the worker based one, since that requires more
> things to setup) @
> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
>
> -Josh
>
>
>> It seems in celery, we can have these tasks run as multiple processes so
>> that they are not affected by GIL.
>>
>
> Right, the above starts to show why taskflow is really a super-set of
> celery (for some combination/interpretation of what taskflow or celery do).
>
>
>> Please correct me if my understanding is wrong ??
>>
>> Thanks
>> Eswar
>>
>>
>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow > > wrote:
>>
>> I also updated
>> https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>> that said wiki is no longer active (it was an attempt to back a
>> taskflow engine[1] with celery); although if u are interested in
>> continuing down this path feel free.
>>
>> Hopefully that clears up some 'confusion' around that wiki.
>>
>> [1] http://docs.openstack.org/developer/taskflow/engines.html
>>
>>
>> Joshua Harlow wrote:
>>
>> So actually they are quite different, (although similar at some
>> level),
>>
>> Given that celery isn't really a replacement for taskflow
>> although one
>> could say, from what I've heard from others, that taskflow is a
>> super-set of what celery is so taskflow likely can replace parts
>> of
>> celery (but not vice-versa).
>>
>> Feel free to jump on #openstack-state-management IRC channel if
>> u want
>> to chat in person more about why (it gets into details that
>> might just
>> be easier to explain in person).
>>
>> ESWAR RAO wrote:
>>
>> Hi All,
>>
>> Please let me know whether celery is replacement for taskflow.
>>
>> As per my understanding, task-flow can break jobs into tasks
>> and execute
>> them.
>>
>>  From celery wiki, it also does almost similar behaviour.
>>
>> I guess in most of openstack components taskflow is widely
>> used.
>> Any places where its being replaced with celery ??
>>
>> Celery: https://wiki.openstack.org/wiki/Celery
>> Distributed:
>> https://wiki.openstack.org/wiki/DistributedTaskManagement
>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>
>> Thanks
>> Eswar
>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> 
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> ___
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to : openst...@lists.openstack.org
>> 
>>