Re: [openstack-dev] [Openstack] [celery][taskflow] Reg. celery and task-flow

[Lingxian Kong]

Hi, Eswar,

In case you're interested, there is another choice of 'workflow
engine' you can refer to: http://docs.openstack.org/developer/mistral/

Hope you'll find it interesting :-)

On Sat, Jan 9, 2016 at 10:33 AM, ESWAR RAO  wrote:
> Thanks Joshua for the clear explanation.
>
> On Sat, Jan 9, 2016 at 6:24 AM, Joshua Harlow  wrote:
>>
>> ESWAR RAO wrote:
>>>
>>> Hi Joshua,
>>>
>>> Thanks for the response.
>>>
>>> As you mentioned, I went through below which has taskflow as celery
>>> front end:
>>>
>>> https://etherpad.openstack.org/p/TaskFlowWorkerBasedEngine
>>>
>>> https://blueprints.launchpad.net/taskflow/+spec/distributed-celery
>>
>>
>> I think the idea way-back-when was celery would be a backend, not a
>> front-end, although my memory might be fuzzy in this area (since it was a
>> few years ago).
>>
>>>
>>>
>>> I guess if we have a job = task1 + task2 ; if we execute them through a
>>> taskflow parallel pattern, they will be executed in parallel.
>>>
>>> Just curious to know that since this threading is built on
>>> eventlets/green threads/python threads they will be affected by GIL and
>>> we may not utilize multi-core capability of our systems.
>>
>>
>> So that assumes u are using a taskflow engine which is powered by threads
>> (or greenthreads), there are two types that do not use threads but actually
>> run out of process:
>>
>> http://docs.openstack.org/developer/taskflow/engines.html#types
>>
>> 1. http://docs.openstack.org/developer/taskflow/workers.html (runs out of
>> process, across different machines, using kombu for triggering
>> start/status... of tasks on other machines)
>>
>> 2. A parallel engine that uses a process pool executor
>> (https://docs.python.org/dev/library/concurrent.futures.html#processpoolexecutor)
>> will then execute tasks using out of process child-processes.
>>
>> So both of these don't have the same GIL issue (although greenthreads by
>> there very nature are not affected by the GIL either).
>>
>> The example (aptly named hello world) shows these different types
>> (although it doesn't show the worker based one, since that requires more
>> things to setup) @
>> https://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L82
>>
>> -Josh
>>
>>>
>>> It seems in celery, we can have these tasks run as multiple processes so
>>> that they are not affected by GIL.
>>
>>
>> Right, the above starts to show why taskflow is really a super-set of
>> celery (for some combination/interpretation of what taskflow or celery do).
>>
>>>
>>> Please correct me if my understanding is wrong ??
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> On Sat, Jan 9, 2016 at 3:06 AM, Joshua Harlow >> > wrote:
>>>
>>> I also updated
>>> https://wiki.openstack.org/wiki/DistributedTaskManagement to denote
>>> that said wiki is no longer active (it was an attempt to back a
>>> taskflow engine[1] with celery); although if u are interested in
>>> continuing down this path feel free.
>>>
>>> Hopefully that clears up some 'confusion' around that wiki.
>>>
>>> [1] http://docs.openstack.org/developer/taskflow/engines.html
>>>
>>>
>>> Joshua Harlow wrote:
>>>
>>> So actually they are quite different, (although similar at some
>>> level),
>>>
>>> Given that celery isn't really a replacement for taskflow
>>> although one
>>> could say, from what I've heard from others, that taskflow is a
>>> super-set of what celery is so taskflow likely can replace parts
>>> of
>>> celery (but not vice-versa).
>>>
>>> Feel free to jump on #openstack-state-management IRC channel if
>>> u want
>>> to chat in person more about why (it gets into details that
>>> might just
>>> be easier to explain in person).
>>>
>>> ESWAR RAO wrote:
>>>
>>> Hi All,
>>>
>>> Please let me know whether celery is replacement for
>>> taskflow.
>>>
>>> As per my understanding, task-flow can break jobs into tasks
>>> and execute
>>> them.
>>>
>>>  From celery wiki, it also does almost similar behaviour.
>>>
>>> I guess in most of openstack components taskflow is widely
>>> used.
>>> Any places where its being replaced with celery ??
>>>
>>> Celery: https://wiki.openstack.org/wiki/Celery
>>> Distributed:
>>> https://wiki.openstack.org/wiki/DistributedTaskManagement
>>> TaskFlow: https://wiki.openstack.org/wiki/TaskFlow
>>>
>>> Thanks
>>> Eswar
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> 

[openstack-dev] "Open Containers" group on LinkedIn

[Nitin Agarwal]

Hello Everyone,

A very Happy New Year 2016 !!

I have started a new group "Open Containers" on LinkedIn to provide a
common platform to all the Containers and Docker enthusiasts and passionate
people. In this group, we will be discussing about the Linux and Docker
containers runtimes as well as Docker development, orchestration,
deployment, monitoring, networking and security tools.

Join the group here: https://www.linkedin.com/groups/8451196

Thanks,
Nitin Agarwal
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] "Open Containers" group on LinkedIn

[James Bottomley]

On Sat, 2016-01-09 at 18:11 +0530, Nitin Agarwal wrote:
> Hello Everyone,
> 
> A very Happy New Year 2016 !!
> 
> I have started a new group "Open Containers" on LinkedIn to provide a
> common platform to all the Containers and Docker enthusiasts and
> passionate
> people. In this group, we will be discussing about the Linux and
> Docker
> containers runtimes as well as Docker development, orchestration,
> deployment, monitoring, networking and security tools.
> 
> Join the group here: https://www.linkedin.com/groups/8451196

For discussions, could you just set up a regular mailing list?  The
advantages over linked-in for open source are: open platform, indexed
by the search engines and easy subscriptions.  If you want an existing
one with a reasonable email interface and social attachments, google
groups is OK.

James



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [trove] [sahara] Adding support for HBase in Trove

[michael mccune]

On 01/08/2016 08:34 AM, Amrith Kumar wrote:

As Kevin suggests, I'm adding [sahara] to the subject line.

Others in sahara who now see this thread, apologies for sending you a delayed 
invitation to the party. There's still lots of food and beer so come on in!


Amrith,

thanks for reposting this, some of our team was on holiday during the 
last week (jan 3-8).


regards,
mike


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder]How to create volume type with extra_specs by using cinder-client?

[zhu4236926]

Hi guys,
When creating volume type, extra_specs is optional in body, as can be seen 
from API document, http://developer.openstack.org/api-ref-blockstorage-v2.html
but I notice that in cinder-client, the method of create in 
VolumeTypeManager dose not take extra_specs in body,  
https://github.com/openstack/python-cinderclient/blob/stable/liberty/cinderclient/v2/volume_types.py
so I wonder how could create volume type with extra_specs in body by using 
cinder-client.
Thank you for your help.


By 
Sylvernass




 





 





 





 





 __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][aodh][vitrage] The purpose of notification about alarm updating

[AFEK, Ifat (Ifat)]

Hi Pradeep,

> From: Pradeep Kilambi [mailto:pkila...@redhat.com] 
> Sent: Thursday, January 07, 2016 9:15 PM
>
> > > alternatively, it was discussed that maybe adding a zaqar notifier
> > > would be useful. that way, aodh-notifier would send alarm to zaqar and
> > > you could configure it to requeue or maybe send a smtp.
> >
> > What are the advantages of using Zaqar over using the message bus? Is the 
> > solution you suggested already supported?
>
> This work is currently in progress, I'm hoping to get this support into 
> Mitaka.
 
We want to do the integration with Aodh in the coming weeks, so I guess we 
won't use Zaqar. We will consider using it in Newton.

Thanks,
Ifat.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [telemetry][aodh][vitrage] The purpose of notification about alarm updating

[AFEK, Ifat (Ifat)]

Hi Julien,

I'm a bit confused. This thread started by liusheng asking to remove the 
notifications to the bus[1], and I replied saying that we do want to use this 
capability in Vitrage. What is the notifier plugin that you suggested? Is it 
the same notifier that liusheng referred to, or something else?


[1] https://review.openstack.org/#/c/246727/ 

Thanks,
Ifat.

> -Original Message-
> From: Julien Danjou [mailto:jul...@danjou.info]
> Sent: Friday, January 08, 2016 11:42 AM
> 
> On Thu, Jan 07 2016, AFEK, Ifat (Ifat) wrote:
> 
> > We have two motivations: one is that we want to get notifications on
> > every change, but we don't want to register our webhook to each and
> > every alarm; and the other is that we already listen to the message
> > bus for other openstack components, so we would like to handle aodh
> the same way.
> >
> > If we disable aodh-notifier, won't it have other impacts on the
> > system? What if someone else used a webhook, and won't be notified
> > because we disabled the notifier?
> 
> We could have a notifier plugin that would send a notification using
> oslo.messaging I guess. That shouldn't be a big deal.
> 
> --
> Julien Danjou
> ;; Free Software hacker
> ;; https://julien.danjou.info

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] Should we add 'os-initialize_connection' to Blockstorage API Document?

[liuxinguo]

Hi,

'os-initialize_connection' is a important API for Cinder to complete the volume 
attach operation,
but there is no record in Cinder(Blockstorage) API Document(1).
Also some developers want to find the description for the 
'os-initialize_connection' API.

So should we add this API to the Blockstorage API Document?

Wilson Liu
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][openstackclient] check/gate job to check for duplicate openstackclient commands

[Steve Martinelli]

During the Liberty release the OpenStackClient (OSC) team ran into an issue
that is documented here: [0] and here: [1]. In short, commands were
clobbering each other because they had the same command name.

A longer example is this, OSC has a command for listing compute flavors (os
flavor list). zaqarclient, an OSC plugin, also implemented an `os flavor
list` command. This caused OSC to break (it became unusable because it
couldn't load entry points), and the user had to upgrade their zaqarclient,
which included a renamed command (os messaging flavor list).

In an effort to make sure this doesn't happen against, we did two things:
1) fixed the exception handling upon load at the cliff level, now OSC won't
become unusable, it'll just take the last entrypoint it sees, and 2) we
created a new gate/check job that checks for duplicate commands [2].
(Thanks to ajaeger and dhellmann for their help in this work!)

I've added this job to the OpenStackClient gate (in a non-voting capacity
for now), and would like to get it added to the following projects, again
in a non-voting capacity (since they are all OSC plugins):

  - python-barbicanclient
  - python-congressclient
  - python-cueclient
  - python-designateclient
  - python-heatclient
  - python-ironicclient
  - python-ironic-inspector-client
  - python-mistralclient
  - python-saharaclient
  - python-tuskarclient
  - python-zaqarclient

If the core team for any of those projects objects to me adding a new check
job then reply to this thread or this patch [3]

Regarding the eventual question about the value of a non-voting job:
AFAICT, the new check job is working fine, it's catching valid errors and
succeeded where it should. I'd like to make this voting eventually, but
it's only been running in the OSC gate for about a week, and I'd like a few
non-voting runs in the plugin projects to ensure we don't have any hiccups
before making this a voting job.

[0]
http://lists.openstack.org/pipermail/openstack-dev/2015-October/076272.html
[1] https://bugs.launchpad.net/python-openstackclient/+bug/1503512
[2] https://review.openstack.org/#/c/261828/
[3] https://review.openstack.org/#/c/265608/

Thanks,

Steve Martinelli
OpenStack Keystone Project Team Lead
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev