Re: [openstack-dev] [freezer] Core team updates

[Dieterly, Deklan]

+1
-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




On 8/25/16, 9:33 AM, "Mathieu, Pierre-Arthur"
 wrote:

>Hello, 
>
>I would like to propose some modifications regarding the Freezer core
>team. 
>
>First, the removal of two inactive members:
>  - Fabrizio Fresco: Inactive
>  - Eldar Nugaev: Switched company and is now focusing on other projects.
>Thank you very much for your contributions.
>
>
>Secondly, I would like to propose that we promote Yang Yapeng
>(yangyapeng) core.
>He has been a highly valuable developper for the past few month, mainly
>working on integration with Nova and Cinder.
>His work can be found here: [1]
>And his stackalitics profile here: [2]
>
>
>If you agree with all these change, please approve with a +1 answer or
>explain your opinion on any of these individual modification.
>If there are no objection, I plan on applying these tomorrow evening.
>
>Thanks
>- Pierre, Freezer PTL
>
>[1]  https://review.openstack.org/#/q/owner:%22yapeng+Yang%22
>[2] http://stackalytics.com/?release=all=loc_id=yang-yapeng
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>  
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Freezer] Replace Gnu Tar with DAR

[Dieterly, Deklan]

Then it would not be an incremental backup/restore. This problem arises
when doing incremental backup and restores.
-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




From:  Fausto Marzi <fausto.ma...@gmail.com>
Reply-To:  "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Date:  Wednesday, May 18, 2016 at 5:22 AM
To:  "OpenStack Development Mailing List (not for usage questions)"
<openstack-dev@lists.openstack.org>
Subject:  Re: [openstack-dev] [Freezer] Replace Gnu Tar with DAR


>Hi Deklan,
>
>what happen if the extract is executed without --listed-incremental or
>--incremental options?
>
>
>Does the issue still happen?
>
>
>Thanks,
>
>Fausto
>
>
>On Sat, May 14, 2016 at 12:56 AM, Dieterly, Deklan
><deklan.diete...@hpe.com> wrote:
>
>When using incremental backups, tar will not handle removing a dir and
>then renaming another dir to the removed dir.
>
>
>dek@dek-HP-Z620-Workstation:~/backup-test$ tar --extract
>--listed-incrementa=/dev/null --file backup.2.tar
>tar: Cannot rename Œbackup/dir1¹ to Œbackup/dir2¹: Directory not empty
>tar: Exiting with failure status due to previous errors
>
>
>
>Here are the steps to reproduce.
>
> 1845  mkdir backup
> 1846  mkdir backup/dir1
> 1847  mkdir backup/dir2
> 1848  echo "aa" > backup/dir1/dir1-file1
> 1849  echo "aa" > backup/dir2/dir2-file1
> 1852  tar --create --file=backup.tar --listed-incremental=./listed-incr
>backup
> 1854  rm -rf backup/dir2
> 1855  mv backup/dir1 backup/dir2
> 1856  tar --create --file=backup.2.tar --listed-incremental=./listed-incr
>backup
> 1859  tar --extract --listed-incrementa=/dev/null --file backup.tar
> 1861  tar --extract --listed-incrementa=/dev/null --file backup.2.tar
>
>
>This seems to be a well known, long-standing issue with tar.
>--
>Deklan Dieterly
>
>Senior Systems Software Engineer
>HPE
>
>
>
>
>On 5/13/16, 4:33 PM, "Fox, Kevin M" <kevin@pnnl.gov> wrote:
>
>>Whats the issue?
>>
>>From: Dieterly, Deklan [deklan.diete...@hpe.com]
>>Sent: Friday, May 13, 2016 3:07 PM
>>To: openstack-dev@lists.openstack.org
>>Subject: [openstack-dev] [Freezer] Replace Gnu Tar with DAR
>>
>>Does anybody see any issues if Freezer used DAR instead of Gnu Tar? DAR
>>seems to handle a particular use case that Freezer has while Gnu Tar does
>>not.
>>--
>>Deklan Dieterly
>>
>>Senior Systems Software Engineer
>>HPE
>>
>>
>>_
>>_
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: 
>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
><http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>_
>>_
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: 
>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
><http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: 
>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
><http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] supporting Go

[Dieterly, Deklan]

Python 2.x will not be supported for much longer, and let¹s face it,
Python is easy, but it just does not scale. Nor does Python have the
performance characteristics that large, distributed systems require. Maybe
Java could replace Python in OpenStack as the workhorse language.
-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




On 5/13/16, 7:00 PM, "Adam Young" <ayo...@redhat.com> wrote:

>On 05/13/2016 08:21 PM, Dieterly, Deklan wrote:
>> If we allow Go, then we should also consider allowing JVM based
>>languages.
>Nope.  Don't get me wrong, I've written more than my fair share of Java
>in my career, and I like it, and I miss automated refactoring and real
>threads.  I have nothing against Java (I know a lot of you do).
>
>Java fills the same niche as Python.  We already have one of those, and
>its very nice (according to John Cleese).
>
>So, what I think we are really saying here is "what is our Native
>extension story going to be? Is it the traditional native languages, or
>is it something new that has learned from them?"
>
>Go is a complement to Python to fill in the native stuff.  The
>alternative is C or C++.  Ok Flapper, or Rust.
>
>This is coming from someone  that has done Kernel stuff.  I did C++ in
>both the Windows and Linux worlds.  I've written inversion of control
>stuff in C++ template metaprogramming.  I am not personally afraid of
>writing code in either language. But I don't want to inflict that on
>OpenStack.  Its a question of reducing complexity, not increasing it.
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] supporting Go

[Dieterly, Deklan]

If we allow Go, then we should also consider allowing JVM based languages.
-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




On 5/13/16, 2:10 PM, "Adam Young"  wrote:

>Can we just up and support Go, please?  I'm a C++ and C buff, but I
>would not inflict either of those on other people, nor would I want to
>support their code. Go is designed to be native but readable/writable.
>
>
>There is nothing perfect in this world.
>
>Python for most things.
>Javascript for web out of necessity
>Go for native tuning.
>
>Yes, Flapper, I like Rust, too, but we have to pick something, and I am
>not the one trying to code this.
>
>It makes sense. Go is already packaged for Fedora and Debian.  We can
>deal with it.
>
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Freezer] Replace Gnu Tar with DAR

[Dieterly, Deklan]

When using incremental backups, tar will not handle removing a dir and
then renaming another dir to the removed dir.


dek@dek-HP-Z620-Workstation:~/backup-test$ tar --extract
--listed-incrementa=/dev/null --file backup.2.tar
tar: Cannot rename Œbackup/dir1¹ to Œbackup/dir2¹: Directory not empty
tar: Exiting with failure status due to previous errors



Here are the steps to reproduce.

 1845  mkdir backup
 1846  mkdir backup/dir1
 1847  mkdir backup/dir2
 1848  echo "aa" > backup/dir1/dir1-file1
 1849  echo "aa" > backup/dir2/dir2-file1
 1852  tar --create --file=backup.tar --listed-incremental=./listed-incr
backup
 1854  rm -rf backup/dir2
 1855  mv backup/dir1 backup/dir2
 1856  tar --create --file=backup.2.tar --listed-incremental=./listed-incr
backup
 1859  tar --extract --listed-incrementa=/dev/null --file backup.tar
 1861  tar --extract --listed-incrementa=/dev/null --file backup.2.tar


This seems to be a well known, long-standing issue with tar.
-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




On 5/13/16, 4:33 PM, "Fox, Kevin M" <kevin@pnnl.gov> wrote:

>Whats the issue?
>____
>From: Dieterly, Deklan [deklan.diete...@hpe.com]
>Sent: Friday, May 13, 2016 3:07 PM
>To: openstack-dev@lists.openstack.org
>Subject: [openstack-dev] [Freezer] Replace Gnu Tar with DAR
>
>Does anybody see any issues if Freezer used DAR instead of Gnu Tar? DAR
>seems to handle a particular use case that Freezer has while Gnu Tar does
>not.
>--
>Deklan Dieterly
>
>Senior Systems Software Engineer
>HPE
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Freezer] Replace Gnu Tar with DAR

[Dieterly, Deklan]

Does anybody see any issues if Freezer used DAR instead of Gnu Tar? DAR
seems to handle a particular use case that Freezer has while Gnu Tar does
not.
-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [freezer] Proposing Saad Zaher for freezer core

[Dieterly, Deklan]

+1 on Saad Zaher (szaher) being a core member

-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




On 4/14/16, 8:23 AM, "Ramirez Garcia, Guillermo"
 wrote:

>+1 on Saad Zaher (szaher) being a core member
>
>Regards 
>GRG
>
>From: Mathieu, Pierre-Arthur
>Sent: Thursday, April 14, 2016 2:20 PM
>To: openstack-dev@lists.openstack.org
>Subject: [openstack-dev]  [freezer] Proposing Saad Zaher for freezer core
>
>Hello,
>
>I would like to propose that we make Saad Zaher (szaher) core on freezer.
>He has been a highly valuable developper for the past few month, mainly
>working on integrating oslo component into freezer.
>He has also been helping a lot with feature testing.
>
>His work can be found here: [1]
>
>Unless there is a disagreement I plan to make Saad core by the end of the
>week.
>
>Thanks
>- Pierre
>
>[1] https://review.openstack.org/#/q/owner:%22Saad+Zaher%22
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [freezer] Proposing Saad Zaher for freezer core

[Dieterly, Deklan]

+1 on Saad Zaher (szaher) being a core member

-- 
Deklan Dieterly

Senior Systems Software Engineer
HPE




On 4/14/16, 8:23 AM, "Ramirez Garcia, Guillermo"
 wrote:

>+1 on Saad Zaher (szaher) being a core member
>
>Regards 
>GRG
>
>From: Mathieu, Pierre-Arthur
>Sent: Thursday, April 14, 2016 2:20 PM
>To: openstack-dev@lists.openstack.org
>Subject: [openstack-dev]  [freezer] Proposing Saad Zaher for freezer core
>
>Hello,
>
>I would like to propose that we make Saad Zaher (szaher) core on freezer.
>He has been a highly valuable developper for the past few month, mainly
>working on integrating oslo component into freezer.
>He has also been helping a lot with feature testing.
>
>His work can be found here: [1]
>
>Unless there is a disagreement I plan to make Saad core by the end of the
>week.
>
>Thanks
>- Pierre
>
>[1] https://review.openstack.org/#/q/owner:%22Saad+Zaher%22
>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [monasca] [java]

[Dieterly, Deklan]

The Monasca project currently has three major components written in Java. 
Monasca-persister, monasca-thresh, and monasca-api. These components work with 
Influxdb 0.9.0 and Vertica 7.1. They integrate with Kafka and MySQL. The 
monasca team is currently bringing the Python versions of these components up 
to parity with their Java counterparts. This effort is being undertaken because 
there seems to be considerable friction in introducing Java components into the 
OpenStack community. At this point, Id like to test the waters a bit and 
determine what the larger community’s reaction to having these components 
remain in Java would be. Would there be a general acceptance or would there be 
a visceral rejection? Is the issue more of integration with existing CI/CD 
architecture or is there more of a cultural issue?

The arguments for Java are non-trivial. Monasca has requirements for very high 
throughput. Furthermore, integration with Kafka is better supported with 
Kafka's Java libraries.

We’ve seen that Swift has introduced components in Go. So, this looks like a 
precedent for allowing other languages where deemed appropriate. Before we 
spend many man-hours hacking on the Python components, it seems reasonable to 
determine if there really exists a reason to do so. I’m interested in 
soliciting any feedback from the community be it pleasant or unpleasant.

Thanks.

—
Deklan Dieterly
Software Engineer
HP

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [monasca] [java]

[Dieterly, Deklan]

Thanks, Kevin.

Performance is critical. At this point, we are trying to do 100K
measurements per second.

Yea, Vertica is not open source. Monasca uses either Vertica OR Influxdb
as the backend DB. You get to decide what you want.

Zookeeper is used by Kafka for distributed synchronization and is very
well regarded in the internet-applications realm.

It looks like what you are saying is that the issue goes beyond just Java
vs Python; it¹s an ops issue. There may be issues with supporting Kafka
and Influxdb. That¹s good feedback.

Interesting, at the 2014 Summit in Atlanta, the some members of the
community heavily lobbied for Influxdb. We¹ve seen perf problems with
MySQL in the Ceilometer Project and to wanted to avoid that by using a
scalable open source DB for the backend.

-- 
Deklan Dieterly

Software Engineer
HP




On 5/14/15, 10:34 AM, Fox, Kevin M kevin@pnnl.gov wrote:

The open source version of java is much better off then it use to be, so
I'd say its not out of the question any more. My preference is still
python whenever possible since it tends to be much easer to debug/patch
in the field. Performance critical stuff is another matter.

I would recommend very strongly considering it from the standpoint of
what distro's are willing to support, and how much additional
learning/operations work you are asking of ops to perform though.
OpenStack already pushes an enormous amount of learning onto the ops
folks. This will make or break the project.

yum list | grep -i influxdb | wc -l
0

hmm...
the rpm from the website looks very unusual... The distro folks wont
support a package that looks like that. My gut reaction looking at it as
an op is to wince and hope I don't have to install it. If I were, I'd
have to carefully pull it apart to figure out how to support it long
term. Definitely not a rpm -Uvh and forget.

Vertica doesn't look to be Open Source?

Kafka yet another messaging system... It might be needed, but its yet
another thing for ops to figure out how to deal with. The quickstart says
Kafka needs Zookeeper. Now yet another dependency for an op to deal with.
What does ZooKeeper give that Pacemaker (already used in a lot of clouds)
doesn't?

I might like to deploy Monasca here some day, but it looks like it will
take a large amount of work for me to do so, relative to all the other
OpenStack components I want to install, so I probably cant for a while
because of some of these design decisions.

Thanks,
Kevin


From: Dieterly, Deklan [deklan.diete...@hp.com]
Sent: Thursday, May 14, 2015 8:29 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev]  [monasca] [java]

The Monasca project currently has three major components written in Java.
Monasca-persister, monasca-thresh, and monasca-api. These components work
with Influxdb 0.9.0 and Vertica 7.1. They integrate with Kafka and MySQL.
The monasca team is currently bringing the Python versions of these
components up to parity with their Java counterparts. This effort is
being undertaken because there seems to be considerable friction in
introducing Java components into the OpenStack community. At this point,
Id like to test the waters a bit and determine what the larger
community¹s reaction to having these components remain in Java would be.
Would there be a general acceptance or would there be a visceral
rejection? Is the issue more of integration with existing CI/CD
architecture or is there more of a cultural issue?

The arguments for Java are non-trivial. Monasca has requirements for very
high throughput. Furthermore, integration with Kafka is better supported
with Kafka's Java libraries.

We¹ve seen that Swift has introduced components in Go. So, this looks
like a precedent for allowing other languages where deemed appropriate.
Before we spend many man-hours hacking on the Python components, it seems
reasonable to determine if there really exists a reason to do so. I¹m
interested in soliciting any feedback from the community be it pleasant
or unpleasant.

Thanks.

‹
Deklan Dieterly
Software Engineer
HP

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [new][cloudpulse] Announcing a project to HealthCheck OpenStack deployments

[Dieterly, Deklan]

How is the different/same as Monasca?

Regards.
--
Deklan Dieterly
Hewlett-Packard Company
Sr. Systems Software Engineer
HP Cloud






On 5/12/15, 11:48 AM, Jay Pipes jaypi...@gmail.com wrote:

For operators:

* Nagios
* Icinga
* Zabbix

installed on baremetal machines deployed with the OpenStack and other
infrastructure services.

For tenants:

* Nagios
* Icinga
* Zabbix

installed on their VMs.

Why are we re-inventing excellent open-source implementations of
monitoring systems that have been around for over a decade?

Best,
-jay

p.s. Sorry for top-posting.

On 05/12/2015 01:20 PM, Vinod Pandarinathan (vpandari) wrote:
 Hello,

I'm pleased to announce the development of a new project called
 CloudPulse.  CloudPulse provides Openstack
 health-checking services to both operators, tenants, and applications.
 This project will begin as
 a StackForge project based upon an empty cookiecutter[1] repo.  The
 repos to work in are:
 Server: https://github.com/stackforge/cloudpulse
 Client: https://github.com/stackforge/python-cloudpulseclient

 Please join us via iRC on #openstack-cloudpulse on freenode.

 I am holding a doodle poll to select times for our first meeting the
 week after summit.  This doodle poll will close May 24th and meeting
 times will be announced on the mailing list at that time.  At our first
 IRC meeting,
 we will draft additional core team members, so if your interested in
 joining a fresh new development effort, please attend our first meeting.
 Please take a moment if your interested in CloudPulse to fill out the
 doodle poll here:

 https://doodle.com/kcpvzy8kfrxe6rvb

 The initial core team is composed of
 Ajay Kalambur,
 Behzad Dastur, Ian Wells, Pradeep chandrasekhar, Steven DakeandVinod
 Pandarinathan.
 I expect more members to join during our initial meeting.

   A little bit about CloudPulse:
   Cloud operators need notification of OpenStack failures before a
 customer reports the failure. Cloud operators can then take timely
 corrective actions with minimal disruption to applications.  Many cloud
 applications, including
 those I am interested in (NFV) have very stringent service level
 agreements.  Loss of service can trigger contractual
 costs associated with the service.  Application high availability
 requires an operational OpenStack Cloud, and the reality
 is that occascionally OpenStack clouds fail in some mysterious ways.
 This project intends to identify when those failures
 occur so corrective actions may be taken by operators, tenants, and the
 applications themselves.

 OpenStack is considered healthy when OpenStack API services respond
 appropriately.  Further OpenStack is
 healthy when network traffic can be sent between the tenant networks and
 can access the Internet.  Finally OpenStack
 is healthy when all infrastructure cluster elements are in an
 operational state.

 For information about blueprints check out:
 https://blueprints.launchpad.net/cloudpulse
 https://blueprints.launchpad.net/python-cloudpulseclient

 For more details, check out our Wiki:
 https://wiki.openstack.org/wiki/Cloudpulse

 Plase join the CloudPulse team in designing and implementing a
 world-class Carrier Grade system for checking
 the health of OpenStack clouds.  We look forward to seeing you on IRC on
 #openstack-cloudpulse.

 Regards,
 Vinod Pandarinathan
 [1] https://github.com/openstack-dev/cookiecutter



 
_
_
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

[Dieterly, Deklan]

Given Python’s inherent inability to scale (GIL) relative to other 
languages/platforms, have there been any serious discussions on allowing other 
more scalable languages into the OpenStack ecosystem when 
concurrency/scalability is paramount?

Regards.
--
Deklan Dieterly
Hewlett-Packard Company
Sr. Systems Software Engineer
HP Cloud


From: Eugene Nikanorov enikano...@mirantis.commailto:enikano...@mirantis.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Monday, May 11, 2015 at 6:30 PM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

 All production Openstack applications today are fully serialized to only be 
 able to emit a single query to the database at a time;
True. That's why any deployment configures tons (tens) of workers of any 
significant service.

  When I talk about moving to threads, this is not a won't help or hurt kind 
 of issue, at the moment it's a change that will immediately allow massive 
 improvement to the performance of all Openstack applications instantly.
Not sure If it will give much benefit over separate processes.
I guess we don't configure many worker for gate testing (at least, neutron 
still doesn't do it), so there could be an improvement, but I guess to enable 
multithreading we would need to fix the same issues that prevented us from 
configuring multiple workers in the gate, plus possibly more.

 We need to change the DB library or dump eventlet.
I'm +1 for the 1st option.

Other option, which is multithreading will most certainly bring concurrency 
issues other than database.

Thanks,
Eugene.


On Mon, May 11, 2015 at 4:46 PM, Boris Pavlovic 
bo...@pavlovic.memailto:bo...@pavlovic.me wrote:
Mike,

Thank you for saying all that you said above.

Best regards,
Boris Pavlovic

On Tue, May 12, 2015 at 2:35 AM, Clint Byrum 
cl...@fewbar.commailto:cl...@fewbar.com wrote:
Excerpts from Mike Bayer's message of 2015-05-11 15:44:30 -0700:

 On 5/11/15 5:25 PM, Robert Collins wrote:
 
  Details: Skip over this bit if you know it all already.
 
  The GIL plays a big factor here: if you want to scale the amount of
  CPU available to a Python service, you have two routes:
  A) move work to a different process through some RPC - be that DB's
  using SQL, other services using oslo.messaging or HTTP - whatever.
  B) use C extensions to perform work in threads - e.g. openssl context
  processing.
 
  To increase concurrency you can use threads, eventlet, asyncio,
  twisted etc - because within a single process *all* Python bytecode
  execution happens inside the GIL lock, so you get at most one CPU for
  a CPU bound workload. For an IO bound workload, you can fit more work
  in by context switching within that one CPU capacity. And - the GIL is
  a poor scheduler, so at the limit - an IO bound workload where the IO
  backend has more capacity than we have CPU to consume it within our
  process, you will run into priority inversion and other problems.
  [This varies by Python release too].
 
  request_duration = time_in_cpu + time_blocked
  request_cpu_utilisation = time_in_cpu/request_duration
  cpu_utilisation = concurrency * request_cpu_utilisation
 
  Assuming that we don't want any one process to spend a lot of time at
  100% - to avoid such at-the-limit issues, lets pick say 80%
  utilisation, or a safety factor of 0.2. If a single request consumes
  50% of its duration waiting on IO, and 50% of its duration executing
  bytecode, we can only run one such request concurrently without
  hitting 100% utilisations. (2*0.5 CPU == 1). For a request that spends
  75% of its duration waiting on IO and 25% on CPU, we can run 3 such
  requests concurrently without exceeding our target of 80% utilisation:
  (3*0.25=0.75).
 
  What we have today in our standard architecture for OpenStack is
  optimised for IO bound workloads: waiting on the
  network/subprocesses/disk/libvirt etc. Running high numbers of
  eventlet handlers in a single process only works when the majority of
  the work being done by a handler is IO.

 Everything stated here is great, however in our situation there is one
 unfortunate fact which renders it completely incorrect at the moment.
 I'm still puzzled why we are getting into deep think sessions about the
 vagaries of the GIL and async when there is essentially a full-on
 red-alert performance blocker rendering all of this discussion useless,
 so I must again remind us: what we have *today* in Openstack is *as
 completely un-optimized as you can possibly be*.

 The most GIL-heavy nightmare CPU bound task you can imagine running on
 25 threads on a ten year old Pentium will run better than the Openstack
 we have today, because we are running a C-based, non-eventlet patched DB
 library within a single