[openstack-dev] [quantum] Deadlock on quantum port-create
I'm moving a thread we had with some vmware guys to this list to make it public. We had a problem with quantum deadlocking when it got several requests in quick succession. Aaron suggested we set sql_dbpool_enable = True. We did and it seemed to resolve our issue. What are the downsides of turning on sql_dbpool_enable? Should it be on by default? Thanks, Jay We are currently experience the following problem in our environment: issuing 5 'quantum port-create' commands in parallel effectively deadlocks quantum: $ for n in $(seq 5); do echo 'quantum --insecure port-create stage-net1'; done | parallel An unknown exception occurred. Request Failed: internal server error while processing your request. An unexpected error occurred in the NVP Plugin:Unable to get logical switches On Jun 21, 2013, at 9:36 AM, Aaron Rosen aro...@vmware.com wrote: We've encountered this issue as well. I'd try enabling: # Enable the use of eventlet's db_pool for MySQL. The flags sql_min_pool_size, # sql_max_pool_size and sql_idle_timeout are relevant only if this is enabled. sql_dbpool_enable = True in nvp.ini to see if that helps at all. In our internal cloud we removed the creations of the lports in nvp from the transaction. Salvatore is working on an async back-end to the plugin that will solve this and improve the plugin performance. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Python overhead for rootwrap
I haven't closely looked at rootwrap, but it seems to me that you could use the rootwrap config files to generate a gigantic sudoers config file which would not necessarily be human readable. That would have the flexibility and maintainability of rootwrap with the speed and audibility sudo. On Thu, Jul 25, 2013 at 1:21 PM, Joe Gordon joe.gord...@gmail.com wrote: Hi All, We have recently hit some performance issues with nova-network. It turns out the root cause of this was we do roughly 20 rootwrapped shell commands, many inside of global locks. (https://bugs.launchpad.net/oslo/+bug/1199433 ) It turns out starting python itself, has a fairly significant overhead when compared to the run time of many of the binary commands we execute. For example: $ time python -c print 'test' test real 0m0.023s user 0m0.016s sys 0m0.004s $ time ip a ... real 0m0.003s user 0m0.000s sys 0m0.000s While we have removed the extra overhead of using entry points, we are now hitting the overhead of just shelling out to python. While there are many possible ways to reduce this issue, such as reducing the number of rootwrapped calls and making locks finer grain, I think its worth exploring alternates to the current rootwrap model. Any ideas? I am sending this email out to get the discussion started. best, Joe Gordon ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
On Tue, Aug 6, 2013 at 8:35 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: I think jay your usage also was before anvil started to build all the *missing* dependencies automatically (something u inspired me to get going in the first place) so hopefully said updates to rhel.yaml are only now needed for exceptions and not the common path :) +1 I'm glad to hear that. I don't think any of the problems I saw with Anvil were intractable, but I do think packaging up the venv is much less error prone. My major complaints are issues related to yum. First off, yum can't resolve a version range (see my message to the yum list: http://lists.baseurl.org/pipermail/yum/2013-February/023922.html) which seems like a pretty basic task for a package manager. Also, Yum seems to do dependency resolution in a single pass. So yum can't provide an install solution for cases like this even when a valid one exists: package foo depends on pkgdep = 2.0 package bar depends on pkgdep = 1.0, = 2.0 pkgdep 2.1 and 2.0 are available for install sudo yum install foo when none of those three packages are installed results in an error. Yum tries to install pkgdep 2.1 and then gives up when it sees foo doesn't work with pkgdep 2.1 rather than doing a second pass at building a dependency graph and finding that pkgdep 2.0 would satisfy all dependencies. Cases like these come up regularly with the 100+ python packages that openstack depends on (in requirements.txt) when building packages continuously. I came to the conclusion that yum is bad software and I should avoid it globbing up most everything into one big rpm. Three months into making the switch I'm happy with that decision. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Python overhead for rootwrap
Personally I'm of the opinion that from an architectural POV, use of either rootwrap or sudo is a bad solution, so arguing about which is better is really missing the bigger picture. In Linux, there has been a move away from use of sudo or similar approaches, towards the idea of having privileged separated services. So if you wanted todo stuff related to storage, you'd have some small daemon running privilegd, which exposed APIs over DBus, which the non-privileged thing would call to make storage changes. Operations exposed by the service would have access control configured via something like PolicyKit, and/or SELinux/AppArmour. The more I think about this problem the more I'm convinced that rootwrap simply exists to work around a fundamental design flaw in most (all?) of the components: a single, threaded, process with poor job/workflow mnemonics. If the architecture was such that every operation was a task which could be carried out by any process (even a process running on a different host) and each process publishes job types that it can do, then you could do proper isolation for just those processes that need to do privileged tasks. https://wiki.openstack.org/wiki/TaskFlow looks to be headed in the right direction, but I worry it's got a very long road ahead. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs distro packages
On Tue, Aug 6, 2013 at 12:00 PM, Monty Taylor mord...@inaugust.com wrote: On 08/06/2013 02:44 PM, Mate Lakat wrote: I would say, use a separated virtual environment in devstack - without the --system-site-packages switch, of course, and set it up as a user. Install the packages that are needed in order to be able to pip install them (like libxslt-dev). It's a development environment. I think my email is equivalent to a +1 to (Monty's change + virtualenv). YUP - except we can't do it without system-site-packages, because of nova and libvirt. We should still be ok though with a single venv, as long as all of the installs into the venv use the -U flag to pip, which will override the system env with teh venv one. I'm building my virtualenv without system site-packages. It's super lame, but my nova rpm has a dep on a specific version of libvirt-python and then includes these symlinks: ln -s /usr/lib64/python2.6/site-packages/libvirt.py $(VENV)/lib/python2.6/site-packages/ ln -s /usr/lib64/python2.6/site-packages/libvirt_qemu.py $(VENV)/lib/python2.6/site-packages/ ln -s /usr/lib64/python2.6/site-packages/libvirtmod.so $(VENV)/lib/python2.6/site-packages/ ln -s /usr/lib64/python2.6/site-packages/libvirtmod_qemu.so $(VENV)/lib/python2.6/site-packages/ Why isn't libvirt-python on pypi? AFAICT, nothing is stopping us from uploading it. Maybe we should just stick it on there and this issue will be resolved once and for all. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] General Question about CentOS
On Fri, Aug 16, 2013 at 2:51 PM, Miller, Mark M (EB SW Cloud - RD - Corvallis) mark.m.mil...@hp.com wrote: Is OpenStack supported on CentOS running Python 2.6? Oh, I forgot to mention, keystone's py2.6 support seems to currently be broken because of this bug: https://bugs.launchpad.net/keystone/+bug/1213284/ Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Code review study
On Wed, Aug 14, 2013 at 7:12 PM, Robert Collins robe...@robertcollins.netwrote: Note specifically the citation of 200-400 lines as the knee of the review effectiveness curve: that's lower than I thought - I thought 200 was clearly fine - but no. This is really interesting. I wish they would have explicitly defined lines of code. Is that git show |wc -l? Just the new lines which were added? The sum of the lines changed, removed and added? You can get vastly different numbers depending on how you count it. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Code review study
On Tue, Aug 20, 2013 at 8:02 AM, Mark McLoughlin mar...@redhat.com wrote: On Tue, 2013-08-20 at 11:26 +0100, Mark McLoughlin wrote: The full study is here: http://support.smartbear.com/resources/cc/book/code-review-cisco-case-study.pdf I can't find the data they based their numbers on, nor their definition for a line of code, so I feel like I have to take that study with a grain of salt. We should not enforce rules like this blindly. I agree with the sentiment: less complex patches are easier to review. Reviewers should use their judgement and push back on complex patches to be split up. Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Cookiecutter repo for ease in making new projects
On Thu, Sep 12, 2013 at 10:08 PM, Monty Taylor mord...@inaugust.com wrote: And boom, you'll have a directory all set up with your new project Awesome. I tried it and ran into a couple of small issues. I don't see a launchpad yet, so I'm not sure where to report bugs. Something is stripping all the new lines at EOF, so flake8 fails. Also, this is annoying because vim's default is to add one when it doesn't exist. Also, foo/test/__init__.py needs to call super(TestCase, self).setUp() otherwise you get an error that is something like this when you run tox: ValueError: TestCase.setUp was not called. Have you upcalled all the way up the hierarchy from your setUp? e.g. Call super(TestFoo, self).setUp() from your setUp(). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
