[openstack-dev] sqlalchemy-migrate 0.9.1 release impending - oslo.db and Py3 support
We need a new release of sqlalchemy-migrate (0.9.1) to make commit 93efb62fd100 available, as a step toward fixing some oslo.db issues with sqlite and its limited/optional support for unique constraints. This release will also be the first one with commit a03b141a954, for Python 3 support. I plan on doing the release tomorrow, unless someone objects. Thanks. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova client] gate-python-novaclient-pypy
10_08_19_39_568> | /usr/lib/pypy/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'zip_safe' 2014-03-10 08:19:39.568 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_568> | 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | warnings.warn(msg) 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> |or: -c --help [cmd1 cmd2 ...] 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> |or: -c --help-commands 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> |or: -c cmd --help 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | 2014-03-10 08:19:39.569 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_569> | 2014-03-10 08:19:39.570 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_570> | 2014-03-10 08:19:39.570 <http://logs.openstack.org/74/67074/8/check/gate-python-novaclient-pypy/44c12e7/console.html#_2014-03-10_08_19_39_570> | error: option --single-version-externally-managed not recognized Any ideas? It looks like the pypy gate is uninstalling setuptools, *not* subsequently installing setuptools or distribute, and the mimeparse package's setup.py includes options (entry_points, zip_safe) that require setuptools or distribute (and cause this setup.py to not work with distutils). mimeparse installs fine in pypy on one of my local computers. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][db] Thoughts on making instances.uuid non-nullable?
On 03/10/2014 10:11 AM, Matt Riedemann wrote: On 3/9/2014 9:18 PM, Jay Pipes wrote: On Mon, 2014-03-10 at 10:05 +0800, ChangBo Guo wrote: 2014-03-10 4:47 GMT+08:00 Jay Pipes : > 3. This would make the instances and shadow_instances tables have > different schemas, i.e. instances.uuid would be nullable=False in > instances but nullable=True in shadow_instances. Maybe this doesn't matter. No, I don't think this matters much, to be honest. I'm not entirely sure what the long-term purpose of the shadow tables are in Nova -- perhaps someone could clue me in to whether the plan is to keep them around? As I know the tables shadow_* are used by command ' nova-manage db archive_deleted_rows' , which moves records with "deleted=True" to table shadow_* . That means these tables are used by other process, So, I think we need other tables to store the old records in your migration. Yeah, that's what I understood the shadow tables were used for, I just didn't know what the long-term future of these tables was... curious if I think Joe Gordon was working on something in the hopes of eventually killing the shadow tables but I can't remember exactly what that was now. https://blueprints.launchpad.net/nova/+spec/db-purge-engine It didn't get done for Icehouse. Maybe it will for Juno. Everyone agrees that db-archiving is not ideal, but we probably want something better in place before we rip it out. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate release impending
On 02/26/2014 11:24 AM, David Ripton wrote: I'd like to release a new version of sqlalchemy-migrate in the next couple of days. The only major new feature is DB2 support. If anyone thinks this is a bad time, please let me know. sqlalchemy-migrate 0.9 is now available via pip. Notable new features include DB2 support and compatibility with SQLAlchemy-0.9. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate release impending
On 02/28/2014 04:22 PM, Matt Riedemann wrote: On 2/26/2014 11:34 AM, Sean Dague wrote: On 02/26/2014 11:24 AM, David Ripton wrote: I'd like to release a new version of sqlalchemy-migrate in the next couple of days. The only major new feature is DB2 support. If anyone thinks this is a bad time, please let me know. So it would be nice if someone could actually work through the 0.9 sqla support, because I think it's basically just a change in quoting behavior that's left (mostly where quoting gets called) - https://review.openstack.org/#/c/66156/ Thomas Goirand has a bunch of new reviews up in this area. Hopefully we can release a sqlalchemy-migrate 0.9 with support for sqlalchemy-0.9 soon. Looks like the 0.8.3 tag is up so it's just a matter of time before it shows up on pypi? https://review.openstack.org/gitweb?p=stackforge/sqlalchemy-migrate.git;a=commit;h=21fcdad0f485437d010e5743626c63ab3acdaec5 I pushed a non-signed 0.8.3 tag, and the release machinery only cares about signed tags. Oops. Anyway, I then pushed a signed 0.8.4 tag, and the new release failed tests because of an unconditional import of ibm_db_sa, a module which did not exist on the test machines. (It was mentioned in test-requirements.txt, but devstack / devstack-gate don't actually use that file.) So we did a 0.8.5 release without DB2 support, so that the latest/default version of sqlalchemy-migrate would work again. There are patches in progress to fix the DB2 support and to add a tempest job for sqlalchemy-migrate so we're more likely to notice problems before doing a release. When that work is all done, we can release 0.8.6. (Or maybe skip 0.8.6 and just release 0.9, if the sqlalchemy 0.9 compatibility work is all done by then.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] WARNING: ... This application has not enabled MySQL traditional mode, which means silent data corruption may occur - real issue?
On 03/02/2014 09:42 PM, Sean Dague wrote: Coming in at slightly less than 1 million log lines in the last 7 days: http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVGhpcyBhcHBsaWNhdGlvbiBoYXMgbm90IGVuYWJsZWQgTXlTUUwgdHJhZGl0aW9uYWwgbW9kZSwgd2hpY2ggbWVhbnMgc2lsZW50IGRhdGEgY29ycnVwdGlvbiBtYXkgb2NjdXJcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MzgxNDExMzcyOX0= "This application has not enabled MySQL traditional mode, which means silent data corruption may occur" This is being generated by *.openstack.common.db.sqlalchemy.session in at least nova, glance, neutron, heat, ironic, and savana http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiVGhpcyBhcHBsaWNhdGlvbiBoYXMgbm90IGVuYWJsZWQgTXlTUUwgdHJhZGl0aW9uYWwgbW9kZSwgd2hpY2ggbWVhbnMgc2lsZW50IGRhdGEgY29ycnVwdGlvbiBtYXkgb2NjdXJcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MzgxNDExMzcyOSwibW9kZSI6InNjb3JlIiwiYW5hbHl6ZV9maWVsZCI6Im1vZHVsZSJ9 At any rate, it would be good if someone that understood the details here could weigh in about whether is this really a true WARNING that needs to be fixed or if it's not, and just needs to be silenced. oslo-incubator commit a5841668 just got in, and made traditional the default. Of course it'll take a while for that to be copied everywhere. Once it is, this problem should mostly go away. But, yeah, we should probably remove that warning, since if we let users override the default, we shouldn't fill their log partition for doing so. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Toward SQLAlchemy 0.9.x compatibility everywhere for Icehouse
On 03/02/2014 03:12 AM, Thomas Goirand wrote: On 03/02/2014 12:03 PM, Thomas Goirand wrote: I hope to get support from core reviewers here, so that we can fix-up the SQLA 0.9.x compat ASAP, preferably before b3. Of course, I'll make sure all we do works with both 0.8 and 0.9 version of SQLA. Is there anyone still running with the old 0.7? If yes, then we can try to continue validating OpenStack against it as well. I think we should continue supporting SQLAlchemy 0.7 for a bit longer, as that's what some distributions still package. (Of course it's possible to work around that by pulling from pip instead, but not everyone will.) FYI, the below patches need review: SQLAlchemy-migrate: https://review.openstack.org/#/c/77387/ https://review.openstack.org/#/c/77388/ https://review.openstack.org/#/c/77396/ https://review.openstack.org/#/c/77397/ I'll review all these today. BTW, I just pushed sqlalchemy-migrate 0.8.4 with DB2 support. I agree that we should push a 0.9 as soon as all the SQLAlchemy-0.9 compatibility stuff is in and working. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Hacking and PEP 257: Extra blank line at end of multi-line docstring
On 02/26/2014 11:40 AM, Joe Gordon wrote: This is missing the point about manually enforcing style. If you pass the 'pep8' job there is no need to change any style. In a perfect world, yes. In the real world, there are several things in PEP8 or our project guidelines that the tools don't enforce perfectly. I think it's fine for human reviewers to point such things out. (And then submit a patch to hacking to avoid the need to do so in the future.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] sqlalchemy-migrate release impending
I'd like to release a new version of sqlalchemy-migrate in the next couple of days. The only major new feature is DB2 support. If anyone thinks this is a bad time, please let me know. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack][nova][social-apects] Social aspects shouldn't impact on dev process
On 11/20/2013 02:06 AM, Boris Pavlovic wrote: I faced some social problems in community. We started working on purge engine for DB (before HK summit) This is very important, because at this moment we don't have any working way to purge DB... so admins should make it by hand. And we made this BP (in october) https://blueprints.launchpad.net/nova/+spec/db-purge-engine And made patch that makes this work. But only because our BP wasn't approved we got -2 from Joe Gordon. (https://review.openstack.org/#/c/51523/ ) And there was long discussion to remove this -2. And now after summit David Ripton made the similar BP (probably he didn't know): https://blueprints.launchpad.net/nova/+spec/db-purge2 That is already approved by Joe Gordon. (that already know that we are working on same problem) Why? (btw question about Purge Engine was raised by me on the summit and community accepted that) I discussed this with Boris on IRC yesterday. When I volunteered to write a DB purger at Summit, I wasn't aware that there was already one actively in progress. (So many patches around the end of Havana.) When I went to file a blueprint and noticed the existing db-purge blueprint, I saw that its patch had been -2'd and figured it was dead. But as long as Boris is working to actively improve that patch (he's on vacation now but said he'd probably have something on Monday), I won't submit a patch for the competing blueprint. Instead, I'll work to make sure Boris's code meets everyone's requirements (some that I got from Joe Gordon and Phil Day are mentioned in db-purge2), and when it does I'll withdraw the db-purge2 blueprint and retarget remove-db-archiving to depend on Boris's blueprint instead. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate needs a new release
On 11/15/2013 10:41 AM, David Ripton wrote: sqlalchemy-migrate-0.8.1 is now up on PyPI. Thanks fungi for kicking PyPI for me. So there was a hardcoded version number inside migrate/__init__.py, which broke things when I didn't manually update it to sync with the new release's version number. (DRY violation.) mordred landed a patch to make it dynamic so we don't need to remember to update it before every release in the future. sqlalchemy-migrate-0.8.2 is now on PyPI. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate needs a new release
On 11/14/2013 03:43 PM, David Ripton wrote: On 11/11/2013 03:35 PM, David Ripton wrote: I'll volunteer to do this release. I'll wait 24 hours from the timestamp of this email for input first. So, if anyone has opinions about the timing of this release, please speak up. (In particular, I'd like to do a release *before* Matt Riedermann's DB2 support patch https://review.openstack.org/#/c/55572/ lands, just in case it breaks anything. Of course we could do another release shortly after it gets in, to make folks who use DB2 happy.) Update: There's now a "0.8" tag in Git but that release failed to reach PyPI, so please ignore it. Thanks fungi and mordred for helping debug what went wrong. https://review.openstack.org/#/c/56449/ (a one-liner) should fix the problem. Once it gets approved, I will attempt to push "0.8.1". Update 2: sqlalchemy-migrate-0.8.1 is now up on PyPI. Thanks fungi for kicking PyPI for me. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate needs a new release
On 11/14/2013 03:55 PM, Matt Riedemann wrote: On 11/14/2013 2:43 PM, David Ripton wrote: On 11/11/2013 03:35 PM, David Ripton wrote: I'll volunteer to do this release. I'll wait 24 hours from the timestamp of this email for input first. So, if anyone has opinions about the timing of this release, please speak up. (In particular, I'd like to do a release *before* Matt Riedermann's DB2 support patch https://review.openstack.org/#/c/55572/ lands, just in case it breaks anything. Of course we could do another release shortly after it gets in, to make folks who use DB2 happy.) Update: There's now a "0.8" tag in Git but that release failed to reach PyPI, so please ignore it. Thanks fungi and mordred for helping debug what went wrong. https://review.openstack.org/#/c/56449/ (a one-liner) should fix the problem. Once it gets approved, I will attempt to push "0.8.1". Any particular reason to go with 0.8 rather than 0.7.3 as a bug fix release? New maintainers, 2 years since a release, SQLAlchemy 0.8 compatibility (we hope). Even in projects that use strict semantic versioning, 0.x just means "not ready yet." There are no stability guarantees until 1.0. http://semver.org/ , point 4 -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate needs a new release
On 11/11/2013 03:35 PM, David Ripton wrote: I'll volunteer to do this release. I'll wait 24 hours from the timestamp of this email for input first. So, if anyone has opinions about the timing of this release, please speak up. (In particular, I'd like to do a release *before* Matt Riedermann's DB2 support patch https://review.openstack.org/#/c/55572/ lands, just in case it breaks anything. Of course we could do another release shortly after it gets in, to make folks who use DB2 happy.) Update: There's now a "0.8" tag in Git but that release failed to reach PyPI, so please ignore it. Thanks fungi and mordred for helping debug what went wrong. https://review.openstack.org/#/c/56449/ (a one-liner) should fix the problem. Once it gets approved, I will attempt to push "0.8.1". -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Split of the openstack-dev list
On 11/14/2013 08:21 AM, Julien Danjou wrote: On Thu, Nov 14 2013, Thierry Carrez wrote: Thoughts ? I agree on the need to split, the traffic is getting huge. As I'd have to subscribe to both openstack-dev and stackforge-dev, that would not help me personally, but I think it can be an easy and first step. I don't think it's worth the bother. openstack-dev would still receive most of the traffic. Once you add back the traffic from people cross-posting, posting to the wrong list, yelling at people cross-posting or posting to the wrong list, etc. I'd expect openstack-dev's traffic to stay about the same. It'll just be one more list for most of us to subscribe to. The thing that would help with message volume would be splitting openstack-dev by subproject. (Except for those who would need to follow most of the projects, who would still get just as much mail plus the extra noise from people posting wrong.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [qa] Policy on spelling and grammar
On 11/11/2013 02:28 PM, Tim Bell wrote: As a speaker of the Queen's English, I find flavor to be incorrect. Does that mean I can -1 any patch that does not use flavour ? If the project had a policy that British English is the only acceptable spelling, yes. Since it doesn't, no. (Unless we did use "flavour" everywhere, at which point you could argue from consistency.) At CERN, we are working with 130 countries in a single community. The value of the contribution of non-english speakers far exceeds the occasional misunderstandings. Absolutely. Nobody is saying that a -1 for spelling or grammar negates the value of the contribution, just that it needs a little tweak before it goes in. Giving grammar/spellings -1 excludes major sections of the community from contribution. No it does not. You give the -1 and explain exactly why, the contributor re-submits with the trivial fix, and you change your -1 to a +1. Just like with any other minor problems found in code review. As our aim is meritocracy (in python, computer architecture and design rather than spelling), I'd propose - If someone identifies a need for clarification/correction as part of a review, they also submit the replacement text rather than just -1. - The submitter incorporates that change into a patch Of course. This is what already happens almost all of the time IMX. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy-migrate needs a new release
On 11/11/2013 11:37 AM, Roman Podoliaka wrote: As you may know, in our global requirements list [1] we are currently depending on SQLAlchemy 0.7.x versions (which is 'old stable' branch and will be deprecated soon). This is mostly due to the fact, that the latest release of sqlalchemy-migrate from PyPi doesn't support SQLAlchemy 0.8.x+. At the same time, distros have been providing patches for fixing this incompatibility for a long time now. Moreover, those patches have been merged to sqlalchemy-migrate master too. As we are now maintaining sqlalchemy-migrate, we could make a new release of it. This would allow us to bump the version of SQLAlchemy release we are depending on (as soon as we fix all the bugs we have) and let distros maintainers stop carrying their own patches. This has been discussed at the design summit [2], so we just basically need a volunteer from [3] Gerrit ACL group to make a new release. Is sqlalchemy-migrate stable enough to make a new release? I think, yes. The commits we've merged since we adopted this library, only fix a few issues with SQLAlchemy 0.8.x compatibility and enable running of tests (we are currently testing all new changes on py26/py27, SQLAlchemy 0.7.x/0.8.x, SQLite/MySQL/PostgreSQL). Who wants to help? :) Thanks, Roman [1] https://github.com/openstack/requirements/blob/master/global-requirements.txt [2] https://etherpad.openstack.org/p/icehouse-oslo-db-migrations [3] https://review.openstack.org/#/admin/groups/186,members I'll volunteer to do this release. I'll wait 24 hours from the timestamp of this email for input first. So, if anyone has opinions about the timing of this release, please speak up. (In particular, I'd like to do a release *before* Matt Riedermann's DB2 support patch https://review.openstack.org/#/c/55572/ lands, just in case it breaks anything. Of course we could do another release shortly after it gets in, to make folks who use DB2 happy.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Bad review patterns
On 11/07/2013 07:54 PM, Sean Dague wrote: On 11/08/2013 01:37 AM, Pedro Roque Marques wrote: Radomir, An extra issue that i don't believe you've covered so far is about comment ownership. I've just read an email on the list that follows a pattern that i've heard many complaints about: -1 with a reasonable comment, submitter addresses the comment, reviewer never comes back. Reviewers do need to allocate time to come back and follow up on the answers to their comments. Perhaps there is an issue with the incentive system. You can earn karma by doing a code review... certainly you want to incentivise developers that help the project by improving the code quality. But if the incentive system allows for "drive by shooting" code reviews that can be a problem. It's not really an incentive system problem, this is some place where there are some gerrit limitations (especially when your list of reviewed code is long). Hopefully once we get a gerrit upgrade we can dashboard out some new items like that via the new rest API. I agree that reviewers could be doing better. But definitely also realize that part of this is just that there is *so* much code to review. Realize that most core reviewers aren't ignoring or failing to come back on patches intentionally. There is just *so* much of it. I feel guilty all the time by how big a review queue I have, but I also need a few hours a day not doing OpenStack (incredible to believe). This is where non core reviewers can really help in addressing the first couple of rounds of review to prune and improve the easy stuff. We're all in this together, Is there a way for Gerrit to only send email when action is required, rather than on any change to any review you've touched? If Gerrit sent less mail, it would be easier to treat its mails as a critical call to action to re-review. (There's probably a way to use fancy message filtering to accomplish this, but that would only work for people willing/able to set up such filtering.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Remove vim modelines?
On 10/24/2013 08:38 AM, Joe Gordon wrote: Why remove them? * Modelines aren't supported by default in debian or ubuntu due to security reasons: https://wiki.python.org/moin/Vim * Having modelines for vim means if someone wants we should support modelines for emacs (http://www.gnu.org/software/emacs/manual/html_mono/emacs.html#Specifying-File-Variables) etc. as well. And having a bunch of headers for different editors in each file seems like extra overhead. * There are other ways of making sure tabstop is set correctly for python files, see https://wiki.python.org/moin/Vim. I am a vIm user myself and have never used modelines. * We have vim modelines in only 828 out of 1213 python files in nova (68%), so if anyone is using modelines today, then it only works 68% of the time in nova * Why have the same config 828 times for one repo alone? This violates the DRY principle (Don't Repeat Yourself). Another +1 from a Vim user. These patches are No Fun to review, so anyone who wants these gone, please pitch in. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Hyper-V meeting Minutes
On 10/16/2013 08:59 AM, Alessandro Pilotti wrote: When somebody (especially a core reviewer) puts a -1 and a new patch is committed to address it, I noticed that other reviewers wait for the guy that put the -1 to say something before +1/+2 it. My feeling on this is that if somebody reviews a patch (positively or negatively) he/she should also keep on with it (in a timely manner) until it is merged or clearly stating that there's no interest in reviewing it further. This is especially true for core revs as other reviewers tend to be shy and avoid contradicting a core rev, generating further delays. What do you guys think? Yeah, it's no fun when someone gives you a -1 then goes away. But the people who do a lot of reviews do a lot of reviews, so they can't be immediately responsive to every change to every patch they've reviewed, or they'd never be able to do anything else. The fundamental problem is that the ratio of patches to reviewers, and especially patches to core reviewers, is too high. We either need people to submit fewer patches or do more reviewing. I'm tempted to submit a patch to next-review to give priority to patches from authors who do a lot of reviews. That would provide an incentive for everyone to review more. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Oslo] Change ListOpt and DictOpt default values
On 10/10/2013 09:45 AM, Ben Nemec wrote:
On 2013-10-10 07:40, Flavio Percoco wrote:
Greetings,
I'd like to propose to change both ListOpt and DictOpt default values
to [] and {} respectively. These values are, IMHO, saner defaults than
None for this 2 options and behavior won't be altered - unles `is not
None` is being used.
Since I may be missing some history, I'd like to ask if there's a
reason why None was kept as the default `default` value for this 2
options?
As mentioned above, this change may be backward incompatible in cases
like:
if conf.my_list_opt is None:
Does anyone if there are cases like this?
Also, I know it is possible to do:
cfg.ListOpt('option', default=[])
This is not terrible, TBH, but it doesn't feel right. I've made the
mistake to ignore the `default` keyword myself, although I know `[]`
is not the default option for `ListOpt`. As already said, I'd expect
`[]` to be the default, non-set value for `ListOpt`.
Thoughts?
Cheers,
FF
P.S: I'm not sure I'll make it to tomorrows meeting so, I starting the
discussion here made more sense.
Since this is technically an incompatible API change, would a major
version bump be needed for oslo.config if we did this? Maybe nobody's
relying on the existing behavior, but since oslo.config is a released
library its API is supposed to be stable.
+1. boto just broke our builds by making an incompatible API change in
version 2.14. We can't make every project in the world not do that, but
we sure should avoid doing it ourselves.
--
David Ripton Red Hat drip...@redhat.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] stalled bug fixes for vmware driver
On 09/20/2013 04:11 PM, Dan Wendlandt wrote: Its great that you have dashboards like this, very cool. The interesting thing here is that the patches I am talking about are not waiting on reviews in general, but rather core review. They have plenty of reviews from non-core folks who provide feedback (and they keep getting +1'd again as they are rebased every few days). Perhaps a good additional metric to track would be be items that have spent a lot of time without a negative review, but have not gotten any core reviews. I think that is the root of the issue in the case of the reviews I'm talking about. I feel the pain. Especially when you have a +2 but need to rebase before the second +2 comes in, and lose it. I just sent a pull request to next-review to add --onlyplusone and --onlyplustwo to next-review, to give core reviewers an easy way to focus on already-somewhat-vetted reviews, and leave the new reviews to non-core reviewers. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][oslo] mysql, sqlalchemy and sql_mode
On 09/11/2013 12:28 PM, Monty Taylor wrote: On 09/11/2013 11:09 AM, David Ripton wrote: On 09/11/2013 06:37 AM, Steven Hardy wrote: I'm investigating some issues, where data stored to a text column in mysql is silently truncated if it's too big. It appears that the default configuration of mysql, and the sessions established via sqlalchemy is to simply warn on truncation rather than raise an error. This seems to me to be almost never what you want, since on retrieval the data is corrupt and bad/unexpected stuff is likely. This AFAICT is a mysql specific issue[1], which can be resolved by setting sql_mode to "traditional"[2,3], after which an error is raised on truncation, allowing us to catch the error before the data is stored. My question is, how do other projects, or oslo.db, handle this atm? It seems we either have to make sure the DB enforces the schema/model, or validate every single value before attempting to store, which seems like an unreasonable burden given that the schema changes pretty regularly. Can any mysql, sqlalchemy and oslo.db experts pitch in with opinions on this? Nova has a PostgreSQL devstack gate, which occasionally catches errors that MySQL lets through. For example, https://bugs.launchpad.net/nova/+bug/1217167 Unfortunately we have some MySQL-only code, and PostgreSQL obviously can't catch such errors there. I think we should consider turning off auto-truncation for MySQL on our CI boxes. Should turn it off everywhere - same as how we auto-configure to use InnoDB and not MyISAM, we should definitely set strict sql_modes strings. There is not an operational concern - sql_modes affect app developers, of which we are they. :) If it's our DB, we can configure it however we want. If it's a user's DB, and it's potentially also used by other programs, then we need to be careful. We can set strict mode either globally for the DB server, or per-session. My gut says we should do it per-session, even though it's a bit annoying to run the code every time we start a session rather than once at setup, Just In Case someone is running OpenStack on a MySQL server that also does other things, and might not appreciate excessive global meddling. Anyway, I'll propose a patch for this in Icehouse. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [heat][oslo] mysql, sqlalchemy and sql_mode
On 09/11/2013 06:37 AM, Steven Hardy wrote: I'm investigating some issues, where data stored to a text column in mysql is silently truncated if it's too big. It appears that the default configuration of mysql, and the sessions established via sqlalchemy is to simply warn on truncation rather than raise an error. This seems to me to be almost never what you want, since on retrieval the data is corrupt and bad/unexpected stuff is likely. This AFAICT is a mysql specific issue[1], which can be resolved by setting sql_mode to "traditional"[2,3], after which an error is raised on truncation, allowing us to catch the error before the data is stored. My question is, how do other projects, or oslo.db, handle this atm? It seems we either have to make sure the DB enforces the schema/model, or validate every single value before attempting to store, which seems like an unreasonable burden given that the schema changes pretty regularly. Can any mysql, sqlalchemy and oslo.db experts pitch in with opinions on this? Nova has a PostgreSQL devstack gate, which occasionally catches errors that MySQL lets through. For example, https://bugs.launchpad.net/nova/+bug/1217167 Unfortunately we have some MySQL-only code, and PostgreSQL obviously can't catch such errors there. I think we should consider turning off auto-truncation for MySQL on our CI boxes. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Frustrations with review wait times
On 08/28/2013 05:10 AM, Daniel P. Berrange wrote: IOW we should prioritize review of work whose authors submitted earlier to encourage good practice with early submission. +1. Can we reconfigure Gerrit to show oldest first rather than newest first by default? (next-review does this. next-review is awesome. Everyone should try next-review. But we should try to make Gerrit do the right thing too, just in case some people prefer it to next-review.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][tempest][rdo] looking for opinions on change 43298
On 08/27/2013 03:52 PM, Matt Riedemann wrote: This change: _https://review.openstack.org/#/c/43298/_ Is attempting to fix a bug where a tempest test fails when "nova-manage --version" is different from "nova-manage version" when using a RHEL 6 installation rather than devstack. Pavel points out an RDO bug that was filed back in April to address the issue: _https://bugzilla.redhat.com/show_bug.cgi?id=952811_ That RDO bug hasn't gotten any attention though (I wasn't aware of it when I reported the launchpad bug). So my question is, is this worth changing in Tempest or should we expect that "nova-manage --version" will always equal "nova-manage version"? I'm not even really sure how they are getting their values, one appears to be coming from the python distribution and one from the rpm (looks like argparse must do something there). My opinion is that "nova-manage version" and "nova-manage --version" should return the same thing, because user interfaces should look like they were designed on purpose, and nobody would intentionally design those two almost identical commands to return different things. I think it's a bug that needs to be fixed in "nova-manage" rather than something Tempest should have to work around. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Proposal oslo.db lib
On 08/16/2013 09:52 AM, Boris Pavlovic wrote: We (OpenStack contributors) done a really huge and great work around DB code in Grizzly and Havana to unify it, put all common parts into oslo-incubator, fix bugs, improve handling of sqla exceptions, provide unique keys, and to use this code in different projects instead of custom implementations. (well done!) oslo-incubator db code is already used by: Nova, Neutron, Cinder, Ironic, Ceilometer. In this moment we finished work around Glance: https://review.openstack.org/#/c/36207/ And working around Heat and Keystone. So almost all projects use this code (or planing to use it) Probably it is the right time to start work around moving oslo.db code to separated lib. We (Roman, Viktor and me) will be glad to help to make oslo.db lib: E.g. Here are two drafts: 1) oslo.db lib code: https://github.com/malor/oslo.db 2) And here is this lib in action: https://review.openstack.org/#/c/42159/ Thoughts? +1. Having to manually paste code from oslo-incubator into other projects is error-prone. Of course it's important to get the library versioning right and do releases, but that's a small cost imposed on just the oslo-db folks to make using this code easier for everyone else. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Neutron a quick qpid revert
On 08/13/2013 09:57 AM, Dan Prince wrote: All of my Neutron tests are failing this morning in SmokeStack. We need a quick revert to fix the qpid RPC implementation: https://review.openstack.org/41689 https://bugs.launchpad.net/neutron/+bug/1211778 I figure we may as well revert this quick and then just wait on oslo.messaging to fix the original RPC concern here? Thanks Dan. That's my mistake, for pulling over the entire latest impl_qpid.py rather than just my tiny fix to it. I'll redo the patch. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron][Requirement] SQLAlchemy raises None, causes TypeError
On 07/18/2013 03:48 PM, Nachi Ueno wrote: Sometimes, I got "SQLAlchemy raises None, causes TypeError" This get debugging hard, because it hides real reasons. It looks like the problem of combining eventlet and sqlalchemy. http://stackoverflow.com/questions/9525220/sqlalchemy-raises-none-causes-typeerror Latest version of sqlalchemy looks improved of this exception handling code. The sqlalchemy version looks pinned to sqlalchemy>=0.7.8,<=0.7.99. Because we needed SqlSoup (see [1]), However sqlsoup is no longer used. so I would like to upgrade sqlalchemy, but if the other project needs 0.7 version, please let me know. Heat fails with SQLAlchemy 0.8. https://bugs.launchpad.net/heat/+bug/1199435 -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] headsup - transient test failures on py26 ' cannot import name OrderedDict'
On 07/17/2013 04:54 PM, Robert Collins wrote: On 18 July 2013 08:48, Chris Jones wrote: Hi On 17 July 2013 21:27, Robert Collins wrote: Surely thats fixable by having a /opt/ install of Python2.7 built for RHEL ? That would make life s much easier for all concerned, and is super Possibly not easier for those tasked with keeping OS security patches up to date, which is part of what a RHEL customer is paying Red Hat a bunch of money to do. I totally agree, which is why it would make sense for Red Hat to supply the build of Python 2.7 :). FYI, http://developerblog.redhat.com/2013/06/05/red-hat-software-collections-1-0-beta-now-available/ (TL;DR : Red Hat Software Collections is a way to get newer versions of Python and some other software on RHEL 6. It's still in beta though.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DB][Migrations] Switching to using of Alembic
On 07/16/2013 12:09 PM, Dolph Mathews wrote: On Tue, Jul 16, 2013 at 10:51 AM, Roman Podolyaka mailto:rpodoly...@mirantis.com>> wrote: We are going to implement ALTER support in Alembic for SQLite in the next few weeks. I'm a little lost on this ... sqlite doesn't support ALTER, so what exactly is being added to alembic? Is the alembic community receptive or interested? There is some code in Nova (on its way into Oslo) to work around not being able to alter tables in SQLite. It deletes the old table and adds the modified version as a new table. That's the best you can do without modifying SQLite itself. The Alembic README specifically mentions this SQLite issue and says "we will support these features provided someone takes the initiative to implement and test". So, yeah, he'll take these patches. Means we'll need to use the future version of Alembic with this feature, though. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [DB][Migrations] Switching to using of Alembic
On 07/16/2013 01:58 PM, Boris Pavlovic wrote: There is no "magic" and we have only 2 ways to end up with this problems and bugs that could be caused by "manually" migrations merging and tons of bugs in sqlalchemy-migrate. 1) step by step (openstack method) There are special tests "test_migrations" that runs migrations on real data against all backends. So we should: a) improve this tests to checks all behaviors // there is a lot of hidden bugs b) replace migration (only one small migration) to alembic c) check that in all backends we made the same changes in schema d) Merge all old migrations in one using alembic (automatically). So it could be done in safe way. 2.a) huge 2 steps 1. Merge all migrations in one huge manually (drop all tests in test migrations) e.g. In Nova was patch https://review.openstack.org/#/c/35748/ I don't believe that there are no mistakes in this migration, and nobody is able to check it. // because of tons of hidden bugs in old migrations and sqla-migrate. 2. Replace this migration in Alembic I don't believe that there will be way to check that there is no bugs 2.b) suicide mode (1 big step) Merge and switch in one step=) We have compacted migrations before, and there's a test document for how to verify that the big migration has exactly the same output as the series of small migrations. See https://wiki.openstack.org/wiki/Database_migration_testing Dan Prince is the expert on this. I think the right process is: 1. Wait until the very beginning of Icehouse cycle. (But not after we have new migrations for Icehouse.) 2. Compact all migrations into 2xx_havana.py (for SQLAlchemy-migrate) 3. Test that it's perfect via above test plan plus whatever enhancements we think of. 4. Manually convert 2xx_havana.py (for SQLAlchemy-migrate) into Alembic, and verify that it's still perfect. 5. Deprecate the SQLAlchemy-migrate version and announce that new migrations should be in Alembic. #4 is hard work but not impossible. I have some old code that does 90% of the work, so we only have to do the other 90%. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova] Seperate out 'soft-deleted' instances from 'deleted' ones?
On 07/15/2013 10:03 AM, Matt Riedemann wrote: I have a patch up for review on this: _https://review.openstack.org/#/c/35061/_ However, this doesn't fix the vm_states.SOFT_DELETED mapping in nova.api.openstack.common so if you show an instance with vm_states.SOFT_DELETED, the response status will be 'DELETED'. I'd like to see if there are any opinions on if this should come back as 'SOFT_DELETED' or if everyone is OK with mapping soft-delete to 'DELETED' in the v3 API? I would like to see them merged. Having multiple kinds of deleted records is really confusing, and leads to bugs. The more public we make this, the harder it will be to fix it in the future. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] SQLAlchemy-migrate needs a new maintainer
On 07/12/2013 07:31 AM, Sean Dague wrote: On 07/12/2013 04:29 AM, Thierry Carrez wrote: Monty Taylor wrote: This brings us to the most important question: Who wants to be on the core team? That's the important question indeed. Accepting it (permanently or temporarily) under stackforge is an easy decision. But it's useless unless we have a set of people sufficiently interested in it not bitrotting to volunteer to maintain it... I'd recommend the nova-db subteam folks, like: jog0, dripton, boris-42 as good people to be +2 on this. I'm happy to help, as long as there are at least 2 others. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] DB team meeting Thursday 1900 UTC
On 07/10/2013 03:41 PM, David Ripton wrote: We don't have the DB team meeting every week, but we're having it this week. Thursday at 1900 UTC in #openstack-meeting. Thanks. We had the DB meeting today. Major things discussed were sqlalchemy-migrate maintainership crisis (it sounds like OpenStack will take ownership of the unmaintained project for legacy support reasons, while trying to move things to Alembic as fast as we safely can), moving Nova DB code to Oslo (resolved by removing some db-archiving functions that not everyone likes), and a bunch of patches that we'd really like approved for Havana-2. Full log: http://eavesdrop.openstack.org/meetings/db/2013/db.2013-07-11-19.00.log.html Thanks. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] SQLAlchemy-migrate needs a new maintainer
OpenStack is currently divided. Older projects like Nova use sqlalchemy-migrate. Some newer projects like Neutron use alembic. I'd personally like to see everything in Alembic, but migrating all the Nova scripts in a way that didn't break compatibility will be a big challenge. It's easier for projects with less to port. Another option is to take over maintaining sqlalchemy-migrate and bend it to our needs. (It's mostly okay, but the big issue for me is its use of strictly incrementing integer sequence numbers. That both causes problems when competing patches in review race for the same filename, and when we try to backport some but not all migration scripts to a stable branch.) We already apply some patches to upstream, so having a friendly maintainer who would apply patches that OpenStack needs would be helpful. This will be a topic at the DB meeting today at 1900 UTC (about 20 minutes from when I send this email). So please attend if it's important to you. On 07/11/2013 02:18 PM, Thomas Goirand wrote: Discussing with Jan Dittberner, who is upstream for sqlalchemy-migrate, it appears that he doesn't have time to maintain it. Is the OpenStack project willing to take over? Jan is ok to hand over everything, moving to Github, give access to Pypi, etc. Below is his reply to me when I asked him. Or is the OpenStack project moving toward Alembic as well? Thoughts anyone? Thomas Goirand (zigo) On 07/12/2013 01:27 AM, Jan Dittberner wrote: I would be very happy to hand over the maintenance of sqlalchemy-migrate to a team that actually uses it. At the moment I take care of the Google Code [1] project for sqlalchemy-migrate and maintain a Jenkins instance at http://jenkins.gnuviech-server.de/. I'm all in favour of moving to github, Google Code was just choosen because it was available at the time the project moved from the initial developer's (Evan Rosson) personal server. I can also give access to the PyPI project page [2] to a prospective new maintainer/team. I wrote some sphinx documentation and improved the tests a while ago but I have no time to maintain it properly. I switched to alembic for my small personal projects. [1] https://code.google.com/p/sqlalchemy-migrate/ [2] https://pypi.python.org/pypi/sqlalchemy-migrate Best regards Jan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Review] Use of exception for non-exceptional cases
On 07/10/2013 02:01 PM, Nachi Ueno wrote: HI folks I would like to ask the review criteria in the community. Should we use exception for non-exceptional cases when we can use parameter checking? Example1: Default value for array index try: value = list[5] except IndexError: value = 'default_value' This can be also written as, list_a[3] if len(list_a) > 3 else 'default_value' Both of these are fine. Neither deserves to be banned. But LBYL is often naive in the face of concurrency. Just because something was true a microsecond ago doesn't mean it's still true. Exceptions are often more robust. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] DB team meeting Thursday 1900 UTC
We don't have the DB team meeting every week, but we're having it this week. Thursday at 1900 UTC in #openstack-meeting. Thanks. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] sqlalchemy 0.8 and Grizzly: heat and cinder failing
On 07/09/2013 12:46 PM, Thomas Goirand wrote: On 07/08/2013 08:32 PM, Sean Dague wrote: On 07/08/2013 04:50 AM, Mark McLoughlin wrote: On Mon, 2013-07-08 at 15:53 +0800, Thomas Goirand wrote: Hi, Since python-sqlalchemy 0.8.2 has been uploaded to Sid, Quantum is uninstallable there right now (see #715294). I am wondering: what's wrong with sqlalchemy >= 0.8, so that it is written explicitly in the requirements that we shouldn't use it? Is there a chance that having such a version of sqlalchemy will make all of OpenStack grizzly fail? What are the consequences? Would it be safe to simply patch the requirements file and ignore this? Don't really have a comment on the specifics, but ... The history of the 0.8 cap was the fact that 0.8beta1 (or something equiv) was uploaded near a freeze point. <0.8 didn't stop 0.8beta1 from being used (go version numbers). Also in 0.8 a piece was spun out as a separate library (I don't remember exactly which), which causes some build fails. Because it was around a freeze the cap was the right approach. However, projects really should be getting themselves on 0.8 in the Havana time frame. AFAIK it's really minor changes to work, so should be a simple review to bump it up. -Sean Indeed, most projects seem to work with the new SQLAlchemy. Though heat fails with a python crash dump: File "/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/models.py", line 32, in class Json(types.TypeDecorator, types.MutableType): AttributeError: 'module' object has no attribute 'MutableType' Indeed, MutableType is gone in SQLAlchemy >= 0.8. I'm therefore unsure what to do to fix the heat package in Sid... :( Any help would be appreciated. Yeah, someone who understands the Heat model code well needs to make class Json no longer inherit from MutableType. I hope it would be possible to do that in a backward-compatible way so it kept working with SQLAlchemy 0.7 in addition to working with 0.8. There's also a big problem with Cinder: File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 91, in runchange change.run(self.engine, step) File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 145, in run script_func(engine) File "/root/src/cinder/build-area/cinder-2013.1.2/cinder/db/sqlalchemy/migrate_repo/versions/002_quota_class.py", line 42, in upgrade _warn_on_bytestring=False), TypeError: __init__() got an unexpected keyword argument 'assert_unicode' Unit tests aren't run at all, and cinder refuses to install (because I'm doing the db_sync in the postinst, which fails). Help there would also be appreciated. AFAICT, these are the only packages affected by the SQLAlchemy upgrade. Old openstack DB migrations contained a lot of "convert_unicode=True", "unicode_error=None" and "_warn_on_bytestring=False" in the Column creation code. Dan Prince removed these from the Nova migrations in commit 93dec58156e , when he squashed all the migrations for Grizzly. Other projects still have them here and there, and that appears to be what is causing the above error. I suspect, but haven't proven, that it's possible to get rid of them (because Nova did) and that getting rid of them will fix this problem (because Nova doesn't have the problem.) Note that we don't like to modify database migrations because of compatibility concerns, so a change like this would need to receive a extra review scrutiny to prove it couldn't break anything. I threw up a patch to excise these arguments from Cinder DB migrations. https://review.openstack.org/36302 I will add some comments about how scary this type of change is to the review. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Work around DB in OpenStack (Oslo, Nova, Cinder, Glance)
On 07/08/2013 08:15 AM, Nikola Đipanov wrote: This is only true if you have one table with no relations that need to be considered. Here is an example of when it gets tricky - Say you have a table T1 and a migration that adds a column c1 that relies on some data from table T2 and T1 has a FK that points to T2. And say for the sake of argument that objects that are represented by rows in T1 and T2 have different life-times in the system (think instances and devices, groups, quotas, networks... this is common in our data model). In order to properly migrate and assign values to the newly created c1 you will need to: * Add the column c1 to the live T1 * join on live T2 *and* shadow T2 to get the data needed and populate the new column. * Add the column c1 to the shadow T1 * join on live T2 *and* shadow T2 to get the data needed and populate the new column. Hence - exponentially more joins, as I stated in my previous email. Now - this was the *simplest* possible example - things get potentially much more complicated if the new column relies on previous state of data (say - counters of some sort), if you need to get data from a third table (think many-to-many relationships) etc. If you need a real example - take a look at migration 186 in the current trunk. As I said in the previous email, and based on the examples above - this design decision (unconstrained rows) makes it difficult to reason about data in the system! I personally - as a developer working on the codebase - am not happy making this trade-off in favour of archiving in this way - and would like to see some design decisions changed, or at the very least a more broad consensus, that the state as-is is actually OK and we don't need to worry about it. I agree that it's a mess. However, the current archiving code just does the simplest thing possible -- move what can be moved to shadow tables, and try again later if foreign key constraints prevent that. It's certainly possible to do something more clever, but that would require the archiving code to know more about all the other tables in the system, which sounds difficult to maintain. Unfortunately, soft-deleted rows still satisfy FK constraints, so rows that point to soft-deleted rows never get deleted, so there is junk permanently left behind in some tables (like nova's instances). Soft-deletes let people get away with being sloppy, so people are sloppy. What I'd really like to see is for the entire soft-delete idea to go away, and just delete rows when it's time to delete them. Does anyone remember why soft-deletes got added in the first place? -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [vmware] VMwareAPI sub-team status update
On 07/03/2013 11:51 AM, Shawn Hartsock wrote: Work in progress: * https://review.openstack.org/#/c/35502/ <- I can't click the "Work in progress button" I'm not sure how else to signal that I'm still working... help? It's your patch. You should be able to. (I just successfully put one of my reviews into WIP state and then back to Ready for Review, so it's not globally broken.) Are you logged into Gerrit? -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone][Nova] Why migrate_version not InnoDB?
On 07/01/2013 12:49 PM, Brant Knudson wrote: 'Stackers - I've got a review up in Keystone that converts tables from MyISAM to InnoDB [0], which I patterned after a change in Nova. One of the comments in the review is suggesting that the migrate_version table should also be changed. The reason I didn't include migrate_version is because that's the way Nova did it, but other than that I don't know why migrate_version should not be converted. The Nova code is pretty explicit that migrate_version isn't changed [1]. Maybe somebody who knows MySQL or SQLAlchemy-migrate better than I do can come up with a reason why migrate_version shouldn't be changed from MyISAM to InnoDB. [0] https://review.openstack.org/#/c/33102/ - Use InnoDB for MySQL [1] https://github.com/openstack/nova/blob/master/nova/tests/db/test_migrations.py#L331 sqlalchemy-migrate relies on the migrate_versions table, so modifying it from within a sqlalchemy-migrate script is scary. And it's a tiny table that's only used during DB migrations, so I doubt you'd see any actual benefit. -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gerrit's Jenkins should stop running tests after first failure
On 06/18/2013 12:43 PM, Martina Kollarova wrote: Jenkins keeps running all the tests, even if the basic pep8 test fails, and runs all of the (very slow) Tempest Quantum tests, even though almost all of them are failing. I propose that it should fail and stop all of the other tests once there is a failure in a voting test. For non-voting tests, it should stop only itself, not the others. This would decrease the feedback loop and we wouldn't have to wait for the non-voting Quantum tests to see that they failed as always. -1 In addition to the other objections, we currently get a lot of false positives (fail, retry, fail, retry, succeed), and it would be harder to debug these problems if the output was truncated differently each time. Is anyone working on fixing the perma-failing Quantum test? When the Postgres test was perma-failing, one of the infrastructure folks gave us an ultimatum that if nobody fixed it soon, it would be disabled. (Happy ending: Mauro fixed it before it got disabled.) -- David Ripton Red Hat drip...@redhat.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
