Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
On 10/18/2013 02:06 AM, Clint Byrum wrote: A link to instructions on setting up a wheezy box for this testing would be helpful. Just install a minimal Wheezy machine, add my repositories (using the Jenkins one at *.pkgs.enovance.com), then do apt-get install openstack-toaster. I'll be trying to provide a new preseed script so that just running the script will be enough. I have that already, though it needs to be updated for Havana. I'll post instructions when it is ready. No unit test failure / error (in both Sid and Wheezy). However, Heat doesn't include a -P option in ./run_tests.sh, and insists on running the PEP8 checks, which fails because Sid has pyflakes 0.7.3, and Heat wants 0.7.2: run_tests.sh is there for _your_ convenience. And by you I mean packagers and others who want to run the tests with system python. And it's been very helpful, though there's room for improvement. IMO you'd be better off adding a --no-venv option to tox to run the prescribed commands with system python, or even just parsing tox.ini to do that yourself. You'll find that the commands in tox.ini are better maintained, since they are gated. Anyway, I'd gladly accept a patch to add -P. I wonder: are the different run_tests.sh scripts maintained in Oslo? It feels strange that do we have different options for different OpenStack projects (like missing -P and -N options sometimes). On the nit-picking details side, an option to disable color output would be great (it's ugly to read [32mOK[0m[33m 0.03[0m ...). Different projects have different behaviors in this regard. (for example, Glance always display in colors, while Cinder doesn't) Also, heat uses /usr/bin/coverage. Please switch to python -m coverage (IMO, the best way...), or check if /usr/bin/coverage and if not, use /usr/bin/python-coverage. I appreciate this message, however, the bug tracker is over there - https://bugs.launchpad.net/heat/+filebug ACK: https://bugs.launchpad.net/heat/+bug/1241330 Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
On 2013-10-18 03:37, Thomas Goirand wrote: On 10/18/2013 02:06 AM, Clint Byrum wrote: A link to instructions on setting up a wheezy box for this testing would be helpful. Just install a minimal Wheezy machine, add my repositories (using the Jenkins one at *.pkgs.enovance.com), then do apt-get install openstack-toaster. I'll be trying to provide a new preseed script so that just running the script will be enough. I have that already, though it needs to be updated for Havana. I'll post instructions when it is ready. No unit test failure / error (in both Sid and Wheezy). However, Heat doesn't include a -P option in ./run_tests.sh, and insists on running the PEP8 checks, which fails because Sid has pyflakes 0.7.3, and Heat wants 0.7.2: run_tests.sh is there for _your_ convenience. And by you I mean packagers and others who want to run the tests with system python. And it's been very helpful, though there's room for improvement. IMO you'd be better off adding a --no-venv option to tox to run the prescribed commands with system python, or even just parsing tox.ini to do that yourself. You'll find that the commands in tox.ini are better maintained, since they are gated. Anyway, I'd gladly accept a patch to add -P. I wonder: are the different run_tests.sh scripts maintained in Oslo? It feels strange that do we have different options for different OpenStack projects (like missing -P and -N options sometimes). On the nit-picking details side, an option to disable color output would be great (it's ugly to read [32mOK[0m[33m 0.03[0m ...). Different projects have different behaviors in this regard. (for example, Glance always display in colors, while Cinder doesn't) I believe there was some work done around getting run_tests.sh into Oslo, but I don't know that it was ever finished, and if it was it obviously isn't being used in the other projects. Glancing through oslo-incubator I don't see anything that looks related so I'm guessing it just didn't happen. -Ben ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
Hey Thomas, this is good information to know about. It seems like all these issues could be bug reports? On 17 October 2013 20:16, Thomas Goirand z...@debian.org wrote: Hi, ... These appear in both Sid and Wheezy backports. * Nova == Only a single: FAIL: nova.tests.db.test_migrations.TestNovaMigrations.test_walk_versions in Sid. The relevant part is: OperationalError: (OperationalError) table instances has no column named cleaned u'CREATE INDEX instances_host_deleted_cleaned_idx ON instances (host, deleted, cleaned)' () It would really be nice if that one could be fixed, and that's the failure which worries me the most. Note that this is using sqlite3. There's no error at all under Wheezy backport, which leads me to believe the error is linked to SQLAlchemy 0.8.2 in Sid. Global requirements still lists SQLAlchemy=0.7.8,=0.7.99 So you'll need an older version. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
Hi, Thanks Robert for this quick reply. On 10/17/2013 07:31 PM, Robert Collins wrote: Hey Thomas, this is good information to know about. It seems like all these issues could be bug reports? My intention was to first warn the list, as it is well possible that there was some mistake that I did in the packaging, and I wanted to avoid reporting some false positive. On 17 October 2013 20:16, Thomas Goirand z...@debian.org wrote: Hi, ... These appear in both Sid and Wheezy backports. * Nova == Only a single: FAIL: nova.tests.db.test_migrations.TestNovaMigrations.test_walk_versions in Sid. The relevant part is: OperationalError: (OperationalError) table instances has no column named cleaned u'CREATE INDEX instances_host_deleted_cleaned_idx ON instances (host, deleted, cleaned)' () It would really be nice if that one could be fixed, and that's the failure which worries me the most. Note that this is using sqlite3. There's no error at all under Wheezy backport, which leads me to believe the error is linked to SQLAlchemy 0.8.2 in Sid. Global requirements still lists SQLAlchemy=0.7.8,=0.7.99 So you'll need an older version. I have stressed it many, many times, though it seems that the message still didn't reach everyone. Debian Sid has SQLAlchemy 0.8.2, and there's nothing we can do about it. Setting =0.7.8,=0.7.99 in the requirements.txt doesn't fix the issue, it makes it worse: some OpenStack developers believe it is a fix, while it prevents building. So yes, we need a fix for the above problem, and Nova needs to support SQLAlchemy 0.8.2 which is what we have in Sid [1]. If I remember well, there's also the problem on RPM based systems. BTW, it looks like it is an isolated unique problem, and it seems to be the only one left in this release of OpenStack. It is also new (Nova Havana B3 didn't have the issue). So I really think it shouldn't be hard to fix the problem. Also, note that the issue isn't only in Debian, it's also a problem for Ubuntu 2013.10, which also has SQLAlchemy 0.8.2. [2] I really hope that for the next release cycle, the above will be taken into consideration, and that someone will come up with a fix quickly for this release. Maybe *now* is time to switch the gate to SQLAlchemy 0.8.2 for the Icehouse branch? Cheers, Thomas Goirand (zigo) P.S: I'll be filling bug reports if I don't get any more comments. [1] http://packages.debian.org/sid/python-sqlalchemy [2] http://packages.ubuntu.com/saucy/python-sqlalchemy ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
On 10/17/2013 08:57 AM, Thomas Goirand wrote: Hi, Thanks Robert for this quick reply. On 10/17/2013 07:31 PM, Robert Collins wrote: Hey Thomas, this is good information to know about. It seems like all these issues could be bug reports? My intention was to first warn the list, as it is well possible that there was some mistake that I did in the packaging, and I wanted to avoid reporting some false positive. On 17 October 2013 20:16, Thomas Goirand z...@debian.org wrote: Hi, ... These appear in both Sid and Wheezy backports. * Nova == Only a single: FAIL: nova.tests.db.test_migrations.TestNovaMigrations.test_walk_versions in Sid. The relevant part is: OperationalError: (OperationalError) table instances has no column named cleaned u'CREATE INDEX instances_host_deleted_cleaned_idx ON instances (host, deleted, cleaned)' () It would really be nice if that one could be fixed, and that's the failure which worries me the most. Note that this is using sqlite3. There's no error at all under Wheezy backport, which leads me to believe the error is linked to SQLAlchemy 0.8.2 in Sid. Global requirements still lists SQLAlchemy=0.7.8,=0.7.99 So you'll need an older version. I have stressed it many, many times, though it seems that the message still didn't reach everyone. Debian Sid has SQLAlchemy 0.8.2, and there's nothing we can do about it. Setting =0.7.8,=0.7.99 in the requirements.txt doesn't fix the issue, it makes it worse: some OpenStack developers believe it is a fix, while it prevents building. I understand what you are saying and I also understand your frustration. However, OpenStack does not, as of yet, support SQLAlchemy 0.8, and as you can see, the requirements file does, in fact, communicate reality, we depend on 0.8. I support the bug being fixed and the requirement being raised, but it has to happen in that order. So yes, we need a fix for the above problem, and Nova needs to support SQLAlchemy 0.8.2 which is what we have in Sid [1]. If I remember well, there's also the problem on RPM based systems. BTW, it looks like it is an isolated unique problem, and it seems to be the only one left in this release of OpenStack. It is also new (Nova Havana B3 didn't have the issue). So I really think it shouldn't be hard to fix the problem. Also, note that the issue isn't only in Debian, it's also a problem for Ubuntu 2013.10, which also has SQLAlchemy 0.8.2. [2] Fascinating. What have they done to make that work? And why has none of that work made it into OpenStack so that we could raise the SQLAlchemy requirement? I really hope that for the next release cycle, the above will be taken into consideration, and that someone will come up with a fix quickly for this release. Maybe *now* is time to switch the gate to SQLAlchemy 0.8.2 for the Icehouse branch? Can't change the req until it works - but yes, I agree with you, we clearly need to upgrade. Thomas Goirand (zigo) P.S: I'll be filling bug reports if I don't get any more comments. Thanks for you help and attention Thomas! I appreciate all the work!!! ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
On 10/17/2013 09:11 PM, Monty Taylor wrote: I understand what you are saying and I also understand your frustration. However, OpenStack does not, as of yet, support SQLAlchemy 0.8, and as you can see, the requirements file does, in fact, communicate reality, we depend on 0.8. It does, at the exception of this little issue when running in Sid with SQLite 3. I support the bug being fixed and the requirement being raised, but it has to happen in that order. Sure! So yes, we need a fix for the above problem, and Nova needs to support SQLAlchemy 0.8.2 which is what we have in Sid [1]. If I remember well, there's also the problem on RPM based systems. BTW, it looks like it is an isolated unique problem, and it seems to be the only one left in this release of OpenStack. It is also new (Nova Havana B3 didn't have the issue). So I really think it shouldn't be hard to fix the problem. Also, note that the issue isn't only in Debian, it's also a problem for Ubuntu 2013.10, which also has SQLAlchemy 0.8.2. [2] Fascinating. What have they done to make that work? I'm not sure what you are talking about, since I don't agree that OpenStack doesn't work with SQLAlchemy 0.8.2 (apart from this one bug). And why has none of that work made it into OpenStack so that we could raise the SQLAlchemy requirement? If you didn't know, I already have Grizzly (currently in Sid) working with SQLAlchemy 0.8.2... (I just backported from Havana to Grizzly a few patches from upstream, namely for Cinder and Heat IIRC.) These were reported by me during this summer, and fixed in July for all projects, and just right before Havana B3 for Heat. I really hope that for the next release cycle, the above will be taken into consideration, and that someone will come up with a fix quickly for this release. Maybe *now* is time to switch the gate to SQLAlchemy 0.8.2 for the Icehouse branch? Can't change the req until it works - but yes, I agree with you, we clearly need to upgrade. Let's have that one fixed, and let's switch... Thanks for you help and attention Thomas! I appreciate all the work!!! My pleasure! :) Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
Hi all, Being a bit familiar with both SQLAlchemy and sqlalchemy-migrate, I decided to check the issue with running of migrations tests in Nova with SQLAlchemy 0.8.x. Long story is here: https://bugs.launchpad.net/sqlalchemy-migrate/+bug/1241038 TL;DR 1. It's really the issue with sqlalchemy-migrate and has nothing to do with Nova. 2. sqlalchemy-migrate DOES seem to support SQLAlchemy 0.8.x, but, I believe, we are going to face more and more similar issues. 3. It's downgrade, so it should be a minor issue for packaging of new OpenStack release in Debian. 4. Everyone's life would be much easier, if we dropped migrations support for SQLite. Alembic doesn't support ALTER for SQLite at all on purpose. And I would really like us to switch to using of Alembic in the following releases. I'll try to fix this in sqlalchemy-migrate and maybe we could release the fix later. Thanks, Roman On Thu, Oct 17, 2013 at 6:31 PM, Thomas Goirand z...@debian.org wrote: On 10/17/2013 09:11 PM, Monty Taylor wrote: I understand what you are saying and I also understand your frustration. However, OpenStack does not, as of yet, support SQLAlchemy 0.8, and as you can see, the requirements file does, in fact, communicate reality, we depend on 0.8. It does, at the exception of this little issue when running in Sid with SQLite 3. I support the bug being fixed and the requirement being raised, but it has to happen in that order. Sure! So yes, we need a fix for the above problem, and Nova needs to support SQLAlchemy 0.8.2 which is what we have in Sid [1]. If I remember well, there's also the problem on RPM based systems. BTW, it looks like it is an isolated unique problem, and it seems to be the only one left in this release of OpenStack. It is also new (Nova Havana B3 didn't have the issue). So I really think it shouldn't be hard to fix the problem. Also, note that the issue isn't only in Debian, it's also a problem for Ubuntu 2013.10, which also has SQLAlchemy 0.8.2. [2] Fascinating. What have they done to make that work? I'm not sure what you are talking about, since I don't agree that OpenStack doesn't work with SQLAlchemy 0.8.2 (apart from this one bug). And why has none of that work made it into OpenStack so that we could raise the SQLAlchemy requirement? If you didn't know, I already have Grizzly (currently in Sid) working with SQLAlchemy 0.8.2... (I just backported from Havana to Grizzly a few patches from upstream, namely for Cinder and Heat IIRC.) These were reported by me during this summer, and fixed in July for all projects, and just right before Havana B3 for Heat. I really hope that for the next release cycle, the above will be taken into consideration, and that someone will come up with a fix quickly for this release. Maybe *now* is time to switch the gate to SQLAlchemy 0.8.2 for the Icehouse branch? Can't change the req until it works - but yes, I agree with you, we clearly need to upgrade. Let's have that one fixed, and let's switch... Thanks for you help and attention Thomas! I appreciate all the work!!! My pleasure! :) Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
On 10/17/2013 04:54 PM, Dolph Mathews wrote: On Thu, Oct 17, 2013 at 10:31 AM, Thomas Goirand z...@debian.org wrote: On 10/17/2013 09:11 PM, Monty Taylor wrote: I understand what you are saying and I also understand your frustration. However, OpenStack does not, as of yet, support SQLAlchemy 0.8, and as you can see, the requirements file does, in fact, communicate reality, we depend on 0.8. It does, at the exception of this little issue when running in Sid with SQLite 3. I support the bug being fixed and the requirement being raised, but it has to happen in that order. Sure! So yes, we need a fix for the above problem, and Nova needs to support SQLAlchemy 0.8.2 which is what we have in Sid [1]. If I remember well, there's also the problem on RPM based systems. BTW, it looks like it is an isolated unique problem, and it seems to be the only one left in this release of OpenStack. It is also new (Nova Havana B3 didn't have the issue). So I really think it shouldn't be hard to fix the problem. Also, note that the issue isn't only in Debian, it's also a problem for Ubuntu 2013.10, which also has SQLAlchemy 0.8.2. [2] Fascinating. What have they done to make that work? I'm not sure what you are talking about, since I don't agree that OpenStack doesn't work with SQLAlchemy 0.8.2 (apart from this one bug). This issue in sqlalchemy-migrate is a blocker (and probably just the first of several..) for adopting sqlalchemy =0.8 in at least keystone heat, AFAIK: https://code.google.com/p/sqlalchemy-migrate/issues/detail?id=156 This is handled in the sqlalchemy-migrate fork which we're now maintaining within openstack: https://github.com/stackforge/sqlalchemy-migrate/commits/master https://review.openstack.org/#/q/project:stackforge/sqlalchemy-migrate,n,z To reiterate, we've been patching these _small_ issues in Debian and Fedora since version 19 (grizzly), and consolidating them upstream so we can officially state sqlalchemy 0.8 is supported. Sqlalchemy really only removed a few long deprecated things. thanks, Pádraig. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
On 10/18/2013 12:05 AM, Roman Podolyaka wrote: 4. Everyone's life would be much easier, if we dropped migrations support for SQLite. Alembic doesn't support ALTER for SQLite at all on purpose. And I would really like us to switch to using of Alembic in the following releases. Except that if this happens, we will have no way to run piuparts [1] install and upgrade tests. These tests are run regularly by the Debian Q/A team, and I *must* support it. I believe it's very good and helpful tests which we shouldn't drop. Yes, life is hard... :) Thomas [1] http://packages.debian.org/sid/piuparts ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Building on Debian: Havana unit tests at build time report
Excerpts from Thomas Goirand's message of 2013-10-17 00:16:22 -0700:
Hi,
As you know, I maintain OpenStack both for Debian Sid, and a backport
for Wheezy (non-official) repository (since using the backports
repository for OpenStack wouldn't be practical). Below is the result of
the building OpenStack Havana (latest RC) under both these environments.
I would really appreciate if people from each of the concerned projects
could pay attention to this report, and provide fixes. It doesn't have
to be merged into the final release (I of course understand that we are
running out of time), though if there's a proposed patch, even if it's
not yet merged and still with work in progress status, that's fine for
me. I can use in a Debian specific patch (under debian/patches), and
upload later on.
Another thing which struck me: OpenStack really works well in Sid, even
with SQLAlchemy 0.8.2, apart from this Nova glitch which *must* be
fixed. I really hope someone will come with a patch for it.
I'd like also to point out that if I am publishing this report, it
really is because I expect others to work on it. Remember that I'm alone
doing the Debian packaging work (as opposed to a full team for Ubuntu),
and that you cannot expect that *I* fix (all) things. I simply don't
have enough time to work on it myself, unfortunately. So help wanted... :)
Last thing: please test and use Debian (Sid or my Wheezy backport
repository) for OpenStack!
A link to instructions on setting up a wheezy box for this testing would
be helpful.
* Heat
==
No unit test failure / error (in both Sid and Wheezy). However, Heat
doesn't include a -P option in ./run_tests.sh, and insists on running
the PEP8 checks, which fails because Sid has pyflakes 0.7.3, and Heat
wants 0.7.2:
run_tests.sh is there for _your_ convenience. And by you I mean
packagers and others who want to run the tests with system python.
IMO you'd be better off adding a --no-venv option to tox to run the
prescribed commands with system python, or even just parsing tox.ini to
do that yourself. You'll find that the commands in tox.ini are better
maintained, since they are gated.
Anyway, I'd gladly accept a patch to add -P.
pkg_resources.VersionConflict: (pyflakes 0.7.3
(/usr/lib/python2.7/dist-packages), Requirement.parse('pyflakes==0.7.2'))
I had the same problem with Sid, with the PEP8 1.4.4 which I maintain as
backport (and Heat expects 1.4.5). I could work it out, though I don't
think it's worse the trouble (as it will not change the useability of
the package), so I just gave up on it.
Also, heat uses /usr/bin/coverage. Please switch to python -m coverage
(IMO, the best way...), or check if /usr/bin/coverage and if not, use
/usr/bin/python-coverage.
I appreciate this message, however, the bug tracker is over there -
https://bugs.launchpad.net/heat/+filebug
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
