Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On 01/27/2015 10:51 AM, Doug Hellmann wrote: > The infra team has been working hard to update our Python 3 testing for all > projects to run on 3.4 instead of 3.3. Two of the last projects to be able to > shift are oslo.messaging and oslo.rootwrap. The test suites for both projects > trigger a segfault bug in the 3.4 interpreter as it is shipped on Ubuntu > Trusty. The fix for the segfault is already available upstream, and the team > at Canonical is working on packaging a new release, but our schedules are out > of sync. Maintaining a separate image and pool of testing nodes for 3.3 > testing of just these two projects is going to be a bit of a burden, and so > the infra team has asked if we’re willing to turn off the 3.3 jobs for the > two projects, leaving us without 3.x testing in the gate until the 3.4 > interpreter on Trusty is updated. > > The latest word from Canonical is that they plan to package Python 3.4.3, due > to be released in about a month. It will take some additional time to put it > through their release process, and so there’s some uncertainty about how long > we would be without 3.x gate jobs, but it doesn’t look like it will be > indefinitely. > > To mitigate that risk, fungi has suggested starting to work on Debian Jessie > worker images, which would include a version of Python 3.4 that doesn’t have > the segfault issue. His goal is to have something working by around the end > of March. That gives Canonical up to a month to release the 3.4.3 package > before we would definitely move those tests to Debian. Whether we move any of > the other projects, or would move anyway if fungi gets Debian working more > quickly than he expects, would remain to be seen. > > Although we do have some risk of introducing Python 3 regressions into the > two libraries, I am inclined to go along with the infra team’s request and > disable the tests for a short period of time. The rootwrap library doesn’t > see a lot of changes, and we can rely on the messaging lib devs to run tests > locally for a little while. > > Before I give the go-ahead, I want to hear concerns from the rest of the > team. Let’s try to have an answer by the 29th (Thursday). > > Doug As long as 1) this is short-term, which it sounds like it is and 2) we make sure to at least run the py3 tests locally before doing a release, I think this will be fine. A bug sneaking into the source isn't the end of the world, but releasing with one would be bad. -Ben __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On 27/01/15 13:22 -0500, Doug Hellmann wrote: On Tue, Jan 27, 2015, at 12:44 PM, Julien Danjou wrote: On Tue, Jan 27 2015, Clark Boylan wrote: > So the issue is that the garbage collector segfaults on null objects in > the to be garbage collected list. Which means that by the time garbage > collection breaks you don't have the info you need to know what > references lead to the segfault. I spent a bit of time in gdb debugging > this and narrowed it down enough to realize what the bug was and find it > was fixed in later python releases but didn't have the time to sort out > how to figure out specifically which references in oslo.messaging caused > the garbage collector to fall over. ╯‵Д′)╯彡┻━┻ Ok, then let's disable it I guess. If there's a chance to keep something has even a non-voting job, that'd be cool, but I'm not even sure that's an option if it just doesn't work and we can't keep py33. I did think about a non-voting job, but there's not much point. We expect it to fail with the segfault, so we would just be wasting resources. :-/ +1 for disabling it for now, although this means we may have some incompatibilities when we put it back. Fla -- Julien Danjou ;; Free Software hacker ;; http://julien.danjou.info __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Email had 1 attachment: + signature.asc 1k (application/pgp-signature) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgp7v874u54Bf.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On Tue, Jan 27, 2015, at 12:44 PM, Julien Danjou wrote: > On Tue, Jan 27 2015, Clark Boylan wrote: > > > So the issue is that the garbage collector segfaults on null objects in > > the to be garbage collected list. Which means that by the time garbage > > collection breaks you don't have the info you need to know what > > references lead to the segfault. I spent a bit of time in gdb debugging > > this and narrowed it down enough to realize what the bug was and find it > > was fixed in later python releases but didn't have the time to sort out > > how to figure out specifically which references in oslo.messaging caused > > the garbage collector to fall over. > > ╯‵Д′)╯彡┻━┻ > > Ok, then let's disable it I guess. If there's a chance to keep something > has even a non-voting job, that'd be cool, but I'm not even sure that's > an option if it just doesn't work and we can't keep py33. I did think about a non-voting job, but there's not much point. We expect it to fail with the segfault, so we would just be wasting resources. :-/ > > -- > Julien Danjou > ;; Free Software hacker > ;; http://julien.danjou.info > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > Email had 1 attachment: > + signature.asc > 1k (application/pgp-signature) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On Tue, Jan 27, 2015, at 12:06 PM, victor stinner wrote: > Hi, > > What is the Python bug? Do you have a reference to the bug report and the > patch? http://bugs.python.org/issue21435 > > Python 3.4.3 release schedule: > "3.4.3rc1 will be tagged Saturday February 7 and released Sunday February > 8. 3.4.3 final will follow two weeks later, tagged Saturday February 21 > and released Sunday February 22." > https://mail.python.org/pipermail/python-dev/2015-January/137773.html > > Victor > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On Tue, Jan 27 2015, Clark Boylan wrote: > So the issue is that the garbage collector segfaults on null objects in > the to be garbage collected list. Which means that by the time garbage > collection breaks you don't have the info you need to know what > references lead to the segfault. I spent a bit of time in gdb debugging > this and narrowed it down enough to realize what the bug was and find it > was fixed in later python releases but didn't have the time to sort out > how to figure out specifically which references in oslo.messaging caused > the garbage collector to fall over. ╯‵Д′)╯彡┻━┻ Ok, then let's disable it I guess. If there's a chance to keep something has even a non-voting job, that'd be cool, but I'm not even sure that's an option if it just doesn't work and we can't keep py33. -- Julien Danjou ;; Free Software hacker ;; http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On Tue, Jan 27, 2015, at 09:13 AM, Julien Danjou wrote: > On Tue, Jan 27 2015, Doug Hellmann wrote: > > > The infra team has been working hard to update our Python 3 testing for all > > projects to run on 3.4 instead of 3.3. Two of the last projects to be able > > to > > shift are oslo.messaging and oslo.rootwrap. The test suites for both > > projects > > trigger a segfault bug in the 3.4 interpreter as it is shipped on Ubuntu > > Trusty. The fix for the segfault is already available upstream, and the > > team at > > Canonical is working on packaging a new release, but our schedules are out > > of > > sync. Maintaining a separate image and pool of testing nodes for 3.3 > > testing of > > just these two projects is going to be a bit of a burden, and so the infra > > team > > has asked if we’re willing to turn off the 3.3 jobs for the two projects, > > leaving us without 3.x testing in the gate until the 3.4 interpreter on > > Trusty > > is updated. > > Isn't there any way to have an ugly work-around in those libs that > wouldn't trigger the Python 3.4 segfault or at least disable the > responsible tests until Python 3.4 gets fixed? > So the issue is that the garbage collector segfaults on null objects in the to be garbage collected list. Which means that by the time garbage collection breaks you don't have the info you need to know what references lead to the segfault. I spent a bit of time in gdb debugging this and narrowed it down enough to realize what the bug was and find it was fixed in later python releases but didn't have the time to sort out how to figure out specifically which references in oslo.messaging caused the garbage collector to fall over. For oslo.rootwrap it affected any tests that ran with logging due to the way rootwrap does logging iirc. Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On Tue, Jan 27, 2015, at 09:06 AM, victor stinner wrote: > Hi, > > What is the Python bug? Do you have a reference to the bug report and the > patch? > https://bugs.launchpad.net/ubuntu/trusty/+source/python3.4/+bug/1367907 Is the bug I filed with the ubuntu package and it has links back to the upstream python bug. https://bugs.launchpad.net/ubuntu/trusty/+source/python3.4/+bug/1382607 is less problematic but also an issue for rootwrap. > Python 3.4.3 release schedule: > "3.4.3rc1 will be tagged Saturday February 7 and released Sunday February > 8. 3.4.3 final will follow two weeks later, tagged Saturday February 21 > and released Sunday February 22." > https://mail.python.org/pipermail/python-dev/2015-January/137773.html > > Victor > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
On Tue, Jan 27 2015, Doug Hellmann wrote: > The infra team has been working hard to update our Python 3 testing for all > projects to run on 3.4 instead of 3.3. Two of the last projects to be able to > shift are oslo.messaging and oslo.rootwrap. The test suites for both projects > trigger a segfault bug in the 3.4 interpreter as it is shipped on Ubuntu > Trusty. The fix for the segfault is already available upstream, and the team > at > Canonical is working on packaging a new release, but our schedules are out of > sync. Maintaining a separate image and pool of testing nodes for 3.3 testing > of > just these two projects is going to be a bit of a burden, and so the infra > team > has asked if we’re willing to turn off the 3.3 jobs for the two projects, > leaving us without 3.x testing in the gate until the 3.4 interpreter on Trusty > is updated. Isn't there any way to have an ugly work-around in those libs that wouldn't trigger the Python 3.4 segfault or at least disable the responsible tests until Python 3.4 gets fixed? -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
Hi, What is the Python bug? Do you have a reference to the bug report and the patch? Python 3.4.3 release schedule: "3.4.3rc1 will be tagged Saturday February 7 and released Sunday February 8. 3.4.3 final will follow two weeks later, tagged Saturday February 21 and released Sunday February 22." https://mail.python.org/pipermail/python-dev/2015-January/137773.html Victor __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
Doug Hellmann wrote: > The infra team has been working hard to update our Python 3 testing for all > projects to run on 3.4 instead of 3.3. Two of the last projects to be able to > shift are oslo.messaging and oslo.rootwrap. The test suites for both projects > trigger a segfault bug in the 3.4 interpreter as it is shipped on Ubuntu > Trusty. The fix for the segfault is already available upstream, and the team > at Canonical is working on packaging a new release, but our schedules are out > of sync. Maintaining a separate image and pool of testing nodes for 3.3 > testing of just these two projects is going to be a bit of a burden, and so > the infra team has asked if we’re willing to turn off the 3.3 jobs for the > two projects, leaving us without 3.x testing in the gate until the 3.4 > interpreter on Trusty is updated. > > The latest word from Canonical is that they plan to package Python 3.4.3, due > to be released in about a month. It will take some additional time to put it > through their release process, and so there’s some uncertainty about how long > we would be without 3.x gate jobs, but it doesn’t look like it will be > indefinitely. > > To mitigate that risk, fungi has suggested starting to work on Debian Jessie > worker images, which would include a version of Python 3.4 that doesn’t have > the segfault issue. His goal is to have something working by around the end > of March. That gives Canonical up to a month to release the 3.4.3 package > before we would definitely move those tests to Debian. Whether we move any of > the other projects, or would move anyway if fungi gets Debian working more > quickly than he expects, would remain to be seen. > > Although we do have some risk of introducing Python 3 regressions into the > two libraries, I am inclined to go along with the infra team’s request and > disable the tests for a short period of time. The rootwrap library doesn’t > see a lot of changes, and we can rely on the messaging lib devs to run tests > locally for a little while. > > Before I give the go-ahead, I want to hear concerns from the rest of the > team. Let’s try to have an answer by the 29th (Thursday). I’m not involved with those two subprojects but as an Oslo member I’d +1 disabling python3.x for now. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap
The infra team has been working hard to update our Python 3 testing for all projects to run on 3.4 instead of 3.3. Two of the last projects to be able to shift are oslo.messaging and oslo.rootwrap. The test suites for both projects trigger a segfault bug in the 3.4 interpreter as it is shipped on Ubuntu Trusty. The fix for the segfault is already available upstream, and the team at Canonical is working on packaging a new release, but our schedules are out of sync. Maintaining a separate image and pool of testing nodes for 3.3 testing of just these two projects is going to be a bit of a burden, and so the infra team has asked if we’re willing to turn off the 3.3 jobs for the two projects, leaving us without 3.x testing in the gate until the 3.4 interpreter on Trusty is updated. The latest word from Canonical is that they plan to package Python 3.4.3, due to be released in about a month. It will take some additional time to put it through their release process, and so there’s some uncertainty about how long we would be without 3.x gate jobs, but it doesn’t look like it will be indefinitely. To mitigate that risk, fungi has suggested starting to work on Debian Jessie worker images, which would include a version of Python 3.4 that doesn’t have the segfault issue. His goal is to have something working by around the end of March. That gives Canonical up to a month to release the 3.4.3 package before we would definitely move those tests to Debian. Whether we move any of the other projects, or would move anyway if fungi gets Debian working more quickly than he expects, would remain to be seen. Although we do have some risk of introducing Python 3 regressions into the two libraries, I am inclined to go along with the infra team’s request and disable the tests for a short period of time. The rootwrap library doesn’t see a lot of changes, and we can rely on the messaging lib devs to run tests locally for a little while. Before I give the go-ahead, I want to hear concerns from the rest of the team. Let’s try to have an answer by the 29th (Thursday). Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
