Re: [openstack-dev] [puppet] today was a bad day for CI

2016-09-09 Thread James Page 
On Fri, 9 Sep 2016 at 01:17 Emilien Macchi  wrote:
[...]

> 3) Disable Ironic testing on Ubuntu. Packages are broken in recent
> Newton upgrade. They are working on it.
>

Ironic packaging issue fixed and released to newton-updates; that will
teach me to spend a morning tidying up old bugs :).


> 4) Enable br_netfilter kernel module on Ubuntu.
> See https://bugs.launchpad.net/cloud-archive/+bug/1621651
> Even if it's not critical, it's a nice-to-have because it removed the
> ERROR that we had in neutron logs. We'll see how the bug report evolve
> and maybe remove this workaround.
>

I've moved this to:

  https://bugs.launchpad.net/cloud-archive/+bug/1621854

as its a separate issue to the os-vif version problem (see below).

6) Disable linuxbridge on scenario003 for Ubuntu
> Also see https://bugs.launchpad.net/cloud-archive/+bug/1621651


os-vif updated to 1.2.1 and released to newton-updates; not detected by our
version checker as its was not in upper-constraints until recently.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] today was a bad day for CI

2016-09-08 Thread Emilien Macchi 
On Thu, Sep 8, 2016 at 8:15 PM, Emilien Macchi  wrote:
> I'm writing this email because it's end of day for me and I want our
> community know the status of our CI.
>
> I spent my day trying to bring our CI back and we found new blockers
> almost at every hour:
> https://review.openstack.org/367551
>
> 1) Add 2GB swap by default
> Add some swap when running CI jobs, so we can reduce the number of
> failures in CI related to missing memory.
> By default 2GB, can be disabled or increased/decreased with parameters.
>
> 2) Promote RDO repository to have Oslo DB from master
> python-oslo-db 4.13.x releases are known to break Gnocchi.
> https://review.openstack.org/367221 will fix it, but it is not yet
> in any release, so RDO unpinned oslo.db until that happens.
>
> 3) Disable Ironic testing on Ubuntu. Packages are broken in recent
> Newton upgrade. They are working on it.
>
> 4) Enable br_netfilter kernel module on Ubuntu.
> See https://bugs.launchpad.net/cloud-archive/+bug/1621651
> Even if it's not critical, it's a nice-to-have because it removed the
> ERROR that we had in neutron logs. We'll see how the bug report evolve
> and maybe remove this workaround.
>
> 5) Export lsmod in logs, easier for us to debug later.
>
> 6) Disable linuxbridge on scenario003 for Ubuntu
> Also see https://bugs.launchpad.net/cloud-archive/+bug/1621651
>
>
> And now, the best one (we don't have any fix yet):
> Puppetlabs released a new version of Puppet that required an uncertain
> version of redhat-release.
> http://logs.openstack.org/51/367551/8/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/fc2c291/console.html#_2016-09-09_00_05_47_317332
>
> They created a new dependency on puppetlabs-release that breaks
> everything. I reported the bug here:
> https://tickets.puppetlabs.com/browse/CPR-378
>
> So CI is now broken and there is nothing more we can do I guess. Any
> help from Puppetlabs is welcome.

Puppetlabs was super responsive and they are fixing the problem here:
https://github.com/puppetlabs/puppetlabs-release/pull/71

In the meantime, we will pin the package in our CI and it should work again.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] today was a bad day for CI

2016-09-08 Thread Emilien Macchi 
I'm writing this email because it's end of day for me and I want our
community know the status of our CI.

I spent my day trying to bring our CI back and we found new blockers
almost at every hour:
https://review.openstack.org/367551

1) Add 2GB swap by default
Add some swap when running CI jobs, so we can reduce the number of
failures in CI related to missing memory.
By default 2GB, can be disabled or increased/decreased with parameters.

2) Promote RDO repository to have Oslo DB from master
python-oslo-db 4.13.x releases are known to break Gnocchi.
https://review.openstack.org/367221 will fix it, but it is not yet
in any release, so RDO unpinned oslo.db until that happens.

3) Disable Ironic testing on Ubuntu. Packages are broken in recent
Newton upgrade. They are working on it.

4) Enable br_netfilter kernel module on Ubuntu.
See https://bugs.launchpad.net/cloud-archive/+bug/1621651
Even if it's not critical, it's a nice-to-have because it removed the
ERROR that we had in neutron logs. We'll see how the bug report evolve
and maybe remove this workaround.

5) Export lsmod in logs, easier for us to debug later.

6) Disable linuxbridge on scenario003 for Ubuntu
Also see https://bugs.launchpad.net/cloud-archive/+bug/1621651


And now, the best one (we don't have any fix yet):
Puppetlabs released a new version of Puppet that required an uncertain
version of redhat-release.
http://logs.openstack.org/51/367551/8/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/fc2c291/console.html#_2016-09-09_00_05_47_317332

They created a new dependency on puppetlabs-release that breaks
everything. I reported the bug here:
https://tickets.puppetlabs.com/browse/CPR-378

So CI is now broken and there is nothing more we can do I guess. Any
help from Puppetlabs is welcome.
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev