Re: [openstack-dev] [requirements][vitrage][infra] SQLAlchemy-Utils version 0.33.6 breaks Vitrage gate

2018-10-23 Thread Ian Wienand
On Thu, Oct 18, 2018 at 01:17:13PM +, Jeremy Stanley wrote:
> It's been deleted (again) and the suspected fix approved so
> hopefully it won't recur.

Unfortunately the underlying issue is still a mystery.  It recurred
once after the suspected fix was merged [1], and despite trying to
replicate it mostly in-situ we could not duplicate the issue.

Another change [2] has made our builds use a modified pip [3] which
logs the sha256 hash of the .whl outputs.  If this reappears, we can
look at the logs and the final (corrupt) wheel and see if the problem
is coming from pip, or something after that as we copy the files.

If looking at hexdumps of zip files is your idea of a good time, there
are some details on the corruption in the comments of [2].  Any
suggestions welcome :) Also any corruption reports welcome too, and we
can continue investigation.

Thanks,

-i

[1] https://review.openstack.org/611444
[2] https://review.openstack.org/612234
[3] https://github.com/pypa/pip/pull/5908

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Release-job-failures] Tag of openstack/python-neutronclient failed

2018-09-10 Thread Ian Wienand
> On Mon, Sep 10, 2018 at 05:13:35AM +, z...@openstack.org wrote:
>> Build failed.
>>
>> - publish-openstack-releasenotes 
>> http://logs.openstack.org/c8/c89ca61fdcaf603a10750b289228b7f9a3597290/tag/publish-openstack-releasenotes/fbbd0fa/
>>  : FAILURE in 4m 03s

The line that is causing this is

  - Add OSC plugin support for the “Networking Service Function Chaining” ...

see if you can find the unicode :)

I did replicate it by mostly doing what the gate does; make a python2
vitualenv and install everything, then run

 ./env/bin/sphinx-build -a -E -W -d releasenotes/build/doctrees/ \
   -b html releasenotes/source/ releasenotes/build/html/

In the gate, it doesn't use "tox -e releasenotes" ... which passes
because it's python3 and everything is unicode already.

I think this is a reno problem, and I've proposed

  https://review.openstack.org/601432 Use unicode for debug string

Thanks

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [networking-odl][networking-bgpvpn][Telemetry] all requirement updates are currently blocked

2018-09-09 Thread Ian Wienand

On 09/10/2018 09:39 AM, Tony Breeds wrote:

Julien, Do you mind me arranging for at least the following versions to
be published to pypi?


For this particular case, I think our best approach is to have an
admin manually upload the tar & wheels from tarballs.openstack.org to
pypi.  All other options seem to be sub-optimal:

 - if we re-ran the release pipeline, I *think* it would all be
   idempotent and the publishing would happen, but there would be
   confusing duplicate release emails sent.

 - we could make a special "only-publish" template that avoids
   notification jobs; switch ceilometer to this, re-run the releases,
   then switch back.  urgh, especially if something goes wrong.

 - ceilometer could make "no-op" releases on each branch to trigger a
   fresh release & publish; but releases that essentially do nothing
   are I imagine an annoyance for users and distributors who track
   stable branches.

It would look like

  https://test.pypi.org/project/ceilometer/

The pypi hashes will all line up with the .asc files we publish, so we
know there's no funny business going on.

Thanks,

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all]-ish : Updates required for readthedocs publishers

2018-09-05 Thread Ian Wienand

On 09/06/2018 02:10 AM, Doug Hellmann wrote:

Those instructions and the ones linked at
https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html#project_template-docs-on-readthedocs
say to "generate a web hook URL".


I think you got the correct answers, thanks Dmitry.  Note it is also
illustrated at

 https://imgur.com/a/Pp4LH31

Thanks

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all]-ish : Updates required for readthedocs publishers

2018-09-05 Thread Ian Wienand
Hello,

If you're interested in the projects mentioned below, you may have
noticed a new, failing, non-voting job
"your-readthedocs-job-requires-attention".  Spoiler alert: your
readthedocs job requires attention.  It's easy to miss because
publishing happens in the post pipeline and people don't often look
at the results of these jobs.

Please see the prior email on this

 http://lists.openstack.org/pipermail/openstack-dev/2018-August/132836.html

for what to do (if you read the failing job logs, it also points you
to this).

I (or #openstack-infra) can help, but only once the openstackci user
is given permissions to the RTD project by its current owner.

Thanks,

-i

The following projects have this job now:

openstack-infra/gear
openstack/airship-armada
openstack/almanach
openstack/ansible-role-bindep
openstack/ansible-role-cloud-launcher
openstack/ansible-role-diskimage-builder
openstack/ansible-role-cloud-fedmsg
openstack/ansible-role-cloud-gearman
openstack/ansible-role-jenkins-job-builder
openstack/ansible-role-logrotate
openstack/ansible-role-ngix
openstack/ansible-role-nodepool
openstack/ansible-role-openstacksdk
openstack/ansible-role-shade
openstack/ansible-role-ssh
openstack/ansible-role-sudoers
openstack/ansible-role-virtualenv
openstack/ansible-role-zookeeper
openstack/ansible-role-zuul
openstack/ara
openstack/bareon
openstack/bareon-allocator
openstack/bareon-api
openstack/bareon-ironic
openstack/browbeat
openstack/downpour
openstack/fuel-ccp
openstack/fuel-ccp-installer
openstack/fuel-noop-fixtures
openstack/ironic-staging-drivers
openstack/k8s-docker-suite-app-murano
openstack/kloudbuster
openstack/nerd-reviewer
openstack/networking-dpm
openstack/nova-dpm
openstack/ooi
openstack/os-faults
openstack/packetary
openstack/packetary-specs
openstack/performa
openstack/poppy
openstack/python-almanachclient
openstack/python-jenkins
openstack/rally
openstack/solar
openstack/sqlalchemy-migrate
openstack/stackalytics
openstack/surveil
openstack/swauth
openstack/turbo-hipster
openstack/virtualpdu
openstack/vmtp
openstack/windmill
openstack/yaql

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all][docs] ACTION REQUIRED for projects using readthedocs

2018-08-02 Thread Ian Wienand
Hello,

tl;dr : any projects using the "docs-on-readthedocs" job template
to trigger a build of their documentation in readthedocs needs to:

 1) add the "openstackci" user as a maintainer of the RTD project
 2) generate a webhook integration URL for the project via RTD
 3) provide the unique webhook ID value in the "rtd_webhook_id" project
variable

See

 
https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html#project_template-docs-on-readthedocs

--

readthedocs has recently updated their API for triggering a
documentation build.  In the old API, anyone could POST to a known URL
for the project and it would trigger a build.  This end-point has
stopped responding and we now need to use an authenticated webhook to
trigger documentation builds.

Since this is only done in the post and release pipelines, projects
probably haven't had great feedback that current methods are failing
and this may be a surprise.  To check your publishing, you can go to
the zuul builds page [1] and filter by your project and the "post"
pipeline to find recent runs.

There is now some setup required which can only be undertaken by a
current maintainer of the RTD project.

In short; add the "openstackci" user as a maintainer, add a "generic
webhook" integration to the project, find the last bit of the URL from
that and put it in the project variable "rtd_webhook_id".

Luckily OpenStack infra keeps a team of highly skilled digital artists
on retainer and they have produced a handy visual guide available at

  https://imgur.com/a/Pp4LH31

Once the RTD project is setup, you must provide the webhook ID value
in your project variables.  This will look something like:

 - project:
templates:
  - docs-on-readthedocs
  - publish-to-pypi
vars:
  rtd_webhook_id: '12345'
check:
  jobs:
  ...

For actual examples; see pbrx [2] which keeps its config in tree, or
gerrit-dash-creator which has its configuration in project-config [3].

Happy to help if anyone is having issues, via mail or #openstack-infra

Thanks!

-i

p.s. You don't *have* to use the jobs from the docs-on-readthedocs
templates and hence add infra as a maintainer; you can setup your own
credentials with zuul secrets in tree and write your playbooks and
jobs to use the generic role [4].  We're always happy to discuss any
concerns.

[1] https://zuul.openstack.org/builds.html
[2] https://git.openstack.org/cgit/openstack/pbrx/tree/.zuul.yaml#n17
[3] 
https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d/projects.yaml
[4] https://zuul-ci.org/docs/zuul-jobs/roles.html#role-trigger-readthedocs

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all] Ongoing spam in Freenode IRC channels

2018-07-31 Thread Ian Wienand

Hello,

It seems freenode is currently receiving a lot of unsolicited traffic
across all channels.  The freenode team are aware [1] and doing their
best.

There are not really a lot of options.  We can set "+r" on channels
which means only nickserv registered users can join channels.  We have
traditionally avoided this, because it is yet one more barrier to
communication when many are already unfamiliar with IRC access.
However, having channels filled with irrelevant messages is also not
very accessible.

This is temporarily enabled in #openstack-infra for the time being, so
we can co-ordinate without interruption.

Thankfully AFAIK we have not needed an abuse policy on this before;
but I guess we are the point we need some sort of coordinated
response.

I'd suggest to start, people with an interest in a channel can request
+r from an IRC admin in #openstack-infra and we track it at [2]

Longer term ... suggestions welcome? :)

-i

[1] https://freenode.net/news/spambot-attack
[2] https://etherpad.openstack.org/p/freenode-plus-r-08-2018

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] OpenStack lagging behind 2 major python versions: we need a Python 3.7 gate

2018-07-17 Thread Ian Wienand
On 07/13/2018 06:38 AM, Thomas Goirand wrote:
> Now, both Debian and Ubuntu have Python 3.7. Every package which I
> upload in Sid need to support that. Yet, OpenStack's CI is still
> lagging with Python 3.5.

OpenStack's CI is rather broad -- I'm going to assume we're talking
about whole-system devstack-ish based functional tests.  Yes most
testing is on Xenial and hence Python 3.5

We have Python 3.6 available via Bionic nodes.  I think current talk
is to look at mass-updates after the next release.  Such updates, from
history, are fairly disruptive.

> I'm aware that there's been some attempts in the OpenStack infra to
> have Debian Sid (which is probably the distribution getting the
> updates the faster).

We do not currently build Debian sid images, or mirror the unstable
repos or do wheel builds for Debian.  diskimage-builder also doesn't
test it in CI.  This is not to say it can't be done.

> If it cannot happen with Sid, then I don't know, choose another
> platform, and do the Python 3-latest gating...

Fedora has been consistently updated in OpenStack Infra for many
years.  IMO, and from my experience, six-monthly-ish updates are about
as frequent as can be practically handled.

The ideal is that a (say) Neutron dev gets a clear traceback from a
standard Python error in their change and happily fixes it.  The
reality is probably more like this developer gets a tempest
failure due to nova failing to boot a cirros image, stemming from a
detached volume due to a qemu bug that manifests due to a libvirt
update (I'm exaggerating, I know :).

That sort of deeply tangled platform issue always exists; however it
is armortised across the lifespan of the testing.  So several weeks
after we update all these key components, a random Neutron dev can be
pretty sure that submitting their change is actually testing *their*
change, and not really a defacto test of every other tangentially
related component.

A small, but real example; uwsgi wouldn't build with the gcc/glibc
combo on Fedora 28 for two months after its release until uwsgi's
2.0.17.1.  Fedora carried patches; but of course there were a lot
previously unconsidered assumptions in devstack around deployment that
made using the packaged versions difficult [1] (that stack still
hasn't received any reviews).

Nobody would claim diskimage-builder is the greatest thing ever, but
it does produce our customised images in a wide variety of formats
that runs in our very heterogeneous clouds.  It's very reactive -- we
don't know about package updates until they hit the distro, and
sometimes that breaks assumptions.  It's largely taken for granted in
our CI, but it takes a constant sustained effort across the infra team
to make sure we have somewhere to test.

I hear myself sounding negative, but I think it's a fundamental
problem.  You can't be dragging in the latest of everything AND expect
that you won't be constantly running off fixing weird things you never
even knew existed.  We can (and do) get to the bottom of these things,
but if the platform changes again before you've even fixed the current
issue, things start piling up.

If the job is constantly broken it gets ignored -- if a non-voting
job fails in the woods, does it make a sound? :)

> When this happens, moving faster with Python 3 versions will be
> mandatory for everyone, not only for fools like me who made the
> switch early.

This is a long way of saying that - IMO - the idea of putting out a
Debian sid image daily (to a lesser degree Buster images) and throwing
a project's devstack runs against it is unlikely to produce a good
problems-avoided : development-resources ratio.  However, prove me
wrong :)

If people would like to run their master against Fedora (note
OpenStack's stable branch lifespan is generally longer than a given
Fedora release is supported, so it is not much good there) you have
later packages, but still a fairly practical 6-month-ish stability
cadence.  I'm happy to help (some projects do already).

> 

With my rant done :) ... there's already discussion around multiple
python versions, containers, etc in [2].  While I'm reserved about the
idea of full platform functional tests, essentially having a
wide-variety of up-to-date tox environments using some of the methods
discussed there is, I think, a very practical way to be cow-catching
some of the bigger issues with Python version updates.  If we are to
expend resources, my 2c worth is that pushing in that direction gives
the best return on effort.

-i

[1] https://review.openstack.org/#/c/565923/
[2] http://lists.openstack.org/pipermail/openstack-dev/2018-July/132152.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [openstack-infra] How to take over a project?

2018-04-18 Thread Ian Wienand

On 04/19/2018 01:19 AM, Ian Y. Choi wrote:

By the way, since the networking-onos-release group has no neutron
release team group, I think infra team can help to include neutron
release team and neutron release team can help to create branches
for the repo if there is no reponse from current
networking-onos-release group member.


This seems sane and I've added neutron-release to
networking-onos-release.

I'm hesitant to give advice on branching within a project like neutron
as I'm sure there's stuff I'm not aware of; but members of the
neutron-release team should be able to get you going.

Thanks,

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [openstack-infra] How to take over a project?

2018-04-17 Thread Ian Wienand
On 04/17/2018 12:00 PM, Sangho Shin wrote:
> I would like to know how to take over an OpenStack project.  I am a
> committer of the networking-onos project
> (https://github.com/openstack/networking-onos
> ), and I would like to
> take over the project.

> The current maintainer (cc’d) has already agreed with that.

> Please let me know the process to take over (or change the
> maintainer of) the project.

Are you talking about the github project or the gerrit project?
Github is a read-only mirror of the project from gerrit.

You appear to already be a member of networking-onos-core [1] so you
have permissions to approve and reject changes.

> BTW, it looks like even the current maintainer cannot create a new
> branch of the codes. How can we get the authority to create a new
> branch?

Are you following something like [2]?

-i

[1] https://review.openstack.org/#/admin/groups/1001,members
[2] https://docs.openstack.org/infra/manual/drivers.html#feature-branches

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack][infra] pip vs psutil

2018-04-16 Thread Ian Wienand

On 04/15/2018 09:32 PM, Gary Kotton wrote:

The gate is currently broken with
 https://launchpad.net/bugs/1763966. https://review.openstack.org/#/c/561427/
 Can unblock us in the short term. Any other ideas?


I'm thinking this is probably along the lines of the best idea.  I
left a fairly long comment on this in [1], but the root issue here is
that if a system package is created using distutils (rather than
setuptools) we end up with this problem with pip10.

That means the problem occurs when we a) try to overwrite a system
package and b) that package has been created using distutils.  This
means it is a small(er) subset of packages that cause this problem.
Ergo, our best option might be to see if we can avoid such packages on
a one-by-one basis, like here.

In some cases, we could just delete the .egg-info file, which is
approximately what was happening before anyway.

In this particular case, the psutils package is used by glance & the
peakmem tracker.  Under USE_PYTHON3, devstack's pip_install_gr only
installs the python3 library; however the peakmem tracker always uses
python2 -- leaing to missing library the failures in [2].  I have two
thoughts; either install for both python2 & 3 always [3] or make
peakmem tracker obey USE_PYTHON3 [4].  We can discuss the approach in
the reviews.

The other option is to move everything to virtualenv's, so we never
conflict with a system package, as suggested by clarkb [5] or
pabelanger [6].  These are more invasive changes, but also arguably
more correct.

Note diskimage-builder, and hence our image generation for some
platforms, is also broken.  Working on that in [7].

-i


[1] https://github.com/pypa/pip/issues/4805#issuecomment-340987536
[2] https://review.openstack.org/561427
[3] https://review.openstack.org/561524
[4] https://review.openstack.org/561525
[5] https://review.openstack.org/558930
[6] https://review.openstack.org/#/c/552939
[7] https://review.openstack.org/#/c/561479/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Asking for ask.openstack.org

2018-04-04 Thread Ian Wienand

On 04/05/2018 10:23 AM, Zane Bitter wrote:

On 04/04/18 17:26, Jimmy McArthur wrote:
Here's the thing: email alerts. They're broken.


This is the type of thing we can fix if we know about it ... I will
contact you off-list because the last email to what I presume is you
went to an address that isn't what you've sent from here, but it was
accepted by the remote end.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Asking for ask.openstack.org

2018-04-04 Thread Ian Wienand
On 04/05/2018 08:30 AM, Paul Belanger wrote:
> We likely need to reduce the number of days we retain database
> backups / http logs or look to attach a volume to increase storage.

We've long had problems with this host and I've looked at it before
[1].  It often drops out.

It seems there's enough interest we should dive a bit deeper.  Here's
what I've found out:

askbot
--

Of the askbot site, it seems under control, except for an unbounded
session log file.  Proposed [2]

 root@ask:/srv# du -hs *
 2.0G   askbot-site
 579M   dist

overall
---

The major consumer is /var; where we've got

 3.9G   log
 5.9G   backups
 9.4G   lib

backups
---

The backup seem under control at least; we're rotating them out and we
keep 10, and the size is pretty consistently 500mb:

 root@ask:/var/backups/pgsql_backups# ls -lh
 total 5.9G
 -rw-r--r-- 1 root root 599M Apr  5 00:03 askbotdb.sql.gz
 -rw-r--r-- 1 root root 598M Apr  4 00:03 askbotdb.sql.gz.1
 ...

We could reduce the backup rotations to just one if we like -- the
server is backed up nightly via bup, so at any point we can get
previous dumps from there.  bup should de-duplicate everything, but
still, it's probably not necessary.

The db directory was sitting at ~9gb

 root@ask:/var/lib/postgresql# du -hs
 8.9G   .

AFAICT, it seems like the autovacuum is running OK on the busy tables

 askbotdb=# select relname,last_vacuum, last_autovacuum, last_analyze, 
last_autoanalyze from pg_stat_user_tables where last_autovacuum is not NULL;
  relname  | last_vacuum |last_autovacuum| 
last_analyze  |   last_autoanalyze
 
--+-+---+---+---
  django_session   | | 2018-04-02 17:29:48.329915+00 | 2018-04-05 
02:18:39.300126+00 | 2018-04-05 00:11:23.456602+00
  askbot_badgedata | | 2018-04-04 07:19:21.357461+00 |  
 | 2018-04-04 07:18:16.201376+00
  askbot_thread| | 2018-04-04 16:24:45.124492+00 |  
 | 2018-04-04 20:32:25.845164+00
  auth_message | | 2018-04-04 12:29:24.273651+00 | 2018-04-05 
02:18:07.633781+00 | 2018-04-04 21:26:38.178586+00
  djkombu_message  | | 2018-04-05 02:11:50.186631+00 |  
 | 2018-04-05 02:14:45.22926+00

Out of interest I did run a manual

 su - postgres -c "vacuumdb --all --full --analyze"

We dropped something

 root@ask:/var/lib/postgresql# du -hs
 8.9G   .
 (after)
 5.8G   

I installed pg_activity and watched for a while; nothing seemed to be
really stressing it.

Ergo, I'm not sure if there's much to do in the db layers.

logs


This leaves the logs

 1.1G   jetty
 2.9G   apache2

The jetty logs are cleaned regularly.  I think they could be made more
quiet, but they seem to be bounded.

Apache logs are rotated but never cleaned up.  Surely logs from 2015
aren't useful.  Proposed [3]

Random offline
--

[3] is an example of a user reporting the site was offline.  Looking
at the logs, it seems that puppet found httpd not running at 07:14 and
restarted it:

 Apr  4 07:14:40 ask puppet-user[20737]: (Scope(Class[Postgresql::Server])) 
Passing "version" to postgresql::server is deprecated; please use 
postgresql::globals instead.
 Apr  4 07:14:42 ask puppet-user[20737]: Compiled catalog for ask.openstack.org 
in environment production in 4.59 seconds
 Apr  4 07:14:44 ask crontab[20987]: (root) LIST (root)
 Apr  4 07:14:49 ask puppet-user[20737]: 
(/Stage[main]/Httpd/Service[httpd]/ensure) ensure changed 'stopped' to 'running'
 Apr  4 07:14:54 ask puppet-user[20737]: Finished catalog run in 10.43 seconds

Which first explains why when I looked, it seemed OK.  Checking the
apache logs we have:

 [Wed Apr 04 07:01:08.144746 2018] [:error] [pid 12491:tid 140439253419776] 
[remote 176.233.126.142:43414] mod_wsgi (pid=12491): Exception occurred 
processing WSGI script '/srv/askbot-site/config/django.wsgi'.
 [Wed Apr 04 07:01:08.144870 2018] [:error] [pid 12491:tid 140439253419776] 
[remote 176.233.126.142:43414] IOError: failed to write data
 ... more until ...
 [Wed Apr 04 07:15:58.270180 2018] [:error] [pid 17060:tid 140439253419776] 
[remote 176.233.126.142:43414] mod_wsgi (pid=17060): Exception occurred 
processing WSGI script '/srv/askbot-site/config/django.wsgi'.
 [Wed Apr 04 07:15:58.270303 2018] [:error] [pid 17060:tid 140439253419776] 
[remote 176.233.126.142:43414] IOError: failed to write data

and the restart logged

 [Wed Apr 04 07:14:48.912626 2018] [core:warn] [pid 21247:tid 140439370192768] 
AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown 
of previous Apache run?
 [Wed Apr 04 07:14:48.913548 2018] [mpm_event:notice] [pid 21247:tid 
140439370192768] AH00489: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f mod_wsgi/3.4 
Python/2.7.6 configured -- resuming normal operations
 [Wed Apr 04 

Re: [openstack-dev] [tripleo][infra][dib] Gate "out of disk" errors and diskimage-builder 2.12.0

2018-03-21 Thread Ian Wienand

On 03/21/2018 03:39 PM, Ian Wienand wrote:

We will prepare dib 2.12.1 with the fix.  As usual there are
complications, since the dib gate is broken due to unrelated triple-o
issues [2].  In the mean time, probably avoid 2.12.0 if you can.



[2] https://review.openstack.org/554705


Since we have having issues getting this verified due to some
instability in the tripleo gate, I've proposed a temporary removal of
the jobs for dib in [1].

[1] https://review.openstack.org/555037

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra][dib] Gate "out of disk" errors and diskimage-builder 2.12.0

2018-03-20 Thread Ian Wienand

Hi,

We had a small issue with dib's 2.12.0 release that means it creates
the root partition with the wrong partition type [1].  The result is
that a very old check in sfdisk fails, and growpart then can not
expand the disk -- which means you may have seen jobs that usually
work fine run out of disk space.

This slipped by because our functional testing doesn't test growpart;
an oversight we will correct in due course.

The bad images should have been removed, so a recheck should work.

We will prepare dib 2.12.1 with the fix.  As usual there are
complications, since the dib gate is broken due to unrelated triple-o
issues [2].  In the mean time, probably avoid 2.12.0 if you can.

Thanks,

-i

[1] https://review.openstack.org/554771
[2] https://review.openstack.org/554705

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra][all] Anyone using our ubuntu-mariadb mirror?

2018-03-14 Thread Ian Wienand
Hello,

We discovered an issue with our mariadb package mirroring that
suggests it hasn't been updating for some time.

This would be packages from

 http://mirror.X.Y.openstack.org/ubuntu-mariadb/10.<1|2>

This was originally added in [1].  AFAICT from codesearch, it is
currently unused.  We export the top-level directory in the mirror
config scripts as NODEPOOL_MARIADB_MIRROR, which is not referenced in
any jobs [2], and I couldn't find anything setting up apt repos
pointing to it.

Thus since it's not updating and nothing seems to reference it, I am
going to assume it is unused and remove it next week.  If not, please
respond and we can organise a fix.

-i

[1] https://review.openstack.org/#/c/307831/
[2] 
http://codesearch.openstack.org/?q=NODEPOOL_MARIADB_MIRROR=nope==

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] Jens Harbott added to core

2018-03-04 Thread Ian Wienand
Hello,

Jens Harbott (frickler) has agreed to take on core responsibilities in
devstack, so feel free to bug him about reviews :)

We have also added the members of qa-release in directly to
devstack-core, just for visibility (they already had permissions via
qa-release -> devstack-release -> devstack-core).

We have also added devstack-core as grenade core to hopefully expand
coverage there.

---

Always feel free to give a gentle ping on reviews that don't seem have
received sufficient attention.

But please also take a few minutes to compose a commit message!  I
think sometimes devs have been deep in the weeds with their cool
change and devstack requires just a few tweaks.  It's easy to forget
not all reviewers may have this same context.  A couple of
well-crafted sentences can avoid pulling projects and "git blame"
archaeological digs, which gets everything going faster!

Thanks,

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Zanata upgrade to version 4

2018-02-28 Thread Ian Wienand

On 02/27/2018 09:32 PM, Frank Kloeker wrote:

We will take the chance now to upgrade our translation platform to a
new version.


This has been completed and translate.o.o is now running 4.3.3.  For
any issues reply, or catch any infra-root in #openstack-infra

Thanks

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Release-job-failures] release-post job for openstack/releases failed

2018-02-08 Thread Ian Wienand
On 02/09/2018 02:35 PM, Tony Breeds wrote:
>> - tag-releases 
>> http://logs.openstack.org/bd/bd802368fe546a891b89f78fec89d3ea9964c155/release-post/tag-releases/ffc68e7/
>>  : TIMED_OUT in 32m 19s
>> - publish-static publish-static : SKIPPED
> 
> Can we please re-run these jobs.

Done with [1]

-i

[1] 
http://logs.openstack.org/bd/bd802368fe546a891b89f78fec89d3ea9964c155/release-post/tag-releases/2cdfded/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa][requirements] CentOS libvirt versus newton/ocata libvirt-python

2018-01-11 Thread Ian Wienand

On 01/12/2018 02:53 PM, Matthew Thode wrote:

First, about newton, it's dead (2017-10-11).


Yeah, there were a few opt-outs, which is why I think devstack still
runs it.  Not worth a lot of effort.


Next, about ocata, it looks like it can support newer libvirt, but
just because a distro updated a library doesn't mean we have to
update.  IIRC, for ubuntu they use cloud-archives to get the right
version of libvirt, does something like that exist for
centos/redhat?


Well cloud-archives is ports of more recent things backwards, whereas
I think we're in a situation of having too recent libraries in the
base platform.  The CentOS 7.3 v 7.4 situation is a little more subtle
than Trusty v Xenial, say, but fundamentally the same I guess.  The
answer may be "Ocata not supported on 7.4".

p.s. I hope I'm understanding the python-libvirt compat story
correctly.  AIUI any newer python-binding release will build against
older versions of libvirt.  But an old version of python-libvirt may
not build against a newer release of the C libraries?

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [qa][requirements] CentOS libvirt versus newton/ocata libvirt-python

2018-01-11 Thread Ian Wienand
Hi,

So I guess since CentOS included libvirt 3.2 (7-1708, or around RHEL
7.4), it's been incompatible with libvirt-python requirements of 2.1.0
in newton [1] and 2.5.0 in ocata [2] (pike, at 3.5.0, works).

Do we want to do anything about this?  I can think of several options

* bump the libvirt-python versions on older branches

* Create an older centos image (can't imagine we have the person
  bandwidth to maintain this)

* Hack something in devstack (seems rather pointless to test
  something so far outside deployments).

* Turn off CentOS testing for old devstack branches

None are particularly appealing...

(I'm sorry if this has been discussed, I have great déjà vu about it,
maybe we were talking about it at summit or something).

-i

[1] 
http://logs.openstack.org/48/531248/2/check/legacy-tempest-dsvm-neutron-full-centos-7/80fa903/logs/devstacklog.txt.gz#_2018-01-09_05_14_40_960
[2] 
http://logs.openstack.org/50/531250/2/check/legacy-tempest-dsvm-neutron-full-centos-7/1c711f5/logs/devstacklog.txt.gz#_2018-01-09_20_43_08_833

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [requirements][vitrage] Networkx version 2.0

2018-01-07 Thread Ian Wienand

On 12/21/2017 02:51 AM, Afek, Ifat (Nokia - IL/Kfar Sava) wrote:

There is an open bug in launchpad about the new release of Networkx
2.0, that is backward incompatible with versions 1.x [1].


From diskimage-builder's POV, we can pretty much switch whenever
ready, just a matter of merging [2] after constraints is bumped.

It's kind of annoying in the code supporting both versions at once.
If we've got changes ready to go with all the related projects in [1]
bumping *should* be minimal disruption.

-i


[1] https://bugs.launchpad.net/diskimage-builder/+bug/1718576

[2] https://review.openstack.org/#/c/506524/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra][all] Removal of packages from bindep-fallback

2017-11-15 Thread Ian Wienand
Hello,

Some time ago we started the process of moving towards projects being
more explicit about thier binary dependencies using bindep [1]

To facilitate the transition, we created a "fallback" set of
dependencies [2] which are installed when a project does not specifiy
it's own bindep dependencies.  This essentially replicated the rather
ad-hoc environment provided by CI images before we started the
transition.

This list has acquired a few packages that cause some problems in
various situations today.  Particularly packages that aren't in the
increasing number of distributions we provide, or packages that come
from alternative repositories.

To this end, [3,4] proposes the removal of

 liberasurecode-*
 mongodb-*
 python-zmq
 redis
 zookeeper
 ruby-*

from the fallback packages.  This has a small potential to affect some
jobs that tacitly rely on these packages.

NOTE: this does *not* affect devstack jobs (devstack manages it's own
dependencies outside bindep) and if you want them back, it's just a
matter of putting them into the bindep file in your project (and as a
bonus, you have better dependency descriptions for your code).

We should be able to then remove centos-release-openstack-* from our
centos base images too [5], which will make life easier for projects
such as triple-o who have to work-around that.

If you have concerns, please reach out either via mail or in
#openstack-infra

Thank you,

-i

[1] https://docs.openstack.org/infra/bindep/
[2] 
https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt
[3] https://review.openstack.org/519533
[4] https://review.openstack.org/519534
[5] https://review.openstack.org/519535

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Building Openstack Trove Images

2017-11-07 Thread Ian Wienand
On 11/07/2017 05:40 PM, Ritesh Vishwakarma wrote:
> as the *dib-lint* file is there instead of the mentioned
> *disk-image-create *and when executed just verifies the other
> elements.

Those instructions unfortunately look out of date for master
diskimage-builder.  I will try to get a minute to parse them and
update later.

You will probably have a lot more success installing diskimage-builder
into a virtualenv; see [1] ... then activate the virtualenv and use
disk-image-create from there.  Likely the rest will work.

If diskimage-builder is the problem, feel free to jump into
#openstack-dib (best during .au hours to catch me) and we can help.

[1] 
https://docs.openstack.org/diskimage-builder/latest/user_guide/installation.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Distutils][pbr][devstack][qa] Announcement: Pip 10 is coming, and will move all internal APIs

2017-10-22 Thread Ian Wienand

On 10/22/2017 12:18 AM, Jeremy Stanley wrote:

Right, on Debian/Ubuntu it's not too terrible (cloud-init's
dependencies are usually the biggest issue there and we manage to
avoid them by building our own images with no cloud-init), but on
Red Hat derivatives there are a lot of deep operating system
internals built on top of packaged Python libraries which simply
can't be uninstalled cleanly nor safely.


Also note though, if it can be uninstalled, we have often had problems
with the packages coming back and overwriting the pip installed
version, which leads to often very obscure problems.  For this reason
in various bits of devstack/devstack-gate/dib's pip install etc we
often install and pin packages to let pip overwrite them.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Distutils][pbr] Announcement: Pip 10 is coming, and will move all internal APIs

2017-10-22 Thread Ian Wienand

On 10/21/2017 07:14 AM, Clark Boylan wrote:

The current issue this change is facing can be seen at
http://logs.openstack.org/25/513825/4/check/legacy-tempest-dsvm-py35/c31deb2/logs/devstacklog.txt.gz#_2017-10-20_20_07_54_838.
The tl;dr is that for distutils installed packages (basically all the
distro installed python packges) pip refuses to uninstall them in order
to perform upgrades because it can't reliably determine where all the
files are. I think this is a new pip 10 behavior.

In the general case I think this means we can not rely on global pip
installs anymore. This may be a good thing to bring up with upstream
PyPA as I expect it will break a lot of people in a lot of places (it
will break infra for example too).


deja-vu!  pip 8 tried this and quickly reverted.  I wrote a long email
with all the details, but then figured that's not going to help much
so translated it into [1].

-i

[1] https://github.com/pypa/pip/issues/4805

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] Short gerrit / zuul outage 2017-10-20 20:00UTC

2017-10-20 Thread Ian Wienand

On 10/20/2017 03:46 PM, Ian Wienand wrote:

We plan a short outage (<30 minutes) of gerrit and zuul on 2017-10-20
20:00UTC to facilitate project rename requests.


Note this has been postponed to a future (TBD) date

Thanks,

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra] Short gerrit / zuul outage 2017-10-20 20:00UTC

2017-10-19 Thread Ian Wienand

Hello,

We plan a short outage (<30 minutes) of gerrit and zuul on 2017-10-20
20:00UTC to facilitate project rename requests.

In flight jobs should be restarted, but if something does go missing a
"recheck" comment will work.

Thanks,

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition

2017-10-12 Thread Ian Wienand

On 10/12/2017 04:28 PM, Ian Wienand wrote:

- logs issues

Should be behind us.  The logs partition ran out of inodes, causing
log upload failures.  Pruning jobs should have rectified this.


This time it's true :)  But please think about this with your jobs, and
don't upload hundreds of little files unnecessarily.


- Ubuntu package issues

You may notice a range of issues with Ubuntu packages.  The root cause
is that our mirror is behind due a broken reprepro.


Thanks to the efforts of jeblair and pabelanger, the ubuntu mirror
has been restored.  There should be no more issues relating to out
of date mirrors.


- system-config breakage


resolved


- devstack-gate cache copying


resolved

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Zuul v3 Rollout Update - devstack-gate issues edition

2017-10-11 Thread Ian Wienand
There are still significant issues

- logs issues

Should be behind us.  The logs partition ran out of inodes, causing
log upload failures.  Pruning jobs should have rectified this.

- Ubuntu package issues

You may notice a range of issues with Ubuntu packages.  The root cause
is that our mirror is behind due a broken reprepro.  Unfortunately, we
build our daily images against an external upstream mirror, so they
have been built using later packages than our un-updated region
mirrors provide, leading apt to great confusion.  Some debugging notes
on reprepro at [1], but I have to conclude the .db files are corrupt
and I have no idea how to recreate these other than to start again.

I think the most expedient solution here will be to turn /ubuntu on
mirrors into a caching reverse proxy for upstream.  However;

- system-config breakage

The system-config gate is broken due to an old pip pin with [2].
However, despite this merging several hours ago, zuulv2 doesn't seem
to want to reload to pick this up.  I have a suspicion that because it
was merged by zuulv3 maybe zuulv2 missed it?  I'm not sure, and don't
think even turning the jobs -nv will help.

- devstack-gate cache copying

This means the original devstack-gate cache issues [3] remain unmerged
at this point.

[1] 
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-10-12.log.html#t2017-10-12T04:04:16
[2] https://review.openstack.org/511360
[3] https://review.openstack.org/511260

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [DIB] DIB Meetings

2017-10-05 Thread Ian Wienand

On 10/06/2017 02:19 AM, Andreas Scheuring wrote:

seems like there is some confusing information about the DIB
meetings in the wiki [1]. The meeting is alternating between 15:00
and 20:00 UTC.  But whenever the Text says 15:00 UTC, the link
points to a 20:00 UTC worldclock site and vice versa.



What is the correct meeting time? At least today 15:00 UTC no one
was there...


Sorry about that, the idea was to alternate every 2 weeks between an
EU time and a APAC/USA time.  But as you noted I pasted everything in
backwards causing great confusion :) Thanks to tonyb we're fixed up
now.


I put an item on the agenda for today's meeting but can't make 20:00
UTC today. It would be great if you could briefly discuss it and
provide feedback on the patch (it's about adding s390x support to
DIB). I'm also open for any offline discussions.


Sorry, with all going on this fell down a bit.  I'll comment there

Thanks,

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] zuulv3 gate status; LIBS_FROM_GIT failures

2017-09-28 Thread Ian Wienand

On 09/29/2017 03:37 PM, Ian Wienand wrote:

I'm not aware of issues other than these at this time


Actually, that is not true.  legacy-grenade-dsvm-neutron-multinode is
also failing for unknown reasons.  Any debugging would be helpful,
thanks.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] zuulv3 gate status; LIBS_FROM_GIT failures

2017-09-28 Thread Ian Wienand

Hi,

There's a few issues with devstack and the new zuulv3 environment

LIBS_FROM_GIT is broken due to the new repos not having a remote
setup, meaning "pip freeze" doesn't give us useful output.  [1] just
disables the test as a quick fix for this; [2] is a possible real fix
but should be tried a bit more carefully in case there's corners I
missed.  This will be affecting other projects.

However, before we can get this in, we need to fix the gate.  The
"updown" tests have missed a couple of requirement projects due to
them setting flags that were not detected during migration.  [3] is a
fix for that and seems to work.

For some reason, the legacy-tempest-dsvm-nnet job is running against
master, and failing as nova-net is deprecated there.  I'm clutching at
straws to understand this one, as it seems like the branch filters are
setup correctly; [4] is one guess?

I'm not aware of issues other than these at this time

-i

[1] https://review.openstack.org/508344
[2] https://review.openstack.org/508366
[3] https://review.openstack.org/508396
[4] https://review.openstack.org/508405

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] Why do we apt-get install NEW files/debs/general at job time ?

2017-09-19 Thread Ian Wienand

On 09/20/2017 09:30 AM, David Moreau Simard wrote:

At what point does it become beneficial to build more than one image per OS
that is more aggressively tuned/optimized for a particular purpose ?


... and we can put -dsvm- in the jobs names to indicate it should run
on these nodes :)

Older hands than myself will remember even more issues, but the
"thicker" the base-image has been has traditionally just lead to a lot
more corners for corner-cases can hide in.  We saw this all the time
with "snapshot" images where we'd be based on upstream images that
would change ever so slightly and break things, leading to
diskimage-builder and the -minimal build approach.

That said, in a zuulv3 world where we are not caching all git and have
considerably smaller images, a nodepool that has a scheduler that
accounts for flavor sizes and could conceivably understand similar for
images, and where we're building with discrete elements that could
"bolt-on" things like a list-of-packages install sanely to daily
builds ... it's not impossible to imagine.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] Why do we apt-get install NEW files/debs/general at job time ?

2017-09-19 Thread Ian Wienand

On 09/19/2017 11:03 PM, Jeremy Stanley wrote:

On 2017-09-19 14:15:53 +0200 (+0200), Attila Fazekas wrote:
[...]

The jobs does 120..220 sec apt-get install and packages defined
/files/debs/general are missing from the images before starting the job.



Is the time spent at this stage mostly while downloading package
files (which is what that used to alleviate) or is it more while
retrieving indices or installing the downloaded packages (things
having them pre-retrieved on the images never solved anyway)?


As you're both aware, but others may not be, at the end of the logs
devstack does keep a timing overview that looks something like

=
DevStack Component Timing
=
Total runtime1352

run_process   15
test_with_retry4
apt-get-update 2
pip_install  270
osc  365
wait_for_service  29
dbsync23
apt-get  137
=

That doesn't break things down into download v install, but apt does
have download summary that can be grepped for

---
$ cat devstacklog.txt.gz | grep Fetched
2017-09-19 17:52:45.808 | Fetched 39.3 MB in 1s (26.3 MB/s)
2017-09-19 17:53:41.115 | Fetched 185 kB in 0s (3,222 kB/s)
2017-09-19 17:54:16.365 | Fetched 23.5 MB in 1s (21.1 MB/s)
2017-09-19 17:54:25.779 | Fetched 18.3 MB in 0s (35.6 MB/s)
2017-09-19 17:54:39.439 | Fetched 59.1 kB in 0s (0 B/s)
2017-09-19 17:54:40.986 | Fetched 2,128 kB in 0s (40.0 MB/s)
2017-09-19 17:57:37.190 | Fetched 333 kB in 0s (1,679 kB/s)
2017-09-19 17:58:17.592 | Fetched 50.5 MB in 2s (18.1 MB/s)
2017-09-19 17:58:26.947 | Fetched 5,829 kB in 0s (15.5 MB/s)
2017-09-19 17:58:49.571 | Fetched 5,065 kB in 1s (3,719 kB/s)
2017-09-19 17:59:25.438 | Fetched 9,758 kB in 0s (44.5 MB/s)
2017-09-19 18:00:14.373 | Fetched 77.5 kB in 0s (286 kB/s)
---

As mentioned, we setup the package manager to point to a region-local
mirror during node bringup.  Depending on the i/o situation, it is
probably just as fast as coming off disk :) Note (also as mentioned)
these were never pre-installed, just pre-downloaded to an on-disk
cache area (as an aside, I don't think dnf was ever really happy with
that situation and kept being too smart and clearing it's caches).

If you're feeling regexy you could maybe do something similar with the
pip "Collecting" bits in the logs ... one idea for investigation down
that path is if we could save time by somehow collecting larger
batches of requirements and doing less pip invocations?

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [diskimage-builder] Does anyone use "fedora" target?

2017-08-29 Thread Ian Wienand

Hi,

The "fedora" element -- the one that downloads the upstream .qcow2 and
re-packages it -- is currently broken as the links we use have
disappeared [1].  Even allowing for this, it's still broken with some
changes to the kernel install scripts [2].  AFAICT, the only thing
noticing this is our CI.

fedora-minimal takes a different approach of building the system
within a blank chroot.  It's what we use to create the upstream
images.

I believe the octavia jobs switched to fedora-minimal?

Is there anyone still using these image-based jobs?  Is there any
reason why you can't use fedora-minimal?  I don't really see this as
being that useful, and our best path forward might be just to retire
it.

Thanks,

-i

[1] https://review.openstack.org/497734
[2] https://bugs.launchpad.net/diskimage-builder/+bug/1713381

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all] Python 3.6 testing is available on Fedora 26 nodes

2017-08-24 Thread Ian Wienand
Hello,

In a recent discussion [1] I mentioned we could, in theory, use Fedora
26 for Python 3.6 testing (3.6.2, to be exact).  After a few offline
queries we have put theory into practice, sorted out remaining issues
and things are working.

For unit testing (tox), you can use the
'gate-{name}-python36-{node}-nv' job template with fedora-26 nodes.
For an example, see [2] (which, I'm happy to report, found a real
issue [3] :).  You may need to modify your bindep.txt files to install
correct build pacakges for RPM platforms; in terms of general
portability this is probably not a bad thing anyway.

I have an up/down devstack test working with a minor change [4].  I
will work on getting this more stable and more complete, but if this
is of interest, reach out.  In general, I track the centos & fedora
jobs fairly closely at [5] to try and keep up with any systemic
issues.

Although it is not exactly trivial, there is fairly complete
instructions within [6] to help build a Fedora image that looks like
the infra ones for testing purposes.  You can also reach out and we
can do things like place failed nodes on hold if there are hard to
debug issues.

Thanks,

-i

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-August/120888.html
[2] 
https://git.openstack.org/cgit/openstack-infra/project-config/commit/?id=5fe3ba95616136709a319ae1cd3beda38a299a13
[3] https://review.openstack.org/496054
[4] https://review.openstack.org/496098
[5] http://people.redhat.com/~iwienand/devstack-status/
[6] 
https://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][infra]Plan to support Python 3.6 ?

2017-08-10 Thread Ian Wienand

On 08/09/2017 11:25 PM, ChangBo Guo wrote:

We received Python 3.6 related Bug recently [1][2]. That let me think
what's the plan to support Python 3.6 for OpenStack in the future.   Python
3.6 was released on December 23, 2016, has some different behaviors from
Python 3.5[3]. talked with cdent in the IRC, would like to discuss this
through mailing list , and suggest a discussion at the PTG[3]

1. what's the time to support Python 3.6 ?

2. what 's the  plan or process ?


If you really want to live on the edge, Fedora 26 CI nodes are
available and include Python 3.6.  I'll be happy to help if you're not
familiar with using different nodes for jobs, or with any issues.

I've had devstack going in experimental successfully.  I probably
wouldn't recommend throwing it in as a voting gate job straight away,
but everything should be there :)

-i


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [heat][infra] Help needed! high gate failure rate

2017-08-10 Thread Ian Wienand
On 08/10/2017 06:18 PM, Rico Lin wrote:
> We're facing a high failure rate in Heat's gates [1], four of our gate
> suffering with fail rate from 6 to near 20% in 14 days. which makes most of
> our patch stuck with the gate.

There have been a confluence of things causing some problems recently.
The loss of OSIC has distributed more load over everything else, and
we have seen an increase in job timeouts and intermittent networking
issues (especially if you're downloading large things from remote
sites).  There have also been some issues with the mirror in rax-ord
[1]

> gate-heat-dsvm-functional-convg-mysql-lbaasv2-ubuntu-xenial(19.67%)
> gate-heat-dsvm-functional-convg-mysql-lbaasv2-non-apache-ubuntu-xenia(9.09%)
> gate-heat-dsvm-functional-orig-mysql-lbaasv2-ubuntu-xenial(8.47%)
> gate-heat-dsvm-functional-convg-mysql-lbaasv2-py35-ubuntu-xenial(6.00%)

> We still try to find out what's the cause but (IMO,) seems it might be some
> thing wrong with our infra. We need some help from infra team, to know if
> any clue on this failure rate?

The reality is you're just going to have to triage this and be a *lot*
more specific with issues.  I find opening an etherpad and going
through the failures one-by-one helpful (e.g. I keep [2] for centos
jobs I'm interested in).

Looking at the top of the console.html log you'll have the host and
provider/region stamped in there.  If it's timeouts or network issues,
reporting to infra the time, provider and region of failing jobs will
help.  If it's network issues similar will help.  Finding patterns is
the first step to understanding what needs fixing.

If it's due to issues with remote transfers, we can look at either
adding specific things to mirrors (containers, images, packages are
all things we've added recently) or adding a caching reverse-proxy for
them ([3],[4] some examples).

Questions in #openstack-infra will usually get a helpful response too

Good luck :)

-i

[1] https://bugs.launchpad.net/openstack-gate/+bug/1708707/
[2] https://etherpad.openstack.org/p/centos7-dsvm-triage
[3] https://review.openstack.org/491800
[4] https://review.openstack.org/491466

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra][devstack] DIB builds after mysql.qcow2 removal

2017-07-17 Thread Ian Wienand

On 07/18/2017 10:01 AM, Tony Breeds wrote:

It wasn't forgotten as suchi, there are jobs still using it/them.  If
keeping the branches around cuases bigger probelsm then EOLing them is
fine.  I'll try to generate a list of the affected projects/jobs and
turn them off.


Thanks; yeah this was pointed out to me later.

I think any jobs can use the -eol tag, rather than the
branch if required (yes, maybe easier said than done depending on how
many layers of magic there are :).  There doesn't seem to be much
point in branches we can't commit to due to broken CI, and I doubt
anyone is keen to maintain it.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra][devstack] DIB builds after mysql.qcow2 removal

2017-07-17 Thread Ian Wienand
Hi,

The removal of the mysql.qcow2 image [1] had a flow-on effect noticed
first by Paul in [2] that the tools/image_list.sh "sanity" check was
not updated, leading to DIB builds failing in a most unhelpful way as
it tries to cache the images for CI builds.

So while [2] fixes the problem; one complication here is that the
caching script [3] loops through the open devstack branches and tries
to collect the images to cache.

Now it seems we hadn't closed the liberty or mitaka branches.  This
causes a problem, because the old branches refer to the old image, but
we can't actually commit a fix to change them because the branch is
broken (such as [4]).

I have taken the liberty of EOL-ing stable/liberty and stable/mitaka
for devstack.  I get the feeling it was just forgotten at the time.
Comments in [4] support this theory.  I have also taken the liberty of
approving backports of the fix to newton and ocata branches [5],[6].

A few 3rd-party CI people using dib have noticed this failure.  As the
trio of [4],[5],[6] move through, your builds should start working
again.

Thanks,

-i

[1] https://review.openstack.org/482600
[2] https://review.openstack.org/484001
[3] 
http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos
[4] https://review.openstack.org/482604
[5] https://review.openstack.org/484299
[6] https://review.openstack.org/484298

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra][nova] Corrupt nova-specs repo

2017-06-30 Thread Ian Wienand
Hi,

Unfortunately it seems the nova-specs repo has undergone some
corruption, currently manifesting itself in an inability to be pushed
to github for replication.

Upon examination, it seems there's a problem with a symlink and
probably jgit messing things up making duplicate files.  I have filed
a gerrit bug at [1] (although it's probably jgit, but it's just a
start).

Anyway, that leaves us the problem of cleaning up the repo into a
pushable state.  Here's my suggestion after some investigation:

The following are corrupt

---
$ git fsck
Checking object directories: 100% (256/256), done.
error in tree a494151b3c661dd9b6edc7b31764a2e2995bd60c: contains duplicate file 
entries
error in tree 26057d370ac90bc01c1cfa56be8bd381618e2b3e: contains duplicate file 
entries
error in tree 57423f5165f0f1f939e2ce141659234cbb5dbd4e: contains duplicate file 
entries
error in tree 05fd99ef56cd24c403424ac8d8183fea33399970: contains duplicate file 
entries
---

After some detective work [2], I related all these bad objects to the
refs that hold them.  It look as follows

---
fsck-bad: a494151b3c661dd9b6edc7b31764a2e2995bd60c
 -> 5fa34732b45f4afff3950253c74d7df11b0a4a36 refs/changes/26/463526/9

fsck-bad: 26057d370ac90bc01c1cfa56be8bd381618e2b3e
 -> 47128a23c2aad12761aa0df5742206806c1dfbb8 refs/changes/26/463526/8
 -> 7cf8302eb30b722a00b4d7e08b49e9b1cd5aacf4 refs/changes/26/463526/7
 -> 818dc055b971cd2b78260fd17d0b90652fb276fb refs/changes/26/463526/6

fsck-bad: 57423f5165f0f1f939e2ce141659234cbb5dbd4e

 -> 25bd72248682b584fb88cc01061e60a5a620463f refs/changes/26/463526/3
 -> c7e385eaa4f45b92e9e51dd2c49e799ab182ac2c refs/changes/26/463526/4
 -> 4b8870bbeda2320564d1a66580ba6e44fbd9a4a2 refs/changes/26/463526/5

fsck-bad: 05fd99ef56cd24c403424ac8d8183fea33399970
 -> e8161966418dc820a4499460b664d87864c4ce24 refs/changes/26/463526/2
---

So you may notice this is refs/changes/26/463526/[2-9]

Just deleting these refs and expiring the objects might be the easiest
way to go here, and seems to get things purged and fix up fsck

---
$ for i in `seq 2 9`; do
>  git update-ref -d refs/changes/26/463526/$i
> done

$ git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 44756, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (43850/43850), done.
Writing objects: 100% (44756/44756), done.
Total 44756 (delta 31885), reused 12846 (delta 0)

$ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (44756/44756), done.
---

I'm thinking if we then force push that to github, we're pretty much
OK ... a few intermediate reviews will be gone but I don't think
they're important in this context.

I had a quick play with "git ls-tree", edit the file, "git mktree",
"git replace" and then trying to use filter-branch, but couldn't get
it to work.  Suggestions welcome; you can play with the repo from [1]
I would say.

Thanks,

-i

[1] https://bugs.chromium.org/p/gerrit/issues/detail?id=6622
[2] "git log --all --format=raw --raw -t --no-abbrev" and search for
the change sha, then find it in "git show-refs"

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] diskimage builder works for trusty but not for xenial

2017-06-21 Thread Ian Wienand

On 06/21/2017 04:44 PM, Ignazio Cassano wrote:

* Connection #0 to host cloud-images.ubuntu.com left intact
Downloaded and cached
http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz,
having forced upstream caches to revalidate
xenial-server-cloudimg-amd64-root.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match



Are there any problems on http://cloud-images.ubuntu.com ?


There was [1] which is apparently fixed.

As Paul mentioned, the -minimal builds take a different approach and
build the image from debootstrap, rather than modifying the upstream
image.  They are generally well tested just as a side-effect of infra
relying on them daily.  You can use DIB_DISTRIBUTION_MIRROR to set
that to a local mirror and eliminate another source of instability
(however, that leaves the mirror in the final image ... a known issue.
Contributions welcome :)

-i

[1] https://bugs.launchpad.net/cloud-images/+bug/1699396


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra] shutting down pholio.openstack.org

2017-06-12 Thread Ian Wienand
Hello,

We will be shutting down pholio.openstack.org in the next few days.

As discussed at the last #infra meeting [1], in short, "the times they
are a changin'" and the Pholio services have not been required.

Of course the original deployment puppet, etc, remains (see [2]), so
you may reach out to the infra team if this is required in the future.

Thanks,

-i

[1] 
http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-06-06-19.03.html
[2] https://specs.openstack.org/openstack-infra/infra-specs/specs/pholio.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra] openstack-ubuntu-testing-bot -- please turn off

2017-06-09 Thread Ian Wienand

Hi,

If you know of someone in control of whatever is trying to use this
account, running on 91.189.91.27 (a canonical IP), can you please turn
it off.  It's in a tight loop failing to connect to gerrit, which
probably isn't good for either end :)

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [diskimage-builder] Restarting bi-weekly meeting

2017-05-25 Thread Ian Wienand

Hi,

We've let this meeting [1] lapse, to our communications detriment.  I
will restart it, starting next week [2].  Of course agenda items are
welcome, otherwise we will use it as a session to make sure patches
are moving in the right direction.

If the matter is urgent, and not getting attention, an agenda item in
the weekly infra meeting would be appropriate.

Ping me off list if you're interested but this time doesn't work.  If
there's a few, we can move it.

Thanks,

-i

[1] http://eavesdrop.openstack.org/#Diskimage_Builder_Team_Meeting
[2] https://review.openstack.org/468270

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] Stop enabling EPEL mirror by default

2017-04-04 Thread Ian Wienand
On 04/05/2017 03:02 AM, Paul Belanger wrote:
> Recently we've been running into some issues keeping our EPEL mirror
> properly sync'd. We are working to fix this, however we'd also like
> to do the following:

>   Stop enabling EPEL mirror by default
>   https://review.openstack.org/#/c/453222/

> For the most part, we enable EPEL for our image build process, this
> to install haveged.  However, it is also likely the majority of
> centos-7 projects don't actually need EPEL.  I know specifically
> both RDO and TripleO avoid using the EPEL repository because of how
> unstable it is.

I agree this is the step to turn it off in our gate, but I've been
trying to excise this so we move to a white-list method during builds,
which is more complicated.  This needs to be done, however, so that
3rd party CI who don't use our mirror scripts don't get EPEL hanging
around from the build too.

I'd appreciate reviews

Firstly, we need to ensure the image build EPEL dependencies we have
are flexible to changes in default status.

 * https://review.openstack.org/439294 : don't install ccache
 * https://review.openstack.org/439911 : allow "--enablerepo" options
 for haveged install
 * https://review.openstack.org/439917 : install haveged from EPEL

Then we need a way to install EPEL, but disabled, during image builds

 * https://review.openstack.org/439926 : Add flag to disable EPEL

Then stop installing EPEL as part of the puppet install, and switch to
installing it from dib in disabled state

 * https://review.openstack.org/453322 : Add epel element (with disabled flag)
 * https://review.openstack.org/439248 : Don't install EPEL during puppet

At this point, our base images should be coming up with only
whitelisted EPEL packages (haveged, unless I find anything else I've
missed) and the repo disabled.

-i

p.s. tangential; but related is

 *  https://review.openstack.org/453325 : use Centos openstack repos, not RDO

This could probably be also moved into DIB as an element, if
we feel strongly about it (or infra-package-needs ... but I wasn't
100% sure that's early enough yet)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Switching gate testing to use Ubuntu Cloud Archive

2017-04-03 Thread Ian Wienand
On 04/04/2017 09:06 AM, Clark Boylan wrote:
> I have pushed a change to devstack [3] to enable using UCA which pulls
> in new Libvirt and mostly seems to work. I think we should consider
> switching to UCA as this may fix our Libvirt problems and if it doesn't,
> we will be closer to a version of Libvirt that upstream should be
> willing to fix.

I'm not 100% sure where this leaves the
devstack-plugin-additional-pkg-repos work [1]; although to be honest
I've always thought that was a little unspecific.  We also have
"devstack-plugin-tar-installer" [2] which was working on installing
libvirt from upstream tarballs IIRC but looks dead?  There was quite
some discussion about this in [3] at the time.

> Finally it is worth noting that we will get newer packages of other
> software as well, most notably openvswitch will be version 2.6.1 instead
> of 2.5.0.

Just to write down the centos side of things -- we have the RDO repo
installed on all centos CI nodes.  That brings in the centos virt-sig
rebuild of qemu (qemu-kvm-ev) which is definitely required to run
trunk nova [4].  We are just using libvirt from standard updates
(2.0.0 based).  RDO provides openvswitch, which is 2.6.1 too.

> Then have specific jobs (like devstack) explicitly opt into the UCA
> repo appropriate for them (if any).

The idea being, presumably, that when devstack branches, it is
essentially pinned to whatever version of UCA it was using at the time
for its lifespan.

I do wonder if installing the latest version on the base images
simplifies things (so that by default, "apt-get install libvirt" in
any context gets the UCA version).  To handle the above, a devstack
branch would have to essentially do the opposite -- override the
default to whatever version it is pinned to.  More work at branch
point, however has the advantage that we don't have to constantly
communicate to people they need to "opt-in".

-i

[1] 
https://git.openstack.org/cgit/openstack/devstack-plugin-additional-pkg-repos
[2] https://git.openstack.org/cgit/openstack/devstack-plugin-tar-installer
[3] https://review.openstack.org/#/c/108714/
[4] http://mirror.centos.org/centos/7/virt/x86_64/kvm-common/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Your next semi weekly gate status report

2017-03-29 Thread Ian Wienand

On 03/28/2017 08:57 AM, Clark Boylan wrote:

1. Libvirt crashes: http://status.openstack.org/elastic-recheck/#1643911
and http://status.openstack.org/elastic-recheck/#1646779



Libvirt is randomly crashing during the job which causes things to fail
(for obvious reasons). To address this will likely require someone with
experience debugging libvirt since it's most likely a bug isolated to
libvirt. We're looking for someone familiar with libvirt internals to
drive the effort to fix this issue,


Ok, from the bug [1] we're seeing malloc() corruption.

While I agree that a coredump is not that likely to help, I would also
like to come to that conclusion after inspecting a coredump :) I've
found things in the heap before that give clues as to what real
problems are.

To this end, I've proposed [2] to keep coredumps.  It's a little
hackish but I think gets the job done. [3] enables this and saves any
dumps to the logs in d-g.

As suggested, running under valgrind would be great but probably
impractical until we narrow it down a little.  Another thing I've had
some success with is electric fence [4] which puts boundaries around
allocations so out-of-bounds access hits at the time of access.  I've
proposed [5] to try this out, but it's not looking particularly
promising unfortunately.  I'm open to suggestions, for example maybe
something like tcalloc might give us a different failure and could be
another clue.  If we get something vaguely reliable here, our best bet
might be to run a parallel non-voting job on all changes to see what
we can pick up.

-i

[1] https://bugs.launchpad.net/nova/+bug/1643911
[2] https://review.openstack.org/451128
[3] https://review.openstack.org/451219
[4] http://elinux.org/Electric_Fence
[5] https://review.openstack.org/451136

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib] diskimage-builder v2 RC1 release; request for test

2017-03-20 Thread Ian Wienand

On 03/21/2017 03:10 AM, Mikhail Medvedev wrote:

On Fri, Mar 17, 2017 at 3:23 PM, Andre Florath  wrote:
Submitted the bug https://bugs.launchpad.net/diskimage-builder/+bug/1674402


Thanks; some updates there.


Would adding a third-party CI job help? I can put together a
functional job on ppc64. I assume we want a job based on
gate-dib-dsvm-functests-*?


As discussed in #openstack-dib we have this reporting on a group of
the functional tests.  My only concern is biting off more than we can
chew initially and essentially training people that the results are
unreliable.  Once we get over this initial hurdle we can look at
expanding it and voting.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-17 Thread Ian Wienand
On 03/17/2017 03:46 AM, Steven Hardy wrote:
> (undercloud) [stack@undercloud ~]$ rpm -qf /usr/bin/dib-run-parts
> dib-utils-0.0.11-1.el7.noarch
> (undercloud) [stack@undercloud ~]$ rpm -qf /bin/dib-run-parts
> dib-utils-0.0.11-1.el7.noarch

/bin is a link to /usr/bin?  So I think this is the same and this is
the dib-run-parts as pacakged by dib-utils.

> (undercloud) [stack@undercloud ~]$ rpm -qf 
> /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts
> diskimage-builder-2.0.1-0.20170314023517.756923c.el7.centos.noarch

This is dib's "private" copy.  As I mentioned in the other mail, the
intention was to vendor this so we could re-write for dib-specific
needs if need be (given future requirements such as running in
restricted container environments).  I believe having dib exporting
this was an (my) oversight.  I have proposed [1] to remove this.

> (undercloud) [stack@undercloud ~]$ rpm -qf /usr/local/bin/dib-run-parts
> file /usr/local/bin/dib-run-parts is not owned by any package

This would come from the image build process.  We copy dib-run-parts
into the chroot to run in-target scripts [2] but we never actually
remove it.  This seems to me to also be a bug and I've proposed [3] to
run this out of /tmp and clean it up.

> But the exciting thing from a rolling-out-bugfixes perspective is that the
> one actually running via o-r-c isn't either of the packaged versions (doh!)
> so we probably need to track down which element is installing it.
>
> This is a little OT for this thread (sorry), but hopefully provides more
> context around my concerns about creating another fork etc.

I don't want us to get a little too "left-pad" [4] with this 95ish
lines of shell :) I think this stack clears things up.

tl;dr

 - dib version should be vendored; not in path & not exported [1]
 - unnecessary /usr/local/bin version removed [3]
 - dib-utils provides /usr/bin/ version

Cross-ports between the vendored dib version and dib-utils should be
trivial given what it is.  If dib wants to rewrite it's vendored
version, or remove it completely, this will not affect anyone
depending on dib-utils.

Thanks,

-i

[1] https://review.openstack.org/446285 (dib: do not provide dib-run-parts)
[2] 
https://git.openstack.org/cgit/openstack/diskimage-builder/tree/diskimage_builder/elements/dib-run-parts/root.d/90-base-dib-run-parts
[3] https://review.openstack.org/446769 (dib: run chroot dib-run-parts out of 
/tmp)
[4] http://left-pad.io/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

2017-03-15 Thread Ian Wienand

On 03/16/2017 08:22 AM, Ben Nemec wrote:

Anyway, I don't know that anything is broken at the moment since I
believe dib-run-parts was brought over unchanged, but the retirement of
dib-utils was proposed in https://review.openstack.org/#/c/445617 and I
would like to resolve this question before we do anything like that.


The underlying motivation behind this was to isolate dib so we could
do things like re-implement dib-run-parts in posixy shell (for busybox
environments) or python, etc.

So my idea was we'd just leave dib-utils alone.  But it raises a good
point that both dib-utils and diskimage-builder are providing
dib-run-parts.  I think this is probably the main oversight here.

I've proposed [1] that makes dib use dib-run-parts from its private
library dir (rather than any globally installed version) and stops it
exporting the script to avoid conflict with dib-utils.  I think this
should allow everything to live in harmony?

-i

[1] https://review.openstack.org/#/c/446285/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] PBR 2.0.0 release *may* cause gate failures

2017-03-01 Thread Ian Wienand

On 03/02/2017 01:13 AM, Doug Hellmann wrote:

Tony identified caps in 5 OpenStack community projects (see [1]) as well
as powervm and python-jsonpath-rw-ext. Pull requests to those other
projects are linked from the bug [2].



[1] https://review.openstack.org/#/q/topic:bug/1668848


Am I nuts or was pbr itself the only one forgotten?

I filed

 https://review.openstack.org/#/c/440010

under the same topic

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [all] Moving coverage jobs to check queue

2017-02-13 Thread Ian Wienand
Hello,

In a prior thread we discussed moving coverage jobs from "post" jobs
to the check queue [1].

Firstly, if you have no idea what I'm talking about, the coverage jobs
run the unit tests under the coverage tool [2] and produce some output
like [3] which identifies which parts of the code have been executed.

It seems the history of these jobs in "post" was that they took a
little too long in the check queue with added measurement overhead.
Looking at, for example, nova jobs, the coverage job is currently in
the 10 minute range.  I'm not sure if the coverage tool has got better
or nova reduced the job time; both are likely.  This has led to much
inconsistency as to where we run the coverage jobs -- it mostly looks
like where they run depends on which other project you used as a
template.  Some projects run in both check and post which seems
unnecessary.

The coverage job results in post are quite hard to find.  You need to
firstly know the job even runs, then find the correct commit sha
(which is probably the merge commit, not the one in gerrit) and then
know how to manually navigate the logs.openstack.org file-hierarchy.
It's probably no surprise that according to apache logs, nobody has
accessed a post coverage-job output at all within about the last
month.  Also, as per the prior email, if the job is actually failing
you get no notification at all.

A recent change has made "-nv" (non-voting) coverage jobs available
[4] which simplifies this somewhat, as there is no need to put special
regexes in to stop voting.

I have proposed [5] which moves all coverage post jobs to non-voting
check queue jobs.  It also renames them with our standard "gate-" for
consistency.  I believe this will improve usage of the tool and also
clean-up our layout.yaml a bit.

Feel free to raise concerns in [5]

Thanks,

-i

[1] http://lists.openstack.org/pipermail/openstack-dev/2016-July/099491.html
[2] https://coverage.readthedocs.io/en/coverage-4.3.4/
[3] 
http://logs.openstack.org/c3/c3671ee7da154e251c2915d4aced2b1a2bd8dfa9/post/nova-coverage-db-ubuntu-xenial/bc21639/cover/
[4] https://review.openstack.org/431783
[5] https://review.openstack.org/432836

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [dib] diskimage-builder v2 RC1 release; request for test

2017-02-08 Thread Ian Wienand
Hi

diskimage-builder has been working on a feature/v2 branch
incorporating some largely internal changes to the way it finds and
calls elements, enhancements to partitioning and removal of some
long-deprecated elements.  We have just tagged 2.0.0rc1 and are
requesting testing by interested parties.

You can view the branch at

 https://git.openstack.org/cgit/openstack/diskimage-builder/log/?h=feature/v2

For any issues it is probably best to file a bug.  Developers are
standing by in #openstack-dib.

=== User facing changes

If you use dib exclusively via the command-line disk-image-create
installed from a package or via pypi you are unlikely to notice any
difference (if you run it directly from a git-tree checkout, you're in
development territory so read on).

v2 includes a new framework for partitioning contributed by Andreas
Florath.  This should allow for creating multiple partitions, images
with encryption, LVM support and flexibility for multiple-devices, all
of which are currently not supported.  Please check the v2
documentation, specs and reach out if these features interest you
(some parts still in review).

Element override is now supported.  If you have an element of the same
name earlier in the ELEMENTS_PATH, it will override later instances
(previously, the behaviour was undefined).

A number of long-deprecated elements have been removed in v2, which
are to the best of our knowledge unused.

 - partitioning-sfdisk
 - deploy-ironic-element
 - ironc-discovered-ramdisk
 - serial-console-element
 - map-services

=== Developer changes

Developers, or those that are doing more "interesting" things in
deployment or CI, might like to read below to follow the history of
the branch and hopefully do some testing with the feature/v2 branch
ASAP so we can address any concerns.

dib started out mostly as a shell script collection that used setup.py
to install itself.  Over time it has made more and more use of Python.
The current instructions say to checkout the source tree and run
"./bin/disk-image-create"; this causes a number of issues for the
internal Python components.  We carry hacks to try and figure out of
we're being called uninstalled from a source tree, or installed in pip
develop mode, or installed on the system.

For purposes of both users and development we want dib to be as
"pythonic" as possible and behave like all other projects.  Two major
visible changes are:

 - command-line scripts are entry points (i.e. need to be installed)
 - elements have moved under diskimage_create module

The result of the first is that "./bin/disk-image-create" from the
source tree is no longer there.  Like all other projects, you should
install dib into a virtualenv (if you're developing, use pip -e) and
"disk-image-create" will "just work".  FYI, on the back-end, this
entry-point essentially exec()s the old shell-script driver; but we
reserve the right to move more of the logic into python over time.
Also, instead of having to os.system(/path/disk-image-create), we can
allow direct calls like a real library.

A side-effect of this is that we have removed and deprecated the
dib-utils package.  This was intended to be a more generic repository
of tools that might be useful outside dib, but that did not eventuate
and it has been folded back into dib for simplicity.

The second change, moving the inbuilt elements under the
diskimage_create module, is a simplification so we always have a
canonical path to our elements.  Currently the elements are installed
via "data_files" in setup.cfg, which means setup.py puts the elements
in /usr/share... when installing, or leaves them in the tree when
installed via pip -e.  Again we have path guessing hacks trying to
guess what situation we're in.  Elements have moved from elements/ to
diskimage_builder/elements in the source tree (we did try to minimise
this with symlinks, but setuptools doesn't like it, it complicates
things when git merging across the branches and is legacy cruft to go
wrong in the future.  It was better to make a clean break).

Since we always know where elements are relative to the imported
diskimage_builder module we can drop all the path guessing complexity.
This has other good flow-on effects such as testr being able to find
unit-tests for elements in the normal fashion and having imports work
as usual.

We are aware there are a number of tools that like to take dib
elements and do things with them.  This is fine, I guess, although of
course if we don't know about how you use elements we're liable to
break them.  Thus your use may be confused that the elements have
moved.

Reading some of the dib source you may find there is a canonical way
to find out the included dib elements path -- ask dib itself,
something like:

 DIB_ELEMENTS=$(python -c '
 import diskimage_builder.paths;
 diskimage_builder.paths.show_path("elements")')

Note you probably do not want this.  As mentioned, another feature of
v2 is override elements -- an element that appears 

Re: [openstack-dev] [ironic-pyhon-agent][DIB] IPA failed to start on Ubuntu because of modprobe path

2017-01-22 Thread Ian Wienand

On 01/21/2017 09:31 PM, Moshe Levi wrote:

Added ExecStartPre=/usr/sbin/modprobe vfat to the
ironic-python-agent systemd script The problem is that modprobe
location in Ubuntu / sbin/modprobe (the /usr/sbin/modprobe vfat
works for redhat)



I have opened this bug in Launchpad

> https://bugs.launchpad.net/diskimage-builder/+bug/1658297


What is the best way to fix this?


This should be just "/sbin"; I've proposed [1].  There may be even
better ways to do this which anyone is welcome to propose :)

-i

[1] https://review.openstack.org/423893

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra][Neutron] Running out of memory on gate for linux bridge job

2017-01-18 Thread Ian Wienand
On 01/14/2017 02:48 AM, Jakub Libosvar wrote:
> recently I noticed we got oom-killer in action in one of our jobs [1]. 

> Any other ideas?

I spent quite a while chasing down similar things with centos a while
ago.  I do have some ideas :)

The symptom is probably that mysql gets chosen by the OOM killer but
it's unlikely to be mysql's fault, it's just big and a good target.

If the system is going offline, I added the ability to turn on the
netconsole in devstack-gate with [1].  As the comment mentions, you
can put little tests that stream data in /dev/kmsg and they will
generally get off the host, even if ssh has been killed.  I found this
very useful for getting the initial oops data (i've used this several
times for other gate oopses, including other kernel issues we've
seen).

For starting to pin down what is really consuming the memory, the
first thing I did was wrote a peak-memory usage tracker that gave me
stats on memory growth during the devstack run [2].  You have to
enable this with "enable_service peakmem_tracker".  This starts to
give you the big picture of where memory is starting to go.

At this point, you should have a rough idea of the real cause, and
you're going to want to start dumping /proc/pid/smaps of target
processes to get an idea of where the memory they're allocating is
going, or at the very least what libraries might be involved.  The
next step is going to depend on what you need to target...

If it's python, it can get a bit tricky to see where the memory is
going but there's a number of approaches.  At the time, despite it
being mostly unmaintained but I had some success with guppy [1].  In
my case, for example, I managed to hook into swift's wsgi startup and
run that under guppy, giving me the ability to get some heap stats.
from my notes [4] that looked something like

---
import signal, os
from guppy import hpy

def handler(signum, frame):
f = open('/tmp/heap.txt', 'w+')
f.write("testing\n")
hp = hpy()
f.write(str(hp.heap()))
f.close()

if __name__ == '__main__':
conf_file, options = parse_options()
signal.signal(signal.SIGUSR1, handler)

sys.exit(run_wsgi(conf_file, 'object-server',
  global_conf_callback=server.global_conf_callback,
  **options))
---

There are of course other tools from gdb to malloc tracers, etc.

But that was enough that I could try different things and compare the
heap usage.  Once you've got the smoking gun ... well then the hard
work starts of fixing it :) In my case it was pycparser and we came up
with a good solution [5].

Hopefully that's some useful tips ... #openstack-infra can of course
help holding vms etc as required.

-i

[1] 
http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate-wrap.sh#n438
[2] 
https://git.openstack.org/cgit/openstack-dev/devstack/tree/tools/peakmem_tracker.sh
[3] https://pypi.python.org/pypi/guppy/
[4] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[5] https://github.com/eliben/pycparser/issues/72

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Gerrit downtime on Thursday 2017-01-12 at 20:00 UTC

2017-01-10 Thread Ian Wienand
Hi everyone,

On Thursday, January 12th from approximately 20:00 through 20:30 UTC
Gerrit will be unavailable while we complete project renames.

Currently, we plan on renaming the following projects:

 Nomad -> Cyborg
  - openstack/nomad -> openstack/cyborg

 Nimble -> Mogan 
  - openstack/nimble -> openstack/mogan
  - openstack/python-nimbleclient -> openstack/python-moganclient
  - openstack/nimble-specs -> openstack/mogan-specs

Existing reviews, project watches, etc, for these projects will all be
carried over.

This list is subject to change. If you need a rename, please be sure
to get your project-config change in soon so we can review it and add
it to 
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames

If you have any questions about the maintenance, please reply here or
contact us in #openstack-infra on freenode.

-i 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [gerrit] dell-afc-ci disabled but constantly connecting to gerrit

2016-12-14 Thread Ian Wienand
Hi,

There's a zombie account "dell-afc-ci" which was disabled in April [1]
that has been trying to connect to gerrit every 5 seconds since

 ...
 [2016-12-15 06:27:44,223 +] 946a4473 dell-afc-ci - AUTH FAILURE FROM 
143.166.116.138 inactive-account
 [2016-12-15 06:27:49,544 +] 9497a4aa dell-afc-ci - AUTH FAILURE FROM 
143.166.116.138 inactive-account
 [2016-12-15 06:27:54,766 +] 794789ee dell-afc-ci - AUTH FAILURE FROM 
143.166.116.138 inactive-account
 [2016-12-15 06:28:00,030 +] 0fac03fa dell-afc-ci - AUTH FAILURE FROM 
143.166.116.138 inactive-account
 [2016-12-15 06:28:05,329 +] 205ebb27 dell-afc-ci - AUTH FAILURE FROM 
143.166.116.138 inactive-account
 ...

That IP appears to be in Texas (maybe) if that helps.

If somebody Dell-ish could stop this, that would be great.

-i

[1] 
http://lists.openstack.org/pipermail/third-party-announce/2016-April/000301.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder] [third-party][ci] devstack failures

2016-12-04 Thread Ian Wienand
On 12/05/2016 04:43 PM, Apoorva Deshpande wrote:
> [1] http://openstack-ci.tintri.com/tintri/refs-changes-23-405223-4/

This is failing at

 2016-12-05 01:19:17.356 | + functions-common:git_timed:598   :   
timeout -s SIGINT 0 git clone git://git.openstack.org/openstack/horizon.git 
/opt/stack/horizon --branch master
 2016-12-05 01:19:17.358 | Cloning into '/opt/stack/horizon'...
 2016-12-05 03:20:41.160 | fatal: read error: Connection reset by peer
 2016-12-05 03:20:41.165 | fatal: early EOF
 2016-12-05 03:20:41.165 | fatal: index-pack failed

> [2] http://openstack-ci.tintri.com/tintri/refs-changes-95-393395-7/

The difference to [2] seems to be we merged 399550 [3] to pass in the
--branch parameter to do the checkout all at once, rather than in a
separate step.

It doesn't seem that much different to me ... but maybe you have a
proxy involved that doesn't like this for some obscure reason?

-i

[3] https://review.openstack.org/#/c/399550/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Gerrit downtime on Tuesday 2016-11-22 at 21:00UTC

2016-11-15 Thread Ian Wienand

Hi everyone,

On Tuesday, November 22nd from approximately 21:00 through 21:30 UTC
Gerrit will be unavailable while we complete project renames.

Currently, we plan on renaming the following projects:

 rsc -> valence

Existing reviews, project watches, etc, for these projects will all be
carried over.

This list is subject to change. If you need a rename, please be sure
to get your project-config change in soon so we can review it and add
it to 
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Upcoming_Project_Renames


If you have any questions about the maintenance, please reply here or
contact us in #openstack-infra on freenode.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [release][diskimage-builder] v2 branch request

2016-09-01 Thread Ian Wienand

Hi,

diskimage-builder would like to request a feature/v2 branch.

We have a backlog of changes that we would like to give more extensive
testing together before we merge to master [1].

We've discussed this in our meeting, and we feel like
series-independent releases in general work well for us, given we
often have to react quickly to changes outside our control such as
upstream packaging issues or breakages.  After a 2.0 release we'll
consider what's working, what's not and what is still in the pipe-line
and consider our model.

Thanks,

-i

[1] https://etherpad.openstack.org/p/dib2.0

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra][neutron] - best way to load 8021q kernel module into cirros

2016-08-08 Thread Ian Wienand

On 08/09/2016 02:10 AM, Jeremy Stanley wrote:

I haven't personally tested the CirrOS build instructions, but have
a feeling writing a diskimage-builder element wrapper for that
wouldn't be particularly challenging.


I'm not exactly sure it fits that well into dib; it seems like
"bundle" has it mostly figured out.  I say that based on [1] where we
are discussing a similar thing for cirros images with watchdog
support.

As mentioned we can easily build these and store them, and put them on
mirrors if we need them closer to nodes.  What I mentioned in [1] and
didn't particularly like is if the build of these images is totally
removed from where they're actually used (e.g. a custom script inside
a job in project-config, where basically anyone outside infra can't
easily replicate the build for a local test).  But if several projects
are building slightly different cusomised cirros images, it might be
worth consolidating.

-i

[1] https://review.openstack.org/#/c/338167/2/tools/build-watchdog-images.sh


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO] [diskimage-builder] Howto refactor?

2016-06-01 Thread Ian Wienand

On 06/01/2016 02:10 PM, Andre Florath wrote:

My long term goal is, to add some functionality to the DIB's block
device layer, like to be able to use multiple partitions, logical
volumes and mount points.


Some thoughts...

There's great specific info in the readme's of the changes you posted
... but I'm missing a single big picture context of what you want to
build on-top of all this and how the bits fit into it.  We don't have
a formalised spec or blueprint process, but something that someone who
knows *nothing* about this can read and follow through will help; I'd
suggest an etherpad, but anything really.  At this point, you are
probably the world-expert on dib's block device layer, you just need
to bring the rest of us along :)

There seems to be a few bits that are useful outside the refactor.
Formalising python elements, extra cleanup phases, dib-run-parts
fixes, etc.  Splitting these out, we can get them in quicker and it
reduces the cognitive load for the rest.  I'd start there.

#openstack-infra is probably fine to discuss this too, as other dib
 knowledgeable people hang out there.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [infra] Jobs failing : "No matching distribution found for "

2016-05-10 Thread Ian Wienand
So it seems the just released pip 8.1.2 has brought in a new version
of setuptools with it, which creates canonical names per [1] by
replacing "." with "-".

The upshot is that pip is now looking for the wrong name on our local
mirrors.  e.g.

---
 $ pip --version
pip 8.1.2 from /tmp/foo/lib/python2.7/site-packages (python 2.7)
$ pip --verbose  install --trusted-host mirror.ord.rax.openstack.org -i 
http://mirror.ord.rax.openstack.org/pypi/simple 'oslo.config>=3.9.0'
Collecting oslo.config>=3.9.0
  1 location(s) to search for versions of oslo.config:
  * http://mirror.ord.rax.openstack.org/pypi/simple/oslo-config/
  Getting page http://mirror.ord.rax.openstack.org/pypi/simple/oslo-config/
  Starting new HTTP connection (1): mirror.ord.rax.openstack.org
  "GET /pypi/simple/oslo-config/ HTTP/1.1" 404 222
  Could not fetch URL 
http://mirror.ord.rax.openstack.org/pypi/simple/oslo-config/: 404 Client Error: 
Not Found for url: http://mirror.ord.rax.openstack.org/pypi/simple/oslo-config/ 
- skipping
  Could not find a version that satisfies the requirement oslo.config>=3.9.0 
(from versions: )
---

(note olso-config, not oslo.config).  Compare to

---
$ pip --verbose install --trusted-host mirror.ord.rax.openstack.org -i 
http://mirror.ord.rax.openstack.org/pypi/simple 'oslo.config>=3.9.0'
You are using pip version 6.0.8, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting oslo.config>=3.9.0
  Getting page http://mirror.ord.rax.openstack.org/pypi/simple/oslo.config/
  Starting new HTTP connection (1): mirror.ord.rax.openstack.org
  "GET /pypi/simple/oslo.config/ HTTP/1.1" 200 2491
---

I think infra jobs that run on bare-precise are hitting this
currently, because that image was just built.  Other jobs *might* be
isolated from this for a bit, until the new pip gets out there on
images, but "winter is coming", as they say...

There is [2] available to make bandersnatch use the new names.
However, I wonder if this might have the effect of breaking the
mirrors for old versions of pip that ask for the "."?

pypi proper does not seem affected, just our mirrors.

I think probably working with bandersnatch to get a fixed version ASAP
is probably the best way forward, rather than us trying to pin to old
pip versions.

-i

[1] https://www.python.org/dev/peps/pep-0503/
[2] 
https://bitbucket.org/pypa/bandersnatch/pull-requests/20/fully-implement-pep-503-normalization/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][releases] Remove diskimage-builder from releases

2016-04-19 Thread Ian Wienand

On 04/20/2016 06:09 AM, Fox, Kevin M wrote:

I've seen dib updated and broken things.



I've seen dib elements updated and things broke (centos6 removal in
particular hurt.)


By the time it gets to a release, however, anything we've broken is
already baked in.  Any changes in there have already passed review and
whatever CI we have.

(not to say we can't do better CI to break stuff less.  But that's
outside the release team's responsibility)

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][releases] Remove diskimage-builder from releases

2016-04-19 Thread Ian Wienand

On 04/20/2016 03:25 AM, Doug Hellmann wrote:

It's not just about control, it's also about communication. One of
the most frequent refrains we hear is "what is OpenStack", and one
way we're trying to answer that is to publicize all of the things
we release through releases.openstack.org.


So for dib, this is mostly about documentation?

We don't have the issues around stable branches mentioned in the
readme, nor do we worry about the requirements/constraints (proposal
bot has always been sufficient?).


Centralizing tagging also helps us ensure consistent versioning
rules, good timing, good release announcements, etc.


We so far haven't had issues keeping the version number straight.

As mentioned, the timing has extra constraints due to use in periodic
infra jobs that I don't think the release team want to be involved
with.  It's not like the release team will be going through the
changes in a new release and deciding if they seem OK or not (although
they're welcome to do dib reviews, before things get committed :) So I
don't see what timing constraints will be monitored in this case.

When you look at this from my point of view, dib was left/is in an
unreleasable state that I've had to clean up [1], we've now missed yet
another day's build [2] and I'm not sure what's different except I now
have to add probably 2 days latency to the process of getting fixes
out there.

To try and be constructive : is what we want a proposal-bot job that
polls for the latest release and adds it to the diskimage-builder.yaml
file?  That seems to cover the documentation component of this.

Or, if you want to give diskimage-builder-release group permissions on
the repo, so we can +2 changes on the diskimage-builder.yaml file, we
could do that. [3]

-i

[1] https://review.openstack.org/#/c/306925/
[2] https://review.openstack.org/#/c/307542/
[3] my actual expectation of this happening is about zero

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [tripleo][releases] Remove diskimage-builder from releases

2016-04-18 Thread Ian Wienand

Hi,

diskimage-builder has fallen under the "centralised release tagging"
mechanism [1], presumably because it is under tripleo.  I'd like to
propose that we don't do that.

Firstly, dib doesn't have any branches to manage.

dib's other main function is as part of the daily CI image builds.
This means to get a fix into the CI images in a somewhat timely
fashion, we approve the changes and make sure our releases happen
before 14:00 UTC, and monitor the build results closely in nodepool.

I don't expect the stable release team to be involved with all this;
but if we miss windows then we're left either going to efforts getting
one of a handful of people with permissions to do manual rebuilds or
waiting yet another day to get something fixed.  Add some timezones
into this, and simple fixes are taking many days to get into builds.
Thus adding points where we can extend this by another 24 hours
really, well, sucks.

I have previously suggested running dib from git directly to avoid the
release shuffle, but it was felt this was not the way to go [2].  I've
proposed putting the release group back with [3] and cleaning up with
[4].

Thanks,

-i

[1] https://review.openstack.org/298866
[2] https://review.openstack.org/283877
[3] https://review.openstack.org/307531
[4] https://review.openstack.org/307534

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] create periodic-ci-reports mailing-list

2016-04-13 Thread Ian Wienand

On 04/14/2016 03:22 AM, Jeremy Stanley wrote:

Mentioned in IRC as well, but would an RSS/ATOM feed be a good
compromise between active notification and focus on the dashboard as
an entry point to researching job failures?


For myself, simply ordering by date on the log page as per [1] would
make it one step easier to write a local cron job to pick up the
latest.

-i

[1] https://review.openstack.org/#/c/301989/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] pip 8 no longer over-installs system packages [was: Gate failure]

2016-01-20 Thread Ian Wienand
On 01/20/2016 06:21 PM, Andreas Jaeger wrote:
> Now docs, pep8, and python27 are broken as well here:
> https://review.openstack.org/#/c/268687/

Ok, so this is a weird one.  On trusty, argparse is sort of in, and
sort of out of the virtualenv.  I think it has to do with [1]

---
(test)ubuntu@trusty:/tmp$ pip install argparse
Requirement already satisfied (use --upgrade to upgrade): argparse in 
/usr/lib/python2.7
Cleaning up...
(test)ubuntu@trusty:/tmp$ pip install --upgrade argparse
Downloading/unpacking argparse from 
https://pypi.python.org/packages/2.7/a/argparse/argparse-1.4.0-py2.py3-none-any.whl#md5=c37216a954c8669054e2b2c54853dd49
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: argparse
  Found existing installation: argparse 1.2.1
Not uninstalling argparse at /usr/lib/python2.7, outside environment 
/tmp/test
Successfully installed argparse
Cleaning up...
---

However, this has now turned into an error

---
(test)ubuntu@trusty:/tmp$ pip install --upgrade pip
Downloading/unpacking pip from 
https://pypi.python.org/packages/py2.py3/p/pip/pip-8.0.0-py2.py3-none-any.whl#md5=7b1da5eba510e1631791dcf300657916
  Downloading pip-8.0.0-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed pip
Cleaning up...
(test)ubuntu@trusty:/tmp$ pip install -U argparse
Collecting argparse
  Downloading argparse-1.4.0-py2.py3-none-any.whl
Installing collected packages: argparse
  Found existing installation: argparse 1.2.1
Detected a distutils installed project ('argparse') which we cannot uninstall. 
The metadata provided by distutils does not contain a list of files which have 
been installed, so pip does not know which files to uninstall.
---

So, yeah :(

-i

[1] https://github.com/pypa/pip/issues/1570

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] pip 8 no longer over-installs system packages [was: Gate failure]

2016-01-19 Thread Ian Wienand

On 01/20/2016 04:14 PM, Ian Wienand wrote:

On 01/20/2016 12:53 PM, Robert Collins wrote:

I suspect we'll see fallout in unit tests too, once new images are
built.


If the images can build ...


yeah, dib is not happy about this either


Just removing the directory as pip
used to do has been enough to keep things going.


To be clear, what happens is that pip removes the egg-info file and
then overwrites the system installed files.  This is, of course,
unsafe, but we generally get away with it.


Presume we can't remove the system python-* packages for these tools
because other bits of the system rely on it.  We've been down the path
of creating dummy packages before, I think ... that never got very
far.


Another option would be for us to just keep a list of egg-info files
to remove within devstack and more or less do what pip was doing
before.


Would pip accept maybe a environment flag to restore the old ability
to remove based on the egg-info?  Is it really so bad given what
devstack is doing?


I proposed a revert in [1] which I'm sure people will have opinions
on.

[1] https://github.com/pypa/pip/pull/3389

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] pip 8 no longer over-installs system packages [was: Gate failure]

2016-01-19 Thread Ian Wienand

On 01/20/2016 12:53 PM, Robert Collins wrote:

I suspect we'll see fallout in unit tests too, once new images are
built.


If the images can build ...

This was marked as deprecated, I understand, but the removal is very
unfortunate [1] considering it's really just a
shoot-yourself-in-the-foot operation.

From the latest runs, on ubuntu we are using pip to over-install
system packages of

 six
 requests
 netaddr
 PyYAML
 PyOpenSSL
 jsonpointer
 urllib3
 PyYAML
 pyOpenSSL

On CentOS it is

 requests
 PyYAML
 enum34
 ipaddress
 numpy

The problem is that we can't remove these system packages with the
package-manager from the base images, because other packages we need
rely on having them installed.  Just removing the directory as pip
used to do has been enough to keep things going.

So, what to do?  We can't stay at pip < 8 forever, because I'm sure
there will be some pip problem we need to patch soon enough.

Presume we can't remove the system python-* packages for these tools
because other bits of the system rely on it.  We've been down the path
of creating dummy packages before, I think ... that never got very
far.

I really don't know how with the world of devstack plugins we'd deploy
a strict global virtualenv.  Heaven knows what "creative" things
plugins are going to come up with if someone hits this (not that I've
proposed anything elegant)...

Would pip accept maybe a environment flag to restore the old ability
to remove based on the egg-info?  Is it really so bad given what
devstack is doing?

-i

[1] 
https://github.com/pypa/pip/commit/6afc718307fea36b9ffddd376c1395ee1061795c


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack][QA] Changing logging format to match documented defaults

2015-12-22 Thread Ian Wienand
On 12/23/2015 05:55 AM, Ronald Bradford wrote:
> I have observed that devstack uses custom logging formatting that
> differs from the documented defaults. An example is for nova.  which
> is defined in [1]

The original point mentioned there of using "*_name" for extra
verbosity still seems relevant [1]

> logging_context_format_string = %(asctime)s.%(msecs)03d %(levelname)s
> %(name)s [%(request_id)s *%(user_identity)s*] %(instance)s%(message)s

user_identity is still just "{user} {tenant} ..." (which is id's, I
believe?) [2].

> This logging variable is also defined differently across 4
> projects (nova, cinder, keystone and glance) so ultimately the goal would
> be to ensure they may present documented configuration defaults.

Since it's hard enough to debug already when tempest gets going with
multiple users/tenants, I think keeping the verbose logging is worth
it.  However, if a bunch of copy-paste setting of this can be
refactored, I'm sure we'd consider it favourably.

-i

[1] 
http://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=6f13ba33d84b95808fc2a7672f332c1f0494e741
[2] 
https://git.openstack.org/cgit/openstack/oslo.context/tree/oslo_context/context.py#n43

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][DIB] diskimage-builder and python 2/3 compatibility

2015-12-09 Thread Ian Wienand
On 12/09/2015 07:15 AM, Gregory Haynes wrote:
> We ran in to a couple issues adding Fedora 23 support to
> diskimage-builder caused by python2 not being installed by default.
> This can be solved pretty easily by installing python2, but given that
> this is eventually where all our supported distros will end up I would
> like to find a better long term solution (one that allows us to make
> images which have the same python installed that the distro ships by
> default).

So I wonder if we're maybe hitting premature optimisation with this

> We use +x and a #! to specify a python
> interpreter, but this needs to be python3 on distros which do not ship a
> python2, and python elsewhere.

> Create a symlink in the chroot from /usr/local/bin/dib-python to
> whatever the apropriate python executable is for that distro.

This is a problem for anyone wanting to ship a script that "just
works" across platforms.  I found a similar discussion about a python
launcher at [1] which covers most points and is more or less what
is described above.

I feel like contribution to some sort of global effort in this regard
might be the best way forward, and then ensure dib uses it.

-i

[1] https://mail.python.org/pipermail/linux-sig/2015-October/00.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] How to source openrc?

2015-11-22 Thread Ian Wienand
On 11/23/2015 03:17 PM, Wilence Yao wrote:
> source openrc admin admin
> openrc:90: unknown condition: -v

I'm pretty sure you're using zsh -- we only support bash (stack.sh
checks for bash, maybe we should add that to openrc)

Anyway, you can probably follow [1] to source it; other things might
break though

-i

[1] 
http://docs.openstack.org/developer/devstack/faq.html#can-i-at-least-source-openrc-with-zsh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][infra] Getting a bleeding edge libvirt gate job running

2015-11-17 Thread Ian Wienand

On 11/18/2015 06:10 AM, Markus Zoeller wrote:
 This

was a trigger to see if we can create a gate job which utilizes the
latest, bleeding edge, version of libvirt to test such features.



* Is already someone working on something like that and I missed it?


I believe the closest we have got is probably [1]; pulling apart some
of the comments there might be helpful

In short, a devstack plugin that installs the latest libvirt is
probably the way to go.

Ongoing, the only issue I see is that we do things to base devstack
that conflict/break this plugin, as we are more-or-less assuming the
distro version (see things like [2], stuff like this comes up every
now and then).


* If 'no', is there already a repo which contains the very latest
   libvirt builds which we can utilize?


For Fedora, there is virt-preview  [3] at least

-i

[1] https://review.openstack.org/#/c/108714/
[2] https://review.openstack.org/246501
[3] https://fedoraproject.org/wiki/Virtualization_Preview_Repository

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [devstack] package dependency installs after recent clean-ups

2015-11-08 Thread Ian Wienand

devstack maintainers, et al.

If your CI is failing with missing packages (xml bindings failing to
build, postgres bindings, etc), it may be due to some of the issues
covered below.

I believe some of the recent changes around letting pip build wheels
and cleaning up some of the package dependencies have revealed that
devstack is not quite installing build pre-reqs as we thought it was.

[1] fixes things so we actually install the packages listed in
"general"

[2] is a further cleanup of the "devlib" packages, which are no longer
installed since we removed tools/build_wheels.sh

I believe a combination of what was removed in [3] and [2] was hiding
the missing installs from [1].  Thus we can clean up some of the
dependencies via [4].

Stacked on that are some less important further clean-ups

Reviews appreciated, because it seems to have broken some CI, see [5]

-i

[1] https://review.openstack.org/242891
[2] https://review.openstack.org/242894
[3] https://review.openstack.org/242895
[4] https://review.openstack.org/242895
[5] https://review.openstack.org/242536

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [requirements] [infra] speeding up gate runs?

2015-11-04 Thread Ian Wienand

On 11/05/2015 01:43 AM, Matthew Thode wrote:

python wheel repo could help maybe?


So I think we've (i.e. greghaynes) got that mostly in place, we just
got a bit side-tracked.

[1] adds mirror slaves, that build the wheels using pypi-mirror [2],
and then [3] adds the jobs.

This should give us wheels of everything in requirements

I think this could be enhanced by using bindep to install
build-requirements on the mirrors; in chat we tossed around some ideas
of making this a puppet provider, etc.

-i

[1] https://review.openstack.org/165240
[2] https://git.openstack.org/cgit/openstack-infra/pypi-mirror
[3] https://review.openstack.org/164927
[4] https://git.openstack.org/cgit/openstack-infra/bindep

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] DevStack errors...

2015-11-02 Thread Ian Wienand

On 11/03/2015 10:51 AM, Thales wrote:

I'm trying to get DevStack to work, but am getting errors.  Is this
a good list to ask questions for this?  I can't seem to get answers
anywhere I look.  I tried the openstack list, but it kind of moves
slow.


Best to file a bug in launchpad, and *attach the full log*.  Very
often with devstack the root cause of a problem happens well before
the issue manifests itself as an error.

I won't say every bug gets attention immediately, but it's something
useful you can ping people with in irc, etc.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][heat] Which repo to use in docs -- git.openstack.org or github.com?

2015-10-20 Thread Ian Wienand

On 10/21/2015 04:20 AM, Christopher Aedo wrote:

On the other hand, the fact that github renders the README nicely
(including images) leads to a much more inviting first impression.


I think it's nice to just have a standard docs target, even if it just
includes the README; [1] is an example.  Then you've got the framework
to actually document something in the future, should you wish.

[1] https://review.openstack.org/#/c/228694/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] Upgrade to Gerrit 2.11

2015-10-14 Thread Ian Wienand

On 10/14/2015 11:08 AM, Zaro wrote:

We are soliciting feedback so please let us know what you think.


Since you asked :)

Mostly it's just different which is fine.  Two things I noticed when
playing around, shown in [1]

When reviewing, the order "-1 0 +1" is kind of counter-intuitive to
the usual dialog layout of the "most positive" thing on the left;
e.g. [OK] [Cancel] dialogs.  I just found it odd to interact with.

Maybe people default themselves to -1 though :)

The colours for +1/-1 seem to be missing.  You've got to think a lot
more to parse the +1/-1 rather than just glance at the colours.

-i

[1] http://imgur.com/QWXOMen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [murano] Fix order of arguments in assertEqual

2015-09-28 Thread Ian Wienand

On 09/24/2015 08:18 PM, Andrey Kurilin wrote:

I agree that wrong order of arguments misleads while debugging errors, BUT
how we can prevent regression?


Spell it out and use keyword args?

  assertEqual(expected="foo", observed=...)

is pretty hard to mess up

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Make libguestfs available on pypi

2015-07-29 Thread Ian Wienand
On 07/30/2015 04:55 AM, Kris G. Lindgren wrote:
 The following bug has already been created over a year ago [1], and
 it looks like most of the work on the libguestfs side is already
 done [2].  It seems something about a complaint of licensing per
 the bug report.

I think best to follow up in that bug

On the license front, to quote from an internal email I
saw fly-by about the pypi sign-up terms under question from Nick on
20-Jul-2015:

---
 Van started drafting some amendments back in February:
 https://bitbucket.org/vanl/pypi/commits/all

 Key changes are here:
 
https://bitbucket.org/vanl/pypi/commits/8df8e0295c0a719e963f7c3ce430284179f03b1f

 Further clarifications at
 
https://bitbucket.org/vanl/pypi/commits/734b1f49776d1f7f5d0671306f61a90aad713e5d
 and
 
https://bitbucket.org/vanl/pypi/commits/0e94b169e81306607936912ecc3c42312aac5eb7

 I'll ping the Board list about next steps in getting those amendments
 formally approved and submitted as a PR to the main PyPI repo.
---

So it is being looked at, but I'm not sure of the time-frame.

-i

 [1] - https://bugzilla.redhat.com/show_bug.cgi?id=1075594

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [CI] No image was created after running nodepoold -d $DAEMON_ARGS

2015-07-29 Thread Ian Wienand

On 07/30/2015 01:51 PM, Ian Wienand wrote:

Nothing jumps out


Something I just thought of that has caused problems is check your
users; I think running things by hand as root and then switching back
to a unprivileged user can cause problems as the second run hits
things it can't modify.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [CI] No image was created after running nodepoold -d $DAEMON_ARGS

2015-07-29 Thread Ian Wienand

On 07/30/2015 12:34 PM, Xie, Xianshan wrote:

DEBUG nodepool.NodePool: Finished node launch calculation
INFO nodepool.DiskImageBuilderThread: Running disk-image-create ...


So, nothing after this?

Nothing jumps out; first thought was to check if disk-image-create is
running and go from there.  If it's stuck, pointing strace() at it
might give a clue.

Full logs and config help, but my best suggestion is to
jump on irc in #openstack-infra; during USA awake hours you'll get the
best response.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][infra] CI System is broken

2015-07-29 Thread Ian Wienand

On 07/29/2015 07:33 PM, Andreas Jaeger wrote:

Currently Zuul is stuck and not processing any events at all, thus no
jobs are checked or gated.


I think whatever happened has happened again; if jhesketh is out it
might be a few hours from this email before people with the right
access are back online to fix it.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [puppet] [infra] issues with beaker/centos7 job

2015-07-01 Thread Ian Wienand
On 07/02/2015 11:52 AM, Emilien Macchi wrote:
 details. I tried to install deltarpm before, but it says : No Presto
 metadata available for rdo-release.

So I looked into this with Gilles, and that error is a red-herring
(it's just saying the rdo repos don't create the presto/deltarpm
stuff); the real issue is when python-requests fails to install [1] a
bit later due to [2] (mentioned in comments of [3]).

So this is really a upstream packaging issue and not an issue with the
nodepool images.

-i

[1] 
http://logs.openstack.org/83/197783/8/check/gate-puppet-neutron-puppet-beaker-rspec-upgrade-dsvm-centos7/9f628b4/console.html
[2] https://bugzilla.redhat.com/show_bug.cgi?format=multipleid=1212145
[3] https://review.openstack.org/#/c/197782/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all][infra][tc][ptl] Scaling up code review process (subdir cores)

2015-06-02 Thread Ian Wienand

On 06/03/2015 07:24 AM, Boris Pavlovic wrote:

Really it's hard to find cores that understand whole project, but
it's quite simple to find people that can maintain subsystems of
project.


  We are made wise not by the recollection of our past, but by the
  responsibility for our future.
   - George Bernard Shaw

Less authorities, mini-kingdoms and
turing-complete-rule-based-gerrit-subtree-git-commit-enforcement; more 
empowerment of responsible developers and building trust.


-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] What's required to accept a new distro rlease?

2015-05-14 Thread Ian Wienand
On 05/15/2015 01:05 PM, Tony Breeds wrote:
 I'm wondering what are the requirements for accepting something
 like:

 -if [[ ! ${DISTRO} =~ 
 (precise|trusty|7.0|wheezy|sid|testing|jessie|f20|f21|rhel7) ]]; then
 +if [[ ! ${DISTRO} =~ 
 (precise|trusty|vivid|7.0|wheezy|sid|testing|jessie|f20|f21|rhel7) ]]; then

Having a CI story makes it a no-brainer...  anyone working on getting
vivid nodes up in nodepool?  In the past we've generally accepted if
people have a convincing story that it works (i.e. they're using it
and tempest is working, etc).

Honestly I doubt 7.0|wheezy|sid|testing|jessie all work anyway -- they
don't have any CI (I know of) -- so we're probably overselling it
anyway.

 I figure there must be more to it as utopic was never added.

I think that was approved, failed a test and was never re-merged [1].
So I guess it's more nobody cared?

-i

[1] https://review.openstack.org/#/c/98844/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Infra] Gerrit downtime and upgrade on Saturday 2015-05-09 at 1600 UTC

2015-05-10 Thread Ian Wienand
On 05/10/2015 05:20 AM, James E. Blair wrote:
 If you encounter any problems, please let us know here or in
 #openstack-infra on Freenode.

One minor thing is that after login you're redirected to
https://review.openstack.org//; (note double //, which then messes up
various relative-links when trying to use gerrit).

I think this is within gerrit's openid provider; I don't see any
obvious problems outside that.  When I watch with firebug, Gerrit
sends me off on a 401 page to launchpad with a form it POSTs -- this
has a good-looking return_to

input name=openid.return_to type=hidden
 value=https://review.openstack.org/OpenID?gerrit.mode=SIGN_IN;

I go through the login, and get a 302 from

 https://login.launchpad.net/NUxdqUgd7EbI5Njo/+decide

which correctly 302's me back to the return_to

 
https://review.openstack.org/OpenID?gerrit.mode=SIGN_INopenid.assoc_handle=[...
 openid stuff follows ...]

Which then further 302's back to

  Location: https://review.openstack.org//

Which is followed to the incorrect URL.  So yeah, something within the
OpenID end-point I think.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] The Evolution of core developer to maintainer?

2015-04-01 Thread Ian Wienand

On 04/02/2015 09:02 AM, Jeremy Stanley wrote:

but since parties who don't understand our mostly non-hierarchical
community can see those sets of access controls, they cling to them
as a sign of importance and hierarchy of the people listed within.


There is no hierarchy for submitting code -- that is good.  We all know
situations in a traditional company where people say that's foo's
area, we don't work on that.

Once code is submitted, there *is* a hierarchy.  The only way
something gets merged in OpenStack is by Brownian motion of this
hierarchy.  These special cores float around and as a contributor
you just hope that two of them meet up and decide your change is
ready.  You have zero insight into when this might happen, if at all.
The efficiency is appalling but somehow we get there in the end.

IMO requiring two cores to approve *every* change is too much.  What
we should do is move the responsibility downwards.  Currently, as a
contributor I am only 1/3 responsible for my change making it through.
I write it, test it, clean it up and contribute it; then require the
extra 2/3 to come from the hierarchy.  If you only need one core,
then core and myself share the responsibility for the change.  In my
mind, this better recognises the skill of the contributor -- we are
essentially saying we trust you.

People involved in openstack are not idiots.  If a change is
controversial, or a reviewer isn't confident, they can and will ask
for assistance or second opinions.  This isn't a two-person-key system
in a nuclear missile silo; we can always revert.

If you want cores to be less special then talking about it or
calling them something else doesn't help -- the only way is to make
them actually less special.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [swift] swift memory usage in centos7 devstack jobs

2015-04-01 Thread Ian Wienand

Note; I haven't finished debugging the glusterfs job yet.  This
relates to the OOM that started happening on Centos after we moved to
using as much pip-packaging as possible.  glusterfs was still failing
even before this.

On 04/01/2015 07:58 PM, Deepak Shetty wrote:

1) So why did this happen on rax VM only, the same (Centos job)on hpcloud
didn't seem to hit it even when we ran hpcloud VM with 8GB memory.


I am still not entirely certain that hp wasn't masking the issue when
we were accidentally giving hosts 32gb RAM.  We can get back to this
once these changes merge.


2) Should this also be sent to centos-devel folks so that they don't
upgrade/update the pyopenssl in their distro repos until the issue
is resolved ?


I think let's give the upstream issues a little while to play-out,
then we decide our next steps around use of the library based on that
information.

thanks

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [swift] swift memory usage in centos7 devstack jobs

2015-03-31 Thread Ian Wienand

On 03/27/2015 08:47 PM, Alan Pevec wrote:

But how come that same recent pyOpenSSL doesn't consume more memory on Ubuntu?


Just to loop back on the final status of this ...

pyOpenSSL 0.14 does seem to use about an order of magnitude more
memory than 0.13 (2mb - 20mb).  For details see [1].

This is due to the way it now goes through cryptography (the
package, not the concept :) which binds to openssl using cffi.  This
ends up parsing a bunch of C to build up the ABI representation, and
it seems pycparser's model of this consumes most of the memory [2].
If that is a bug or not remains to be seen.

Ubuntu doesn't notice this in our CI environment because it comes with
python-openssl 0.13 pre-installed in the image.  Centos started
hitting this when I merged my change to start using as many libraries
from pip as possible.

I have a devstack workaround for centos out (preinstall the package)
[3] and I think a global solution of avoiding it in requirements [4]
(reviews appreciated).

I'm also thinking about how we can better monitor memory usage for
jobs.  Being able to see exactly what change pushed up memory usage by
a large % would have made finding this easier.  We keep some overall
details for devstack runs in a log file, but there is room to do
better.

-i

[1] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[2] https://github.com/eliben/pycparser/issues/72
[3] https://review.openstack.org/168217
[4] https://review.openstack.org/169596

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [swift] swift memory usage in centos7 devstack jobs

2015-03-27 Thread Ian Wienand

On 03/27/2015 08:47 PM, Alan Pevec wrote:

But how come that same recent pyOpenSSL doesn't consume more memory
on Ubuntu?


Because we don't use it in CI; I believe the packaged version is
installed before devstack runs on our ubuntu CI vm's.  It's probably a
dependency of some base package there, or something we've
pre-installed.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [swift] swift memory usage in centos7 devstack jobs

2015-03-26 Thread Ian Wienand
On 03/26/2015 04:07 PM, Ian Wienand wrote:
 See [1] for some more details; but the short story is that the various
 swift processes -- even just sitting around freshly installed from
 devstack before anything happens -- take up twice as much space on
 centos as ubuntu
 
 --- swift (% total system memory) ---
 ubuntu  6.6%
 centos  12%

So after more investigation, it turns out that pyOpenSSL has rewritten
itself in python; necessitating dependencies on the cryptography
package and cffi  pycparser [1].  Examining the heap shows where the
memory has gone missing :

Partition of a set of 205366 objects. Total size = 30969040 bytes.
 Index  Count   % Size   % Cumulative  % Kind (class / dict of class)
 0  67041  33  5712560  18   5712560  18 str
 1  10260   5  2872800   9   8585360  28 dict of pycparser.plyparser.Coord
 2  27765  14  2367552   8  10952912  35 tuple
 3   1215   1  2246760   7  13199672  43 dict (no owner)
 4   1882   1  1972336   6  15172008  49 dict of pycparser.c_ast.Decl
 5  16085   8  1736232   6  16908240  55 list
 6360   0  1135296   4  18043536  58 dict of module
 7   4041   2  1131480   4  19175016  62 dict of pycparser.c_ast.TypeDecl
 8   4021   2  1125880   4  20300896  66 dict of 
pycparser.c_ast.IdentifierType
 9   6984   3   893952   3  21194848  68 types.CodeType
413 more rows. Type e.g. '_.more' to view.

If I reinstall the packaged version of pyOpenSSL, all that drops out
and we're back to a more reasonable usage

Partition of a set of 95591 objects. Total size = 12500080 bytes.
 Index  Count   % Size   % Cumulative  % Kind (class / dict of class)
 0  45837  48  3971040  32   3971040  32 str
 1  22843  24  1943416  16   5914456  47 tuple
 2298   0   978160   8   6892616  55 dict of module
 3   6065   6   776320   6   7668936  61 types.CodeType
 4551   1   742184   6   8411120  67 dict (no owner)
 5805   1   725520   6   9136640  73 type
 6   5876   6   705120   6   9841760  79 function
 7805   1   666232   5  10507992  84 dict of type
 8289   0   279832   2  10787824  86 dict of class
 9152   0   159296   1  10947120  88 dict of pkg_resources.Distribution
310 more rows. Type e.g. '_.more' to view.

The end result of this is that swift-* processes go from consuming
about 6% of a CI VM's 8gb to 12%.  This 500mb is enough to push the
host into OOM when tempest gets busy.  For more see [2].  a workaround is [3]

I'll spend a bit more time on this -- I haven't determined if it's
centos or swift specific yet -- but in the mean-time, beware of
recent pyOpenSSL

-i

[1] 
https://github.com/pyca/pyopenssl/commit/fd193a2f9dd8be80d9f42d8dd8068de5f5ac5e67
 
[2] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[3] https://review.openstack.org/#/c/168217/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [swift] swift memory usage in centos7 devstack jobs

2015-03-25 Thread Ian Wienand

Hi,

We've been having an issue with Centos 7 jobs and the host running
out of memory.  After some investigation; one likely angle seems to be
the memory usage by swift.

See [1] for some more details; but the short story is that the various
swift processes -- even just sitting around freshly installed from
devstack before anything happens -- take up twice as much space on
centos as ubuntu

--- swift (% total system memory) ---
ubuntu  6.6%
centos  12%

In general memory usage is higher on centos, but this one is an
outlier.  Unfortunately, the main difference between the two appears
to be heap allocations (see comments in [1]) which doesn't give a lot
of clues.

The end result is that the host ends up just plain running out of
memory; the OOM killer kicks in and then everything starts
collapsing. I had the host sending me telemetry while it was running;
the last entry before things fell over was [2] and we can see that
it's not just one thing that comes along and sucks up memory, but
death by a thousand cuts.  I think the Centos 7 job is struggling to
fit into the 8gb available so we're susceptible to finding memory
issues first.

Any swift people have some ideas on this?

Thanks

-i

[1] https://etherpad.openstack.org/p/oom-in-rax-centos7-CI-job
[2] http://paste.openstack.org/show/196769/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] Overriding settings file for devstack plugin

2015-03-25 Thread Ian Wienand

On 03/25/2015 09:28 PM, Sean Dague wrote:

I would instead do the following:
1) CINDER_ENABLED_BACKENDS+=,glusterfs:glusterfs


This is what I was about to suggest.  I'd be willing to believe
ordering could still get tangled depending on exactly what you want --
I think at that point best to follow up in a bug and we can pull apart
the specifics.

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] Overriding settings file for devstack plugin

2015-03-24 Thread Ian Wienand
On 03/24/2015 03:17 PM, Deepak Shetty wrote:
 For eg: Look at [1]
 [1] 
 https://github.com/stackforge/devstack-plugin-glusterfs/blob/master/devstack/settings

 I would like ability to change these while I use the enable_plugin
 apporach to setup devstack w/ GlusterFS per my local glusterfs setup

So I think the plugin should do

CINDER_ENABLED_BACKENDS=${CINDER_ENABLED_BACKENDS:-glusterfs:glusterfs,lvm:lvm1}

i.e. provide a default only if the variable is unset.

This seems like one of those traps for new players and is one
concern I have with devstack plugins -- that authors keep having to
find out lessons learned independently.  I have added a note on this
to the documentation in [1].

-i

[1] https://review.openstack.org/#/c/167375/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [devstack] Overriding settings file for devstack plugin

2015-03-23 Thread Ian Wienand

On 03/23/2015 09:20 PM, Deepak Shetty wrote:

Hi all,
   I was wondering if there was a neat way to override the settings file
present in the devstack plugin stackforge project.

For eg: stackforge/devstack-plugin-glusterfs

I plan to use `enable_plugin glusterfs repo` in my local to setup
GlusterFS backend for openstack

But I am forced to use the settings that the above repo has.


Can you explain more what you mean?  The glusterfs plugin should have
access to anything defined by the local.conf?

-i

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Testtools 1.7.0 may error if you installed it before reading this email

2015-03-11 Thread Ian Wienand
On 03/11/2015 08:10 AM, Robert Collins wrote:
 The wheel has been removed from PyPI and anyone installing testtools
 1.7.0 now will install from source which works fine.

I noticed the centos7 job failed with the source version.

The failing job was [1] where the back-trace looks like ~45 songs on
Python's greatest-hits album (pasted below).  The next run [2] it got
the 1.7.1 wheel and just worked.

Maybe this jumps out at someone as a known issue ...

-i

[1] 
http://logs.openstack.org/49/163249/1/check/check-tempest-dsvm-centos7/8dceac8/logs/devstacklog.txt.gz
[2] 
http://logs.openstack.org/49/163249/1/check/check-tempest-dsvm-centos7/f3b86d5/logs/devstacklog.txt.gz

---
Colecting testtools=0.9.22 (from 
fixtures=0.3.14-oslo.concurrency=1.4.1-keystone==2015.1.dev395)
Downloading 
http://pypi.IAD.openstack.org/packages/source/t/testtools/testtools-1.7.0.tar.gz
 (202kB)

 Installed 
/tmp/easy_install-mV2rSm/unittest2-1.0.0/.eggs/traceback2-1.4.0-py2.7.egg

 Installed 
/tmp/easy_install-mV2rSm/unittest2-1.0.0/.eggs/linecache2-1.0.0-py2.7.egg
 /usr/lib/python2.7/site-packages/setuptools/dist.py:291: UserWarning: The 
version specified (__main__.late_version instance at 0x34654d0) is an invalid 
version, this may not work as expected with newer versions of setuptools, pip, 
and PyPI. Please see PEP 440 for more details.
   details. % self.metadata.version
 Traceback (most recent call last):
   File string, line 20, in module
   File /tmp/pip-build-aGC1zC/testtools/setup.py, line 92, in module
 setup_requires=deps,
   File /usr/lib64/python2.7/distutils/core.py, line 112, in setup
 _setup_distribution = dist = klass(attrs)
   File /usr/lib/python2.7/site-packages/setuptools/dist.py, line 265, in 
__init__
 self.fetch_build_eggs(attrs['setup_requires'])
   File /usr/lib/python2.7/site-packages/setuptools/dist.py, line 310, in 
fetch_build_eggs
 replace_conflicting=True,
   File /usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 799, 
in resolve
 dist = best[req.key] = env.best_match(req, ws, installer)
   File /usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 
1049, in best_match
 return self.obtain(req, installer)
   File /usr/lib/python2.7/site-packages/pkg_resources/__init__.py, line 
1061, in obtain
 return installer(requirement)
   File /usr/lib/python2.7/site-packages/setuptools/dist.py, line 377, in 
fetch_build_egg
 return cmd.easy_install(req)
   File /usr/lib/python2.7/site-packages/setuptools/command/easy_install.py, 
line 620, in easy_install
 return self.install_item(spec, dist.location, tmpdir, deps)
   File /usr/lib/python2.7/site-packages/setuptools/command/easy_install.py, 
line 650, in install_item
 dists = self.install_eggs(spec, download, tmpdir)
   File /usr/lib/python2.7/site-packages/setuptools/command/easy_install.py, 
line 835, in install_eggs
 return self.build_and_install(setup_script, setup_base)
   File /usr/lib/python2.7/site-packages/setuptools/command/easy_install.py, 
line 1063, in build_and_install
 self.run_setup(setup_script, setup_base, args)
   File /usr/lib/python2.7/site-packages/setuptools/command/easy_install.py, 
line 1049, in run_setup
 run_setup(setup_script, args)
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 240, in 
run_setup
 raise
   File /usr/lib64/python2.7/contextlib.py, line 35, in __exit__
 self.gen.throw(type, value, traceback)
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 193, in 
setup_context
 yield
   File /usr/lib64/python2.7/contextlib.py, line 35, in __exit__
 self.gen.throw(type, value, traceback)
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 164, in 
save_modules
 saved_exc.resume()
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 139, in 
resume
 compat.reraise(type, exc, self._tb)
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 152, in 
save_modules
 yield saved
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 193, in 
setup_context
 yield
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 237, in 
run_setup
 DirectorySandbox(setup_dir).run(runner)
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 267, in 
run
 return func()
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 236, in 
runner
 _execfile(setup_script, ns)
   File /usr/lib/python2.7/site-packages/setuptools/sandbox.py, line 46, in 
_execfile
 exec(code, globals, locals)
   File /tmp/easy_install-mV2rSm/unittest2-1.0.0/setup.py, line 87, in 
module
 'testtools.tests.matchers',
   File /usr/lib64/python2.7/distutils/core.py, line 152, in setup
 dist.run_commands()
   File /usr/lib64/python2.7/distutils/dist.py, line 953, in run_commands
 self.run_command(cmd)
   File /usr/lib64/python2.7/distutils/dist.py, line 971, in run_command
 cmd_obj.ensure_finalized()
  

  1   2   >