Re: [openstack-dev] [nova] Creating VM error: Insufficient compute resources

[Stephen Finucane]

On Thu, 2016-08-25 at 14:11 +0400, Fawaz Mohammed wrote:
> Have you enabled huge page on the host level?
> Do you have enough vn.nr_hugepages?
> As per your requirements, you need a host with 512 hugepage (1G/2M).
> Check your host's /etc/sysctl.conf file and see vm.nr_hugepages
> value.

As a more immediate test, you can unset the 'hw:mem_page_size'
parameter and see if it boots. If it does, it's the hugepages as
suggested above.

Stephen

> On Aug 25, 2016 1:15 PM, "zhi"  wrote:
> > hi, all
> > 
> >     I plan to create VM with huge page. And I created a new flavor
> > like this:
> > 
> > $ nova flavor-show ed8dccd2-adbe-44ee-9e4f-391d045d3653
> > ++-
> > -
> > ---+
> > | Property                   | Value                              
> >                                                                    
> >     |
> > ++-
> > -
> > ---+
> > | OS-FLV-DISABLED:disabled   | False                              
> >                                                                    
> >     |
> > | OS-FLV-EXT-DATA:ephemeral  | 0                                  
> >                                                                    
> >     |
> > | disk                       | 30                                  
> >                                                                    
> >    |
> > | extra_specs                |
> > {"aggregate_instance_extra_specs:pinned": "true", "hw:cpu_policy":
> > "dedicated", "hw:mem_page_size": "2048"} |
> > | id                         | ed8dccd2-adbe-44ee-9e4f-391d045d3653 
> >                                                                    
> >    |
> > | name                       | m1.vm_2                            
> >                                                                    
> >     |
> > | os-flavor-access:is_public | True                                
> >                                                                    
> >    |
> > | ram                        | 1024                                
> >                                                                    
> >    |
> > | rxtx_factor                | 1.0                                
> >                                                                    
> >     |
> > | swap                       |                                    
> >                                                                    
> >     |
> > | vcpus                      | 4                                  
> >                                                                    
> >     |
> > ++-
> > -
> > ---+
> > 
> > Then I create a VM by using this flavor and creating fail. The
> > error message is :
> > "
> > {"message": "Build of instance ada7ac22-1052-44e1-b4a5-c21221dbab87 
> > was re-scheduled: Insufficient compute resources: Requested
> > instance NUMA topology cannot fit the given
> >  host NUMA topology.", "code": 500, "details": "  File
> > \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line
> > 1905, in _do_build_and_run_instance
> > "
> > 
> > And, my compute node's numa info is:
> > 
> > $ numactl --hardware
> > available: 2 nodes (0-1)
> > node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38
> > node 0 size: 32543 MB
> > node 0 free: 28307 MB
> > node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
> > node 1 size: 32768 MB
> > node 1 free: 29970 MB
> > node distances:
> > node   0   1
> >   0:  10  21
> >   1:  21  10
> > 
> > Qemu version is "QEMU emulator version 2.1.2 (qemu-kvm-ev-2.1.2-
> > 23.el7.1)". And libvirtd version is "1.2.17". 
> > 
> > 
> > Did anyone meet the same error like me?
> > 
> > 
> > 
> > B.R.
> > 
> > ___
> > ___
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsu
> > bscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Proposal for Nova Integration tests

[Stephen Finucane]

On Fri, 2016-09-23 at 13:33 -0400, Prasanth Anbalagan wrote:
> Adding the project to email subject.
> 
> Thanks
> Prasanth Anbalagan
> 
> On Fri, 2016-09-23 at 12:56 -0400, Prasanth Anbalagan wrote:
> > 
> > Hi,
> > 
> > Continuing the topic on the need for integration style tests for
> > Nova
> > (brought up earlier during the weekly meeting at #openstack-
> > meeting,
> > Sep 22). The proposal is for a project to hold integration tests
> > that
> > includes low-level testing and runs against a devstack backend. I
> > have
> > included more details here -
> > https://etherpad.openstack.org/p/integration-tests
> > 
> > Please comment on the need for the project, whether or not any
> > similar
> > efforts are in place, approaches suggested, taking forward the
> > initiative, etc.

I missed that conversation, so to clarify, these would be additional
integration tests (as opposed to functional, unit tests) but kept
outside of tempest, correct? If so, how do these differ from what
third-party CIs already provide? [1] If they do provide something
different (i.e. they're run upstream), could these be kept in-tree
(like neutron's scenario tests [2]) rather than in a different project?

Stephen

[1] http://docs.openstack.org/developer/nova/test_strategy.html#infra-v
s-third-party
[2] https://github.com/openstack/neutron/blob/master/TESTING.rst#scenar
io-tests

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Anyone interested in writing a policy generator sphinx extension?

[Stephen Finucane]

On Wed, 2016-09-21 at 11:16 -0400, Doug Hellmann wrote:
> Excerpts from Matt Riedemann's message of 2016-09-21 09:49:29 -0500:
> > 
> > Nova has policy defaults in code now and we can generate the
> > sample 
> > using oslopolicy-sample-generator but we'd like to get the default 
> > policy sample in the Nova developer documentation also, like we
> > have for 
> > nova.conf.sample.
> > 
> > I see we use the sphinxconfiggen extension for building the 
> > nova.conf.sample in our docs, but I don't see anything like that
> > for 
> > generating docs for a sample policy file.
> > 
> > Has anyone already started working on that, or is interested in
> > working 
> > on that? I've never written a sphinx extension before but I'm
> > guessing 
> > it could be borrowed a bit from how sphinxconfiggen was written in 
> > oslo.config.
> > 
> 
> I don't have time to do it myself, but I can help get someone else
> started and work with them on code reviews in oslo.policy.

I can take a look at this, though I'll only persue a "sphinxpolicygen"
module for now (we don't use oslo_config's "sphinxext" module yet).

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Follow up on BCN review cadence discussions

[Stephen Finucane]

On Mon, 2016-11-07 at 11:50 +, Matthew Booth wrote:
> I'd like to follow up on the discussions we had in Barcelona around
> review cadence. I took a lot away from these discussions, and I
> thought they were extremely productive. I think the summary of the
> concerns was:
> 
>   Nova is a complex beast, very few people know even most of it well.
>   There are areas of Nova where mistakes are costly and hard to
> rectify later.
>   A large amount of good code does not merge quickly.
>   The pool of active core reviewers is very much smaller than the
> total pool of contributors.
>   The barrier to entry for Nova core is very high.
> 
> There was more, but I think most people agreed on the above.
> 
> Just before summit I pitched a subsystem maintainer model here:
>   http://lists.openstack.org/pipermail/openstack-dev/2016-September/1
> 04093.html
> 
> Reading over this again, I still think this is worth trialling: it
> pushes the burden of initial detailed review further out, whilst
> ensuring that a core will still check that no wider issues have been
> missed. Bearing in mind that the primary purpose is to reduce the
> burden on core reviewers of any individual patch, I think this will
> work best if we aggressively push initial review down as far as
> possible.
> 
> I'm still not sure what the best description of 'domain' is, though.
> Other projects seem to have defined this as: the reviewer should, in
> good faith, know when they're competent to give a +2 and when they're
> not. I suspect this is too loose for us, but I'm sure we could come
> up with something.
> 
> I'd expect the review burden of a maintainer of an active area of
> code to be as heavy as a core reviewer's, so this probably shouldn't
> be taken lightly. If we were to trial it, I'd also recommend starting
> with a small number of maintainers (about 3, perhaps), preferably
> technical-leaders of active sub-teams. Any trial should be the topic
> of a retrospective at the PTG.
> 
> I'd like to re-iterate that what I'd personally like to achieve is:
> 
>   "Good code should merge quickly."
> 
> There are likely other ways to achieve this, and if any of them are
> better than this one then I'm in favour. However, I think we can try
> this one with limited risk and initial up-front effort.

I'm just going to leave this here...

https://github.com/torvalds/linux/blob/master/MAINTAINERS
http://dpdk.org/browse/dpdk/tree/MAINTAINERS

We probably don't want to/can't go down the path of hard subtrees
(though we might be already simulating that with the various 'os-'
projects). I do imagine, however, that most folks who have been working
on nova for long enough have a list of domain experts in their heads
already. Would actually putting that on paper really hurt?

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][SR-IOV] update the SR-IOV meeting to bi-weekly

[Stephen Finucane]

On Thu, 2016-12-01 at 14:48 +, Moshe Levi wrote:
> Hi all,
> 
> I would like to update the frequency of the SR-IOV meeting to be  bi-
> weekly [1].
> Does anyone see value in keeping it as weekly meeting?
> 
> [1] - https://review.openstack.org/#/c/405415/

Seeing as we don't have any specs during this short cycle, I think
moving back to bi-weekly is a good idea. We can reassess come February
and the Pike cycle.

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Continued reduction in NFV tech debt - input?

[Stephen Finucane]

nova has worked hard over the last few cycles to identify and reduce
the amount of tech debt we're carrying. One such item identified early
on was the lack of automated testing and upstream documentation for NFV
features, such as CPU pinning, realtime, hugepages, SR-IOV, and PCI
passthrough. We discussed this at the Austin (M) summit and have been
slowly chugging away at it since through a combination of weekly SR-IOV 
meetings [1] and changes to the docs [2][3][4][5], code and in- [6] and
out-of-tree tests [7].

This work will continue for this cycle, however, we'd like to ask is
there any of these features where developers or operators can still
identify significant holes in either validation or (upstream)
documentation? If so, please give us a shout. This is something we'll
probably double back on in a PTG session, but we'd like to make full
use of the few months between now and then to really build on these
features.

I'd also like to take the opportunity to promote the weekly SR-IOV
meeting. We regularly address all things NFV at this meeting, so if
you've ideas on how to improve or build upon these features, this is as
good a place to bring said ideas up.

Cheers,
Stephen (sfinucan)

[1] http://eavesdrop.openstack.org/#SR-IOV/PCI_Passthrough_Meeting
[2] http://docs.openstack.org/networking-guide/config-sriov.html
[3] http://docs.openstack.org/admin-guide/compute-pci-passthrough.html
[4] http://docs.openstack.org/admin-guide/compute-cpu-topologies.html
[5] http://docs.openstack.org/admin-guide/compute-huge-pages.html
[6] https://github.com/openstack/nova/tree/master/nova/tests/functional
/libvirt
[7] https://github.com/openstack/intel-nfv-ci-tests

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] How should we go about removing legacy VIF types in Queens?

[Stephen Finucane]

os-vif has been integrated into nova since the newton cycle. With the
integration of os-vif, the expectation is that all the old, non-os-vif
plugging/unplugging code found in [1] will be replaced by code that harnesses
os-vif plugins [2]. This has happened for a few of the VIF types, and newer
VIFs are being added in this manner [3]. However, there are quite a few VIFs
that are still using the legacy path, and I think it's about time we started
moving things forward. Doing so allows us to continue to progress on passing
os-vif objects from neutron and remove the large swathes of legacy code still
found in nova. 

I've opened a bug against networking-bigswitch [4] for one of these VIF types,
IVS, and I'm thinking I'll do the same for a lot of the other VIF types where I
can find definite vendors. Is there anything else we can do though? At some
point we're going to have to just start deleting code and I'd like to avoid
leaving operators in the lurch.

Cheers,
Stephen

[1] https://github.com/openstack/nova/blob/6205a3f8c/nova/virt/libvirt/vif.py#L
599-L764
[2] https://github.com/openstack/nova/blob/6205a3f8c/nova/network/os_vif_util.p
y#L346-L403
[3] https://github.com/Juniper/contrail-nova-vif-driver
[4] https://bugs.launchpad.net/networking-bigswitch/+bug/1704129

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [openstack-doc] Attending PTG?

[Stephen Finucane]

On Fri, 2017-07-21 at 10:33 -0400, Doug Hellmann wrote:
> Excerpts from Alexandra Settle's message of 2017-07-21 13:41:58 +:
> > Hey everyone,
> > 
> > Put your hand in the air* if you’re attending the PTG!
> > 
> > Myself and the potential PTL will need to start planning shortly, and it
> > would be good to get an idea of numbers, and who we are planning for.
> > 
> > I will be looking at hosting a working bee or hack session for migration
> > related issues throughout the week.
> > 
> > Cheers,
> > 
> > Alex
> > 
> > *reply to this ML thread
> 
> Count me in.

I'll be there too.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] How should we go about removing legacy VIF types in Queens?

[Stephen Finucane]

On Thu, 2017-07-13 at 07:54 -0600, Kevin Benton wrote:
> On Thu, Jul 13, 2017 at 7:26 AM, Stephen Finucane <sfinu...@redhat.com>
wrote:
> 
> > os-vif has been integrated into nova since the newton cycle. With the
> > integration of os-vif, the expectation is that all the old, non-os-vif
> > plugging/unplugging code found in [1] will be replaced by code that
> > harnesses
> > os-vif plugins [2]. This has happened for a few of the VIF types, and newer
> > VIFs are being added in this manner [3]. However, there are quite a few
> > VIFs
> > that are still using the legacy path, and I think it's about time we
> > started
> > moving things forward. Doing so allows us to continue to progress on
> > passing
> > os-vif objects from neutron and remove the large swathes of legacy code
> > still
> > found in nova.
> > 
> > I've opened a bug against networking-bigswitch [4] for one of these VIF
> > types,
> > IVS, and I'm thinking I'll do the same for a lot of the other VIF types
> > where I
> > can find definite vendors. Is there anything else we can do though? At some
> > point we're going to have to just start deleting code and I'd like to avoid
> > leaving operators in the lurch.
>
> Some of the stuff like '802.1qbh' isn't particularly vendor specific so I'm
> not sure who will host it and a repo just for that seems like a bit much.
> Should we just bite the bullet and convert them in the nova tree or put them
> in os-vif?

That VIF type actually seems to be a CISCO-only option [1][2] but I get what
you're saying. I think we can definitely move some of them, though (IVS, for a
start). Perhaps moving the ones that *do* have clear owners to their respective
packages is the way to go?

Stephen

[1] http://codesearch.openstack.org/?q=802.1qbh=nope==
[2] https://git.openstack.org/cgit/openstack/networking-cisco/tree/networking_c
isco/plugins/ml2/drivers/cisco/ucsm/constants.py

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][docs] Concerns with docs migration

[Stephen Finucane]

On Wed, 2017-08-02 at 09:28 -0600, Chris Friesen wrote:
> On 08/02/2017 09:22 AM, Stephen Finucane wrote:
> > On Wed, 2017-08-02 at 09:55 -0500, Matt Riedemann wrote:
> > > 3. The patch for the import of the admin guide [8] is missing some CLI
> > > specific pages which are pretty useful given they aren't documented
> > > anywhere else, like the forced_host part of the compute API [9].
> > > Basically anything that's cli-nova-* in the admin guide should be in the
> > > Nova docs. It's also missing the compute-flavors page [10] which is
> > > pretty important for using OpenStack at all.
> > 
> > This is a tricky one. Based on previous discussions with dhellmann, the
> > plan
> > seems to be to replace any references to 'nova xxx' or 'openstack xxx'
> > commands
> > (i.e. commands using python-novaclient or python-openstackclient) in favour
> > of
> > 'curl'-based requests. The idea here is that the Python clients are not the
> > only clients available, and we shouldn't be "mandating" their use by
> > referencing them in the docs. I get this, though I don't fully agree with
> > it
> > (who really uses curl?)
>
> Are we going to document the python clients elsewhere then?

I think the docs would go into the respective python clients docs?

> Personally I find it highly useful to have complete examples of how to do
> things with python-novaclient or python-openstackclient.

I agree. Personally, I'd like to see something like Stripe's API docs [1],
maybe through a not-yet-existant 'sphinx-slate' extension [2], but that seems a
lot of work for very little gain and would need serious maintaining.

> Given that any users of the raw HTTP API are likely going to be developers, 
> while users of the CLI tools may not be, it seems more important to give
> good examples of using the CLI tools.  Any developer should be able to figure
> out the underlying HTTP (using the --debug option of the CLI tool if
> necessary).

I think the main idea is that the 'python-*client's are not the only game in
town and should not be treated as such. Who these other clients are is a
question I don't personally know the answer to, however.

In any case, this is surely a post-Pike issue, as noted above.

Stephen

[1] https://stripe.com/docs/api
[2] https://github.com/lord/slate

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][docs] Concerns with docs migration

[Stephen Finucane]

On Wed, 2017-08-02 at 09:55 -0500, Matt Riedemann wrote:
> Now that Stephen Finucane is back from enjoying his youth and 
> gallivanting all over Europe, and we talked about a few things in IRC 
> this morning on the docs migration for Nova, I wanted to dump my 
> concerns here for broader consumption.
> 
> 1. We know we have to fix a bunch of broken links by adding in redirects 
> [1] which sdague started here [2]. However, that apparently didn't catch 
> everything, e.g. [3], so I'm concerned we're missing other broken links. 
> Is there a way to find out?

We knew this was going to be an issue, but I wasn't aware of any way to fix
this. The solution looks good though - nice work.

Unfortunately I can't think of any cleverer way to identify these broken links
than manual inspection. I'll do this again for all the patches I've authored
and try to do them for any others I encounter.

> 2. The bottom change in the docs migration series for Nova is a massive 
> refactor of the layout of the Nova devref [4]. That's something I don't 
> want to do in Pike for two reasons:
> 
> a) It's a huge change and we simply don't have the time to invest in 
> properly assessing and reviewing it before Pike RC1.
> 
> b) I think that if we're going to refactor the Nova devref home page to 
> be a certain format, then we should really consider doing the same thing 
> in the other projects, because today they are all different formats 
> [5][6][7]. This is likely a cross-project discussion for the Queens PTG 
> to determine if the home page for the projects should look similar. It 
> seems they should given the uniformity that the Foundation has been 
> working toward lately.

This is fair. I've been working on this with asettle and I personally agree
with a lot of the changes/design decisions made there. However, I understand
the time constraints so we can surely split this out. I'll work with asettle on
this too.

> 3. The patch for the import of the admin guide [8] is missing some CLI 
> specific pages which are pretty useful given they aren't documented 
> anywhere else, like the forced_host part of the compute API [9]. 
> Basically anything that's cli-nova-* in the admin guide should be in the 
> Nova docs. It's also missing the compute-flavors page [10] which is 
> pretty important for using OpenStack at all.

This is a tricky one. Based on previous discussions with dhellmann, the plan
seems to be to replace any references to 'nova xxx' or 'openstack xxx' commands
(i.e. commands using python-novaclient or python-openstackclient) in favour of
'curl'-based requests. The idea here is that the Python clients are not the
only clients available, and we shouldn't be "mandating" their use by
referencing them in the docs. I get this, though I don't fully agree with it
(who really uses curl?). In any case though, this would surely be a big rewrite
and, per your concerns in #2 above,  doesn't sound like something we should be
doing in Pike. I guess a basic import is the way to go for now. I'll take care
of this.

> 4. Similar to #3, but we don't have a patch yet for importing the user 
> guide and there are several docs in the user guide that are Nova 
> specific so I'd like to make sure we include those, like [11][12].

As above.

Stephen

> [1] 
> http://lists.openstack.org/pipermail/openstack-dev/2017-August/120418.html
> [2] https://review.openstack.org/#/c/489650/
> [3] https://review.openstack.org/#/c/489641/
> [4] https://review.openstack.org/#/c/478485/
> [5] https://docs.openstack.org/cinder/latest/
> [6] https://docs.openstack.org/glance/latest/
> [7] https://docs.openstack.org/neutron/latest/
> [8] https://review.openstack.org/#/c/477497/
> [9] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/admin-gu
> ide/source/cli-nova-specify-host.rst
> [10] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/admin-gu
> ide/source/compute-flavors.rst
> [11] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/user-gui
> de/source/cli-launch-instances.rst
> [12] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/user-gui
> de/source/cli-delete-an-instance.rst
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][docs] Concerns with docs migration

[Stephen Finucane]

On Wed, 2017-08-02 at 09:55 -0500, Matt Riedemann wrote:
> Now that Stephen Finucane is back from enjoying his youth and 
> gallivanting all over Europe, and we talked about a few things in IRC 
> this morning on the docs migration for Nova, I wanted to dump my 
> concerns here for broader consumption.
> 
> 1. We know we have to fix a bunch of broken links by adding in redirects 
> [1] which sdague started here [2]. However, that apparently didn't catch 
> everything, e.g. [3], so I'm concerned we're missing other broken links. 
> Is there a way to find out?

Manual inspection is probably the easiest way to go. 

> 2. The bottom change in the docs migration series for Nova is a massive 
> refactor of the layout of the Nova devref [4]. That's something I don't 
> want to do in Pike for two reasons:
> 
> a) It's a huge change and we simply don't have the time to invest in 
> properly assessing and reviewing it before Pike RC1.
> 
> b) I think that if we're going to refactor the Nova devref home page to 
> be a certain format, then we should really consider doing the same thing 
> in the other projects, because today they are all different formats 
> [5][6][7]. This is likely a cross-project discussion for the Queens PTG 
> to determine if the home page for the projects should look similar. It 
> seems they should given the uniformity that the Foundation has been 
> working toward lately.
> 
> 3. The patch for the import of the admin guide [8] is missing some CLI 
> specific pages which are pretty useful given they aren't documented 
> anywhere else, like the forced_host part of the compute API [9]. 
> Basically anything that's cli-nova-* in the admin guide should be in the 
> Nova docs. It's also missing the compute-flavors page [10] which is 
> pretty important for using OpenStack at all.
> 
> 4. Similar to #3, but we don't have a patch yet for importing the user 
> guide and there are several docs in the user guide that are Nova 
> specific so I'd like to make sure we include those, like [11][12].
> 
> [1] 
> http://lists.openstack.org/pipermail/openstack-dev/2017-August/120418.html
> [2] https://review.openstack.org/#/c/489650/
> [3] https://review.openstack.org/#/c/489641/
> [4] https://review.openstack.org/#/c/478485/
> [5] https://docs.openstack.org/cinder/latest/
> [6] https://docs.openstack.org/glance/latest/
> [7] https://docs.openstack.org/neutron/latest/
> [8] https://review.openstack.org/#/c/477497/
> [9] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/admin-gu
> ide/source/cli-nova-specify-host.rst
> [10] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/admin-gu
> ide/source/compute-flavors.rst
> [11] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/user-gui
> de/source/cli-launch-instances.rst
> [12] 
> https://github.com/openstack/openstack-manuals/blob/stable/ocata/doc/user-gui
> de/source/cli-delete-an-instance.rst
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][docs] Concerns with docs migration

[Stephen Finucane]

On Wed, 2017-08-02 at 09:28 -0700, Clark Boylan wrote:
> On Wed, Aug 2, 2017, at 07:55 AM, Matt Riedemann wrote:
> > Now that Stephen Finucane is back from enjoying his youth and 
> > gallivanting all over Europe, and we talked about a few things in IRC 
> > this morning on the docs migration for Nova, I wanted to dump my 
> > concerns here for broader consumption.
> > 
> > 1. We know we have to fix a bunch of broken links by adding in redirects 
> > [1] which sdague started here [2]. However, that apparently didn't catch 
> > everything, e.g. [3], so I'm concerned we're missing other broken links. 
> > Is there a way to find out?
> 
> The infra team can generate lists of 404 urls fairly easily on the docs
> server. This won't show you everything but will show you what urls
> people are finding/using that 404.

I'd appreciate that. Have been going through things manually this afternoon but
it's slow work. Can you do this (or know who could)?

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposing Balazs Gibizer for nova-core

[Stephen Finucane]

On Tue, 2017-08-22 at 20:18 -0500, Matt Riedemann wrote:
> I'm proposing that we add gibi to the nova core team. He's been around 
> for awhile now and has shown persistence and leadership in the 
> multi-release versioned notifications effort, which also included 
> helping new contributors to Nova get involved which helps grow our 
> contributor base.
> 
> Beyond that though, gibi has a good understanding of several areas of 
> Nova, gives thoughtful reviews and feedback, which includes -1s on 
> changes to get them in shape before a core reviewer gets to them, 
> something I really value and look for in people doing reviews who aren't 
> yet on the core team. He's also really helpful with not only reporting 
> and triaging bugs, but writing tests to recreate bugs so we know when 
> they are fixed, and also works on fixing them - something I expect from 
> a core maintainer of the project.
> 
> So to the existing core team members, please respond with a yay/nay and 
> after about a week or so we should have a decision (knowing a few cores 
> are on vacation right now).

Yay, for sure.

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [storyboard] Should New Projects Be Using Storyboard?

[Stephen Finucane]

On Tue, 2017-10-03 at 12:57 -0700, Mike Perez wrote:
> I noticed that the project creator [1] and cookiecutter [2] promote using
> launchpad. If we're migrating projects to storyboard today, should we stop
> promoting launchpad for new projects?

Not to hijack the thread but, we're moving projects to storyboard? The only
thing I can find about this is that the Watcher project considering switching
[1]. Is there a spec or something I can look at about this?

Stephen

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/117963.html

> [1] - https://docs.openstack.org/infra/manual/creators.html
> [2] - https://git.openstack.org/cgit/openstack-dev/cookiecutter/tree/cookiecu
> tter.json#n6
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [neutron] Should we continue providing FQDNs for instance hostnames?

[Stephen Finucane]

On Tue, 2017-09-05 at 12:30 -0500, Ben Nemec wrote:
> On 09/04/2017 07:48 AM, Stephen Finucane wrote:
> > On Mon, 2017-09-04 at 11:51 +, Gary Kotton wrote:
> > > On 9/4/17, 11:18 AM, "Stephen Finucane" <sfinu...@redhat.com> wrote:
> > > > Nova has a feature whereby it will provide instance host names that
> > > > cloud-init can extract and use inside the guest, i.e. this won't happen
> > > > without cloud-init. These host names are fully qualified domain names
> > > > (FQDNs) based upon the instance name and local domain name. However, as
> > > > noted in bug #1698010 [1], the domain name part of this is based up
> > > > nova's 'dhcp_domain' option, which is a nova-network option that has
> > > > been deprecated [2].
> > > >  
> > > > I've started work on a fix [3] that will allow us to retrieve this
> > > > information from neutron instead, uncoupling us from this legacy
> > > > option. However, some commentators have pointed out that this may not
> > > > necessarily be what we want to do: a FQDN is a hostname and domain
> > > > name, and using one as a hostname may not be that clever nor correct.
> > > > 
> > > > My networking fu isn't strong enough to deliver a verdict so I'm hoping
> > > > someone else can make my mind up for me: do we want to migrate this
> > > > feature to neutron, or do we want to stop using FQDN and just use
> > > > instance names?
> > > 
> > > Hi,
> > > 
> > > Your patch https://review.openstack.org/#/c/480616/ requires that neutron
> > > expose the ‘DNS Integration’ extension be support by neutron and the
> > > relevant networking plugin. If it does not then will be a regression and
> > > things will not work.
> > > 
> > > In addition to this that is per network and not global so there will also
> > > be a regression for running instances if nova is updated.
> > 
> > OK, so ultimately this isn't something we can rely on from neutron? Does
> > that mean we should abandon the idea of providing FQDN when using neutron?
> > Mores to the original point, is there any reason we would want to/should
> > use FQDN?
> 
> TripleO gets its FQDNs from this feature, which is how I noticed it was 
> still using the deprecated nova.conf value.  So we need some way to 
> maintain that functionality.  I'm not sure what would happen if 
> cloud-init stopped getting an FQDN - would it just set the hostname but 
> leave the domain as whatever was provided via DHCP?  If so I think that 
> would be acceptable for us.

OK, and we do want to use a FQDN as the hostnames inside the guest? That's my
key question. Random ServerFault questions seems to suggest no clear answer to
this [1].

> On the other hand, since it turns out this configuration option isn't 
> only used with nova-network maybe it should just be un-deprecated? 
> Perhaps with an update so that it is only used when Neutron doesn't 
> provide an FQDN?

You're correct in pointing out that it's not only used by nova-network, but
this seemed to be a mistake. The main issue I saw was that the nova-network
option is completely divorced from the neutron configuration and is static.
However, if we can't guarantee that this information will be available from
neutron, then perhaps a static configuration option is the only way to go? It's
quite the pickle.

Stephen

[1] https://serverfault.com/q/331936/

> > > > [1] https://bugs.launchpad.net/nova/+bug/1698010
> > > > [2] https://review.openstack.org/#/c/395683/
> > > > [3] https://review.openstack.org/#/c/480616/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [neutron] Should we continue providing FQDNs for instance hostnames?

[Stephen Finucane]

On Wed, 2017-09-06 at 15:06 +, Jeremy Stanley wrote:
> On 2017-09-06 10:40:14 +0100 (+0100), Stephen Finucane wrote:
> [...]
> > OK, and we do want to use a FQDN as the hostnames inside the
> > guest? That's my key question. Random ServerFault questions seems
> > to suggest no clear answer to this [1].
> 
> [...]
> 
> As a data point, the servers maintained by the Infra team are
> consistently built with their FQDN as the instance name itself, and
> then the hostname and domainname set from that value in order to
> avoid the dreaded "foo.novalocal" sorts of FQDNs prevalent in so
> many OpenStack environments. At least for us, having OpenStack
> specify the domain is more of a nuisance to be worked around than
> anything.

That's great feedback.

> You may also want to consider asking a similar question on the
> operators ML, or perhaps sending them a pointer to this discussion
> so you can get more real-world feedback.

Good call. I'll re-post this to the operators mailing list (with the above
point) and see what feedback I receive.

Cheers,
Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [neutron] Adding neutron VIF NUMA locality support

[Stephen Finucane]

Hey,

NUMA locality matters as much for NICs used e.g for Open vSwitch as for SR-IOV
devices. At the moment, nova support NUMA affinity for PCI passthrough devices
and SR-IOV devices, but it makes no attempt to do the same for other NICs. In
the name of NFV enablement, we should probably close this gap.

I have some ideas around how this could work, but they're fuzzy enough and
involve exchanging os-vif objects between nova and neutron. This is probably
the most difficult path as we've been trying to get os-vif objects over the
nova-neutron wire for a while now, to no success.

Anyone else keen on such a feature? Given that there are a significant amount
of people from nova, neutron, and general NFV backgrounds at the PTG next
week, we have a very good opportunity to talk about this (either in the nova-
neutron sync, if that's not already full, or in some hallway somewhere).

At this point in the day, this is probably very much a Rocky feature, but we
could definitely put in whatever groundwork is necessary this cycle to make the
work in Rocky as easy possible.

Cheers,
Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Running large instances with CPU pinning and OOM

[Stephen Finucane]

On Mon, 2017-09-25 at 17:36 +0200, Jakub Jursa wrote:
> Hello everyone,
> 
> We're experiencing issues with running large instances (~60GB RAM) on
> fairly large NUMA nodes (4 CPUs, 256GB RAM) while using cpu pinning. The
> problem is that it seems that in some extreme cases qemu/KVM can have
> significant memory overhead (10-15%?) which nova-compute service doesn't
> take in to the account when launching VMs. Using our configuration as an
> example - imagine running two VMs with 30GB RAM on one NUMA node
> (because we use cpu pinning) - therefore using 60GB out of 64GB for
> given NUMA domain. When both VMs would consume their entire memory
> (given 10% KVM overhead) OOM killer takes an action (despite having
> plenty of free RAM in other NUMA nodes). (the numbers are just
> arbitrary, the point is that nova-scheduler schedules the instance to
> run on the node because the memory seems 'free enough', but specific
> NUMA node can be lacking the memory reserve).
> 
> Our initial solution was to use ram_allocation_ratio < 1 to ensure
> having some reserved memory - this didn't work. Upon studying source of
> nova, it turns out that ram_allocation_ratio is ignored when using cpu
> pinning. (see
> https://github.com/openstack/nova/blob/mitaka-eol/nova/virt/hardware.py#L859
> and
> https://github.com/openstack/nova/blob/mitaka-eol/nova/virt/hardware.py#L821
> ). We're running Mitaka, but this piece of code is implemented in Ocata
> in a same way.
> We're considering to create a patch for taking ram_allocation_ratio in
> to account.
> 
> My question is - is ram_allocation_ratio ignored on purpose when using
> cpu pinning? If yes, what is the reasoning behind it? And what would be
> the right solution to ensure having reserved RAM on the NUMA nodes?

Both 'ram_allocation_ratio' and 'cpu_allocation_ratio' are ignored when using
pinned CPUs because they don't make much sense: you want a high performance VM
and have assigned dedicated cores to the instance for this purpose, yet you're
telling nova to over-schedule and schedule multiple instances to some of these
same cores.

What you're probably looking for is the 'reserved_host_memory_mb' option. This
defaults to 512 (at least in the latest master) so if you up this to 4192 or
similar you should resolve the issue.

Hope this helps,
Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] [neutron] Should we continue providing FQDNs for instance hostnames?

[Stephen Finucane]

Nova has a feature whereby it will provide instance host names that cloud-init
can extract and use inside the guest, i.e. this won't happen without cloud-
init. These host names are fully qualified domain names (FQDNs) based upon the
instance name and local domain name. However, as noted in bug #1698010 [1], the
domain name part of this is based up nova's 'dhcp_domain' option, which is a
nova-network option that has been deprecated [2].

I've started work on a fix [3] that will allow us to retrieve this information
from neutron instead, uncoupling us from this legacy option. However, some
commentators have pointed out that this may not necessarily be what we want to
do: a FQDN is a hostname and domain name, and using one as a hostname may not
be that clever nor correct.

My networking fu isn't strong enough to deliver a verdict so I'm hoping someone
else can make my mind up for me: do we want to migrate this feature to neutron,
or do we want to stop using FQDN and just use instance names?

Cheers,
Stephen

[1] https://bugs.launchpad.net/nova/+bug/1698010
[2] https://review.openstack.org/#/c/395683/
[3] https://review.openstack.org/#/c/480616/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [neutron] Should we continue providing FQDNs for instance hostnames?

[Stephen Finucane]

On Mon, 2017-09-04 at 11:51 +, Gary Kotton wrote:
> On 9/4/17, 11:18 AM,
"Stephen Finucane" <sfinu...@redhat.com> wrote:
> > Nova has a feature whereby
it will provide instance host names that cloud-
> > init can extract and use
inside the guest, i.e. this won't happen without
> > cloud-init. These host
names are fully qualified domain names (FQDNs) based
> > upon the instance name
and local domain name. However, as noted in bug
> > #1698010 [1], the domain
name part of this is based up nova's 'dhcp_domain'
> > option, which is a nova-
network option that has been deprecated [2].
> > 
> > I've started work on a
fix [3] that will allow us to retrieve this
> > information from neutron
instead, uncoupling us from this legacy option.
> > However, some commentators
have pointed out that this may not necessarily 
> > be what we want to do: a
FQDN is a hostname and domain name, and using one 
> > as a hostname may not be
that clever nor correct.
> >
> > My networking fu isn't strong enough to deliver
a verdict so I'm hoping
> > someone else can make my mind up for me: do we want
to migrate this feature
> > to neutron, or do we want to stop using FQDN and
just use instance names?
>
> Hi,
>
> Your patch https://review.openstack.org/#/c/48
0616/ requires that neutron
> expose the ‘DNS Integration’ extension be support
by neutron and the relevant
> networking plugin. If it does not then will be a
regression and things will
> not work.
>
> In addition to this that is per network
and not global so there will also be
> a regression for running instances if
nova is updated.

OK, so ultimately this isn't something we can rely on from neutron? Does that
mean we should abandon the idea of providing FQDN when using neutron? Mores to
the original point, is there any reason we would want to/should use FQDN?

Stephen

> > [1] https://bugs.launchpad.net/nova/+bug/1698010
> > [2] https://review.openstack.org/#/c/395683/
> > [3] https://review.openstack.org/#/c/480616/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [pbr] Migration to Storyboard

[Stephen Finucane]

The Python packaging ecosystem is shifting under our feet and we have a couple
of things in pbr we need to look at over the next few months (Pipfiles, pyenv,
changes in setuptools setup.cfg, ...). We should probably track these somewhere
and Storyboard [1] seems like a better candidate for doing this than Launchpad
(plus, I get to play with Storyboard).

I can initiate the transfer (it seems pretty easy to do [2]) but I'd like to
make sure there are no objections before doing so.

Speak now or forever hold your tongue.

Stephen

[1] https://docs.openstack.org/infra/storyboard/
[2] https://docs.openstack.org/infra/storyboard/migration.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Free documentation using Sphinx extensions

[Stephen Finucane]

As part of the documentation onboarding session this morning, we gave a small
talk on how people can actually avoid writing documentation (or rather use the
documentation they have already stored in their code). There seems to be a
general lack of awareness about these tools and I've uploaded the slides to
SpeakerDeck in the hopes of closing this gap. You can find them here:

  https://speakerdeck.com/stephenfin/openstack-plus-sphinx-in-a-tree

Feel free to reach out to me on IRC (stephenfin) or contact any of the docs
team on #openstack-doc if you've any questions or suggestions for these tools.

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-vif] [nova] Changes to os-vif cores

[Stephen Finucane]

On Tue, 2017-10-24 at 15:32 +0100, Stephen Finucane wrote:
> Hey,
> 
> I'm not actually sure what the protocol is for adding/removing cores to a
> library project without a PTL, so I'm just going to put this out there: I'd
> like to propose the following changes to the os-vif core team.
> 
> - Add 'nova-core'
> 
>   os-vif makes extensive use of objects and we've had a few hiccups around
>   versionings and the likes recently [1][2]. I'd the expertise of some of the
>   other nova cores here as we roll this out to projects other than nova, and 
>   I trust those not interested/knowledgeable in this area to stay away :)
> 
> - Remove Russell Bryant, Maxime Leroy
> 
>   These folks haven't been active on os-vif  [3][4] for a long time and I 
>   think they can be safely removed.
> 
> To the existing core team members, please respond with a yay/nay and we'll
> wait a week before doing anything.

Both of these are done now. Thanks for the feedback, folks.

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [os-vif] [nova] Changes to os-vif cores

[Stephen Finucane]

Hey,

I'm not actually sure what the protocol is for adding/removing cores to a
library project without a PTL, so I'm just going to put this out there: I'd
like to propose the following changes to the os-vif core team.

- Add 'nova-core'

  os-vif makes extensive use of objects and we've had a few hiccups around
  versionings and the likes recently [1][2]. I'd the expertise of some of the
  other nova cores here as we roll this out to projects other than nova, and I
  trust those not interested/knowledgeable in this area to stay away :)

- Remove Russell Bryant, Maxime Leroy

  These folks haven't been active on os-vif  [3][4] for a long time and I think
  they can be safely removed.

To the existing core team members, please respond with a yay/nay and we'll wait
a week before doing anything.

Cheers,
Stephen

[1] https://review.openstack.org/#/c/508498/
[2] https://review.openstack.org/#/c/509107/
[3] https://review.openstack.org/#/q/reviewedby:%22Russell+Bryant+%253Crbryant%
2540redhat.com%253E%22+project:openstack/os-vif
[4] https://review.openstack.org/#/q/reviewedby:%22Maxime+Leroy+%253Cmaxime.ler
oy%25406wind.com%253E%22+project:openstack/os-vif

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] nova-manage cell_v2 map_instances uses invalid UUID as marker in the db

[Stephen Finucane]

On Tue, 2018-05-08 at 11:49 +0200, Balázs Gibizer wrote:
> Hi,
> 
> The oslo UUIDField emits a warning if the string used as a field value 
> does not pass the validation of the uuid.UUID(str(value)) call [3]. All 
> the offending places are fixed in nova except the nova-manage cell_v2 
> map_instances call [1][2]. That call uses markers in the DB that are 
> not valid UUIDs. If we could fix this last offender then we could merge 
> the patch [4] that changes the this warning to an exception in the nova 
> tests to avoid such future rule violations.
> 
> However I'm not sure it is easy to fix. Replacing 
> 'INSTANCE_MIGRATION_MARKER' at [1] to 
> '----' might work but I don't know what to 
> do with instance_uuid.replace(' ', '-') [2] to make it a valid uuid. 
> Also I think that if there is an unfinished mapping in the deployment 
> and then the marker is changed in the code that leads to 
> inconsistencies.
> 
> I'm open to any suggestions.
> 
> Cheers,
> gibi

This is a somewhat complicated issue. I haven't got any ideas to solve
this (edleafe tried and failed) but I have submitted a patch to explain
why we do this, pending a real resolution.

   https://review.openstack.org/567597

Stephen

> 
> [1] 
> https://github.com/openstack/nova/blob/09af976016a83288df22ac6ed1cce1676c2294cc/nova/cmd/manage.py#L1168
> [2] 
> https://github.com/openstack/nova/blob/09af976016a83288df22ac6ed1cce1676c2294cc/nova/cmd/manage.py#L1180
> [3] 
> https://github.com/openstack/oslo.versionedobjects/blob/29e643e4a9866b33965b68fc8dfb8acf30fa/oslo_versionedobjects/fields.py#L359
> [4] https://review.openstack.org/#/c/540386
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] psycopg2 wheel packaging issues

[Stephen Finucane]

On Tue, 2018-05-15 at 07:24 -0400, Doug Hellmann wrote:
> Excerpts from Stephen Finucane's message of 2018-05-15 11:44:11 +0100:
> > I imagine most people have been seeing warnings like the one below
> > raised by various openstack packages recently:
> > 
> >   .tox/py27/lib/python2.7/site-packages/psycopg2/__init__.py:144: 
> > UserWarning: The psycopg2
> >   wheel package will be renamed from release 2.8; in order to keep 
> > installing from binary
> >   please use "pip install psycopg2-binary" instead. For details see:
> >   .
> > 
> > Based on this warning, I had done what seemed to be the obvious thing
> > to do and proposed adding psycopg2-binary to the list of global
> > requirements [1]. This would allow us to replace all references to
> > psycopg2 with psycopg2-wheel in individual projects. However, upon
> > further investigation it seems this is not really an option since the
> > two packages exist in the same namespace and will clobber each other.
> > I've now abandoned this patch.
> > 
> > Does anyone with stronger Python packaging-fu than I have a better
> > solution for the psycopg2 folks? There's a detailed description of why
> > this was necessary on GitHub [2] along with some potential resolutions,
> > none of which seem to be acceptable. If nothing better is possible, it
> > seems we'll simply have to live with (or silence) these warnings in
> > psycopg2 2.7.x and start installing libpg again once 2.8 is released.
> > 
> > Cheers,
> > Stephen
> > 
> > [1] https://review.openstack.org/#/c/561924/
> > [2] https://github.com/psycopg/psycopg2/issues/674
> > 
> 
> Bundling an SSL library seems like a particularly bad situation, but if
> its ABI isn't stable it may be all they can do.
> 
> Perhaps some of the folks in the community who actually use Postgresql
> can get involved with helping the upstream maintainers of psycopg and
> libpg sort things out.

Yes, this would be my hope.

> In the mean time, is there any reason we can't just continue to
> install psycopg2 from source in our gate jobs after 2.8? If the
> wheel packages for psycopg2 2.7.x are bad perhaps we can come up
> with a way to pass --no-binary when installing it, but it's not
> clear if we need to. Does the bug affect us?

The only reason we might have issues is the libpq dependency. This was
required in 2.6 and will be required once again in 2.8. If this hasn't
been dropped from the list of requirements then we won't see any
breakages. If we do, we know where the issue lies.

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] psycopg2 wheel packaging issues

[Stephen Finucane]

I imagine most people have been seeing warnings like the one below
raised by various openstack packages recently:

  .tox/py27/lib/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: 
The psycopg2
  wheel package will be renamed from release 2.8; in order to keep installing 
from binary
  please use "pip install psycopg2-binary" instead. For details see:
  .

Based on this warning, I had done what seemed to be the obvious thing
to do and proposed adding psycopg2-binary to the list of global
requirements [1]. This would allow us to replace all references to
psycopg2 with psycopg2-wheel in individual projects. However, upon
further investigation it seems this is not really an option since the
two packages exist in the same namespace and will clobber each other.
I've now abandoned this patch.

Does anyone with stronger Python packaging-fu than I have a better
solution for the psycopg2 folks? There's a detailed description of why
this was necessary on GitHub [2] along with some potential resolutions,
none of which seem to be acceptable. If nothing better is possible, it
seems we'll simply have to live with (or silence) these warnings in
psycopg2 2.7.x and start installing libpg again once 2.8 is released.

Cheers,
Stephen

[1] https://review.openstack.org/#/c/561924/
[2] https://github.com/psycopg/psycopg2/issues/674

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][ptl][python3][help-wanted] starting work on "python 3 first" transition

[Stephen Finucane]

On Wed, 2018-06-06 at 16:04 -0400, Doug Hellmann wrote:
> I have started submitting a series of patches to fix up the tox.ini
> settings for projects as a step towards running "python3 first"
> [1]. The point of doing this now is to give teams a head start on
> understanding the work involved as we consider whether to make this
> a community goal.
> 
> The current patches are all mechanically generated changes to the
> basepython value for environments that seem to be likely candidates.
> They're basically the "easy" part of the transition. I've left any
> changes that will need more discussion alone for now.
> 
> In particular, I've skipped over any tox environments with "functional"
> in the name, since I thought those ran functional tests. Teams will
> need to decide whether to change those job definitions, or duplicate
> them and run them under python 2 and 3. Since we are not dropping
> python 2 support until the U cycle, I suggest going ahead and running
> the jobs twice.
> 
> Note that changing the tox settings won't actually change some of the
> jobs. For example, with our current PTI definition, the documentation
> and releasenotes jobs do not run under tox. That means those will need
> to be changed by editing the zuul configuration for the repository.
> 
> I have started to make notes for tracking the work in
> https://etherpad.openstack.org/p/python3-first -- including some notes
> about taking the next step to update the zuul job definitions and common
> issues we've already encountered to help folks debug job failures.
> 
> I could use some help keeping an eye on these changes and getting
> them through the gate. If you are interested in helping, please
> leave a comment on the review you are willing to shepherd.
> 
> Doug

I banged through nearly everything I could +2/+W this morning (mostly
stuff under the oslo or nova teams' remit). I saw very few issues and
no failing CIs so it was mostly a case of sanity checking and
approving.

If anyone is seeing issues with their "docs" target, feel free to give
me a shout (IRC: stephenfin) as we had to work through similar issues
in nova not too long ago. It's amazing where monkey patching will get
you :)

Stephen

> [1] 
> https://review.openstack.org/#/q/topic:python3-first+(status:open+OR+status:merged)
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Thu, 2018-04-26 at 15:49 +0100, Stephen Finucane wrote:
> On Wed, 2018-04-25 at 12:06 -0500, Sean McGinnis wrote:
> > > > > 
> > > > > [1] https://review.openstack.org/#/c/564232/
> > > > > 
> > > > 
> > > > The only concern I have is that it may slow the transition to the
> > > > python 3 version of the jobs, since someone would have to actually
> > > > fix the warnings before they could add the new job. I'm not sure I
> > > > want to couple the tasks of fixing doc build warnings with also
> > > > making those docs build under python 3 (which is usually quite
> > > > simple).
> > > > 
> > > > Is there some other way to enable this flag independently of the move to
> > > > the python3 job?
> > > 
> > > The existing proposal is:
> > > 
> > > https://review.openstack.org/559348
> > > 
> > > TL;DR if you still have a build_sphinx section in setup.cfg then defaults
> > > will remain the same, but when removing it as part of the transition to 
> > > the
> > > new PTI you'll have to eliminate any warnings. (Although AFAICT it doesn't
> > > hurt to leave that section in place as long as you need, and you can still
> > > do the rest of the PTI conversion.)
> > > 
> > > The hold-up is that the job in question is also potentially used by other
> > > Zuul users outside of OpenStack - including those who aren't using pbr at
> > > all (i.e. there's no setup.cfg). So we need to warn those folks to 
> > > prepare.
> > > 
> > > cheers,
> > > Zane.
> > > 
> > 
> > Ah, I had looked but did not find an existing proposal. Looks like that 
> > would
> > work too. I am good either way, but I will leave my approach out there just 
> > as
> > another option to consider. I'll abandon that if folks prefer this way.
> 
> Yeah, I reviewed your patch but assumed you'd seen mine already and
> were looking for a quicker alternative.
> 
> I've started the process of adding this to zuul-jobs by posting the
> warning to zuul-announce (though it's waiting moderation by corvus). We
>  only need to wait two weeks after sending that message before we can
> merge the patch to zuul-jobs, so I guess we should go that way now?
> 
> Stephen

This is now merged:

  https://review.openstack.org/#/c/559348/

For anyone that has migrated to the new PTI for documentation (namely,
not using 'python setup.py build_sphinx'), you may find your docs
builds are now broken. This will likely be a result of regressions
introduced since the migration to the new PTI and should be trivial to
resolve. If you have any issues, feel free to jump on #openstack-doc
where one of the lovely docs people will be happy to help.

Cheers,
Stephen

PS: I do realize this has likely incurred some extra work for some
projects. No one likes busy work and I'm sorry to have forced that upon
people but this change was essential to ensure every project could rely
on the gate to actually validate their documentation.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Updated PTI for documentation

[Stephen Finucane]

There have been a couple of threads about an updated "PTI" for
documentation bouncing around the mailing list of late.

 * http://lists.openstack.org/pipermail/openstack-dev/2018-March/128817
   .html
 * http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594
   .html

I've been told the reasoning behind this change and what is required
has not been made clear so here goes my attempt at explaining it. In
short, there are two problems we're trying to work around with this
change.

 * The legacy 'build_sphinx' setuptools command provided by pbr has
   been found to be lacking. It's buggy as hell, frequently breaks with
   Sphinx version bumps, and is generally a PITA to maintain. We (the
   oslo team) want to remove this feature to ease our maintenance
   burden.
 * The recent move to zuul v3 has changed how documentation is built in
   the gate. Previously, zuul called the 'docs' target in tox (e.g.
   'tox -e docs'), which would run whatever the project team had
   defined for that target. With zuul v3, zuul no longer calls this.
   Instead, it call either 'python setup.py build_sphinx' or 'sphinx-
   build' (more on this below). This means everything you wish to do as
   part of the documentation build must now be done via Sphinx
   extensions.

Both the oslo and infra teams have a strong incentive to drop support
for the 'build_sphinx' command (albeit for different reasons) but doing
so isn't simply a case of calling 'sphinx-build' instead. In order to
migrate, some steps are required:

   1. pbr's 'build_sphinx' setuptools command provides some additional
  functionality on top of 'sphinx-build'. This must be replaced by
  Sphinx extensions.
   2. Calls to 'python setup.py build_sphinx' must be replaced by
  additional calls to 'sphinx-build'
   3. Documentation requirements must be moved to 'doc/requirements.txt'
  to avoid having to install every requirement of a project simply to
  build documentation.

The first of these has already be achieved: 'openstackdocstheme'
recently gained support for automatically configuring the project name
and version in generated documentation [1], which replaced that aspect
of the 'build_sphinx' command. Similarly, the 'sphinxcontrib-apidoc'
Sphinx extension [2] was produced in order to provide a way to
automatically generate API documentation as part of 'sphinx-build'
rather than by having to make a secondary call to 'sphinx-apidoc'
(which the gate, which, once again, no longer runs anything but
'sphinx-build' or 'python setup.py build_sphinx', would not do).

The second step is the troublesome bit and has been the reason for most
of the individual patches to various projects. The steps necessary to
make this change have been documented multiple times on the list but
they're listed here once again for posterity:

 * If necessary, enable 'sphinxcontrib.apidoc' as described at [3].
 * Make sure you're using 'openstackdocstheme', assuming your project
   is an official OpenStack one.
 * Remove the 'build_sphinx' section from 'setup.cfg' (this is
   described at [3] but applies whether you need that or not).
 * Update your doc/releasenotes/api-guide targets in 'tox.ini' so
   you're using the same commands as the gate.

The third change should be self-explanatory and infra have reasons for
requesting it. It's generally easiest to do this as part of the above.

Hopefully this clears things up for people. If anyone has any
questions, feel free to reach out to me on IRC (stephenfin) and I'll be
happy to help.

Cheers,
Stephen

PS: For those that curious, the decision on whether to run 'python
setup.py build_sphinx' command or 'sphinx-build' in the gate is based
on the presence of a 'build_sphinx' section in 'setup.cfg'. If present,
the former is run. If not, we use 'sphinx-build'. This is why it's
necessary to remove that section from 'setup.cfg'.

[1] https://docs.openstack.org/openstackdocstheme/latest/#using-the-theme
[2] https://pypi.org/project/sphinxcontrib-apidoc/
[3] https://pypi.org/project/sphinxcontrib-apidoc/#migration-from-pbr

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cyborg] [Nova] Backup plan without nested RPs

[Stephen Finucane]

On Mon, 2018-06-04 at 10:49 -0700, Nadathur, Sundar wrote:
> Hi,
> 
>  Cyborg needs to create RCs and traits for accelerators. The
> original plan was to do that with nested RPs. To avoid rushing
> the
> Nova developers, I had proposed that Cyborg could start by
> applying
> the traits to the compute node RP, and accept the resulting
> caveats
> for Rocky, till we get nested RP support. That proposal did not
> find
> many takers, and Cyborg has essentially been in waiting mode.
> 
> 
> 
> Since it is June already, and there is a risk of not delivering
> anything meaningful in Rocky, I am reviving my older proposal,
> which
> is summarized as below:
> 
> 
>   Cyborg shall create the RCs and traits as per spec
> (https://review.openstack.org/#/c/554717/), both in Rocky and
> beyond. Only the RPs will change post-Rocky.
> 
>   
>   In Rocky:
>   
> Cyborg will not create nested RPs. It shall apply the device
>   traits to the compute node RP.
> Cyborg will document the resulting caveat, i.e., all devices
>   in the same host should have the same traits. In
> particular,
>   we cannot have a GPU and a FPGA, or 2 FPGAs of different
>   types, in the same host.
> Cyborg will document that upgrades to post-Rocky releases
>   will require operator intervention (as described below).
> 
> 
>   
>For upgrade to post-Rocky world with nested RPs:
>   
> The operator needs to stop all running instances that use an
>   accelerator.
> The operator needs to run a script that removes the Cyborg
>   traits and the inventory for Cyborg RCs from compute node
> RPs.
> The operator can then perform the upgrade. The new Cyborg
>   agent/driver(s) shall created nested RPs and publish
>   inventory/traits as specified.
>   
> 
> IMHO, it is acceptable for Cyborg to do this because it is new
>   and we can set expectations for the (lack of) upgrade plan. The
>   alternative is that potentially no meaningful use cases get
>   addressed in Rocky for Cyborg. 
> 
> 
> 
> Please LMK what you think.

I thought nested resource providers were already supported by
placement? To the best of my knowledge, what is not supported is virt
drivers using these to report NUMA topologies but I doubt that affects
you. The placement guys will need to weigh in on this as I could be
missing something but it sounds like you can start using this
functionality right now.

Stephen

> 
> 
> Regards,
> 
> Sundar
> 
>   
> 
> _
> _OpenStack Development Mailing List (not for usage
> questions)Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack
> .org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Wed, 2018-04-25 at 12:06 -0500, Sean McGinnis wrote:
> > > > 
> > > > [1] https://review.openstack.org/#/c/564232/
> > > > 
> > > 
> > > The only concern I have is that it may slow the transition to the
> > > python 3 version of the jobs, since someone would have to actually
> > > fix the warnings before they could add the new job. I'm not sure I
> > > want to couple the tasks of fixing doc build warnings with also
> > > making those docs build under python 3 (which is usually quite
> > > simple).
> > > 
> > > Is there some other way to enable this flag independently of the move to
> > > the python3 job?
> > 
> > The existing proposal is:
> > 
> > https://review.openstack.org/559348
> > 
> > TL;DR if you still have a build_sphinx section in setup.cfg then defaults
> > will remain the same, but when removing it as part of the transition to the
> > new PTI you'll have to eliminate any warnings. (Although AFAICT it doesn't
> > hurt to leave that section in place as long as you need, and you can still
> > do the rest of the PTI conversion.)
> > 
> > The hold-up is that the job in question is also potentially used by other
> > Zuul users outside of OpenStack - including those who aren't using pbr at
> > all (i.e. there's no setup.cfg). So we need to warn those folks to prepare.
> > 
> > cheers,
> > Zane.
> > 
> 
> Ah, I had looked but did not find an existing proposal. Looks like that would
> work too. I am good either way, but I will leave my approach out there just as
> another option to consider. I'll abandon that if folks prefer this way.

Yeah, I reviewed your patch but assumed you'd seen mine already and
were looking for a quicker alternative.

I've started the process of adding this to zuul-jobs by posting the
warning to zuul-announce (though it's waiting moderation by corvus). We
 only need to wait two weeks after sending that message before we can
merge the patch to zuul-jobs, so I guess we should go that way now?

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-vif]

[Stephen Finucane]

On Tue, 2018-01-09 at 16:30 +0530, Sriharsha Basavapatna wrote:
> On Tue, Jan 9, 2018 at 2:20 PM, Sriharsha Basavapatna
>  wrote:
> > Hi Andreas,
> > 
> > On Tue, Jan 9, 2018 at 12:04 PM, Andreas Jaeger 
> > wrote:
> > > On 2018-01-09 07:00, Sriharsha Basavapatna wrote:
> > > > Hi,
> > > > 
> > > > I've uploaded a patch for review:
> > > > https://review.openstack.org/#/c/531674/
> > > > 
> > > > This is the first time I'm submitting a patch on openstack. I'd
> > > > like
> > > 
> > > Welcome to OpenStack, Harsha.
> > 
> > Thank you.
> > 
> > > Please read
> > > https://docs.openstack.org/infra/manual/developers.html if you
> > > haven't.
> > 
> > Ok, i'll read it.
> > > 
> > > I see that your change fails the basic tests, you can run these
> > > locally
> > > as follows to check that your fixes will pass:
> > > 
> > > tox -e pep8
> > > tox -e py27
> > 
> > I was wondering if there's a way to catch these errors without
> > having
> > to submit it for gerrit review.  I fixed the ones that were
> > reported
> > in patch-set-1; looks like there's some new ones in the second
> > patch-set. I'll run the above commands to verify the fix locally.
> > 
> > Thanks,
> > -Harsha
> 
> I installed python-pip and tox.  But when I run "tox -e pep8", I'm
> seeing some errors:
> 
> building 'netifaces' extension
> gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
> --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
> -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall
> -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
> --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
> -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DNETIFACES_VERSION=0.10.6
> -DHAVE_GETIFADDRS=1 -DHAVE_GETNAMEINFO=1 -DHAVE_NETASH_ASH_H=1
> -DHAVE_NETATALK_AT_H=1 -DHAVE_NETAX25_AX25_H=1
> -DHAVE_NETECONET_EC_H=1
> -DHAVE_NETIPX_IPX_H=1 -DHAVE_NETPACKET_PACKET_H=1
> -DHAVE_LINUX_IRDA_H=1 -DHAVE_LINUX_ATM_H=1 -DHAVE_LINUX_LLC_H=1
> -DHAVE_LINUX_TIPC_H=1 -DHAVE_LINUX_DN_H=1 -DHAVE_SOCKADDR_AT=1
> -DHAVE_SOCKADDR_AX25=1 -DHAVE_SOCKADDR_IN=1 -DHAVE_SOCKADDR_IN6=1
> -DHAVE_SOCKADDR_IPX=1 -DHAVE_SOCKADDR_UN=1 -DHAVE_SOCKADDR_ASH=1
> -DHAVE_SOCKADDR_EC=1 -DHAVE_SOCKADDR_LL=1 -DHAVE_SOCKADDR_ATMPVC=1
> -DHAVE_SOCKADDR_ATMSVC=1 -DHAVE_SOCKADDR_DN=1 -DHAVE_SOCKADDR_IRDA=1
> -DHAVE_SOCKADDR_LLC=1 -DHAVE_PF_NETLINK=1 -I/usr/include/python2.7 -c
> netifaces.c -o build/temp.linux-x86_64-2.7/netifaces.o
> netifaces.c:1:20: fatal error: Python.h: No such file or
> directory
>  #include 
> ^
> compilation terminated.
> error: command 'gcc' failed with exit status 1
> 
> 
> Command "/home/harshab/os-vif/.tox/pep8/bin/python2 -u -c "import
> setuptools, tokenize;__file__='/tmp/pip-build-
> OibnHO/netifaces/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install
> --record /tmp/pip-3Hu__1-record/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /home/harshab/os-vif/.tox/pep8/include/site/python2.7/netifaces"
> failed with error code 1 in /tmp/pip-build-OibnHO/netifaces/
> 
> ERROR: could not install deps
> [-r/home/harshab/os-vif/requirements.txt,
> -r/home/harshab/os-vif/test-requirements.txt]; v =
> InvocationError('/home/harshab/os-vif/.tox/pep8/bin/pip install -U
> -r/home/harshab/os-vif/requirements.txt
> -r/home/harshab/os-vif/test-requirements.txt (see
> /home/harshab/os-vif/.tox/pep8/log/pep8-1.log)', 1)
> ___ summary
> 
> ERROR:   pep8: could not install deps
> [-r/home/harshab/os-vif/requirements.txt,
> -r/home/harshab/os-vif/test-requirements.txt]; v =
> InvocationError('/home/harshab/os-vif/.tox/pep8/bin/pip install -U
> -r/home/harshab/os-vif/requirements.txt
> -r/home/harshab/os-vif/test-requirements.txt (see
> /home/harshab/os-vif/.tox/pep8/log/pep8-1.log)', 1)
> 
> Thanks,
> -Harsha

That's happening because the 'pep8' target is installing all the
requirements for the project in a virtualenv, and one of them needs
Python development headers. What Linux distro are you using? On Fedora
you can fix this like so:

sudo dnf install python-devel

On Ubuntu, I think it's something like this:

sudo apt-get install python-dev

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [os-api-ref][doc] Adding openstack-doc-core to os-api-ref

[Stephen Finucane]

I'm not sure what the procedure for this is but here goes.

I've noticed that the 'os-api-ref' project seems to have its own group
of cores [1], many of whom are no longer working on OpenStack (at
least, not full-time), and has a handful of open patches against it
[2]. Since the doc team has recently changed its scope from writing
documentation to enabling individual projects to maintain their own
docs, we've become mainly responsible for projects like 'openstack-doc-
theme'. Given that the 'os-api-ref' project is a Sphinx thing required
for multiple OpenStack projects, it seems like something that
could/should fall into the doc team's remit.

I'd like to move this project into the remit of the 'openstack-doc-
core' team, by way of removing the 'os-api-ref-core' group or adding
'openstack-doc-core' to the list of included groups. In both cases,
existing active cores will be retained. Do any of the existing 'os-api-
ref' cores have any objections to this?

Stephen

PS: I'm not sure how this affects things from a release management
perspective. Are there PTLs for these sorts of projects?

[1] https://review.openstack.org/#/admin/groups/1391,members
[2] https://review.openstack.org/#/q/project:openstack/os-api-ref+statu
s:open

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [os-api-ref][doc] Adding openstack-doc-core to os-api-ref

[Stephen Finucane]

On Thu, 2018-01-04 at 15:39 +, Stephen Finucane wrote:
> I'm not sure what the procedure for this is but here goes.
> 
> I've noticed that the 'os-api-ref' project seems to have its own group
> of cores [1], many of whom are no longer working on OpenStack (at
> least, not full-time), and has a handful of open patches against it
> [2]. Since the doc team has recently changed its scope from writing
> documentation to enabling individual projects to maintain their own
> docs, we've become mainly responsible for projects like 'openstack-doc-
> theme'. Given that the 'os-api-ref' project is a Sphinx thing required
> for multiple OpenStack projects, it seems like something that
> could/should fall into the doc team's remit.
> 
> I'd like to move this project into the remit of the 'openstack-doc-
> core' team, by way of removing the 'os-api-ref-core' group or adding
> 'openstack-doc-core' to the list of included groups. In both cases,
> existing active cores will be retained. Do any of the existing 'os-api-
> ref' cores have any objections to this?

It's been two weeks with no -1s, so we've gone ahead and done this.
Thanks, everyone.

Stephen

> Stephen
> 
> PS: I'm not sure how this affects things from a release management
> perspective. Are there PTLs for these sorts of projects?

Turns out this project was already listed as a doc team deliverable.
Who knew?!

> [1] https://review.openstack.org/#/admin/groups/1391,members
> [2] https://review.openstack.org/#/q/project:openstack/os-api-ref+statu
> s:open

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [pbr] support v_version

[Stephen Finucane]

On Tue, 2018-01-09 at 10:25 +0100, Gaetan wrote:
> Hello
> 
> I have submitted this patch ([1]) that add support for v_version in
> PBR. Basically I can tag v1.0.0 instead of 1.0.0 to release 1.0.0.
> 
> However, after rework it appears PBR does not behaves well, even if
> the unit tests pass:
> On tag for instance v1.0.0, the result packages in named
> `-1.0.0.dev1`.
> 
> Do you know where I need to hack PBR to fix it?

So 'pbr' correctly parses the prefixed tags, but it's just the output
packages (sdists, wheels) that always unversioned? If so, this sounds
correct. Python packaging, as defined in PEP-440 [1], doesn't use the
'v' prefixes, so it doesn't really make sense to stick them in here.
Out of curiosity, what's your rationale for modifying the package name?

> Second point, to go to the end of the logic of my change, I would
> like to propose an optional way (in setup.cfg?) to **prevent** any
> tag without the 'v' prefix, ie, where a bare version tag like `1.0.0`
> is not to be considered as a valid version.
> That way, on system such as gitlab or github:
> - repository owners "protect" tags with pattern "v*", ie, all tags
> for release such as "v1.0.0", ... cannot be pushed by anyone but the
> owners/masters
> - other developer can still push other tags for other purpose

So this could be used to prevent pbr reading the tags, but it won't
stop anyone from creating them in the first place (i.e. "protect"
tags). We can do this but it would be a separate feature and, to be
honest, I'd suggest using Git hooks or some form of access control as a
better way to do this (Note: it seems GitLab already supports something
similar [2]).

Stephen

[1] https://www.python.org/dev/peps/pep-0440/
[2] https://gitlab.com/gitlab-org/gitlab-ce/issues/18471

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Adding Takashi Natsume to python-novaclient core

[Stephen Finucane]

On Fri, 2018-02-09 at 09:01 -0600, Matt Riedemann wrote:
> I'd like to add Takashi to the python-novaclient core team.
> 
> python-novaclient doesn't get a ton of activity or review, but Takashi 
> has been a solid reviewer and contributor to that project for quite 
> awhile now:
> 
> http://stackalytics.com/report/contribution/python-novaclient/180
> 
> He's always fast to get new changes up for microversion support and help 
> review others that are there to keep moving changes forward.
> 
> So unless there are objections, I'll plan on adding Takashi to the 
> python-novaclient-core group next week.

Easy +1 from me. Would be good to have him on the team.

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] [nova] [os-vif] [vif_plug_ovs] Support for OVS DB tcp socket communication.

[Stephen Finucane]

On Wed, 2018-07-25 at 15:22 +0530, pranab boruah wrote:
> Hello folks,
> I have filed a bug in os-vif: 
> https://bugs.launchpad.net/os-vif/+bug/1778724 and working on a
> patch. Any feedback/comments from you guys would be extremely
> helpful. 
> Bug details:
> OVS DB server has the feature of listening over a TCP socket for
> connections rather than just on the unix domain socket. [0]
> 
> If the OVS DB server is listening over a TCP socket, then the ovs-
> vsctl commands should include the ovsdb_connection parameter:
> # ovs-vsctl --db=tcp:IP:PORT ...
> eg:
> # ovs-vsctl --db=tcp:169.254.1.1:6640 add-port br-int eth0
> Neutron supports running the ovs-vsctl commands with the
> ovsdb_connection parameter. The ovsdb_connection parameter is
> configured in openvswitch_agent.ini file. [1]
> While adding a vif to the ovs bridge(br-int), Nova(os-vif) invokes
> the ovs-vsctl command. Today, there is no support to pass the
> ovsdb_connection parameter while invoking the ovs-vsctl command. The
> support should be added. This would enhance the functionality of os-
> vif, since it would support a scenario when OVS DB server is
> listening on a TCP socket connection and on functional parity with
> Neutron.
> [0] http://www.openvswitch.org/support/dist-docs/ovsdb-server.1.html
> [1] 
> https://docs.openstack.org/neutron/pike/configuration/openvswitch-agent.html
> 
> 
> TIA,Pranab

Perhaps not the same thing, but would the patches mentioned in the
below mail work for this too?

  
http://lists.openstack.org/pipermail/openstack-dev/2018-March/127907.html

Cheers,
Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Setting-up NoVNC 1.0.0 with nova

[Stephen Finucane]

On Sun, 2018-05-20 at 09:33 -0700, Matt Riedemann wrote:
> On 5/20/2018 6:37 AM, Thomas Goirand wrote:
> > The novnc package in Debian and Ubuntu is getting very old. So I thought
> > about upgrading to 1.0.0, which has lots of very nice newer features,
> > like the full screen mode, and so on.
> > 
> > All seemed to work, however, when trying to connect to the console of a
> > VM, NoVNC attempts to connect tohttps://example.com:6080/websockify  and
> > then fails (with a 404).
> > 
> > So I was wondering: what's missing in my setup so that there's a
> > /websockify URL? Is there some missing code in the nova-novncproxy so
> > that it would forward this URL to /usr/bin/websockify? If so, has anyone
> > started working on it?
> > 
> > Also, what's the status of NoVNC with Python 3? I saw lots of print
> > statements which are easy to fix, though I even wonder if the code in
> > the python-novnc package is useful. Who's using it? Nova-novncproxy?
> > That's unlikely, since I didn't package a Python 3 version for it.
> 
> Stephen Finucane (stephenfin on irc) would know best at this point, but 
> I know he ran into some issues with configuring nova when using novnc 
> 1.0.0, so check your novncproxy_base_url config option value:
> 
> https://docs.openstack.org/nova/latest/configuration/config.html#vnc.novncproxy_base_url
> 
> Specifically:
> 
> "If using noVNC >= 1.0.0, you should use vnc_lite.html instead of 
> vnc_auto.html."

We've got a patch up to resolve this in DevStack [1]. As Matt notes,
the issue is because a path was renamed in noVNC 1.0.0. You could
resolve this by including a symlink to the path in your package but it
might be better long-term to simply ensure the deployment tools take
care of this. We can eventually change the default in nova once noVNC
1.0.0 gains enough momentum. There's a WIP patch up for this too [2].

Let me know if you need more info,
Stephen

[1] https://review.openstack.org/#/c/550172/6
[2] https://review.openstack.org/#/c/550173/4


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] proposing Moisés Guimarães for oslo.config core

[Stephen Finucane]

On Wed, 2018-08-01 at 09:27 -0400, Doug Hellmann wrote:
> Moisés Guimarães (moguimar) did quite a bit of work on oslo.config
> during the Rocky cycle to add driver support. Based on that work,
> and a discussion we have had since then about general cleanup needed
> in oslo.config, I think he would make a good addition to the
> oslo.config review team.
> 
> Please indicate your approval or concerns with +1/-1.
> 
> Doug

+1. The more the merrier.

Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Paste unmaintained

[Stephen Finucane]

tl;dr: It seems Paste [1] may be entering unmaintained territory and we
may need to do something about it.

I was cleaning up some warning messages that nova was issuing this
morning and noticed a few coming from Paste. I was going to draft a PR
to fix this, but a quick browse through the Bitbucket project [2]
suggests there has been little to no activity on that for well over a
year. One particular open PR - "Python 3.7 support" - is particularly
concerning, given the recent mailing list threads on the matter.

Given that multiple projects are using this, we may want to think about
reaching out to the author and seeing if there's anything we can do to
at least keep this maintained going forward. I've talked to cdent about
this already but if anyone else has ideas, please let me know.

Stephen

[1] https://pypi.org/project/Paste/
[2] https://bitbucket.org/ianb/paste/
[3] https://bitbucket.org/ianb/paste/pull-requests/41


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] numa aware vswitch

[Stephen Finucane]

On Fri, 2018-08-24 at 07:55 +, Guo, Ruijing wrote:
> Hi, All,
>  
> I am verifying numa aware vwitch features 
> (https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-vswitches.html).
>  But the result is not my expectation.
>  
> What I missing?
>  
>  
> Nova configuration:
>  
> [filter_scheduler]
> track_instance_changes = False
> enabled_filters = 
> RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,NUMATopologyFilter
>  
> [neutron]
> physnets = physnet0,physnet1
>  
> [neutron_physnet_physnet0]
> numa_nodes = 0
>  
> [neutron_physnet_physnet1]
> numa_nodes = 1
>  
>  
> ml2 configuration:
>  
> [ml2_type_vlan]
> network_vlan_ranges = physnet0,physnet1
> [ovs]
> vhostuser_socket_dir = /var/lib/libvirt/qemu
> bridge_mappings = physnet0:br-physnet0,physnet1:br-physnet1
>  
>  
> command list:
>  
> openstack network create net0 --external --provider-network-type=vlan 
> --provider-physical-network=physnet0 --provider-segment=100
> openstack network create net1 --external --provider-network-type=vlan 
> --provider-physical-network=physnet1 --provider-segment=200
> openstack subnet create --network=net0 --subnet-range=192.168.1.0/24 
> --allocation-pool start=192.168.1.200,end=192.168.1.250 --gateway 192.168.1.1 
> subnet0
> openstack subnet create --network=net1 --subnet-range=192.168.2.0/24 
> --allocation-pool start=192.168.2.200,end=192.168.2.250 --gateway 192.168.2.1 
> subnet1
> openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic 
> net-id=net0 vm0
> openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic 
> net-id=net1 vm1
>  
> vm0 and vm1 are created but numa is not enabled:
>   1
>   
> 1024
>   
 
Using this won't add a NUMA topology - it'll just control how any
topology present will be mapped to the guest. You need to enable
dedicated CPUs or a explicitly request a NUMA topology for this to
work.

openstack flavor set --property hw:numa_nodes=1 1



openstack flavor set --property hw:cpu_policy=dedicated 1


This is perhaps something that we could change in the future, though I
haven't given it much thought yet.

Regards,
Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] numa aware vswitch

[Stephen Finucane]

On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote:
> On 8/24/2018 8:58 AM, Stephen Finucane wrote:
> > Using this won't add a NUMA topology - it'll just control how any
> > topology present will be mapped to the guest. You need to enable
> > dedicated CPUs or a explicitly request a NUMA topology for this to
> > work.
> > 
> > openstack flavor set --property hw:numa_nodes=1 1
> > 
> > 
> > 
> > openstack flavor set --property hw:cpu_policy=dedicated 1
> > 
> > 
> > This is perhaps something that we could change in the future, though I
> > haven't given it much thought yet.
> 
> Looks like the admin guide [1] should be updated to at least refer to 
> the flavor user guide on setting up these types of flavors?
> 
> [1] https://docs.openstack.org/nova/latest/admin/networking.html#numa-affinity

Good idea.

https://review.openstack.org/596393

Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [placement] extraction (technical) update

[Stephen Finucane]

On Mon, 2018-08-27 at 14:23 -0700, melanie witt wrote:
> On Mon, 27 Aug 2018 10:31:50 -0500, Matt Riedemann wrote:
> > On 8/24/2018 7:36 AM, Chris Dent wrote:
> > > 
> > > Over the past few days a few of us have been experimenting with
> > > extracting placement to its own repo, as has been discussed at
> > > length on this list, and in some etherpads:
> > > 
> > > https://etherpad.openstack.org/p/placement-extract-stein
> > > https://etherpad.openstack.org/p/placement-extraction-file-notes
> > > 
> > > As part of that, I've been doing some exploration to tease out the
> > > issues we're going to hit as we do it. None of this is work that
> > > will be merged, rather it is stuff to figure out what we need to
> > > know to do the eventual merging correctly and efficiently.
> > > 
> > > Please note that doing that is just the near edge of a large
> > > collection of changes that will cascade in many ways to many
> > > projects, tools, distros, etc. The people doing this are aware of
> > > that, and the relative simplicity (and fairly immediate success) of
> > > these experiments is not misleading people into thinking "hey, no
> > > big deal". It's a big deal.
> > > 
> > > There's a strategy now (described at the end of the first etherpad
> > > listed above) for trimming the nova history to create a thing which
> > > is placement. From the first run of that Ed created a github repo
> > > and I branched that to eventually create:
> > > 
> > > https://github.com/EdLeafe/placement/pull/2
> > > 
> > > In that, all the placement unit and functional tests are now
> > > passing, and my placecat [1] integration suite also passes.
> > > 
> > > That work has highlighted some gaps in the process for trimming
> > > history which will be refined to create another interim repo. We'll
> > > repeat this until the process is smooth, eventually resulting in an
> > > openstack/placement.
> > 
> > We talked about the github strategy a bit in the placement meeting today
> > [1]. Without being involved in this technical extraction work for the
> > past few weeks, I came in with a different perspective on the end-game,
> > and it was not aligned with what Chris/Ed thought as far as how we get
> > to the official openstack/placement repo.
> > 
> > At a high level, Ed's repo [2] is a fork of nova with large changes on
> > top using pull requests to do things like remove the non-placement nova
> > files, update import paths (because the import structure changes from
> > nova.api.openstack.placement to just placement), and then changes from
> > Chris [3] to get tests working. Then the idea was to just use that to
> > seed the openstack/placement repo and rather than review the changes
> > along the way*, people that care about what changed (like myself) would
> > see the tests passing and be happy enough.
> > 
> > However, I disagree with this approach since it bypasses our community
> > code review system of using Gerrit and relying on a core team to approve
> > changes at the sake of expediency.
> > 
> > What I would like to see are the changes that go into making the seed
> > repo and what gets it to passing tests done in gerrit like we do for
> > everything else. There are a couple of options on how this is done though:
> > 
> > 1. Seed the openstack/placement repo with the filter_git_history.sh
> > script output as Ed has done here [4]. This would include moving the
> > placement files to the root of the tree and dropping nova-specific
> > files. Then make incremental changes in gerrit like with [5] and the
> > individual changes which make up Chris's big pull request [3]. I am
> > primarily interested in making sure there are not content changes
> > happening, only mechanical tree-restructuring type changes, stuff like
> > that. I'm asking for more changes in gerrit so they can be sanely
> > reviewed (per normal).
> > 
> > 2. Eric took a slightly different tack in that he's OK with just a
> > couple of large changes (or even large patch sets within a single
> > change) in gerrit rather than ~30 individual changes. So that would be
> > more like at most 3 changes in gerrit for [4][5][3].
> > 
> > 3. The 3rd option is we just don't use gerrit at all and seed the
> > official repo with the results of Chris and Ed's work in Ed's repo in
> > github. Clearly this would be the fastest way to get us to a new repo
> > (at the expense of bucking community code review and development process
> > - is an exception worth it?).
> > 
> > Option 1 would clearly be a drain on at least 2 nova cores to go through
> > the changes. I think Eric is on board for reviewing options 1 or 2 in
> > either case, but he prefers option 2. Since I'm throwing a wrench in the
> > works, I also need to stand up and review the changes if we go with
> > option 1 or 2. Jay said he'd review them but consider these reviews
> > lower priority. I expect we could get some help from some other nova
> > cores though, maybe not on all changes, but at least some (thinking
> 

Re: [openstack-dev] [oslo] proposing Moisés Guimarães for oslo.config core

[Stephen Finucane]

On Mon, 2018-08-13 at 17:39 -0500, Ben Nemec wrote:
> 
> On 08/08/2018 08:18 AM, Doug Hellmann wrote:
> > Excerpts from Doug Hellmann's message of 2018-08-01 09:27:09 -0400:
> > > Moisés Guimarães (moguimar) did quite a bit of work on oslo.config
> > > during the Rocky cycle to add driver support. Based on that work,
> > > and a discussion we have had since then about general cleanup needed
> > > in oslo.config, I think he would make a good addition to the
> > > oslo.config review team.
> > > 
> > > Please indicate your approval or concerns with +1/-1.
> > > 
> > > Doug
> > 
> > Normally I would have added moguimar to the oslo-config-core team
> > today, after a week's wait. Funny story, though. There is no
> > oslo-config-core team.
> > 
> > oslo.config is one of a few of our libraries that we never set up with a
> > separate review team. It is managed by oslo-core. We could set up a new
> > review team for that library, but after giving it some thought I
> > realized that *most* of the libraries are fairly stable, our team is
> > pretty small, and Moisés is a good guy so maybe we don't need to worry
> > about that.
> > 
> > I spoke with Moisés, and he agreed to be part of the larger core team.
> > He pointed out that the next phase of the driver work is going to happen
> > in castellan, so it would be useful to have another reviewer there. And
> > I'm sure we can trust him to be careful with reviews in other repos
> > until he learns his way around.
> > 
> > So, I would like to amend my original proposal and suggest that we add
> > Moisés to the oslo-core team.
> > 
> > Please indicate support with +1 or present any concerns you have. I
> > apologize for the confusion on my part.
> 
> I'm good with this reasoning, so +1 from me.

As above. +1



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] numa aware vswitch

[Stephen Finucane]

On Mon, 2018-08-27 at 10:24 +0100, Sean Mooney wrote:
> 
> 
> On Mon 27 Aug 2018, 04:20 Guo, Ruijing,  wrote:
> > Hi, Stephen,
> > 
> > After setting flavor, VM was created in node 0 (expect in node1). How to 
> > debug it?
> > 
> > Nova.conf
> > [neutron]
> > physnets = physnet0,physnet1
> > 
> > [neutron_physnet_physnet1]
> > numa_nodes = 1
> 
> Have you enabled the numa topology filter its off by default and without it 
> the numa aware vswitch code is disabled.

Yeah, make sure this is enabled. You should turn on debug-level logging
as this will give you additional information about how things are being
scheduled. Also, is this a new deployment? If not, you're going to need
to upgrade and restart all the nova-* services since there are object
changes which will need to be propagated.

Stephen

> > openstack network create net1 --external --provider-network-type=vlan 
> > --provider-physical-network=physnet1 --provider-segment=200
> > ...
> > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk --nic 
> > net-id=net1 vm1
> > 
> > 
> > 1024
> > 
> > 
> >   
> > 
> > available: 2 nodes (0-1)
> > node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
> > node 0 size: 64412 MB
> > node 0 free: 47658 MB
> > node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
> > node 1 size: 64502 MB
> > node 1 free: 44945 MB
> > node distances:
> > node   0   1 
> >   0:  10  21 
> >   1:  21  10
> > 
> > Thanks,
> > -Ruijing
> > 
> > -Original Message-
> > From: Stephen Finucane [mailto:sfinu...@redhat.com] 
> > Sent: Saturday, August 25, 2018 12:15 AM
> > To: OpenStack Development Mailing List (not for usage questions) 
> > 
> > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch
> > 
> > On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote:
> > > On 8/24/2018 8:58 AM, Stephen Finucane wrote:
> > > > Using this won't add a NUMA topology - it'll just control how any 
> > > > topology present will be mapped to the guest. You need to enable 
> > > > dedicated CPUs or a explicitly request a NUMA topology for this to 
> > > > work.
> > > > 
> > > > openstack flavor set --property hw:numa_nodes=1 1
> > > > 
> > > > 
> > > > 
> > > > openstack flavor set --property hw:cpu_policy=dedicated 1
> > > > 
> > > > 
> > > > This is perhaps something that we could change in the future, though 
> > > > I haven't given it much thought yet.
> > > 
> > > Looks like the admin guide [1] should be updated to at least refer to 
> > > the flavor user guide on setting up these types of flavors?
> > > 
> > > [1] 
> > > https://docs.openstack.org/nova/latest/admin/networking.html#numa-affi
> > > nity
> > 
> > Good idea.
> > 
> > https://review.openstack.org/596393
> > 
> > Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Stephen Finucane]

On Sat, 2018-08-25 at 08:08 +0800, Alex Xu wrote:
> 
> 
> 2018-08-18 20:25 GMT+08:00 Chris Dent :
> > On Fri, 17 Aug 2018, Doug Hellmann wrote:
> > 
> > > If we ignore the political concerns in the short term, are there
> > > other projects actually interested in using placement? With what
> > > technical caveats? Perhaps with modifications of some sort to
> > > support
> > > the needs of those projects?
> > > 
> >  
> > I think ignoring the political concerns (in any term) is not
> > possible. We are a group of interacting humans, politics are always
> > present. Cordial but active debate to determine the best course of
> > action is warranted.
> > 
> > (tl;dr: Let's have existing and potential placement contributors
> > decide its destiny.)
> > 
> > Five topics I think are relevant here, in order of politics, least
> > to most:
> > 
> > 1. Placement has been designed from the outset to have a hard
> > contract between it and the services that use it. Being embedded
> > and/or deeply associated with one other single service means that
> > that contract evolves in a way that is strongly coupled. We made
> > placement have an HTTP API, not use RPC, and not produce or consume
> > notifications because it is supposed to be bounded and independent.
> > Sharing code and human management doesn't enable that. As you'll
> > read below, placement's progress has been overly constrained by
> > compute.
> > 
> > 2. There are other projects actively using placement, not merely
> > interested. If you search codesearch.o.o for terms like "resource
> > provider" you can find them. But to rattle off those that I'm aware
> > of (which I'm certain is an incomplete list):
> > 
> > * Cyborg is actively working on using placement to track FPGA
> >   e.g., https://review.openstack.org/#/c/577438/
> > 
> > * Blazar is working on using them for reservations:
> >   
> > https://review.openstack.org/#/q/status:open+project:openstack/blazar+branch:master+topic:bp/placement-api
> > 
> > * Neutron has been reporting to placement for some time and has
> > work
> >   in progress on minimum bandwidth handling with the help of
> >   placement:
> >   
> > https://review.openstack.org/#/q/status:open+project:openstack/neutron-lib+branch:master+topic:minimum-bandwidth-allocation-placement-api
> > 
> > * Ironic uses resource classes to describe types of nodes
> > 
> > * Mogan (which may or may not be dead, not clear) was intending to
> >   track nodes with placement:
> >   
> > http://git.openstack.org/cgit/openstack/mogan-specs/tree/specs/pike/approved/track-resources-using-placement.rst
> > 
> > * Zun is working to use placement for "unified resource
> > management":
> >   
> > https://blueprints.launchpad.net/zun/+spec/use-placement-resource-management
> > 
> > * Cinder has had discussion about using placement to overcome race
> >   conditions in its existing scheduling subsystem (a purpose to
> >   which placement was explicitly designed).
> > 
> > 3. Placement's direction and progress is heavily curtailed by the
> > choices and priorities that compute wants or needs to make. That
> > means that for the past year or more much of the effort in
> > placement
> > has been devoted to eventually satisfying NFV use cases driven by
> > "enhanced platform awareness" to the detriment of the simple use
> > case of "get me some resource providers". Compute is under a lot of
> > pressure in this area, and is under-resourced, so placement's
> > progress is delayed by being in the (necessarily) narrow engine of
> > compute. Similarly, computes's overall progress is delayed because
> > a
> > lot of attention is devoted to placement.
> > 
> > I think the relevance of that latter point has been under-estimated
> > by the voices that are hoping to keep placement near to nova. The
> > concern there has been that we need to continue iterating in
> > concert
> > and quickly. I disagree with that from two angles. One is that we
> > _will_ continue to work in concert. We are OpenStack, and
> > presumably
> > all the same people working on placement now will continue to do
> > so,
> > and many of those are active contributors to nova. We will work
> > together.
> > 
> > The other angle is that, actually, placement is several months
> > ahead
> > of nova in terms of features and it would be to everyone's
> > advantage if
> > placement, from a feature standpoint, took a time out (to extract)
> > while nova had a chance to catch up with fully implementing shared
> > providers, nested resource providers, consumer generations,
> > resource
> > request groups, using the reshaper properly from the virt drivers,
> > having a fast forward upgrade script talking to PlacementDirect,
> > and
> > other things that I'm not remembering right now. The placement side
> > for those things is in place. The work that it needs now is a
> > _diversity_ of callers (not just nova) so that the features can
> > been
> > fully exercised and bugs and performance problems found.
> > 
> > 

Re: [openstack-dev] [all] [nova] [placement] placement below or beside compute after extraction?

[Stephen Finucane]

On Sat, 2018-08-25 at 08:08 +0800, Alex Xu wrote:
> 
> 
> 2018-08-18 20:25 GMT+08:00 Chris Dent :
> > On Fri, 17 Aug 2018, Doug Hellmann wrote:
> > 
> > > If we ignore the political concerns in the short term, are there
> > > other projects actually interested in using placement? With what
> > > technical caveats? Perhaps with modifications of some sort to
> > > support
> > > the needs of those projects?
> > > 
> >  
> > I think ignoring the political concerns (in any term) is not
> > possible. We are a group of interacting humans, politics are always
> > present. Cordial but active debate to determine the best course of
> > action is warranted.
> > 
> > (tl;dr: Let's have existing and potential placement contributors
> > decide its destiny.)
> > 
> > Five topics I think are relevant here, in order of politics, least
> > to most:
> > 
> > 1. Placement has been designed from the outset to have a hard
> > contract between it and the services that use it. Being embedded
> > and/or deeply associated with one other single service means that
> > that contract evolves in a way that is strongly coupled. We made
> > placement have an HTTP API, not use RPC, and not produce or consume
> > notifications because it is supposed to be bounded and independent.
> > Sharing code and human management doesn't enable that. As you'll
> > read below, placement's progress has been overly constrained by
> > compute.
> > 
> > 2. There are other projects actively using placement, not merely
> > interested. If you search codesearch.o.o for terms like "resource
> > provider" you can find them. But to rattle off those that I'm aware
> > of (which I'm certain is an incomplete list):
> > 
> > * Cyborg is actively working on using placement to track FPGA
> >   e.g., https://review.openstack.org/#/c/577438/
> > 
> > * Blazar is working on using them for reservations:
> >   
> > https://review.openstack.org/#/q/status:open+project:openstack/blazar+branch:master+topic:bp/placement-api
> > 
> > * Neutron has been reporting to placement for some time and has
> > work
> >   in progress on minimum bandwidth handling with the help of
> >   placement:
> >   
> > https://review.openstack.org/#/q/status:open+project:openstack/neutron-lib+branch:master+topic:minimum-bandwidth-allocation-placement-api
> > 
> > * Ironic uses resource classes to describe types of nodes
> > 
> > * Mogan (which may or may not be dead, not clear) was intending to
> >   track nodes with placement:
> >   
> > http://git.openstack.org/cgit/openstack/mogan-specs/tree/specs/pike/approved/track-resources-using-placement.rst
> > 
> > * Zun is working to use placement for "unified resource
> > management":
> >   
> > https://blueprints.launchpad.net/zun/+spec/use-placement-resource-management
> > 
> > * Cinder has had discussion about using placement to overcome race
> >   conditions in its existing scheduling subsystem (a purpose to
> >   which placement was explicitly designed).
> > 
> > 3. Placement's direction and progress is heavily curtailed by the
> > choices and priorities that compute wants or needs to make. That
> > means that for the past year or more much of the effort in
> > placement
> > has been devoted to eventually satisfying NFV use cases driven by
> > "enhanced platform awareness" to the detriment of the simple use
> > case of "get me some resource providers". Compute is under a lot of
> > pressure in this area, and is under-resourced, so placement's
> > progress is delayed by being in the (necessarily) narrow engine of
> > compute. Similarly, computes's overall progress is delayed because
> > a
> > lot of attention is devoted to placement.
> > 
> > I think the relevance of that latter point has been under-estimated
> > by the voices that are hoping to keep placement near to nova. The
> > concern there has been that we need to continue iterating in
> > concert
> > and quickly. I disagree with that from two angles. One is that we
> > _will_ continue to work in concert. We are OpenStack, and
> > presumably
> > all the same people working on placement now will continue to do
> > so,
> > and many of those are active contributors to nova. We will work
> > together.
> > 
> > The other angle is that, actually, placement is several months
> > ahead
> > of nova in terms of features and it would be to everyone's
> > advantage if
> > placement, from a feature standpoint, took a time out (to extract)
> > while nova had a chance to catch up with fully implementing shared
> > providers, nested resource providers, consumer generations,
> > resource
> > request groups, using the reshaper properly from the virt drivers,
> > having a fast forward upgrade script talking to PlacementDirect,
> > and
> > other things that I'm not remembering right now. The placement side
> > for those things is in place. The work that it needs now is a
> > _diversity_ of callers (not just nova) so that the features can
> > been
> > fully exercised and bugs and performance problems found.
> > 
> > 

Re: [openstack-dev] [nova] Reminder on nova-network API removal work

[Stephen Finucane]

On Fri, 2018-07-13 at 15:23 -0500, Matt Riedemann wrote:
> There are currently no open changes for the nova-network API removal 
> tracked here [1] but there are at least two low-hanging fruit APIs to 
> remove:
> 
> * os-floating-ips-bulk
> * os-floating-ips-dns

The two of these are done now.

 * https://review.openstack.org/582912
 * https://review.openstack.org/582943

There's a tempest test that needs to be disabled/removed for the first
one and I'm waiting on CI results for the second.

Looking at the other ones to see if I can figure out if they can/should
be removed.

Stephen

> It would be nice to at least get those removed yet before the feature 
> freeze. See one of the existing linked removal patches in the etherpad 
> for an example of how to do this, and/or read the doc [2].
> 
> [1] https://etherpad.openstack.org/p/nova-network-removal-rocky
> [2] 
> https://docs.openstack.org/nova/latest/contributor/api.html#removing-deprecated-apis
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [Openstack-sigs] [all] Bringing the community together (combine the lists!)

[Stephen Finucane]

On Fri, 2018-08-31 at 10:03 +1000, Tony Breeds wrote:
> On Thu, Aug 30, 2018 at 09:12:57PM +, Jeremy Stanley wrote:
> > On 2018-08-31 01:13:58 +0800 (+0800), Rico Lin wrote:
> > [...]
> > > What needs to be done for this is full topic categories support
> > > under `options` page so people get to filter emails properly.
> > 
> > [...]
> > 
> > Unfortunately, topic filtering is one of the MM2 features the
> > Mailman community decided nobody used (or at least not enough to
> > warrant preserving it in MM3). I do think we need to be consistent
> > about tagging subjects to make client-side filtering more effective
> > for people who want that, but if we _do_ want to be able to upgrade
> > we shouldn't continue to rely on server-side filtering support in
> > Mailman unless we can somehow work with them to help in
> > reimplementing it.
> 
> The suggestion is to implement it as a 3rd party plugin or work with the
> mm community to implement:
> https://wiki.mailman.psf.io/DEV/Dynamic%20Sublists
> 
> So if we decide we really want that in mm3 we have options.
> 
> Yours Tony.

I've tinked with mailman 3 before so I could probably take a shot at
this over the next few week(end)s; however, I've no idea how this
feature is supposed to work. Any chance an admin of the current list
could send me a couple of screenshots of the feature in mailman 2 along
with a brief description of the feature? Alternatively, maybe we could
upload them to the wiki page Tony linked above or, better yet, to the
technical details page for same:

  https://wiki.mailman.psf.io/DEV/Brief%20Technical%20Details

Cheers,
Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron] numa aware vswitch

[Stephen Finucane]

On Thu, 2018-08-30 at 00:15 +, Guo, Ruijing wrote:
> Hi, Stephen, Sean
> 
> It worked as expected.

Good to hear. What were you missing? If there are other gaps in the
documentation, I'd be happy to fix them.

Stephen

> Thanks,
> -Ruijing
> 
> -Original Message-
> From: Stephen Finucane [mailto:sfinu...@redhat.com] 
> Sent: Monday, August 27, 2018 5:37 PM
> To: OpenStack Development Mailing List (not for usage questions) 
> 
> Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch
> 
> On Mon, 2018-08-27 at 10:24 +0100, Sean Mooney wrote:
> > 
> > 
> > On Mon 27 Aug 2018, 04:20 Guo, Ruijing,  wrote:
> > > Hi, Stephen,
> > > 
> > > After setting flavor, VM was created in node 0 (expect in node1). How to 
> > > debug it?
> > > 
> > > Nova.conf
> > > [neutron]
> > > physnets = physnet0,physnet1
> > > 
> > > [neutron_physnet_physnet1]
> > > numa_nodes = 1
> > 
> > Have you enabled the numa topology filter its off by default and without it 
> > the numa aware vswitch code is disabled.
> 
> Yeah, make sure this is enabled. You should turn on debug-level logging as 
> this will give you additional information about how things are being 
> scheduled. Also, is this a new deployment? If not, you're going to need to 
> upgrade and restart all the nova-* services since there are object changes 
> which will need to be propagated.
> 
> Stephen
> 
> > > openstack network create net1 --external 
> > > --provider-network-type=vlan --provider-physical-network=physnet1 
> > > --provider-segment=200 ...
> > > openstack server create --flavor 1 --image=cirros-0.3.5-x86_64-disk 
> > > --nic net-id=net1 vm1
> > > 
> > > 
> > > 1024
> > > 
> > > 
> > >   
> > > 
> > > available: 2 nodes (0-1)
> > > node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 
> > > 64412 MB node 0 free: 47658 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 
> > > 25 26 27 28 29 30 31 node 1 size: 64502 MB node 1 free: 44945 MB 
> > > node distances:
> > > node   0   1 
> > >   0:  10  21 
> > >   1:  21  10
> > > 
> > > Thanks,
> > > -Ruijing
> > > 
> > > -Original Message-
> > > From: Stephen Finucane [mailto:sfinu...@redhat.com]
> > > Sent: Saturday, August 25, 2018 12:15 AM
> > > To: OpenStack Development Mailing List (not for usage questions) 
> > > 
> > > Subject: Re: [openstack-dev] [nova][neutron] numa aware vswitch
> > > 
> > > On Fri, 2018-08-24 at 09:13 -0500, Matt Riedemann wrote:
> > > > On 8/24/2018 8:58 AM, Stephen Finucane wrote:
> > > > > Using this won't add a NUMA topology - it'll just control how 
> > > > > any topology present will be mapped to the guest. You need to 
> > > > > enable dedicated CPUs or a explicitly request a NUMA topology 
> > > > > for this to work.
> > > > > 
> > > > > openstack flavor set --property hw:numa_nodes=1 1
> > > > > 
> > > > > 
> > > > > 
> > > > > openstack flavor set --property hw:cpu_policy=dedicated 1
> > > > > 
> > > > > 
> > > > > This is perhaps something that we could change in the future, 
> > > > > though I haven't given it much thought yet.
> > > > 
> > > > Looks like the admin guide [1] should be updated to at least refer 
> > > > to the flavor user guide on setting up these types of flavors?
> > > > 
> > > > [1]
> > > > https://docs.openstack.org/nova/latest/admin/networking.html#numa-
> > > > affi
> > > > nity
> > > 
> > > Good idea.
> > > 
> > > https://review.openstack.org/596393
> > > 
> > > Stephen
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Replacing pbr's autodoc feature with sphinxcontrib-apidoc

[Stephen Finucane]

On Mon, 2018-04-02 at 19:41 -0400, Zane Bitter wrote:
> On 28/03/18 10:31, Stephen Finucane wrote:
> > As noted last week [1], we're trying to move away from pbr's autodoc
> > feature as part of the new docs PTI. To that end, I've created
> > sphinxcontrib-apidoc, which should do what pbr was previously doing for
> > us by via a Sphinx extension.
> > 
> >https://pypi.org/project/sphinxcontrib-apidoc/
> > 
> > This works by reading some configuration from your documentation's
> > 'conf.py' file and using this to call 'sphinx-apidoc'. It means we no
> > longer need pbr to do this for.
> > 
> > I have pushed version 0.1.0 to PyPi already but before I add this to
> > global requirements, I'd like to ensure things are working as expected.
> > smcginnis was kind enough to test this out on glance and it seemed to
> > work for him but I'd appreciate additional data points. The
> > configuration steps for this extension are provided in the above link.
> > To test this yourself, you simply need to do the following:
> > 
> > 1. Add 'sphinxcontrib-apidoc' to your test-requirements.txt or
> >doc/requirements.txt file
> > 2. Configure as noted above and remove the '[pbr]' and '[build_sphinx]'
> >configuration from 'setup.cfg'
> > 3. Replace 'python setup.py build_sphinx' with a call to 'sphinx-build'
> > 4. Run 'tox -e docs'
> > 5. Profit?
> > 
> > Be sure to let me know if anyone encounters issues. If not, I'll be
> > pushing for this to be included in global requirements so we can start
> > the migration.
> 
> Thanks Stephen! I tried it out with no problems:
> 
> https://review.openstack.org/558262
> 
> However, there are a couple of differences compared to how pbr did things.
> 
> 1) pbr can generate an 'autoindex' file with a flat list of modules 
> (this appears to be configurable with the autodoc_index_modules option), 
> but apidoc only generates a 'modules' file with a hierarchical list of 
> modules. This is easy to work around, but I guess it needs to be added 
> to the instructions to check that you're not relying on it.

Yup, smcginnis and I discussed this at some point. PBR has two
different ways of generating API documentation: 'autodoc_tree', which
is based on 'sphinx-apidoc', and 'autodoc', which is custom (and
presumably legacy). This extension replaces the former of those but, as
you note below, it seems 'sphinx-apidoc' can be wrangled into
generating something approaching the latter.

> 2) pbr generates a page per module; this plugin generates a page per 
> package. This results in wy too much information on a page to be 
> able to navigate it comfortably IMHO. To the point where it's easier to 
> read the code. (It also breaks existing links, if you care about that 
> kind of thing.) I sent you a PR to add an option to pass --separate:
> 
> https://github.com/sphinx-contrib/apidoc/pull/1

Thanks for that. I've merged it and will use it as the basis of a 0.2.0
release assuming nothing else pops up in the next day or two. I'm not
sure what we can do about the broken links though - maybe use the
redirect infrastructure to just send everyone to the new place? I guess
I can add this to the guide I'm adding to the README on migrating from
pbr.

Cheers,
Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Replacing pbr's autodoc feature with sphinxcontrib-apidoc

[Stephen Finucane]

On Wed, 2018-03-28 at 15:31 +0100, Stephen Finucane wrote:
> As noted last week [1], we're trying to move away from pbr's autodoc
> feature as part of the new docs PTI. To that end, I've created
> sphinxcontrib-apidoc, which should do what pbr was previously doing for
> us by via a Sphinx extension.
> 
>   https://pypi.org/project/sphinxcontrib-apidoc/
> 
> This works by reading some configuration from your documentation's
> 'conf.py' file and using this to call 'sphinx-apidoc'. It means we no
> longer need pbr to do this for.
> 
> I have pushed version 0.1.0 to PyPi already but before I add this to
> global requirements, I'd like to ensure things are working as expected.
> smcginnis was kind enough to test this out on glance and it seemed to
> work for him but I'd appreciate additional data points. The
> configuration steps for this extension are provided in the above link.
> To test this yourself, you simply need to do the following:
> 
>1. Add 'sphinxcontrib-apidoc' to your test-requirements.txt or
>   doc/requirements.txt file
>2. Configure as noted above and remove the '[pbr]' and '[build_sphinx]'
>   configuration from 'setup.cfg'
>3. Replace 'python setup.py build_sphinx' with a call to 'sphinx-build'
>4. Run 'tox -e docs'
>5. Profit?
> 
> Be sure to let me know if anyone encounters issues. If not, I'll be
> pushing for this to be included in global requirements so we can start
> the migration.
> 
> Cheers,
> Stephen
> 
> [1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html

'sphinxcontrib.apidoc' has now been added to requirements [1]. The
README [2] provides a far more detailed overview of how one can migrate
from the pbr features than I gave above and I'd advise anyone making
changes to their documentation to follow that guide. Feel free to ping
me here or on IRC (stephenfin) if you've any questions.

Next up: deprecating this feature in pbr.

Stephen

[1] https://review.openstack.org/#/c/557532/
[2] https://github.com/sphinx-contrib/apidoc#migration-from-pbr

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Thu, 2018-04-05 at 16:36 -0400, Zane Bitter wrote:
> On 21/03/18 06:49, Stephen Finucane wrote:
> > As noted by Monty in a prior openstack-dev post [2], some projects rely
> > on a pbr extension to the 'build_sphinx' setuptools command which can
> > automatically run the 'sphinx-apidoc' tool before building docs. This
> > is enabled by configuring some settings in the '[pbr]' section of the
> > 'setup.cfg' file [3]. To ensure this continued working, the zuul jobs
> > definitions [4] check for the presence of these settings and build docs
> > using the legacy 'build_sphinx' command if found. **At no point do the
> > jobs call the tox job**. As a result, if you convert a project to use
> > 'sphinx-build' in 'tox.ini' without resolving the autodoc issues, you
> > lose the ability to build docs locally.
> > 
> > I've gone through and proposed a couple of reverts to fix projects
> > we've already broken. However, going forward, there are two things
> > people should do to prevent issues like this popping up.
> > 
> >   * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections
> > from 'setup.cfg' in any patches that aim to convert a project to use
> > the new PTI. This will ensure the gate catches any potential
> > issues.
> 
> How can we enable warning_is_error in the gate with the new PTI? It's 
> easy enough to add the -W flag in tox.ini for local builds, but as you 
> say the tox job is never called in the gate. In the gate zuul checks for 
> it in the [build_sphinx] section of setup.cfg:
> 
> https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.py#n23
> 
> So I think it makes more sense to remove the [pbr] section, but leave 
> the [build_sphinx] section?
> 
> thanks,
> Zane.

I'd be more in favour of changing the zuul job to build with the '-W'
flag. To be honest, there is no good reason to not have this flag
enabled. I'm not sure that will be a popular opinion though as it may
break some projects' builds (correctly, but still).

I'll propose a patch against zuul-jobs and see what happens :)

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Fri, 2018-04-06 at 08:02 -0500, Sean McGinnis wrote:
> > > 
> > > How can we enable warning_is_error in the gate with the new PTI? It's 
> > > easy enough to add the -W flag in tox.ini for local builds, but as you 
> > > say the tox job is never called in the gate. In the gate zuul checks for 
> > > it in the [build_sphinx] section of setup.cfg:
> > > 
> > > https://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/sphinx/library/sphinx_check_warning_is_error.pyLovel#n23
> > > 
> > > [...]
> > 
> > I'd be more in favour of changing the zuul job to build with the '-W'
> > flag. To be honest, there is no good reason to not have this flag
> > enabled. I'm not sure that will be a popular opinion though as it may
> > break some projects' builds (correctly, but still).
> > 
> > I'll propose a patch against zuul-jobs and see what happens :)
> > 
> > Stephen
> > 
> 
> I am in favor of this too. We will probably need to give some teams some time
> to get warnings fixed though. I haven't done any kind of extensive audit of
> projects, but from a few I looked through, there are definitely a few that are
> not erroring on warnings and are likely to be blocked if we suddenly flipped
> the switch and errored on those.
>
> This is a legitimate issue though. In Cinder we had -W in the tox docs job, 
> but
> since that is no longer being enforced in the gate, running "tox -e docs" from
> a fresh clone of master was failing. We really do need some way to enforce 
> this
> so things like that do not happen.

This. While forcing work on teams to do busywork is undeniably A Very
Bad Thing (TM), I do think the longer we leave this, the worse it'll
get. The zuul-jobs [1] patch will probably introduce some pain for
projects but it seems like inevitable pain and we're in the right part
of the cycle in which to do something like this. I'd be willing to help
projects fix issues they encounter, which I expect will be minimal for
most projects.

Stephen

[1] https://review.openstack.org/559348

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Proposing Eric Fried for nova-core

[Stephen Finucane]

+1

On Mon, 2018-03-26 at 19:00 -0700, melanie witt wrote:
> Howdy everyone,
> 
> I'd like to propose that we add Eric Fried to the nova-core team.
> 
> Eric has been instrumental to the placement effort with his work on 
> nested resource providers and has been actively contributing to many 
> other areas of openstack [0] like project-config, gerritbot, 
> keystoneauth, devstack, os-loganalyze, and so on.
> 
> He's an active reviewer in nova [1] and elsewhere in openstack and 
> reviews in-depth, asking questions and catching issues in patches and 
> working with authors to help get code into merge-ready state. These are 
> qualities I look for in a potential core reviewer.
> 
> In addition to all that, Eric is an active participant in the project in 
> general, helping people with questions in the #openstack-nova IRC 
> channel, contributing to design discussions, helping to write up 
> outcomes of discussions, reporting bugs, fixing bugs, and writing tests. 
> His contributions help to maintain and increase the health of our project.
> 
> To the existing core team members, please respond with your comments, 
> +1s, or objections within one week.
> 
> Cheers,
> -melanie
> 
> [0] https://review.openstack.org/#/q/owner:efried
> [1] http://stackalytics.com/report/contribution/nova/90
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Wed, 2018-03-28 at 14:14 -0500, Sean McGinnis wrote:
> On Thu, Mar 22, 2018 at 10:43:45AM +0000, Stephen Finucane wrote:
> > On Wed, 2018-03-21 at 09:57 -0500, Sean McGinnis wrote:
> > > On Wed, Mar 21, 2018 at 10:49:02AM +, Stephen Finucane wrote:
> > > > tl;dr: Make sure you stop using pbr's autodoc feature before converting
> > > > them to the new PTI for docs.
> > > > 
> > > > [snip]
> > > > 
> > 
> > That's unfortunate. What we really need is a migration path from the
> > 'pbr' way of doing things to something else. I see three possible
> > avenues at this point in time:
> > 
> >1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar
> >   things to 'sphinx-apidoc' but it takes the form of an extension.
> >   From my brief experiments, the output generated from this is
> >   radically different and far less comprehensive than what 'sphinx-
> >   apidoc' generates. However, it supports templating so we could
> >   probably configure this somehow and add our own special directive
> >   somewhere like 'openstackdocstheme'
> >2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back
> >   against upstream Sphinx [1]. This essentially does what the PBR
> >   extension does but moves configuration into 'conf.py'. However, this
> >   is currently held up as I can't adequately explain the differences
> >   between this and 'sphinx.ext.autosummary' (there's definite overlap
> >   but I don't understand 'autosummary' well enough to compare them).
> >3. Modify the upstream jobs that detect the pbr integration and have
> >   them run 'sphinx-apidoc' before 'sphinx-build'. This is the least
> >   technically appealing approach as it still leaves us unable to build
> >   stuff locally and adds yet more "magic" to the gate, but it does let
> >   us progress.
> > 
> 
> It's not mentioned here, but I discovered today that Cinder is using the
> sphinx.ext.autodoc module. Is there any issue with using this?
> 

Nope - sphinx-apidoc and the likes use autodoc under the hood. You can
see this by checking the output in 'contributor/api' or the likes.

Stephen

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Replacing pbr's autodoc feature with sphinxcontrib-apidoc

[Stephen Finucane]

On Tue, 2018-04-03 at 12:04 -0400, Zane Bitter wrote:
> On 03/04/18 06:28, Stephen Finucane wrote:
> > On Mon, 2018-04-02 at 19:41 -0400, Zane Bitter wrote:
> > > On 28/03/18 10:31, Stephen Finucane wrote:
> > > > As noted last week [1], we're trying to move away from pbr's autodoc
> > > > feature as part of the new docs PTI. To that end, I've created
> > > > sphinxcontrib-apidoc, which should do what pbr was previously doing for
> > > > us by via a Sphinx extension.
> > > > 
> > > > https://pypi.org/project/sphinxcontrib-apidoc/
> > > > 
> > > > This works by reading some configuration from your documentation's
> > > > 'conf.py' file and using this to call 'sphinx-apidoc'. It means we no
> > > > longer need pbr to do this for.
> > > > 
> > > > I have pushed version 0.1.0 to PyPi already but before I add this to
> > > > global requirements, I'd like to ensure things are working as expected.
> > > > smcginnis was kind enough to test this out on glance and it seemed to
> > > > work for him but I'd appreciate additional data points. The
> > > > configuration steps for this extension are provided in the above link.
> > > > To test this yourself, you simply need to do the following:
> > > > 
> > > >  1. Add 'sphinxcontrib-apidoc' to your test-requirements.txt or
> > > > doc/requirements.txt file
> > > >  2. Configure as noted above and remove the '[pbr]' and 
> > > > '[build_sphinx]'
> > > > configuration from 'setup.cfg'
> > > >  3. Replace 'python setup.py build_sphinx' with a call to 
> > > > 'sphinx-build'
> > > >  4. Run 'tox -e docs'
> > > >  5. Profit?
> > > > 
> > > > Be sure to let me know if anyone encounters issues. If not, I'll be
> > > > pushing for this to be included in global requirements so we can start
> > > > the migration.
> > > 
> > > Thanks Stephen! I tried it out with no problems:
> > > 
> > > https://review.openstack.org/558262
> > > 
> > > However, there are a couple of differences compared to how pbr did things.
> > > 
> > > 1) pbr can generate an 'autoindex' file with a flat list of modules
> > > (this appears to be configurable with the autodoc_index_modules option),
> > > but apidoc only generates a 'modules' file with a hierarchical list of
> > > modules. This is easy to work around, but I guess it needs to be added
> > > to the instructions to check that you're not relying on it.
> > 
> > Yup, smcginnis and I discussed this at some point. PBR has two
> > different ways of generating API documentation: 'autodoc_tree', which
> > is based on 'sphinx-apidoc', and 'autodoc', which is custom (and
> > presumably legacy). This extension replaces the former of those but, as
> > you note below, it seems 'sphinx-apidoc' can be wrangled into
> > generating something approaching the latter.
> 
> That explains quite a lot that was confusing me :D
> 
> > > 2) pbr generates a page per module; this plugin generates a page per
> > > package. This results in wy too much information on a page to be
> > > able to navigate it comfortably IMHO. To the point where it's easier to
> > > read the code. (It also breaks existing links, if you care about that
> > > kind of thing.) I sent you a PR to add an option to pass --separate:
> > > 
> > > https://github.com/sphinx-contrib/apidoc/pull/1
> > 
> > Thanks for that. I've merged it and will use it as the basis of a 0.2.0
> > release assuming nothing else pops up in the next day or two.
> 
> Thanks!
> 
> > I'm not sure what we can do about the broken links though - maybe use the
> > redirect infrastructure to just send everyone to the new place? I guess
> > I can add this to the guide I'm adding to the README on migrating from
> > pbr.
> 
> No links break if you use the apidoc_separate_modules=True option, so I 
> would recommend any projects currently generating a page per module 
> (i.e. using 'autodoc' instead of 'autodoc_tree') should enable that 
> option to keep continuity.

Fancy taking a look at [1], in that case? This should clarify
everything.

[1] https://github.com/sphinx-contrib/apidoc/pull/3

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Replacing pbr's autodoc feature with sphinxcontrib-apidoc

[Stephen Finucane]

As noted last week [1], we're trying to move away from pbr's autodoc
feature as part of the new docs PTI. To that end, I've created
sphinxcontrib-apidoc, which should do what pbr was previously doing for
us by via a Sphinx extension.

  https://pypi.org/project/sphinxcontrib-apidoc/

This works by reading some configuration from your documentation's
'conf.py' file and using this to call 'sphinx-apidoc'. It means we no
longer need pbr to do this for.

I have pushed version 0.1.0 to PyPi already but before I add this to
global requirements, I'd like to ensure things are working as expected.
smcginnis was kind enough to test this out on glance and it seemed to
work for him but I'd appreciate additional data points. The
configuration steps for this extension are provided in the above link.
To test this yourself, you simply need to do the following:

   1. Add 'sphinxcontrib-apidoc' to your test-requirements.txt or
  doc/requirements.txt file
   2. Configure as noted above and remove the '[pbr]' and '[build_sphinx]'
  configuration from 'setup.cfg'
   3. Replace 'python setup.py build_sphinx' with a call to 'sphinx-build'
   4. Run 'tox -e docs'
   5. Profit?

Be sure to let me know if anyone encounters issues. If not, I'll be
pushing for this to be included in global requirements so we can start
the migration.

Cheers,
Stephen

[1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128594.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Thu, 2018-03-29 at 07:47 -0500, Sean McGinnis wrote:
> > > 
> > > It's not mentioned here, but I discovered today that Cinder is using the
> > > sphinx.ext.autodoc module. Is there any issue with using this?
> > > 
> > 
> > Nope - sphinx-apidoc and the likes use autodoc under the hood. You can
> > see this by checking the output in 'contributor/api' or the likes.
> > 
> > Stephen
> > 
> 
> I'm wondering if there is a problem with using this vs. the way being 
> proposed.
> 
> In other words, do we need to switch over to this new sphinxcontrib module, or
> staying with autodoc should be OK. And if so, why not switch current users of
> the pbr method over to use sphinx.ext.autdoc rather than introducing something
> new?

tl;dr: You don't _have_ to automate this stuff, but it helps.

sphinx-apidoc generates stub files containing a whole load of autodoc
directives. As noted above, you can check the output of a sphinx-apidoc 
run and you'll see just this. If I were to guess, Cinder simply checked
in the output of such a run [*] into Git, meaning they don't need to
run it each time. This works but it comes with the downside that your
docs can and will get out of sync with the actual code when, for
example, you either add or remove some modules or functions. Running
sphinx-apidoc on each build, as we've been doing with pbr's autodoc
feature, ensures this out-of-sync issue doesn't happen, at the expense
of increased doc build times.

Stephen

[*] They might also have handwritten this stuff, but I highly doubt
that (it's rather tedious to write).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Wed, 2018-03-21 at 09:57 -0500, Sean McGinnis wrote:
> On Wed, Mar 21, 2018 at 10:49:02AM +0000, Stephen Finucane wrote:
> > tl;dr: Make sure you stop using pbr's autodoc feature before converting
> > them to the new PTI for docs.
> > 
> > [snip]
> > 
> > I've gone through and proposed a couple of reverts to fix projects
> > we've already broken. However, going forward, there are two things
> > people should do to prevent issues like this popping up.
> 
> Unfortunately this will not work to just revert the changes. That may fix
> things locally, but they will not pass in gate by going back to the old way.
> 
> Any cases of this will have to actually be updated to not use the unsupported
> pieces you point out. But the doc builds will still need to be done the way
> they are now, as that is what the PTI requires at this point.

That's unfortunate. What we really need is a migration path from the
'pbr' way of doing things to something else. I see three possible
avenues at this point in time:

   1. Start using 'sphinx.ext.autosummary'. Apparently this can do similar
  things to 'sphinx-apidoc' but it takes the form of an extension.
  From my brief experiments, the output generated from this is
  radically different and far less comprehensive than what 'sphinx-
  apidoc' generates. However, it supports templating so we could
  probably configure this somehow and add our own special directive
  somewhere like 'openstackdocstheme'
   2. Push for the 'sphinx.ext.apidoc' extension I proposed some time back
  against upstream Sphinx [1]. This essentially does what the PBR
  extension does but moves configuration into 'conf.py'. However, this
  is currently held up as I can't adequately explain the differences
  between this and 'sphinx.ext.autosummary' (there's definite overlap
  but I don't understand 'autosummary' well enough to compare them).
   3. Modify the upstream jobs that detect the pbr integration and have
  them run 'sphinx-apidoc' before 'sphinx-build'. This is the least
  technically appealing approach as it still leaves us unable to build
  stuff locally and adds yet more "magic" to the gate, but it does let
  us progress.

Try as I may, I don't really have the bandwidth to work on this for
another few weeks so I'd appreciate help from anyone with sufficient
Sphinx-fu to come up with a long-term solution to this issue.

Cheers,
Stephen

[1] https://github.com/sphinx-doc/sphinx/pull/4101/files

> >  * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections
> >from 'setup.cfg' in any patches that aim to convert a project to use
> >the new PTI. This will ensure the gate catches any potential
> >issues. 
> >  * In addition, if your project uses the pbr autodoc feature, you
> >should either (a) remove these docs from your documentation tree or
> >(b) migrate to something else like the 'sphinx.ext.autosummary'
> >extension [5]. I aim to post instructions on the latter shortly.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

tl;dr: Make sure you stop using pbr's autodoc feature before converting
them to the new PTI for docs.

There have been a lot of patches converting projects to use the new
Project Testing Interface for building docs [1]. Generally, these make
the following changes:

   1. Move any requirements for building docs or release notes from 'test-
  requirements.txt' to 'doc/requirements.txt'
   2. Modify 'tox.ini' to call 'sphinx-build' instead of 'python setup.py
  build_sphinx'

Once done, the idea is that the gate will be able to start building
docs by calling the 'sphinx-build' executable directly instead of using
the 'build_sphinx' setuptools command. Unfortunately, this doesn't
always do what you think and has resulted in a few now-broken projects
(mostly oslo).

As noted by Monty in a prior openstack-dev post [2], some projects rely
on a pbr extension to the 'build_sphinx' setuptools command which can
automatically run the 'sphinx-apidoc' tool before building docs. This
is enabled by configuring some settings in the '[pbr]' section of the
'setup.cfg' file [3]. To ensure this continued working, the zuul jobs
definitions [4] check for the presence of these settings and build docs
using the legacy 'build_sphinx' command if found. **At no point do the
jobs call the tox job**. As a result, if you convert a project to use
'sphinx-build' in 'tox.ini' without resolving the autodoc issues, you
lose the ability to build docs locally.

I've gone through and proposed a couple of reverts to fix projects
we've already broken. However, going forward, there are two things
people should do to prevent issues like this popping up.

 * Firstly, you should remove the '[build_sphinx]' and '[pbr]' sections
   from 'setup.cfg' in any patches that aim to convert a project to use
   the new PTI. This will ensure the gate catches any potential
   issues. 
 * In addition, if your project uses the pbr autodoc feature, you
   should either (a) remove these docs from your documentation tree or
   (b) migrate to something else like the 'sphinx.ext.autosummary'
   extension [5]. I aim to post instructions on the latter shortly.

If anyone has any questions on the above, feel free to reply here or
contact me on IRC (stephenfin).

Cheers,
Stephen

[1] 
https://review.openstack.org/#/q/topic:updated-pti+(status:open+OR+status:merged)
[2] http://lists.openstack.org/pipermail/openstack-dev/2017-December/125710.html
[3] https://docs.openstack.org/pbr/latest/user/using.html#pbr-setup-cfg
[4] 
https://github.com/openstack-infra/zuul-jobs/blob/d75f5d2b/roles/sphinx/tasks/main.yaml
[5] http://www.sphinx-doc.org/en/stable/ext/autosummary.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Following the new PTI for document build, broken local builds

[Stephen Finucane]

On Fri, 2018-03-23 at 12:23 -0400, Doug Hellmann wrote:
> Excerpts from Monty Taylor's message of 2018-03-23 08:03:22 -0500:
> > On 03/22/2018 05:43 AM, Stephen Finucane wrote:
> > > 
> > > That's unfortunate. What we really need is a migration path from the
> > > 'pbr' way of doing things to something else. I see three possible
> > > avenues at this point in time:
> > > 
> > > 1. Start using 'sphinx.ext.autosummary'. Apparently this can do 
> > > similar
> > >things to 'sphinx-apidoc' but it takes the form of an extension.
> > >From my brief experiments, the output generated from this is
> > >radically different and far less comprehensive than what 'sphinx-
> > >apidoc' generates. However, it supports templating so we could
> > >probably configure this somehow and add our own special directive
> > >somewhere like 'openstackdocstheme'
> > > 2. Push for the 'sphinx.ext.apidoc' extension I proposed some time 
> > > back
> > >against upstream Sphinx [1]. This essentially does what the PBR
> > >extension does but moves configuration into 'conf.py'. However, 
> > > this
> > >is currently held up as I can't adequately explain the differences
> > >between this and 'sphinx.ext.autosummary' (there's definite overlap
> > >but I don't understand 'autosummary' well enough to compare them).
> > > 3. Modify the upstream jobs that detect the pbr integration and have
> > >them run 'sphinx-apidoc' before 'sphinx-build'. This is the least
> > >technically appealing approach as it still leaves us unable to 
> > > build
> > >stuff locally and adds yet more "magic" to the gate, but it does 
> > > let
> > >us progress.
> > 
> > I'd suggest a #4:
> > 
> > Take the sphinx.ext.apidoc extension and make it a standalone extension 
> > people can add to doc/requirements.txt and conf.py. That way we don't 
> > have to convince the sphinx folks to land it.
> > 
> > I'd been thinking for a while "we should just write a sphinx extension 
> > with the pbr logic in it" - but hadn't gotten around to doing anything 
> > about it. If you've already written that extension - I think we're in 
> > potentially great shape!
> 
> That also has the benefit that we don't have to wait for a new sphinx
> release to start using it.

I can do this. Where will it live? pbr? openstackdocstheme? Somewhere
else?

Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Shared envdir across tox environments

[Stephen Finucane]

Just a quick heads up that an upcoming change to nova's 'tox.ini' will
change the behaviour of multiple environments slightly.

  https://review.openstack.org/#/c/534382/9

With this change applied, tox will start sharing environment
directories (e.g. '.tox/py27') among environments with identical
requirements and Python versions. This will mean you won't need to
download dependencies for every environment, which should massively
reduce the amount of time taken to (re)initialize many environments and
 save a bit of disk space to boot. This shouldn't affect most people
but it could affect people that use some fancy tooling that depends on
these directories. If this _is_ going to affect you, be sure to make
your concerns known on the review sooner rather than later so we can
resolve said concerns.

Cheers,
Stephen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Sphinx testing fun

[Stephen Finucane]

On Thu, 2018-10-04 at 18:00 -0400, Doug Hellmann wrote:
> Stephen Finucane  writes:
> 
> > On Thu, 2018-10-04 at 07:21 -0400, Doug Hellmann wrote:
> > > Stephen Finucane  writes:
> 
> [snip]
> 
> > > > Anyway, we can go figure out what's changed here and handle it but this
> > > > is, at best, going to be a band aid. The fact is 'sphinx_testing' is
> > > > unmaintained and has been for some time now. The new hotness is
> > > > 'sphinx.testing' [3], which is provided (with zero documentation) as
> > > > part of Sphinx. Unfortunately, this uses pytest fixtures [4] which I'm
> > > > pretty sure Monty (and a few others?) are vehemently against using in
> > > > OpenStack. That leaves us with three options:
> > > > 
> > > >  * Take over 'sphinx_testing' and bring it up-to-date. Maintain
> > > >forever.
> > > >  * Start using 'sphinx.testing' and everything it comes with
> > > >  * Delete any tests that use 'sphinx_testing' and deal with the lack of
> > > >coverage
> > > 
> > > Could we change our tests to use pathlib to wrap app.outdir and get the
> > > same results as before?
> > 
> > That's what I've done [2], which is kind of based on how I fixed this
> > in Sphinx. However, this is at best a stopgap. The fact remains that
> > 'sphinx_testing' is dead and the large changes that Sphinx is
> > undergoing (2.0 will be Python 3 only, with multiple other fixes) will
> > make further breakages more likely. Unless we want a repeat of the Mox
> > situation, I do think we should start thinking about this sooner rather
> > than later.
> 
> Yeah, it sounds like we need to deal with the change.
> 
> It looks like only the os-api-ref repo uses sphinx-testing. How many
> tests are we talking about having to rewrite/update there?
> 
> Doug

That's good news. I'd expected other projects would use it but then
nothing I've worked on does (and that likely constitutes a large
percentage of Sphinx extensions in OpenStack). I see four failing tests
so I guess, if they break again, we can opt for option 3 above and deal
with it. I can't see os-api-ref changing too much in the future
(barring adding PDF support at some point).

Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra] Polygerrit (was: [oslo][taskflow] Thoughts on moving taskflow out of openstack/oslo)

[Stephen Finucane]

On Wed, 2018-10-10 at 13:35 -0500, Greg Hill wrote:
> > If it's just about preferring the pull request workflow versus the 
> > Gerrit rebase workflow, just say so. Same for just preferring the Github 
> > UI versus Gerrit's UI (which I agree is awful).
> 
> I mean, yes, I personally prefer the Github UI and workflow, but that
> was not a primary consideration. I got used to using gerrit well
> enough. It was mostly the  There's also a sense that if a project is
> in the Openstack umbrella, it's not useful outside Openstack, and
> Taskflow is designed to be a general purpose library. The hope is
> that just making it a regular open source project might attract more
> users and contributors. This may or may not bear out, but as it is,
> there's no real benefit to staying an openstack project on this front
> since nobody is actively working on it within the community.

As an aside, are there any plans to enable PolyGerrit [1] in the
OpenStack Gerrit instance? The Gerrit documentation lists the feature
as a beta [2], but I suspect that might be out-of-date given how long
it's been around and folks suggesting the opposite elsewhere [3].

Stephen

[1] https://www.youtube.com/watch?v=WsPhoPGUsss
[2] https://www.gerritcodereview.com/dev-polygerrit.html#
[3] https://gitenterprise.me/2018/04/23/gerrithub-adopts-100-polygerrit/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][taskflow] Thoughts on moving taskflow out of openstack/oslo

[Stephen Finucane]

On Wed, 2018-10-10 at 18:51 +, Jeremy Stanley wrote:
> On 2018-10-10 13:35:00 -0500 (-0500), Greg Hill wrote:
> [...]
> > We plan to still have a CI gatekeeper, probably Travis CI, to make sure PRs
> > past muster before being merged, so it's not like we're wanting to
> > circumvent good contribution practices by committing whatever to HEAD.
> 
> Travis CI has gained the ability to prevent you from merging changes
> which fail testing? Or do you mean something else when you refer to
> it as a "gatekeeper" here?

Yup but it's GitHub feature rather than specifically a Travis CI
feature.

  https://help.github.com/articles/about-required-status-checks/

Doesn't help the awful pull request workflow but that's neither here
nor there.

Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] shall we do a spec review day next tuesday oct 23?

[Stephen Finucane]

On Mon, 2018-10-15 at 10:07 -0700, melanie witt wrote:
> Hey all,
> 
> Milestone s-1 is coming up next week on Thursday Oct 25 [1] and I was 
> thinking it would be a good idea to have a spec review day next week on 
> Tuesday Oct 23 to spend some focus on spec reviews together.
> 
> Spec freeze is s-2 Jan 10, so the review day isn't related to any 
> deadlines, but would just be a way to organize and make sure we have 
> initial review on the specs that have been proposed so far.
> 
> How does Tuesday Oct 23 work for everyone? Let me know if another day 
> works better.
> 
> So far, efried and mriedem are on board when I asked in the 
> #openstack-nova channel. I'm sending this mail to gather more responses 
> asynchronously.
> 
> Cheers,
> -melanie
> 
> [1] https://wiki.openstack.org/wiki/Nova/Stein_Release_Schedule

Good by me.

Stephen


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Sphinx testing fun

[Stephen Finucane]

On Thu, 2018-10-04 at 07:21 -0400, Doug Hellmann wrote:
> Stephen Finucane  writes:
> 
> > The tests in os-api-ref recently broke:
> > 
> >   
> > http://logs.openstack.org/62/606362/1/check/openstack-tox-py35/8b709de/testr_results.html.gz
> > 
> > Specifically, we're seeing errors likes this:
> > 
> >   ft1.1: 
> > os_api_ref.tests.test_basic_example.TestBasicExample.test_rest_method_StringException:
> >  Traceback (most recent call last):
> > File 
> > "/home/zuul/src/git.openstack.org/openstack/os-api-ref/.tox/py35/lib/python3.5/site-packages/sphinx_testing/util.py",
> >  line 143, in decorator
> >func(*(args + (app, status, warning)), **kwargs)
> >  File 
> > "/home/zuul/src/git.openstack.org/openstack/os-api-ref/os_api_ref/tests/test_basic_example.py",
> >  line 41, in setUp
> >self.html = (app.outdir / 'index.html').read_text(encoding='utf-8')
> >TypeError: unsupported operand type(s) for /: 'str' and 'str'
> > 
> > Which is wrong because 'app.outdir' is not supposed to be an instance
> > of 'unicode' but rather 'sphinx_testing.path.path' [1] (which overrides
> > the '/' operator to act as concatenation because operator overloading
> > is always a good idea ) [2].
> 
> Is that a change in Sphinx's API? Or sphinx_testing's?

It's a change in Sphinx, though not in the API [1]. I should really
stop playing with Sphinx. 

> > 
> > Anyway, we can go figure out what's changed here and handle it but this
> > is, at best, going to be a band aid. The fact is 'sphinx_testing' is
> > unmaintained and has been for some time now. The new hotness is
> > 'sphinx.testing' [3], which is provided (with zero documentation) as
> > part of Sphinx. Unfortunately, this uses pytest fixtures [4] which I'm
> > pretty sure Monty (and a few others?) are vehemently against using in
> > OpenStack. That leaves us with three options:
> > 
> >  * Take over 'sphinx_testing' and bring it up-to-date. Maintain
> >forever.
> >  * Start using 'sphinx.testing' and everything it comes with
> >  * Delete any tests that use 'sphinx_testing' and deal with the lack of
> >coverage
> 
> Could we change our tests to use pathlib to wrap app.outdir and get the
> same results as before?

That's what I've done [2], which is kind of based on how I fixed this
in Sphinx. However, this is at best a stopgap. The fact remains that
'sphinx_testing' is dead and the large changes that Sphinx is
undergoing (2.0 will be Python 3 only, with multiple other fixes) will
make further breakages more likely. Unless we want a repeat of the Mox
situation, I do think we should start thinking about this sooner rather
than later.

Stephen

[1] https://github.com/sphinx-doc/sphinx/commit/3a85b3502f
[2] https://review.openstack.org/607984

> Doug



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Sphinx testing fun

[Stephen Finucane]

The tests in os-api-ref recently broke:

  
http://logs.openstack.org/62/606362/1/check/openstack-tox-py35/8b709de/testr_results.html.gz

Specifically, we're seeing errors likes this:

  ft1.1: 
os_api_ref.tests.test_basic_example.TestBasicExample.test_rest_method_StringException:
 Traceback (most recent call last):
File 
"/home/zuul/src/git.openstack.org/openstack/os-api-ref/.tox/py35/lib/python3.5/site-packages/sphinx_testing/util.py",
 line 143, in decorator
   func(*(args + (app, status, warning)), **kwargs)
 File 
"/home/zuul/src/git.openstack.org/openstack/os-api-ref/os_api_ref/tests/test_basic_example.py",
 line 41, in setUp
   self.html = (app.outdir / 'index.html').read_text(encoding='utf-8')
   TypeError: unsupported operand type(s) for /: 'str' and 'str'

Which is wrong because 'app.outdir' is not supposed to be an instance
of 'unicode' but rather 'sphinx_testing.path.path' [1] (which overrides
the '/' operator to act as concatenation because operator overloading
is always a good idea ) [2].

Anyway, we can go figure out what's changed here and handle it but this
is, at best, going to be a band aid. The fact is 'sphinx_testing' is
unmaintained and has been for some time now. The new hotness is
'sphinx.testing' [3], which is provided (with zero documentation) as
part of Sphinx. Unfortunately, this uses pytest fixtures [4] which I'm
pretty sure Monty (and a few others?) are vehemently against using in
OpenStack. That leaves us with three options:

 * Take over 'sphinx_testing' and bring it up-to-date. Maintain
   forever.
 * Start using 'sphinx.testing' and everything it comes with
 * Delete any tests that use 'sphinx_testing' and deal with the lack of
   coverage

For what it's worth, I use 'sphinx.testing' in 'sphinxcontrib-apidoc'
[5] without *too* many issues, but that lives outside the 'openstack'
namespace and isn't bound by upper-constraints and the likes.

Stephen

[1] 
https://github.com/sphinx-doc/sphinx-testing/blob/0.7.2/src/sphinx_testing/path.py#L217
[2] 
https://github.com/sphinx-doc/sphinx-testing/blob/0.7.2/src/sphinx_testing/util.py#L45-L63
[3] https://github.com/sphinx-doc/sphinx/tree/v1.8.0/sphinx/testing
[4] https://docs.pytest.org/en/latest/fixture.html
[5] https://github.com/sphinx-contrib/apidoc/blob/0.3.0/tests/test_ext.py


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][xenapi] can we deprecate the xenapi-specific 'nova-console' service?

[Stephen Finucane]

On Thu, 2018-10-04 at 12:03 +, Bob Ball wrote:
> Hi Melanie,
> 
> We recommend using novncproxy_base_url with
> vncserver_proxyclient_address set to the dom0's management IP
> address.
> 
> We don't currently use nova-console, so deprecation would be the best
> approach.
> 
> Thanks,
> 
> Bob

What about nova-xvpvncproxy [1]? This would be configured using
xvpvncproxy_base_url. This is also Xen-specific (as the name, Xen VNC
Proxy, would suggest). If the noVNC-based console is now recommended,
can we also deprecate the XVP one?

Stephen

[1] 
https://review.openstack.org/#/c/606148/5/doc/source/admin/remote-console-access.rst@313

> -Original Message-
> From: melanie witt [mailto:melwi...@gmail.com] 
> Sent: 03 October 2018 23:08
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>; 
> openstack-operat...@lists.openstack.org
> Subject: [openstack-dev] [nova][xenapi] can we deprecate the xenapi-
> specific 'nova-console' service?
> 
> Greetings Devs and Ops,
> 
> Today I noticed that our code does not handle the 'nova-console'
> service properly in a multi-cell deployment and given that no one has
> complained or reported bugs about it, we're wondering if anyone still
> uses the nova-console service. The documentation [1] says that the
> nova-console service is a "XenAPI-specific service that most recent
> VNC proxy architectures do not use."
> 
> Can anyone from xenapi land shed some light on whether the nova-
> console service is still useful in deployments using the xenapi
> driver, or is it an old relic that we should deprecate and remove?
> 
> Thanks for your help,
> -melanie
> 
> [1] 
> https://docs.openstack.org/nova/latest/admin/remote-console-access.html 
> class="Apple-tab-span" style="white-space:pre">   
>   
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Nominating Chris Dent for placement-core

[Stephen Finucane]

On Fri, 2018-08-31 at 10:45 -0500, Eric Fried wrote:
> The openstack/placement project [1] and its core team [2] have been
> established in gerrit.
> 
> I hereby nominate Chris Dent for membership in the placement-core team.
> He has been instrumental in the design, implementation, and stewardship
> of the placement API since its inception and has shown clear and
> consistent leadership.
> 
> As we are effectively bootstrapping placement-core at this time, it
> would seem appropriate to consider +1/-1 responses from heavy placement
> contributors as well as existing cores (currently nova-core).
> 
> [1] https://review.openstack.org/#/admin/projects/openstack/placement
> [2] https://review.openstack.org/#/admin/groups/1936,members

+1


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg] Post-lunch presentations schedule

[Stephen Finucane]

On Tue, 2018-09-18 at 12:42 +, Bedyk, Witold wrote:
> Stephen,
> 
> could you please share your presentation slides?
> 
> Thanks
> Witek

Yup. They're available here:

https://docs.google.com/presentation/d/1gaRFmUEJEmy-8lCFsxauQLZNB2ch4hPQ0L4yO11vgL0/edit

Stephen

> 
> > -Original Message-
> > From: Thierry Carrez 
> > Sent: Freitag, 24. August 2018 11:21
> > To: OpenStack Development Mailing List  > d...@lists.openstack.org>
> > Subject: [openstack-dev] [ptg] Post-lunch presentations schedule
> 
> 
> 
> > Friday: Lightning talks
> > Fast-paced 5-min segments to talk about anything... Summaries of
> > team
> > plans for Stein encouraged. A presentation of Sphinx in OpenStack
> > by
> > stephenfin will open the show.
> 
> _
> _
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Sphinx 'doctrees' bug recently resolved

[Stephen Finucane]

FYI, Sphinx 1.8.0 contained a minor issue (introduced by yours truly)
which resulted in an incorrect default doctree directory [1]. This
would manifest itself in a 'docs/source/.doctree' directory being
created and would only affects projects that call 'sphinx-build' from
tox without specifying the '-d' parameter.

This has been fixed in Sphinx 1.8.1. I don't think it's a significant
enough issue to blacklist the 1.8.0 version and if you were seeing
issues with this in recent weeks, the issue should now be resolved. You
can still merge patches which explicitly set the '-d' parameter but
they should be unnecessary now.

Stephen

[1] https://github.com/sphinx-doc/sphinx/issues/5418


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3][nova] starting zuul migration for nova repos

[Stephen Finucane]

On Mon, 2018-09-10 at 13:48 -0600, Doug Hellmann wrote:
> Melanie gave me the go-ahead to propose the patches, so here's the list
> of patches for the zuul migration, doc job update, and python 3.6 unit
> tests for the nova repositories.

I've reviewed/+2d all of these on master and think Sylvain will be
following up with the +Ws. I need someone else to handle the
'stable/XXX' patches though.

Here's a query for anyone that wants to jump in here.

https://review.openstack.org/#/q/topic:python3-first+status:open+(openstack/nova+OR+project:openstack/nova-specs+OR+openstack/os-traits+OR+openstack/os-vif+OR+openstack/osc-placement+OR+openstack/python-novaclient)

Stephen

PS: Thanks, Andreas, for the follow-up cleanup patches. Much
appreciated :)

> +--++---+
> > Subject  | Repo 
> >   | Branch|
> 
> +--++---+
> > remove job settings for nova repositories| 
> > openstack-infra/project-config | master|
> > import zuul job settings from project-config | openstack/nova   
> >   | master|
> > switch documentation job to new PTI  | openstack/nova   
> >   | master|
> > add python 3.6 unit test job | openstack/nova   
> >   | master|
> > import zuul job settings from project-config | openstack/nova   
> >   | stable/ocata  |
> > import zuul job settings from project-config | openstack/nova   
> >   | stable/pike   |
> > import zuul job settings from project-config | openstack/nova   
> >   | stable/queens |
> > import zuul job settings from project-config | openstack/nova   
> >   | stable/rocky  |
> > import zuul job settings from project-config | openstack/nova-specs 
> >   | master|
> > import zuul job settings from project-config | openstack/os-traits  
> >   | master|
> > switch documentation job to new PTI  | openstack/os-traits  
> >   | master|
> > add python 3.6 unit test job | openstack/os-traits  
> >   | master|
> > import zuul job settings from project-config | openstack/os-traits  
> >   | stable/pike   |
> > import zuul job settings from project-config | openstack/os-traits  
> >   | stable/queens |
> > import zuul job settings from project-config | openstack/os-traits  
> >   | stable/rocky  |
> > import zuul job settings from project-config | openstack/os-vif 
> >   | master|
> > switch documentation job to new PTI  | openstack/os-vif 
> >   | master|
> > add python 3.6 unit test job | openstack/os-vif 
> >   | master|
> > import zuul job settings from project-config | openstack/os-vif 
> >   | stable/ocata  |
> > import zuul job settings from project-config | openstack/os-vif 
> >   | stable/pike   |
> > import zuul job settings from project-config | openstack/os-vif 
> >   | stable/queens |
> > import zuul job settings from project-config | openstack/os-vif 
> >   | stable/rocky  |
> > import zuul job settings from project-config | openstack/osc-placement  
> >   | master|
> > switch documentation job to new PTI  | openstack/osc-placement  
> >   | master|
> > add python 3.6 unit test job | openstack/osc-placement  
> >   | master|
> > import zuul job settings from project-config | openstack/osc-placement  
> >   | stable/queens |
> > import zuul job settings from project-config | openstack/osc-placement  
> >   | stable/rocky  |
> > import zuul job settings from project-config | openstack/python-novaclient  
> >   | master|
> > switch documentation job to new PTI  | openstack/python-novaclient  
> >   | master|
> > add python 3.6 unit test job | openstack/python-novaclient  
> >   | master|
> > add lib-forward-testing-python3 test job | openstack/python-novaclient  
> >   | master|
> > import zuul job settings from project-config | openstack/python-novaclient  
> >   | stable/ocata  |
> > import zuul job settings from project-config | openstack/python-novaclient  
> >   | stable/pike   |
> > import zuul job settings from project-config | openstack/python-novaclient  
> >   | stable/queens |
> > import zuul job settings from project-config | openstack/python-novaclient  
> >   | stable/rocky  |
> 
> +--++---+
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: