[openstack-dev] [nova][glance][keystone] Philosophy of the service catalog (was: Who needs multiple api_servers?)

[Mike Dorman]

After discussion in the Large Deployments Team session this morning, we wanted 
to follow up on the earlier thread [1,2] about overriding endpoint URLs.

That topic is exposing an underlying implication about the purpose of the 
service catalog.  The LDT position is that the service catalog should be for 
end user clients to do endpoint discovery.  While it can also be used for 
discovery by other OpenStack services, we desire to maintain the ability to 
override (like that which was discussed in the previous thread about Glance.)  
In addition to the Glance to nova-compute use case, the feedback during the LDT 
session surfaced potential use cases for other services.

The point to raise here from LDT is that we would like to avoid a trend toward 
services *only* supporting discovery via the service catalog, with no ability 
to override in config.  I.e., we want to maintain the endpoint_override (and 
similar) options.
Thanks!

[1]  http://lists.openstack.org/pipermail/openstack-dev/2017-April/116028.html 
/ 
http://lists.openstack.org/pipermail/openstack-operators/2017-April/013272.html
[2]  
http://lists.openstack.org/pipermail/openstack-dev/2017-May/thread.html#116133 
/ 
http://lists.openstack.org/pipermail/openstack-operators/2017-May/thread.html#13309
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova][glance] Who needs multiple api_servers?

[Mike Dorman]

l exist. Instead of then deprecating the api_servers list, we keep 
it- but add a big doc warning listing the gotchas and limitations - but 
for those folks for whom they are not an issue, you've got an out.

Alternative Two - Hybrid Approach - optional list of URLs

We go ahead and move to service config being the standard way one lists 
how to consume a service from the catalog. One of the standard options 
for consuming services is "endpoint_override" - which is a way an API 
user can say "hi, please to ignore the catalog and use this endpoint 
I've given you instead". The endpoint in question is a full URL, so 
https/http and ports and whatnot are all handled properly.

We add, in addition, an additional option "endpoint_override_list" which 
allows you to provide a list of URLs (not API servers) and if you 
provide that option, we'll keep the logic of choosing one at random at 
API call time. It's still a poor load balancer, and we'll still put 
warnings in the docs about it not being a featureful load balancing 
solution, but again would be available if needed.

Alternative Three - We ignore you and give you docs

I'm only including this because in the name of completeness. But we 
could write a bunch of docs about a recommended way of putting your 
internal endpoints in a load balancer and registering that with the 
internal endpoint in keystone. (I would prefer to make the operators 
happy, so let's say whatever vote I have is not for this option)

Alternative Four - We update client libs to understand multiple values 
from keystone for endpoints

I _really_ don't like this one - as I think us doing dumb software 
loadbalancing client side is prone to a ton of failures. BUT - right now 
the assumption when consuming endpoints from the catalog is that one and 
only one endpoint will be returned for a given 
service_type/service_name/interface. Rather than special-casing the
url roundrobin in nova, we could move that round-robin to be in the base 
client library, update api consumption docs with round-robin 
recommendations and then have you register the list of endpoints with 
keystone.

I know the keystone team has long been _very_ against using keystone as 
a list of all the endpoints, and I agree with them. Putting it here for 
sake of argument.

Alternative Five - We update keystone to round-robin lists of endpoints

Potentially even worse than four and even more unlikely given the 
keystone team's feelings, but we could have keystone continue to only 
return one endpoint, but have it do the round-robin selection at catalog 
generation time.


Sorry - you caught me in early morning brainstorm mode.

I am neither nova core nor keystone core. BUT:

I think honestly if adding a load balancer in front of your internal 
endpoints is an undue burden and/or the usefulness of the lists 
outweighs the limitations they have, we should go with One or Two. (I 
think three through five are all terrible)

My personal preference would be for Two - the round-robin code winds up 
being the same logic in both cases, but at least in Two folks who want 
to SSL all the way _can_, and it shouldn't be an undue extra burden on 
those of you using the api_servers now. We also don't have to do the 
funky things we currently have to do to turn the api_severs list into 
workable URLs.


On 04/27/2017 11:50 PM, Blair Bethwaite wrote:
> We at Nectar are in the same boat as Mike. Our use-case is a little
> bit more about geo-distributed operations though - our Cells are in
> different States around the country, so the local glance-apis are
> particularly important for caching popular images close to the
> nova-computes. We consider these glance-apis as part of the underlying
> cloud infra rather than user-facing, so I think we'd prefer not to see
> them in the service-catalog returned to users either... is there going
> to be a (standard) way to hide them?
>
> On 28 April 2017 at 09:15, Mike Dorman <mdor...@godaddy.com> wrote:
>> We make extensive use of the [glance]/api_servers list.  We configure 
that on hypervisors to direct them to Glance servers which are more “local” 
network-wise (in order to reduce network traffic across security 
zones/firewalls/etc.)  This way nova-compute can fail over in case one of the 
Glance servers in the list is down, without putting them behind a load 
balancer.  We also don’t run https for these “internal” Glance calls, to save 
the overhead when transferring images.
>>
>> End-user calls to Glance DO go through a real load balancer and then are 
distributed out to the Glance servers on the 

Re: [openstack-dev] [Openstack-operators] [nova][glance] Who needs multiple api_servers?

[Mike Dorman]

Ok.  That would solve some of the problem for us, but we’d still be losing the 
redundancy.  We could do some HAProxy tricks to route around downed services, 
but it wouldn’t handle the case when that one physical box is down.

Is there some downside to allowing endpoint_override to remain a list?   That 
piece seems orthogonal to the spec and IRC discussion referenced, which are 
more around the service catalog.  I don’t think anyone in this thread is 
arguing against the idea that there should be just one endpoint URL in the 
catalog.  But it seems like there are good reasons to allow multiples on the 
override setting (at least for glance in nova-compute.)

Thanks,
Mike



On 4/28/17, 8:05 AM, "Eric Fried" <openst...@fried.cc> wrote:

Blair, Mike-

There will be an endpoint_override that will bypass the service
catalog.  It still only takes one URL, though.

Thanks,
Eric (efried)

On 04/27/2017 11:50 PM, Blair Bethwaite wrote:
> We at Nectar are in the same boat as Mike. Our use-case is a little
> bit more about geo-distributed operations though - our Cells are in
> different States around the country, so the local glance-apis are
> particularly important for caching popular images close to the
> nova-computes. We consider these glance-apis as part of the underlying
> cloud infra rather than user-facing, so I think we'd prefer not to see
> them in the service-catalog returned to users either... is there going
> to be a (standard) way to hide them?
> 
    > On 28 April 2017 at 09:15, Mike Dorman <mdor...@godaddy.com> wrote:
>> We make extensive use of the [glance]/api_servers list.  We configure 
that on hypervisors to direct them to Glance servers which are more “local” 
network-wise (in order to reduce network traffic across security 
zones/firewalls/etc.)  This way nova-compute can fail over in case one of the 
Glance servers in the list is down, without putting them behind a load 
balancer.  We also don’t run https for these “internal” Glance calls, to save 
the overhead when transferring images.
>>
>> End-user calls to Glance DO go through a real load balancer and then are 
distributed out to the Glance servers on the backend.  From the end-user’s 
perspective, I totally agree there should be one, and only one URL.
>>
>> However, we would be disappointed to see the change you’re suggesting 
implemented.  We would lose the redundancy we get now by providing a list.  Or 
we would have to shunt all the calls through the user-facing endpoint, which 
would generate a lot of extra traffic (in places where we don’t want it) for 
image transfers.
>>
>> Thanks,
>> Mike
>>
>>
>>
>> On 4/27/17, 4:02 PM, "Matt Riedemann" <mriede...@gmail.com> wrote:
>>
>> On 4/27/2017 4:52 PM, Eric Fried wrote:
>> > Y'all-
>> >
>> >   TL;DR: Does glance ever really need/use multiple endpoint URLs?
>> >
>> >   I'm working on bp use-service-catalog-for-endpoints[1], which 
intends
>> > to deprecate disparate conf options in various groups, and 
centralize
>> > acquisition of service endpoint URLs.  The idea is to introduce
>> > nova.utils.get_service_url(group) -- note singular 'url'.
>> >
>> >   One affected conf option is [glance]api_servers[2], which 
currently
>> > accepts a *list* of endpoint URLs.  The new API will only ever 
return *one*.
>> >
>> >   Thus, as planned, this blueprint will have the side effect of
>> > deprecating support for multiple glance endpoint URLs in Pike, and
>> > removing said support in Queens.
>> >
>> >   Some have asserted that there should only ever be one endpoint 
URL for
>> > a given service_type/interface combo[3].  I'm fine with that - it
>> > simplifies things quite a bit for the bp impl - but wanted to make 
sure
>> > there were no loudly-dissenting opinions before we get too far 
down this
>> > path.
>> >
>> > [1]
>> > 
https://blueprints.launchpad.net/nova/+spec/use-service-catalog-for-endpoints
>> > [2]
>> > 
https://github.com/openstack/nova/blob/7e7bdb198ed6412273e22dea72e37a6371fce8bd/nova/conf/glance.py#L27-L37
>> > [3]
>> > 
http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-04-27.log.html#t2017-04-27T20:38:29
>> >

Re: [openstack-dev] [nova][glance] Who needs multiple api_servers?

[Mike Dorman]

We make extensive use of the [glance]/api_servers list.  We configure that on 
hypervisors to direct them to Glance servers which are more “local” 
network-wise (in order to reduce network traffic across security 
zones/firewalls/etc.)  This way nova-compute can fail over in case one of the 
Glance servers in the list is down, without putting them behind a load 
balancer.  We also don’t run https for these “internal” Glance calls, to save 
the overhead when transferring images.

End-user calls to Glance DO go through a real load balancer and then are 
distributed out to the Glance servers on the backend.  From the end-user’s 
perspective, I totally agree there should be one, and only one URL.

However, we would be disappointed to see the change you’re suggesting 
implemented.  We would lose the redundancy we get now by providing a list.  Or 
we would have to shunt all the calls through the user-facing endpoint, which 
would generate a lot of extra traffic (in places where we don’t want it) for 
image transfers.

Thanks,
Mike



On 4/27/17, 4:02 PM, "Matt Riedemann"  wrote:

On 4/27/2017 4:52 PM, Eric Fried wrote:
> Y'all-
>
>   TL;DR: Does glance ever really need/use multiple endpoint URLs?
>
>   I'm working on bp use-service-catalog-for-endpoints[1], which intends
> to deprecate disparate conf options in various groups, and centralize
> acquisition of service endpoint URLs.  The idea is to introduce
> nova.utils.get_service_url(group) -- note singular 'url'.
>
>   One affected conf option is [glance]api_servers[2], which currently
> accepts a *list* of endpoint URLs.  The new API will only ever return 
*one*.
>
>   Thus, as planned, this blueprint will have the side effect of
> deprecating support for multiple glance endpoint URLs in Pike, and
> removing said support in Queens.
>
>   Some have asserted that there should only ever be one endpoint URL for
> a given service_type/interface combo[3].  I'm fine with that - it
> simplifies things quite a bit for the bp impl - but wanted to make sure
> there were no loudly-dissenting opinions before we get too far down this
> path.
>
> [1]
> 
https://blueprints.launchpad.net/nova/+spec/use-service-catalog-for-endpoints
> [2]
> 
https://github.com/openstack/nova/blob/7e7bdb198ed6412273e22dea72e37a6371fce8bd/nova/conf/glance.py#L27-L37
> [3]
> 
http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-04-27.log.html#t2017-04-27T20:38:29
>
> Thanks,
> Eric Fried (efried)
> .
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

+openstack-operators

-- 

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] neutron-server high CPU and RAM load when updating DHCP ports

[Mike Dorman]

We noticed an issue in one of our larger clouds (~700 hypervisors and ovs 
agents) where (Liberty) neutron-server CPU and RAM load would spike up quite a 
bit whenever a DHCP agent port was updated.  So much load that processes were 
getting OOM killed on our API servers, and so many queries were going to the 
database that it was affecting performance of other APIs (sharing the same 
database cluster.)

Kris Lindgren determined what was happening: any time a DHCP port is changed, 
Neutron forces a complete refresh of all security group filter rules for all 
ports on the same network as the DHCP port.  We run only provider networks 
which VMs plug directly into, and our largest network has several thousand 
ports.  This was generating an avalanche of RPCs from the OVS agents, thus 
loading up neutron-server with a lot of work.

We only use DHCP has a backup network configuration mechanism in case something 
goes wrong with config-drive, so we are not huge users of it.  But DHCP agents 
are being scheduled and removed often enough, and our networks contain a large 
enough number of ports, that this has begun affecting us quite a bit.

Kevin Benton suggested a minor patch [1] to disable the blanket refresh of 
security group filters on DHCP port changes.  I tested it out in our staging 
environment and can confirm that:

-  Security group filters are indeed not refreshed on DHCP port changes

-  iptables rules generated for regular VM ports still include the 
generic rules to allow the DHCP ports 67 and 68 regardless of the presence of a 
DHCP agent on the network or not.  (This covers the scenario where VM ports are 
created while there are no DHCP agents, and a DHCP agent is added later.)

I think there are some plans to deprecate this behavior.  As far as I know, it 
still exists in master neutron.  I’m happy to put the trivial patch up for 
review if people think this is a useful change to Neutron.

We are a bit of an edge case with such large number of ports per network.  We 
are also considering disabling DHCP altogether since it is really not used.  
But, wanted to share the experience with others in case people are running into 
the same issue.

Thanks,
Mike

[1] https://gist.github.com/misterdorm/37a8997aed43081bac8d12c7f101853b
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron cli json output parsing.

[Mike Dorman]

I wonder if we should just refactor the Neutron provider to support either 
format?  That way we stay independent from whatever the particular 
installation’s cliff/tablib situation is.

We can probably safely assume that none of the Neutron objects will have 
attributes called ‘Field’ or ‘Value’, so it would be fairly easy to detect 
which format is there.

Mike


From: Denis Egorenko >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Thursday, December 31, 2015 at 5:59 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: Re: [openstack-dev] [puppet] [neutron] Serious bug in puppet neutron 
cli json output parsing.

Last I checked, which was quite a while ago, openstackclient didn't support 
everything we were using from the neutron client.

That's true. Openstack client doesn't support all features of neutron client 
[1].

I would prefer use 3) option, but, unfortunately, i also don't see way, how to 
detect stevedore and cliff.

[1] 
https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L329-L339

2015-12-30 19:53 GMT+03:00 Colleen Murphy 
>:


On Wed, Dec 30, 2015 at 8:37 AM, Sofer Athlan-Guyot 
> wrote:
Hi,

I have added neutron people as they may help to find a solution.

After banging my head against the wall for a whole afternoon I
discovered a serious bug in puppet neutron module.

I not going to repeat here the detail of the bug report[1].  Basically:
 - neutron_port
 - neutron_subnet
 - neutron_router
 - neutron_network

may break idempotency randomly and won't work at all when clifftablib is
removed from the package dependency of python-openstackclient
(Mitaka[2])

So the problem is that neutron cli json output on liberty (at least, and
maybe before) is not consistent and may come from cliff or clifftablib.
I didn't test it but the same may apply to openstack cli. As we don't
use the openstack cli json output it's not a issue (for puppet modules)

The available solution I can see are:
 1. go back to parsing csv, shell output (revert [3])
 2. find a way to leverage openstacklib parsing for neutron as well
 3. keep json and parse the right output (cliff) and find a way to make
sure that it is always used by stevedore
 4. ?

Last I checked, which was quite a while ago, openstackclient didn't support 
everything we were using from the neutron client. I would like to reevaluate 
that and go with option 2 if we can. Otherwise option 1 seems reasonable.

From my point of view 3) is not a option, but other may disagree.

The problem is tricky and the fact that the CI cannot detect this is
trickier[4].

So before Mitaka, the json parsing should go.  I would love to see an
interface that all puppet modules would use (solution 2).  The current
openstacklib parses openstack client well enough.  The neutron command
is not that different and I think there is space for code reuse.  This
would be a long term solution.  It would bring the advantage of having
only one interface to change if it was decided to use the API directly
for instance[5]

In the meantime, a quick solution to this bug must be found.

Looking forward to your comments.

Regards,

[1] https://bugs.launchpad.net/puppet-neutron/+bug/1530163
[2] https://bugs.launchpad.net/python-neutronclient/+bug/1529914
[3] https://review.openstack.org/#/c/238156/
[4] https://review.openstack.org/#/c/262223/
[5] http://lists.openstack.org/pipermail/openstack-dev/2015-October/076439.html
--
Sofer Athlan-Guyot

Colleen

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Best Regards,
Egorenko Denis,
Deployment Engineer
Mirantis
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Creating puppet-keystone-core and proposing Richard Megginson core-reviewer

[Mike Dorman]

+1

From: Clayton O'Neill >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Sunday, November 1, 2015 at 5:13 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: Re: [openstack-dev] [puppet] Creating puppet-keystone-core and 
proposing Richard Megginson core-reviewer

+1

Big thanks for Rich for all the work he’s done so far.

On Mon, Nov 2, 2015 at 3:34 AM, Adam Young 
> wrote:
On 10/31/2015 10:55 AM, Emilien Macchi wrote:
At the Summit we discussed about scaling-up our team.
We decided to investigate the creation of sub-groups specific to our modules 
that would have +2 power.

I would like to start with puppet-keystone:
https://review.openstack.org/240666

And propose Richard Megginson part of this group.

Rich is leading puppet-keystone work since our Juno cycle. Without his 
leadership and skills, I'm not sure we would have Keystone v3 support in our 
modules.
He's a good Puppet reviewer and takes care of backward compatibility. He also 
has strong knowledge at how Keystone works. He's always willing to lead our 
roadmap regarding identity deployment in OpenStack.

Having him on-board is for us an awesome opportunity to be ahead of other 
deployments tools and supports many features in Keystone that real deployments 
actually need.

I would like to propose him part of the new puppet-keystone-core group.

As a Keystone developer I have to say I am indebted to Rich for his efforts.  
+1 from me.


Thank you Rich for your work, which is very appreciated.
--
Emilien Macchi



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Large Deployment Team] Action/Work Items Ahead of Summit, please review

[Mike Dorman]

During our meeting [1] today we discussed our agenda [2] and action items for 
the LDT working session [3] at the summit.

We’ve compiled a list of a few things folks can be working on before getting to 
Tokyo to ensure this session is productive:


1. Neutron Segmented/Routed Networks:
   * Review the current WIP spec: [4]
   * Neutron likely will have their own session on this topic [5], so we may 
opt to have most of the discussion there.  Schedule for that is TBD, watch the 
ether pads [2] [5] for updates.

2. Neutron General
   * Dev team is seeking “pain points” feedback, please add yours to the 
etherpad [6]

3. Common Cells v1 Patch sets
   * If you have not already done so, please add any cells patches you are 
carrying to the etherpad [7]
   * We’d like to assign the low-hanging-fruit patches out to folks to get into 
Nova reviews as a step toward merging them into upstream

4. Public Clouds
   * Plan to focus discussion during the session on identifying specific gaps 
and/or RFEs needed by this constituency
   * Please add these to the etherpad [2] and come ready to discuss at the 
summit

5. Glance Asks
   * Similar to Public Clouds, this will focus on current gaps and RFEs
   * Fill them in on the etherpad [8] and come prepared to discuss

6. Performance Issues
   * The cross project group has a session specifically for this [9] [10], so 
we will forego this discussion in LDT in lieu of that


Thanks for your participation in getting these work items moved forward.  We 
have a big agenda with only 90 minutes.  We can accomplish more in Tokyo if we 
prepare some ahead of time!


[1]  
http://eavesdrop.openstack.org/meetings/large_deployments_team_monthly_meeting/2015/large_deployments_team_monthly_meeting.2015-10-15-16.01.html
[2]  https://etherpad.openstack.org/p/TYO-ops-large-deployments-team
[3]  http://sched.co/4Nl4
[4]  https://review.openstack.org/#/c/225384/
[5]  https://etherpad.openstack.org/p/mitaka-neutron-next-network-model
[6]  https://etherpad.openstack.org/p/mitaka-neutron-next-ops-painpoints
[7]  https://etherpad.openstack.org/p/PAO-LDT-cells-patches
[8]  https://etherpad.openstack.org/p/LDT-glance-asks
[9]  http://sched.co/4Qds
[10] 
https://etherpad.openstack.org/p/mitaka-cross-project-performance-team-kick-off
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [Neutron][L3] [Large Deployments Team] Representing a networks connected by routers

[Mike Dorman]

Ok, cool.  We plan to discuss this during the LDT time slot at 1330-1500 
Pacific on Tuesday 8/18.  We can have this as the first agenda item so there’s 
a defined start time for those who are remote.

I'll take ownership of setting up a hangout (or whatever.)  Do people have a 
preference on what videoconference tool to us?  Absent any opinions, I’ll just 
do a Google Hangout.

Thanks!
Mike


From: Kyle Mestery
Date: Tuesday, August 4, 2015 at 8:09 AM
To: Ryan Moats
Cc: Mike Dorman, OpenStack Development Mailing List (not for usage 
questions), OpenStack Operators
Subject: Re: [Openstack-operators] [openstack-dev] [Neutron][L3] [Large 
Deployments Team] Representing a networks connected by routers

Can you also try to have some sort of remote option? I'd like to attend this, 
and I'd like Carl to try and attend as well. Thanks!

On Tue, Aug 4, 2015 at 8:50 AM, Ryan Moats 
rmo...@us.ibm.commailto:rmo...@us.ibm.com wrote:

I will be there for my lightning talk, and I think armax and kevin_benton will 
be there - it would be good to find some time for us to pow-wow, along with 
some teleconference so that carl_baldwin and mestery can join in...

Ryan Moats (regXboi)

Mike Dorman mdor...@godaddy.commailto:mdor...@godaddy.com wrote on 
08/03/2015 10:07:23 PM:

 From: Mike Dorman mdor...@godaddy.commailto:mdor...@godaddy.com
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org,
  OpenStack Operators openstack-
 operat...@lists.openstack.orgmailto:operat...@lists.openstack.org
 Date: 08/03/2015 10:08 PM
 Subject: Re: [Openstack-operators] [openstack-dev] [Neutron][L3]
 [Large Deployments Team] Representing a networks connected by routers


 I hope we can move this idea moving forward.  I was disappointed to see
 the spec abandoned.

 Some of us from the large deployers group will be at the Ops Meetup.  Will
 there be any representation from Neutron there that we could discuss with
 more?

 Thanks,
 Mike





 On 8/3/15, 12:27 PM, Carl Baldwin 
 c...@ecbaldwin.netmailto:c...@ecbaldwin.net wrote:

 Kevin, sorry for the delay in response.  Keeping up on this thread was
 getting difficult while on vacation.
 
 tl;dr:  I think it is worth it to talk through the idea of inserting
 some sort of a subnet group thing in the model to which floating ips
 (and router external gateways) will associate.  It has been on my mind
 for a long time now.  I didn't pursue it because a few informal
 attempts to discuss it with others indicated to me that it would be a
 difficult heavy-lifting job that others may not appreciate or
 understand.  Scroll to the bottom of this message for a little more on
 this.
 
 Carl
 
 On Tue, Jul 28, 2015 at 1:15 AM, Kevin Benton 
 blak...@gmail.commailto:blak...@gmail.com wrote:
 Also, in my proposal, it is more the router that is the grouping
 mechanism.
 
  I can't reconcile this with all of the points you make in the rest of
 your
  email. You want the collection of subnets that a network represents,
 but you
  don't want any other properties of the network.
 
 This is closer to what I'm trying to say but isn't quite there.  There
 are some subnets that should be associated with the segments
 themselves and there are some that should be associated with the
 collection of segments.  I want floating IPs that are not tied the an
 L2 network.  None of the alternate proposals that I'd heard addressed
 this.
 
 that the network object is currently the closest thing we have to
  representing the L3 part of the network.
 
  The L3 part of a network is the subnets. You can't have IP addresses
 without
  the subnets, you can't have floating IPs without the subnets, etc.
 
 You're right but in the current model you can't have IP addresses
 without the network either which is actually the point I'm trying to
 make.
 
  A Neutron network is an L2 construct that encapsulates L3 things. By
  encapsulating them it also provides an implicit grouping. The routed
  networks proposal basically wants that implicit grouping without the
  encapsulation or the L2 part.
 
 This sounds about right.  I think it is wrong to assume that we need
 an L2 network to encapsulate L3 things.  I'm feeling restricted by the
 model and the insistence that a neutron network is a purely L2
 construct.
 
 We don't associate floating ips with a network because we want to arp
 for
  them.  You're taking a consequence of the current model and its
 constraints
  and presenting that as the motivation for the model. We do so because
 there
  is no better L3 object to associate it to.
 
  Don't make assumptions about how people use floating IPs now just
 because it
  doesn't fit your use-case well. When an external network is implemented
 as a
  real Neutron network (leaving external_network_bridge blank like we
 suggest
  in the networking guide), normal ports can be attached and can
  co-exist/communicate with the floating IPs because it behaves as an L2

Re: [openstack-dev] [Neutron][L3] [Large Deployments Team] Representing a networks connected by routers

[Mike Dorman]

I hope we can move this idea moving forward.  I was disappointed to see 
the spec abandoned.

Some of us from the large deployers group will be at the Ops Meetup.  Will 
there be any representation from Neutron there that we could discuss with 
more?

Thanks,
Mike





On 8/3/15, 12:27 PM, Carl Baldwin c...@ecbaldwin.net wrote:

Kevin, sorry for the delay in response.  Keeping up on this thread was
getting difficult while on vacation.

tl;dr:  I think it is worth it to talk through the idea of inserting
some sort of a subnet group thing in the model to which floating ips
(and router external gateways) will associate.  It has been on my mind
for a long time now.  I didn't pursue it because a few informal
attempts to discuss it with others indicated to me that it would be a
difficult heavy-lifting job that others may not appreciate or
understand.  Scroll to the bottom of this message for a little more on
this.

Carl

On Tue, Jul 28, 2015 at 1:15 AM, Kevin Benton blak...@gmail.com wrote:
Also, in my proposal, it is more the router that is the grouping 
mechanism.

 I can't reconcile this with all of the points you make in the rest of 
your
 email. You want the collection of subnets that a network represents, 
but you
 don't want any other properties of the network.

This is closer to what I'm trying to say but isn't quite there.  There
are some subnets that should be associated with the segments
themselves and there are some that should be associated with the
collection of segments.  I want floating IPs that are not tied the an
L2 network.  None of the alternate proposals that I'd heard addressed
this.

that the network object is currently the closest thing we have to
 representing the L3 part of the network.

 The L3 part of a network is the subnets. You can't have IP addresses 
without
 the subnets, you can't have floating IPs without the subnets, etc.

You're right but in the current model you can't have IP addresses
without the network either which is actually the point I'm trying to
make.

 A Neutron network is an L2 construct that encapsulates L3 things. By
 encapsulating them it also provides an implicit grouping. The routed
 networks proposal basically wants that implicit grouping without the
 encapsulation or the L2 part.

This sounds about right.  I think it is wrong to assume that we need
an L2 network to encapsulate L3 things.  I'm feeling restricted by the
model and the insistence that a neutron network is a purely L2
construct.

We don't associate floating ips with a network because we want to arp 
for
 them.  You're taking a consequence of the current model and its 
constraints
 and presenting that as the motivation for the model. We do so because 
there
 is no better L3 object to associate it to.

 Don't make assumptions about how people use floating IPs now just 
because it
 doesn't fit your use-case well. When an external network is implemented 
as a
 real Neutron network (leaving external_network_bridge blank like we 
suggest
 in the networking guide), normal ports can be attached and can
 co-exist/communicate with the floating IPs because it behaves as an L2
 network exactly as implied by the API. The current model works quite 
well if
 your fabric can extend the external network everywhere it needs to be.

Yes, when an external network is implemented as a real Neutron
network all of this is true and my proposal doesn't change any of
this.  I'm wasn't making any such assumptions.  I acknowledge that the
current model works well in this case and didn't intend to change it
for current use cases.  It is precisely that because it does not fit
my use case well that I'm pursuing this.

Notice that a network marked only as external doesn't allow normal
tenants to create ports.  It must also be marked shared to allow it.
Unless tenants are creating regular ports then they really don't care
if arp or anything else L2 is involved because such an external
network is meant to give access external to the cloud where L2 is
really just an implementation detail.  It is the deployer that cares
because of whatever infrastructure (like a gateway router) needs to
work with it.  If the L2 is important, then the deployer will not
attempt to use an L3 only network, she will use the same kinds of
networks as always.

The bad assumption here is that floating IPs need an explicit
association with an L2 only construct:  tenant's allocate a floating
IP by selecting the Neutron network it is recorded in the DB that way.
Tenant's aren't even allowed to see the subnets on an external
network.  This is counter-intuitive to me because I believe that, in
most cases, tenants want a floating IP to get L3 access to the world
(or a part of it) that is external to Openstack.  Yet, they can only
see the L2 object?  These are the factors that make me view the
Neutron network as an L2 + L3 construct.

 If you don't want floating IPs to be reachable on the network they are
 associated with, then let's stop associating them with a 

Re: [openstack-dev] [Neutron][L3] Representing a networks connected by routers

[Mike Dorman]

On 7/23/15, 8:54 AM, Carl Baldwin c...@ecbaldwin.net wrote:

On Thu, Jul 23, 2015 at 8:51 AM, Kevin Benton blak...@gmail.com wrote:
Or, migration scheduling would need to respect the constraint that a
 port may be confined to a set of hosts.  How can be assign a port to a
 different network?  The VM would wake up and what?  How would it know
 to reconfigure its network stack?

 Right, that's a big mess. Once a network is picked for a port I think we
 just need to rely on a scheduler filter that limits the migration to 
where
 that network is available.

+1.  That's where I was going.

Agreed, this seems reasonable to me for the migration scheduling case.

I view the pre-created port scenario as an edge case.  By explicitly 
pre-creating a port and using it for a new instance (rather than letting 
nova create a port for you), you are implicitly stating that you have more 
knowledge about the networking setup.  In so doing, you’re removing the 
guard rails (of nova scheduling the instance to a good network for the 
host it's on), and therefore are at higher risk to crash and burn.  To me 
that’s an acceptable trade-off.

Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][L3] Representing a networks connected by routers

[Mike Dorman]

On 7/23/15, 9:42 AM, Carl Baldwin c...@ecbaldwin.net wrote:

On Wed, Jul 22, 2015 at 3:21 PM, Kevin Benton blak...@gmail.com wrote:
 The issue with the availability zone solution is that we now force
 availability zones in Nova to be constrained to network configuration. 
In
 the L3 ToR/no overlay configuration, this means every rack is its own
 availability zone. This is pretty annoying for users to deal with 
because
 they have to choose from potentially hundreds of availability zones and 
it
 rules out making AZs based on other things (e.g.  current phase, cooling
 systems, etc).

 I may be misunderstanding and you could be suggesting to not expose this
 availability zone to the end user and only make it available to the
 scheduler. However, this defeats one of the purposes of availability 
zones
 which is to let users select different AZs to spread their instances 
across
 failure domains.

I was actually talking with some guys at dinner during the Nova
mid-cycle last night (Andrew ??, Robert Collins, Dan Smith, probably
others I can't remember) about the relationship of these network
segments to AZs and cells.  I think we were all in agreement that we
can't confine segments to AZs or cells nor the other way around.


I just want to +1 this one from the operators’ perspective.  Network 
segments, availability zones, and cells are all separate constructs which 
are used for different purposes.  We prefer to not have any relationships 
forced between them.

Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] puppet-designate POC implementation of virtualenv and docker support.

[Mike Dorman]

I have been meaning to ask you about this, so thanks for posting.

I like the approach.  Definitely a lot cleaner than the somewhat hardcoded 
dependencies and subscriptions that are in the modules now.

Do you envision that long term the docker/venv/whatever else implementation 
(like you have in designate_ext) would actually be part of the upstream Puppet 
module?  Or would we provide the hooks that you describe, and leave it up to 
other modules to handle the non-package-based installs?

Mike


From: Clayton O'Neill
Reply-To: OpenStack Development Mailing List (not for usage questions)
Date: Monday, July 13, 2015 at 8:34 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [puppet] puppet-designate POC implementation of 
virtualenv and docker support.

Last year I put together a virtualenv patch for the Designate puppet module, 
but the patch was too invasive of a change and too opinionated to be practical 
to merge.  I've taken another shot at this with the approach of implementing 
well defined hooks for various phases of the module. This should  allow 
external support for alternative ways of installing and running services (such 
as virtualenv, and docker).  I think this patch is now mostly ready for some 
outside reviews (we'll be running the virtualenv support in production soon).

The puppet-designate change to support this can be found here:  
https://review.openstack.org/#/c/197172/

The supporting puppet-designate_ext module can be found here: 
https://github.com/twc-openstack/puppet-designate_ext

The basic approach is to split the module dependency chain into 3 phases:

 * install begin/end
 * config begin/end
 * service begin/end

Each of these phases have a pair of corresponding anchors that are used 
internally for dependencies and notifications.  This allows external modules to 
hook into this flow without having to change the module.  For example, the 
virtualenv support will build the virtualenv environment between the 
designate::install::begin and designate::install::end anchors.  Additionally, 
the virtualenv support will notify the designate::install::end anchor.  This 
allows other resources to subscribe to this anchor without needing to know if 
the software is being installed as a package, virtualenv, or docker image.

I think this approach could be applied mostly as is to at least some of the 
existing modules with similar levels of changes.  For example, horizon, 
keystone  heat would probably be fairly straightforward.  I suspect this 
approach would need refinement for more complex services like neutron and nova. 
 We would need to work out how to manage things like external packages that 
would still be needed if running a virtualenv based install, but probably not 
needed if running a docker based install.  We would probably also want to 
consider how to be more granular about service notifications.

I'd love to get some feedback on this approach if people have time to look it 
over.  We're still trying to move away from using packages for service installs 
and I'd like to figure out how to do that without carrying heavyweight and 
fragile patches to the openstack puppet modules.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] cross-site console web socket proxies no longer work

[Mike Dorman]

I noticed in Kilo there’s a validation check in the console web socket proxies 
to ensure the hostnames from the Origin and Host headers match.  This was as a 
result of CVE-2015-0259 (https://bugs.launchpad.net/nova/+bug/1409142).  
Effectively it disabled cross-site web socket connections.

This is OK for Horizon, but we also run our own custom UI that’s on a different 
hostname from the console proxy servers.  Therefore we need to have the 
cross-site connections work.  I have opened 
https://bugs.launchpad.net/nova/+bug/1474079 for this.

My thought is to add a new nova configuration parameter which would list 
additional allowed Origin hosts for the proxy servers.  And add those to the 
check at 
https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L116

I will probably go ahead and implement that for us internally, but interested 
in opinions on this approach for upstream Nova purposes.  I’m happy to do the 
work, but want to make sure this is generally in line with what the community 
would accept first.

Thanks,
Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] oslo.messaging version and RabbitMQ heartbeat support

[Mike Dorman]

As a follow up to the discussion during the IRC meeting yesterday, please vote 
for one of these approaches:

1)  Make the default for the rabbit_heartbeat_timeout_threshold parameter 60, 
which matches the default in Kilo oslo.messaging.  This will by default enable 
the RMQ heartbeat feature, which also matches the default in Kilo 
oslo.messaging.  Operators will need to set this parameter to 0 in order to 
disable the feature, which will be documented in the comments within the 
manifest.  We will reevaluate the default value for the Liberty release, since 
the oslo.messaging default most likely will change to 0 for that release.

2)  In addition to #1 above, also add a rabbit_enable_heartbeat parameter, 
which will default to false.  Setting that to true will be needed to explicitly 
enable the RMQ heartbeat feature, regardless of the value of 
rabbit_heartbeat_timeout_threshold.  By default the RMQ heartbeat feature will 
be disabled, which may be a marginally safer approach (due to the 
requirements.txt stuff, see below), but will not match the upstream Kilo 
oslo.messaging default.  This will also turn off the feature for people who 
have already been “getting it for free” if they do nothing and don’t update 
their composition module.

My vote is for #1.

Let’s plan to close the voting by next week’s IRC meeting, so we can come to a 
final conclusion at that time.

Thanks,
Mike




From: Mike Dorman
Reply-To: OpenStack Development Mailing List (not for usage questions)
Date: Tuesday, June 23, 2015 at 5:07 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [puppet] oslo.messaging version and RabbitMQ heartbeat 
support

As a follow up to https://review.openstack.org/#/c/194399/ and the meeting 
discussion earlier today, I’ve determined that everybody (RDU, Ubuntu, Debian) 
is packaging oslo.messaging 1.8.2 or 1.8.3 with the Kilo build.  (This is also 
the version we get on our internal Anvil-based build.)  This is considerably 
lower than 1.11.0 where the default rabbit_heartbeat_timeout_threshold changes 
(from 60 to 0.)

If we go forward using the default rabbit_heartbeat_timeout_threshold value of 
60, that will be the correct default behavior up through oslo.messaging 1.10.x.

When people upgrade to 1.11.0 or higher, we’ll no longer match the upstream 
default behavior.  But, it should maintain the _actual_ behavior (heartbeating 
enabled) for people doing an upgrade.  Once Liberty is cut, we should 
reevaluate to make sure we’re matching whatever the default is at that time.

However, the larger problem I see is that oslo.messaging requirements.txt in 
=1.10.x does not enforce the needed versions of kombu and amqp for heartbeat 
to work:
https://github.com/openstack/oslo.messaging/blob/1.8.2/requirements.txt#L25-L26 
 This is confusing as heartbeat is enabled by default!

I am not sure what the behavior is when heartbeat is enabled with older kombu 
or amqp.  Does anyone know?  If it silently fails, maybe we don’t care.  But if 
enabling heartbeat (by default, rabbit_heartbeat_timeout_threshold=60) actively 
breaks, that would be bad.

I see two options here:

1)  Make default rabbit_heartbeat_timeout_threshold=60 in the Puppet modules, 
to strictly follow the upstream default in Kilo.  Reevaluate this default value 
for Liberty.  Ignore the kombu/amqp library stuff and hope “it just works 
itself out naturally.”

2)  Add a rabbit_enable_heartbeat parameter to explicitly enable/disable the 
feature, and default to disable.  This goes against the current default 
behavior, but will match it for oslo.messaging =1.11.0.  I think this is the 
safest path, as we won’t be automatically enabling heartbeat for people who 
don’t have a new enough kombu or amqp.

Personally, I like #1, because I am going to enable this feature, anyway.  And 
I can’t really imagine why one would _not_ want to enable it.  But I am fine 
implementing it either way, I just want to get it done so I can get off my 
local forks. :)

Thoughts?

Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] oslo.messaging version and RabbitMQ heartbeat support

[Mike Dorman]

As a follow up to https://review.openstack.org/#/c/194399/ and the meeting 
discussion earlier today, I’ve determined that everybody (RDU, Ubuntu, Debian) 
is packaging oslo.messaging 1.8.2 or 1.8.3 with the Kilo build.  (This is also 
the version we get on our internal Anvil-based build.)  This is considerably 
lower than 1.11.0 where the default rabbit_heartbeat_timeout_threshold changes 
(from 60 to 0.)

If we go forward using the default rabbit_heartbeat_timeout_threshold value of 
60, that will be the correct default behavior up through oslo.messaging 1.10.x.

When people upgrade to 1.11.0 or higher, we’ll no longer match the upstream 
default behavior.  But, it should maintain the _actual_ behavior (heartbeating 
enabled) for people doing an upgrade.  Once Liberty is cut, we should 
reevaluate to make sure we’re matching whatever the default is at that time.

However, the larger problem I see is that oslo.messaging requirements.txt in 
=1.10.x does not enforce the needed versions of kombu and amqp for heartbeat 
to work:
https://github.com/openstack/oslo.messaging/blob/1.8.2/requirements.txt#L25-L26 
 This is confusing as heartbeat is enabled by default!

I am not sure what the behavior is when heartbeat is enabled with older kombu 
or amqp.  Does anyone know?  If it silently fails, maybe we don’t care.  But if 
enabling heartbeat (by default, rabbit_heartbeat_timeout_threshold=60) actively 
breaks, that would be bad.

I see two options here:

1)  Make default rabbit_heartbeat_timeout_threshold=60 in the Puppet modules, 
to strictly follow the upstream default in Kilo.  Reevaluate this default value 
for Liberty.  Ignore the kombu/amqp library stuff and hope “it just works 
itself out naturally.”

2)  Add a rabbit_enable_heartbeat parameter to explicitly enable/disable the 
feature, and default to disable.  This goes against the current default 
behavior, but will match it for oslo.messaging =1.11.0.  I think this is the 
safest path, as we won’t be automatically enabling heartbeat for people who 
don’t have a new enough kombu or amqp.

Personally, I like #1, because I am going to enable this feature, anyway.  And 
I can’t really imagine why one would _not_ want to enable it.  But I am fine 
implementing it either way, I just want to get it done so I can get off my 
local forks. :)

Thoughts?

Mike

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Re: duplicate keystone endpoints

[Mike Dorman]

We’ve had this same problem, too, and I’d agree it should fail the Puppet run 
rather than just passing.  Would you mind writing up a bug report for this at 
https://launchpad.net/puppet-openstacklib ?

I have this on my list of stuff to fix when we go to Kilo (soon), so if 
somebody else doesn’t fix it, then I will.

Thanks!


From: Black, Matthew
Reply-To: 
puppet-openst...@puppetlabs.commailto:puppet-openst...@puppetlabs.com
Date: Wednesday, June 17, 2015 at 12:54 PM
To: puppet-openst...@puppetlabs.commailto:puppet-openst...@puppetlabs.com
Subject: duplicate keystone endpoints

I was digging around in the icehouse puppet code and I found what I believe is 
the cause of a duplicate endpoint creation during a short network disruption. 
In my environments the keystone servers do not reside in the same network as 
the regions. It looks like the puppet code fails the first request, sleeps 10 
seconds, tries again and if that fails it then returns with a nil. The code 
then returns an empty array to the provider which then is assumed to mean that 
the endpoint does not exist. If the network blip is over by that point it will 
attempt to create the endpoint and thus a duplicate endpoint in the catalog.

https://github.com/openstack/puppet-keystone/blob/stable/icehouse/lib/puppet/provider/keystone.rb#L139

https://github.com/openstack/puppet-keystone/blob/stable/icehouse/lib/puppet/provider/keystone.rb#L83-L88


Looking at the juno code, which it is using the openstacklib, the issue still 
exists but in a slightly different fashion.

https://github.com/openstack/puppet-openstacklib/blob/master/lib/puppet/provider/openstack.rb#L55-L66

I believe this should be changed that instead of a breaking out of the loop it 
should throw an exception.

--


To unsubscribe from this group and stop receiving emails from it, send an email 
to 
puppet-openstack+unsubscr...@puppetlabs.commailto:puppet-openstack+unsubscr...@puppetlabs.com.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Change abandonment policy

[Mike Dorman]

I vote #2, with a smaller N.

We can always adjust this policy in the future if find we have to manually 
abandon too many old reviews.


From: Colleen Murphy
Reply-To: 
puppet-openst...@puppetlabs.commailto:puppet-openst...@puppetlabs.com
Date: Tuesday, June 2, 2015 at 12:39 PM
To: OpenStack Development Mailing List (not for usage questions), 
puppet-openst...@puppetlabs.commailto:puppet-openst...@puppetlabs.com
Subject: [puppet] Change abandonment policy

In today's meeting we discussed implementing a policy for whether and when core 
reviewers should abandon old patches whose author's were inactive. (This 
doesn't apply to authors that want to abandon their own changes, only for core 
reviewers to abandon other people's changes.) There are a few things we could 
do here, with potential policy drafts for the wiki:

1) Never abandon

```
Our policy is to never abandon changes except for our own.
```

The sentiment here is that an old change in the queue isn't really hurting 
anything by just sitting there, and it is more visible if someone else wants to 
pick up the change.

2) Manually abandon after N months/weeks changes that have a -2 or were fixed 
in a different patch

```
If a change is submitted and given a -1, and subsequently the author becomes 
unresponsive for a few weeks, reviewers should leave reminder comments on the 
review or attempt to contact the original author via IRC or email. If the 
change is easy to fix, anyone should feel welcome to check out the change and 
resubmit it using the same change ID to preserve original authorship. Core 
reviewers will not abandon such a change.

If a change is submitted and given a -2, or it otherwise becomes clear that the 
change can not make it in (for example, if an alternate change was chosen to 
solve the problem), and the author has been unresponsive for at least 3 months, 
a core reviewer should abandon the change.
```

Core reviewers can click the abandon button only on old patches that are 
definitely never going to make it in. This approach has the advantage that it 
is easier for contributors to find changes and fix them up, even if the change 
is very old.

3) Manually abandon after N months/weeks changes that have a -1 that was never 
responded to

```
If a change is submitted and given a -1, and subsequently the author becomes 
unresponsive for a few weeks, reviewers should leave reminder comments on the 
review or attempt to contact the original author via IRC or email. If the 
change is easy to fix, anyone should feel welcome to check out the change and 
resubmit it using the same change ID to preserve original authorship. If the 
author is unresponsive for at least 3 months and no one else takes over the 
patch, core reviewers can abandon the patch, leaving a detailed note about how 
the change can be restored.

If a change is submitted and given a -2, or it otherwise becomes clear that the 
change can not make it in (for example, if an alternate change was chosen to 
solve the problem), and the author has been unresponsive for at least 3 months, 
a core reviewer should abandon the change.
```

Core reviewers can click the abandon button on changes that no one has shown an 
interest in in N months/weeks, leaving a message about how to restore the 
change if the author wants to come back to it. Puppet Labs does this for its 
module pull requests, setting N at 1 month.

4) Auto-abandon after N months/weeks if patch has a -1 or -2

```
If a change is given a -2 and the author has been unresponsive for at least 3 
months, a script will automatically abandon the change, leaving a message about 
how the author can restore the change and attempt to resolve the -2 with the 
reviewer who left it.
```

We would use a tool like this one[1] to automatically abandon changes meeting a 
certain criteria. We would have to decide whether we want to only auto-abandon 
changes with -2's or go as far as to auto-abandon those with -1's. The policy 
proposal above assumes -2. The tool would leave a canned message about how to 
restore the change.


Option 1 has the problem of leaving clutter around, which the discussion today 
seeks to solve.

Option 3 leaves the possibility that a change that is mostly good becomes 
abandoned, making it harder for someone to find and restore it.

 I don't think option 4 is necessary because there are not an overwhelming 
number of old changes (I count 9 that are currently over six months old). In 
working through old changes a few months ago I found that many of them are easy 
to fix up to remove a -1, and auto-abandoning removes the ability for a human 
to make that call. Moreover, if a patch has a procedural -2 that ought to be 
lifted after some point, auto-abandonment has the potential to accidentally 
throw out a change that was intended to be kept (though presumably the core 
reviewer who left the -2 would notice the abandonment and restore it if that 
was the case).

I am in favor of option 2. I think setting N 

Re: [openstack-dev] [puppet] Renaming the IRC channel to #openstack-puppet

[Mike Dorman]

+1 Let’s do it.


From: Matt Fischer
Reply-To: OpenStack Development Mailing List (not for usage questions)
Date: Friday, May 29, 2015 at 1:46 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [puppet] Renaming the IRC channel to 
#openstack-puppet

I would love to do this. +2!

On Fri, May 29, 2015 at 1:39 PM, Mathieu Gagné 
mga...@iweb.commailto:mga...@iweb.com wrote:
Hi,

We recently asked for our IRC channel (#puppet-openstack) to be logged
by the infra team. We happen to be the only channel suffixing the word
openstack instead of prefixing it. [1]

I would like to propose renaming our IRC channnel to #openstack-puppet
to better fit the mold (convention) already in place and be more
intuitive for new comers to discover.

Jemery Stanley (fungi) explained to me that previous IRC channel renames
were done following the Ubuntu procedure. [2] Last rename I remember of
was #openstack-stable to #openstack-release and it went smoothly without
any serious problem.

What do you guys think about the idea?

[1] http://eavesdrop.openstack.org/irclogs/
[2] https://wiki.ubuntu.com/IRC/MovingChannels

Note: I already registered the channel name for safe measures.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Proposal to configure Oslo libraries

[Mike Dorman]

+1 I agree we should do this, etc., etc.

I don’t have a strong preference for #1 or #2, either.  But I do think #1 
is slightly more complicated from a deployer/operator perspective.  It’s 
another module I have to manage, pull in, etc.  Granted this is a trivial 
amount of incremental work.

I confess I am not super familiar with openstacklib, but I don’t 
understand why We have to differentiate *common-in-OpenStack* and 
*common-in-our-modules*.”  To me, openstacklib is for _anything_ that’s 
common.  Maybe you could expand upon your thinking on this a little more, 
just so it’s a little more explicit?

Since others are not chomping at the bit to chime in here, I guess there 
is probably not many major preferences on this.  I would be happy with 
getting this done, regardless of how it’s implemented.

Thanks,
Mike






On 5/8/15, 7:50 AM, Rich Megginson rmegg...@redhat.com wrote:

On 05/08/2015 07:17 AM, Doug Hellmann wrote:
 Excerpts from Ben Nemec's message of 2015-05-07 15:57:48 -0500:
 I don't know much about the puppet project organization so I won't
 comment on whether 1 or 2 is better, but a big +1 to having a common
 way to configure Oslo opts.  Consistency of those options across all
 services is one of the big reasons we pushed so hard for the libraries
 to own their option definitions, so this would align well with the way
 the projects are consumed.

 - -Ben
 Well said, Ben.

 Doug

 On 05/07/2015 03:19 PM, Emilien Macchi wrote:
 Hi,

 I think one of the biggest challenges working on Puppet OpenStack
 modules is to keep code consistency across all our modules (~20).
 If you've read the code, you'll see there is some differences
 between RabbitMQ configuration/parameters in some modules and this
 is because we did not have the right tools to make it properly. A
 lot of the duplicated code we have comes from Oslo libraries
 configuration.

 Now, I come up with an idea and two proposals.

 Idea 

 We could have some defined types to configure oslo sections in
 OpenStack configuration files.

 Something like: define oslo::messaging::rabbitmq( $user, $password
 ) { ensure_resource($name, 'oslo_messaging_rabbit/rabbit_userid',
 {'value' = $user}) ... }

 Usage in puppet-nova: ::oslo::messaging::rabbitmq{'nova_config':
 user = 'nova', password = 'secrete', }

 And patch all our modules to consume these defines and finally
 have consistency at the way we configure Oslo projects (messaging,
 logging, etc).

 Proposals =

 #1 Creating puppet-oslo ... and having oslo::messaging::rabbitmq,
 oslo::messaging::qpid, ..., oslo::logging, etc. This module will be
 used only to configure actual Oslo libraries when we deploy
 OpenStack. To me, this solution is really consistent with how
 OpenStack works today and is scalable as soon we contribute Oslo
 configuration changes in this module.

+1 - For the Keystone authentication options, I think it is important to 
encapsulate this and hide the implementation from the other services as 
much as possible, to make it easier to use all of the different types of 
authentication supported by Keystone now and in the future.  I would 
think that something similar applies to the configuration of other 
OpenStack services.


 #2 Using puppet-openstacklib ... and having
 openstacklib::oslo::messaging::(...) A good thing is our modules
 already use openstacklib. But openstacklib does not configure
 OpenStack now, it creates some common defines  classes that are
 consumed in other modules.


 I personally prefer #1 because: * it's consistent with OpenStack. *
 I don't want openstacklib the repo where we put everything common.
 We have to differentiate *common-in-OpenStack* and
 *common-in-our-modules*. I think openstacklib should continue to be
 used for common things in our modules, like providers, wrappers,
 database management, etc. But to configure common OpenStack bits
 (aka Oslo©), we might want to create puppet-oslo.

 As usual, any thoughts are welcome,

 Best,



 __
 

 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 -BEGIN PGP SIGNATURE-

 
_
_
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for 

Re: [openstack-dev] [puppet][operators] How to specify Keystone v3 credentials?

[Mike Dorman]

We also run all masterless/puppet apply.  And we just populate a bare 
bones keystone.conf on any box that does not have keystone installed, but 
Puppet needs to be able to create keystone resources.

Also agreed on avoiding puppetdb, for the same reasons.

(Something to note for those of us doing masterless today: there are plans 
from Puppet to move more of the manifest compiling functionality to run 
only in the puppet master process.  So at some point, it’s likely that 
masterless setups may not be possible.)

Mike




 If you do not wish to explicitly define Keystone resources for
 Glance on Keystone nodes but instead let Glance nodes manage
 their own resources, you could always use exported resources.

 You let Glance nodes export their keystone resources and then
 you ask Keystone nodes to realize them where admin credentials
 are available. (I know some people don't really like exported
 resources for various reasons)

 I'm not familiar with exported resources.  Is this a viable
 option that has less impact than just requiring Keystone
 resources to be realized on the Keystone node?
 
 I'm not in favor of having exported resources because it requires 
 PuppetDB, and a lot of people try to avoid that. For now, we've
 been able to setup all OpenStack without PuppetDB in TripleO and in
 some other installers, we might want to keep this benefit.
 
 +100
 
 We're looking at using these puppet modules in a bit, but we're also a
 few steps away from getting rid of our puppetmaster and moving to a
 completely puppet apply based workflow. I would be double-plus
 sad-panda if we were not able to use the openstack puppet modules to
 do openstack because they'd been done in such as way as to require a
 puppetmaster or puppetdb.

100% agree.

Even if you had a puppetmaster and puppetdb, you would still end up in
this eventual consistency dance of puppet runs.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][operators] How to specify Keystone v3 credentials?

[Mike Dorman]

Cool, fair enough.  Pretty glad to hear that actually!


From: Colleen Murphy
Reply-To: OpenStack Development Mailing List (not for usage questions)
Date: Wednesday, May 6, 2015 at 5:31 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [puppet][operators] How to specify Keystone v3 
credentials?


On Wed, May 6, 2015 at 4:26 PM, Mike Dorman 
mdor...@godaddy.commailto:mdor...@godaddy.com wrote:
We also run all masterless/puppet apply.  And we just populate a bare
bones keystone.conf on any box that does not have keystone installed, but
Puppet needs to be able to create keystone resources.

Also agreed on avoiding puppetdb, for the same reasons.

(Something to note for those of us doing masterless today: there are plans
from Puppet to move more of the manifest compiling functionality to run
only in the puppet master process.  So at some point, it’s likely that
masterless setups may not be possible.)
I don't think that's true. I think making sure puppet apply works is a priority 
for them, just the implementation as they move to a C++-based agent has yet to 
be figured out.

Colleen

Mike
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] rabbit/kombu settings deprecations

[Mike Dorman]

I feel somewhat responsible for this whole thing, since I landed the first 
review that kicked off all this.  We had gone to a kilo oslo-messaging for 
RMQ improvements, which was what spurred me to patch in order to get rid 
of the deprecation warnings.  I should have actually validated it against 
Juno, known it would break, and called that out.  Sorry about that.  (On 
the other hand, thanks to Gael for hitting up all the other modules that I 
did not do.)

But, I have to say that I’m sympathetic with Matt on this.  We also more 
or less track the master branches, and have the same challenge.

Emilien’s idea below for a bot creating the backport cherry pick is 
intriguing.  Tbh, from a contributor’s perspective, the main reason I 
would not create the cherry pick review is 1) lack of time, and, 2) I’m 
tracking master so I (selfishly) don’t necessarily care about the stable 
branch.  If we had a bot that would automate some of this process, that 
would reduce the resistance somewhat.  But I have no idea what the 
effort/feasibility is of setting up such a thing.  Is there a way in 
Gerrit to make tags more visible when viewing a review?  Like checkboxes 
or something, rather than just having to know the tag and typing it in?

For me, personally, I would be more open to tracking stable branches, too, 
if the backports were better/faster.  Once I was on a stable branch, I 
would be more motivated to do the cherry picks/backports as well.  So 
maybe somewhat of a chicken-and-egg thing.

In any case, definitely a challenge that we should come to some decision 
on.  Then at least there’ll be consistent behavior, one way or another, 
going forward.

Mike

 




On 4/16/15, 12:42 PM, Emilien Macchi emil...@redhat.com wrote:



On 04/16/2015 02:15 PM, Clayton O'Neill wrote:
 On Thu, Apr 16, 2015 at 10:50 AM, Emilien Macchi emil...@redhat.com
 mailto:emil...@redhat.com wrote:
 
 We do our best now to backport what is backportable to stable/juno.
 
 
 This certainly has gotten much better, but I don't think it's 100% there
 yet either.  It's just a ton of work and we probably need better tooling
 around this to expect it to be as good as it should be.
  
 
 FWIW, even without rabbit/kombu topic, master won't work on Juno, 
there
 is plenty of things that are brought in Kilo.
 
 
 That may be the case in some areas, but we're using it without issues
 (until now) on Ubuntu with the features we need.
  
 
 My opinion is we should follow other projects that use stable 
branches
 with doing deprecation for one (or more?) release (currently our 
master)
 and then drop the deprecations after some time.
 
 So I would propose this policy:
 * for new features, patch master with backward compatibility
 
 
 Agreed, I think some of these might also be candidates for back port if
 they're new module features.  For example a new cinder backend that
 existed in the previous release might get back ported if they're just
 adding a new class.
  

A solution could be to add a tag in commits that can be backported?
Something like:

backport-juno
backport-icehouse

or just:
backport-icehouse

And the patch once merged would create the backport auto-magically with
a bot ?

We would have to add a rule in our policy, to ensure a patch has the tag
if needed (core-reviewers will have to take care to see if the tag
deserves to be here or not).
This is a proposal, it could be wrong at all.

 * backports relevant patches from master to stable branches (mainly
 bugs)
 
 Agreed.
  
 
 * in the case of rabbit or any update in OpenStack upstream, update
 master without backward compatibility, except if we accept to have 
a lot
 of if/else in our code, and a lot of backwards to support; I'm not 
in
 favor of that.
 
 
 I think I agree here also.  However, I'd like to see us avoid making
 breaking changes solely to avoid deprecation warnings until x amount of
 time after a release comes out.  If we're able to support some level of
 backwards compatibility, then it also makes upgrading between releases a
 lot easier.  Upgrading all of your packages, db schemas, etc is a lot
 less scary and easier to test than upgrading all that + every OpenStack
 puppet module you use at the same time.

Well, we also rely on OpenStack upstream (oslo, etc), that use to change
configuration parameters. But I agree with you, we should more take care
of this kind of changes.

 
 
_
_
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 

-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [nova] The strange case of osapi_compute_unique_server_name_scope

[Mike Dorman]

I can report that we do use this option (‘global' setting.)  We have to 
enforce name uniqueness for instances’ integration with some external 
systems (namely AD and Spacewalk) which require unique naming.

However, we also do some external name validation which I think 
effectively enforces uniqueness as well.  So if this were deprecated, I 
don’t know if it would directly affect us for our specific situation.

Other operators, anyone else using osapi_compute_unique_server_name_scope?

Mike





On 2/19/15, 3:18 AM, Matthew Booth mbo...@redhat.com wrote:

Nova contains a config variable osapi_compute_unique_server_name_scope.
Its help text describes it pretty well:

When set, compute API will consider duplicate hostnames invalid within
the specified scope, regardless of case. Should be empty, project or
global.

So, by default hostnames are not unique, but depending on this setting
they could be unique either globally or in the scope of a project.

Ideally a unique constraint would be enforced by the database but,
presumably because this is a config variable, that isn't the case here.
Instead it is enforced in code, but the code which does this predictably
races. My first attempt to fix this using the obvious SQL solution
appeared to work, but actually fails in MySQL as it doesn't support that
query structure[1][2]. SQLite and PostgreSQL do support it, but they
don't support the query structure which MySQL supports. Note that this
isn't just a syntactic thing. It looks like it's still possible to do
this if we compound the workaround with a second workaround, but I'm
starting to wonder if we'd be better fixing the design.

First off, do we need this config variable? Is anybody actually using
it? I suspect the answer's going to be yes, but it would be extremely
convenient if it's not.

Assuming this configurability is required, is there any way we can
instead use it to control a unique constraint in the db at service
startup? This would be something akin to a db migration. How do we
manage those?

Related to the above, I'm not 100% clear on which services run this
code. Is it possible for different services to have a different
configuration of this variable, and does that make sense? If so, that
would preclude a unique constraint in the db.

Thanks,

Matt

[1] Which has prompted me to get the test_db_api tests running on MySQL.
See this series if you're interested:
https://review.openstack.org/#/c/156299/

[2] For specifics, see my ramblings here:
https://review.openstack.org/#/c/141115/7/nova/db/sqlalchemy/api.py,cm
line 2547
-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
GPG ID:  D33C3490
GPG FPR: 3733 612D 2D05 5458 8A8A 1600 3441 EA19 D33C 3490

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev