Re: [openstack-dev] [Openstack-sigs] [goals][tc][ptl][uc] starting goal selection for T series

[Mathieu Gagné]

+1 Yes please!

--
Mathieu

On Wed, Sep 26, 2018 at 2:56 PM Tim Bell  wrote:
>
>
> Doug,
>
> Thanks for raising this. I'd like to highlight the goal "Finish moving legacy 
> python-*client CLIs to python-openstackclient" from the etherpad and propose 
> this for a T/U series goal.
>
> To give it some context and the motivation:
>
> At CERN, we have more than 3000 users of the OpenStack cloud. We write an 
> extensive end user facing documentation which explains how to use the 
> OpenStack along with CERN specific features (such as workflows for requesting 
> projects/quotas/etc.).
>
> One regular problem we come across is that the end user experience is 
> inconsistent. In some cases, we find projects which are not covered by the 
> unified OpenStack client (e.g. Manila). In other cases, there are subsets of 
> the function which require the native project client.
>
> I would strongly support a goal which targets
>
> - All new projects should have the end user facing functionality fully 
> exposed via the unified client
> - Existing projects should aim to close the gap within 'N' cycles (N to be 
> defined)
> - Many administrator actions would also benefit from integration (reader 
> roles are end users too so list and show need to be covered too)
> - Users should be able to use a single openrc for all interactions with the 
> cloud (e.g. not switch between password for some CLIs and Kerberos for OSC)
>
> The end user perception of a solution will be greatly enhanced by a single 
> command line tool with consistent syntax and authentication framework.
>
> It may be a multi-release goal but it would really benefit the cloud 
> consumers and I feel that goals should include this audience also.
>
> Tim
>
> -Original Message-
> From: Doug Hellmann 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Date: Wednesday, 26 September 2018 at 18:00
> To: openstack-dev , openstack-operators 
> , openstack-sigs 
> 
> Subject: [openstack-dev] [goals][tc][ptl][uc] starting goal selection for T   
>   series
>
> It's time to start thinking about community-wide goals for the T series.
>
> We use community-wide goals to achieve visible common changes, push for
> basic levels of consistency and user experience, and efficiently improve
> certain areas where technical debt payments have become too high -
> across all OpenStack projects. Community input is important to ensure
> that the TC makes good decisions about the goals. We need to consider
> the timing, cycle length, priority, and feasibility of the suggested
> goals.
>
> If you are interested in proposing a goal, please make sure that before
> the summit it is described in the tracking etherpad [1] and that you
> have started a mailing list thread on the openstack-dev list about the
> proposal so that everyone in the forum session [2] has an opportunity to
> consider the details.  The forum session is only one step in the
> selection process. See [3] for more details.
>
> Doug
>
> [1] https://etherpad.openstack.org/p/community-goals
> [2] https://www.openstack.org/summit/berlin-2018/vote-for-speakers#/22814
> [3] https://governance.openstack.org/tc/goals/index.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> openstack-sigs mailing list
> openstack-s...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-sigs

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3] mixed versions?

[Mathieu Gagné]

On Fri, Sep 14, 2018 at 4:22 PM, Jim Rollenhagen  
wrote:
> On Thu, Sep 13, 2018 at 9:46 PM, Mathieu Gagné  wrote:
>>
>> On Thu, Sep 13, 2018 at 10:14 PM, Doug Hellmann 
>> wrote:
>> > Excerpts from Mathieu Gagné's message of 2018-09-13 20:09:12 -0400:
>> >> On Thu, Sep 13, 2018 at 7:41 PM, Doug Hellmann 
>> >> wrote:
>> >> > Excerpts from Mathieu Gagné's message of 2018-09-13 14:12:56 -0400:
>> >> >> On Wed, Sep 12, 2018 at 2:04 PM, Doug Hellmann
>> >> >>  wrote:
>> >> >> >
>> >> >> > IIRC, we also talked about not supporting multiple versions of
>> >> >> > python on a given node, so all of the services on a node would
>> >> >> > need
>> >> >> > to be upgraded together.
>> >> >> >
>> >> >>
>> >> >> Will services support both versions at some point for the same
>> >> >> OpenStack release? Or is it already the case?
>> >> >>
>> >> >> I would like to avoid having to upgrade Nova, Neutron and Ceilometer
>> >> >> at the same time since all end up running on a compute node and
>> >> >> sharing the same python version.
>> >> >
>> >> > We need to differentiate between what the upstream community supports
>> >> > and what distros support. In the meeting in Vancouver, we said that
>> >> > the community would support upgrading all of the services on a
>> >> > single node together. Distros may choose to support more complex
>> >> > configurations if they choose, and I'm sure patches related to any
>> >> > bugs would be welcome.
>> >>
>> >> We maintain and build our own packages with virtualenv. We aren't
>> >> bound to distribution packages.
>> >
>> > OK, I should rephrase then. I'm talking about the limits on the
>> > tests that I think are useful and reasonable to run upstream and
>> > for the community to support.
>> >
>> >> > But I don't think we can ask the community
>> >> > to support the infinite number of variations that would occur if
>> >> > we said we would test upgrading some services independently of
>> >> > others (unless I'm mistaken, we don't even do that for services
>> >> > all using the same version of python 2, today).
>> >>
>> >> This contradicts what I heard in fishbowl sessions from core reviewers
>> >> and read on IRC.
>> >> People were under the false impression that you need to upgrade
>> >> OpenStack in lock steps when in fact, it has never been the case.
>> >> You should be able to upgrade services individually.
>> >>
>> >> Has it changed since?
>> >
>> > I know that some deployments do upgrade components separately, and
>> > it works in some configurations.  All we talked about in Vancouver
>> > was how we would test upgrading python 2 to python 3, and given
>> > that the community has never, as far as I know, run upgrade tests
>> > in CI that staggered the upgrades of components on a given node,
>> > there seemed no reason to add those tests just for the python 2 to
>> > 3 case.
>> >
>> > Perhaps someone on the QA team can correct me if I'm wrong about the
>> > history there.
>> >
>>
>> Or maybe it's me that misinterpreted the actual impact of not
>> supported 2 versions of Python at the same time.
>>
>> Lets walk through an actual upgrade scenario.
>>
>> I suppose the migration to Python 3 will happen around Stein and
>> therefore affects people upgrading from Rocky to Stein. At this point,
>> an operator should already be running Ubuntu Bionic which supports
>> both Python 2.7 and 3.6.
>>
>> If that operator is using virtualenv (and not distribution packages),
>> it's only a matter a building new virtualenv using Python 3.6 for
>> Stein instead. This means installing both Python 2.7/3.6 on the same
>> node should be enough to upgrade and switch to Python 3.6 on a per
>> project/service basis.
>>
>> My main use case is with the compute node which has multiple services
>> running. Come to think of it, it's a lot less impactful than I
>> thought.
>>
>> Let me know if I got some details wrong. But if the steps are similar
>> to what I described above, I no longer have concerns or objections.
>
>
>
> The plan is to maintain suppor

Re: [openstack-dev] [goals][python3] mixed versions?

[Mathieu Gagné]

On Thu, Sep 13, 2018 at 10:14 PM, Doug Hellmann  wrote:
> Excerpts from Mathieu Gagné's message of 2018-09-13 20:09:12 -0400:
>> On Thu, Sep 13, 2018 at 7:41 PM, Doug Hellmann  wrote:
>> > Excerpts from Mathieu Gagné's message of 2018-09-13 14:12:56 -0400:
>> >> On Wed, Sep 12, 2018 at 2:04 PM, Doug Hellmann  
>> >> wrote:
>> >> >
>> >> > IIRC, we also talked about not supporting multiple versions of
>> >> > python on a given node, so all of the services on a node would need
>> >> > to be upgraded together.
>> >> >
>> >>
>> >> Will services support both versions at some point for the same
>> >> OpenStack release? Or is it already the case?
>> >>
>> >> I would like to avoid having to upgrade Nova, Neutron and Ceilometer
>> >> at the same time since all end up running on a compute node and
>> >> sharing the same python version.
>> >
>> > We need to differentiate between what the upstream community supports
>> > and what distros support. In the meeting in Vancouver, we said that
>> > the community would support upgrading all of the services on a
>> > single node together. Distros may choose to support more complex
>> > configurations if they choose, and I'm sure patches related to any
>> > bugs would be welcome.
>>
>> We maintain and build our own packages with virtualenv. We aren't
>> bound to distribution packages.
>
> OK, I should rephrase then. I'm talking about the limits on the
> tests that I think are useful and reasonable to run upstream and
> for the community to support.
>
>> > But I don't think we can ask the community
>> > to support the infinite number of variations that would occur if
>> > we said we would test upgrading some services independently of
>> > others (unless I'm mistaken, we don't even do that for services
>> > all using the same version of python 2, today).
>>
>> This contradicts what I heard in fishbowl sessions from core reviewers
>> and read on IRC.
>> People were under the false impression that you need to upgrade
>> OpenStack in lock steps when in fact, it has never been the case.
>> You should be able to upgrade services individually.
>>
>> Has it changed since?
>
> I know that some deployments do upgrade components separately, and
> it works in some configurations.  All we talked about in Vancouver
> was how we would test upgrading python 2 to python 3, and given
> that the community has never, as far as I know, run upgrade tests
> in CI that staggered the upgrades of components on a given node,
> there seemed no reason to add those tests just for the python 2 to
> 3 case.
>
> Perhaps someone on the QA team can correct me if I'm wrong about the
> history there.
>

Or maybe it's me that misinterpreted the actual impact of not
supported 2 versions of Python at the same time.

Lets walk through an actual upgrade scenario.

I suppose the migration to Python 3 will happen around Stein and
therefore affects people upgrading from Rocky to Stein. At this point,
an operator should already be running Ubuntu Bionic which supports
both Python 2.7 and 3.6.

If that operator is using virtualenv (and not distribution packages),
it's only a matter a building new virtualenv using Python 3.6 for
Stein instead. This means installing both Python 2.7/3.6 on the same
node should be enough to upgrade and switch to Python 3.6 on a per
project/service basis.

My main use case is with the compute node which has multiple services
running. Come to think of it, it's a lot less impactful than I
thought.

Let me know if I got some details wrong. But if the steps are similar
to what I described above, I no longer have concerns or objections.

I think the only people who could be concerned is those doing rolling
upgrading, which could impact RPC message encoding as described by
Thomas. But you are already addressing it so I will just read and see
where this is going.

Thanks

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3] mixed versions?

[Mathieu Gagné]

On Thu, Sep 13, 2018 at 7:41 PM, Doug Hellmann  wrote:
> Excerpts from Mathieu Gagné's message of 2018-09-13 14:12:56 -0400:
>> On Wed, Sep 12, 2018 at 2:04 PM, Doug Hellmann  wrote:
>> >
>> > IIRC, we also talked about not supporting multiple versions of
>> > python on a given node, so all of the services on a node would need
>> > to be upgraded together.
>> >
>>
>> Will services support both versions at some point for the same
>> OpenStack release? Or is it already the case?
>>
>> I would like to avoid having to upgrade Nova, Neutron and Ceilometer
>> at the same time since all end up running on a compute node and
>> sharing the same python version.
>
> We need to differentiate between what the upstream community supports
> and what distros support. In the meeting in Vancouver, we said that
> the community would support upgrading all of the services on a
> single node together. Distros may choose to support more complex
> configurations if they choose, and I'm sure patches related to any
> bugs would be welcome.

We maintain and build our own packages with virtualenv. We aren't
bound to distribution packages.


> But I don't think we can ask the community
> to support the infinite number of variations that would occur if
> we said we would test upgrading some services independently of
> others (unless I'm mistaken, we don't even do that for services
> all using the same version of python 2, today).

This contradicts what I heard in fishbowl sessions from core reviewers
and read on IRC.
People were under the false impression that you need to upgrade
OpenStack in lock steps when in fact, it has never been the case.
You should be able to upgrade services individually.

Has it changed since?

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3] mixed versions?

[Mathieu Gagné]

On Wed, Sep 12, 2018 at 2:04 PM, Doug Hellmann  wrote:
>
> IIRC, we also talked about not supporting multiple versions of
> python on a given node, so all of the services on a node would need
> to be upgraded together.
>

Will services support both versions at some point for the same
OpenStack release? Or is it already the case?

I would like to avoid having to upgrade Nova, Neutron and Ceilometer
at the same time since all end up running on a compute node and
sharing the same python version.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] A culture change (nitpicking)

[Mathieu Gagné]

Hi Julia,

Thanks for the follow up on this topic.

On Tue, May 29, 2018 at 6:55 AM, Julia Kreger
 wrote:
>
> These things are not just frustrating, but also very inhibiting for
> part time contributors such as students who may also be time limited.
> Or an operator who noticed something that was clearly a bug and that
> put forth a very minor fix and doesn't have the time to revise it over
> and over.
>

What I found frustrating is receiving *only* nitpicks, addressing them
to only receive more nitpicks (sometimes from the same reviewer) with
no substantial review on the change itself afterward.
I wouldn't mind addressing nitpicks if more substantial reviews were
made in a timely fashion.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ironic] ironic_host_manager and baremetal scheduler options removal

[Mathieu Gagné]

On Wed, May 2, 2018 at 1:39 PM, Matt Riedemann  wrote:
>
> I know you're just one case, but I don't know how many people are really
> running the CachingScheduler with ironic either, so it might be rare. It
> would be nice to get other operator input here, like I'm guessing CERN has
> their cells carved up so that certain cells are only serving baremetal
> requests while other cells are only VMs?

I found FilterScheduler to be near impossible to use with Ironic due
to the huge number of hypervisors it had to handle.
Using CachingScheduler made a huge difference, like day and night.

> FWIW, I think we can also backport the data migration CLI to stable branches
> once we have it available so you can do your migration in let's say Queens
> before getting to Rocky.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ironic] ironic_host_manager and baremetal scheduler options removal

[Mathieu Gagné]

On Wed, May 2, 2018 at 12:49 PM, Matt Riedemann <mriede...@gmail.com> wrote:
> On 5/2/2018 11:40 AM, Mathieu Gagné wrote:
>>
>> What's the state of caching_scheduler which could still be using those
>> configs?
>
>
> The CachingScheduler has been deprecated since Pike [1]. We discussed the
> CachingScheduler at the Rocky PTG in Dublin [2] and have a TODO to write a
> nova-manage data migration tool to create allocations in Placement for
> instances that were scheduled using the CachingScheduler (since Pike) which
> don't have their own resource allocations set in Placement (remember that
> starting in Pike the FilterScheduler started creating allocations in
> Placement rather than the ResourceTracker in nova-compute).
>
> If you're running computes that are Ocata or Newton, then the
> ResourceTracker in the nova-compute service should be creating the
> allocations in Placement for you, assuming you have the compute service
> configured to talk to Placement (optional in Newton, required in Ocata).
>
> [1] https://review.openstack.org/#/c/492210/
> [2] https://etherpad.openstack.org/p/nova-ptg-rocky-placement

If one can still run CachingScheduler (even if it's deprecated), I
think we shouldn't remove the above options.
As you can end up with a broken setup and IIUC no way to migrate to
placement since migration script has yet to be written.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ironic] ironic_host_manager and baremetal scheduler options removal

[Mathieu Gagné]

What's the state of caching_scheduler which could still be using those configs?

Mathieu

On Wed, May 2, 2018 at 12:25 PM, Matt Riedemann  wrote:
> The baremetal scheduling options were deprecated in Pike [1] and the
> ironic_host_manager was deprecated in Queens [2] and is now being removed
> [3]. Deployments must use resource classes now for baremetal scheduling. [4]
>
> The large host subset size value is also no longer needed. [5]
>
> I've gone through all of the references to "ironic_host_manager" that I
> could find in codesearch.o.o and updated projects accordingly [6].
>
> Please reply ASAP to this thread and/or [3] if you have issues with this.
>
> [1] https://review.openstack.org/#/c/493052/
> [2] https://review.openstack.org/#/c/521648/
> [3] https://review.openstack.org/#/c/565805/
> [4]
> https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html#scheduling-based-on-resource-classes
> [5] https://review.openstack.org/565736/
> [6]
> https://review.openstack.org/#/q/topic:exact-filters+(status:open+OR+status:merged)
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova] Default scheduler filters survey

[Mathieu Gagné]

Hi Dave,

On Tue, May 1, 2018 at 4:30 AM, Dave Holland <d...@sanger.ac.uk> wrote:
> On Mon, Apr 30, 2018 at 12:41:21PM -0400, Mathieu Gagné wrote:
>> Weighers for baremetal cells:
>> * ReservedHostForTenantWeigher [7]
> ...
>> [7] Used to favor reserved host over non-reserved ones based on project.
>
> Hello Mathieu,
>
> we are considering writing something like this, for virtual machines not
> for baremetal. Our use case is that a project buying some compute
> hardware is happy for others to use it, but when the compute "owner"
> wants sole use of it, other projects' instances must be migrated off or
> killed; a scheduler weigher like this might help us to minimise the
> number of instances needing migration or termination at that point.
> Would you be willing to share your source code please?
>

I'm not sure how battle-tested this code is to be honest but here it is:
https://gist.github.com/mgagne/659ca02e63779802de6f7aec8cda612a

I had to merge 2 files in one (the weigher and the conf) so I'm not
sure if it still works but I think you will get the idea.

To use it, you need to define the "reserved_for_tenant_id" Ironic node
property with the project ID to reserve it. (through Ironic API)

This code also assumes you already filtered out hosts which are
reserved for a different tenant. I included that code in the gist too.

On a side note, our technicians generally use the forced host feature
of Nova to target specific Ironic nodes:
https://docs.openstack.org/nova/pike/admin/availability-zones.html

But if the customer buys and reserves some machines, he should get
them first before the ones in the "public pool".

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova] Default scheduler filters survey

[Mathieu Gagné]

On Mon, Apr 30, 2018 at 1:06 PM, Jay Pipes  wrote:
> Mathieu,
>
> How do you handle issues where compute nodes are associated with multiple
> aggregates and both aggregates have different values for a particular filter
> key?
>
> Is that a human-based validation process to ensure you don't have that
> situation?
>

It's human-based and we are fine with it.

We also have automatic reports which generate stats on those aggregates.
You will visually see it if some hosts are part of multiple aggregates.

If we need an intersection of 2 aggregates, we create a new one and
use it instead.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [nova] Default scheduler filters survey

[Mathieu Gagné]

Hi,

On Sun, Apr 29, 2018 at 5:29 PM, Ed Leafe  wrote:
> On Apr 29, 2018, at 1:34 PM, Artom Lifshitz  wrote:
>>
>> Based on that, we can definitely say that SameHostFilter and
>> DifferentHostFilter do *not* belong in the defaults. In fact, we got
>> our defaults pretty spot on, based on this admittedly very limited
>> dataset. The only frequently occurring filter that's not in our
>> defaults is AggregateInstanceExtraSpecsFilter.
>
> Another data point that might be illuminating is: how many sites use a custom 
> (i.e., not in-tree) filter or weigher? One of the original design tenets of 
> the scheduler was that we did not want to artificially limit what people 
> could use to control their deployments, but inside of Nova there is a lot of 
> confusion as to whether anyone is using anything but the included filters.
>
> So - does anyone out there rely on a filter and/or weigher that they wrote 
> themselves, and maintain outside of OpenStack?

Yes and we have a bunch.

Here are our filters and weighers with explanations.

Filters for cells:
* InstanceTypeClassFilter [0]

Filters for cloud/virtual cells:
* RetryFilter
* AvailabilityZoneFilter
* RamFilter
* ComputeFilter
* AggregateCoreFilter
* ImagePropertiesFilter
* AggregateImageOsTypeIsolationFilter [1]
* AggregateInstanceExtraSpecsFilter
* AggregateProjectsIsolationFilter [2]

Weighers for cloud/virtual cells:
* MetricsWeigher
* AggregateRAMWeigher [3]

Filters for baremetal cells:
* ComputeFilter
* NetworkModelFilter [4]
* TenantFilter [5]
* UserFilter [6]
* RetryFilter
* AvailabilityZoneFilter
* ComputeCapabilitiesFilter
* ImagePropertiesFilter
* ExactRamFilter
* ExactDiskFilter
* ExactCoreFilter

Weighers for baremetal cells:
* ReservedHostForTenantWeigher [7]
* ReservedHostForUserWeigher [8]

[0] Used to scheduler instances based on flavor class found in
extra_specs (virtual/baremetal)
[1] Allows to properly isolated hosts for licensing purposes.
The upstream filter is not strict as per bugs/reviews/specs:
* https://bugs.launchpad.net/nova/+bug/1293444
* https://bugs.launchpad.net/nova/+bug/1677217
* https://review.openstack.org/#/c/56420/
* https://review.openstack.org/#/c/85399/
Our custom implementation for Mitaka:
https://gist.github.com/mgagne/462e7fa8417843055aa6da7c5fd51c00
[2] Similar filter to AggregateImageOsTypeIsolationFilter but for projects.
Our custom implementation for Mitaka:
https://gist.github.com/mgagne/d729ccb512b0434568ffb094441f643f
[3] Allows to change stacking behavior based on the 'ram_weight_multiplier'
aggregate key. (emptiest/fullest)
Our custom implementation for Mitaka:
https://gist.github.com/mgagne/65f033cbc5fdd4c8d1f45e90c943a5f4
[4] Used to filter Ironic nodes based on supported network models as requested
by flavor extra_specs. We support JIT network configuration (flat/bond) and
need to know which nodes support what network models beforehand.
[5] Used to filter Ironic nodes based on the 'reserved_for_tenant_id'
Ironic node property.
This is used to reserve Ironic node to specific projects.
Some customers order lot of machines in advance. We reserve those for them.
[6] Used to filter Ironic nodes based on the 'reserved_for_user_id'
Ironic node property.
This is mainly used when enrolling existing nodes already living
on a different system.
We reserve the node to a special internal user so the customer
cannot reserve
the node by mistake until the process is completed.
Latest version of Nova dropped user_id from RequestSpec. We had to
add it back.
[7] Used to favor reserved host over non-reserved ones based on project.
[8] Used to favor reserved host over non-reserved ones based on user.
Latest version of Nova dropped user_id from RequestSpec. We had to
add it back.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] campaign question: How can we make contributing to OpenStack easier?

[Mathieu Gagné]

On Tue, Apr 24, 2018 at 3:51 PM, Fox, Kevin M  wrote:
> I support 12 factor. But 12 factor only works if you can commit to always 
> deploying on top of 12 factor tools. If OpenStack committed to only ever 
> deploying api services on k8s then my answer might be different. but so far 
> has been unable to do that. Barring that, I think simplifying the operators 
> life so you get more users/contributors has priority over pure 12 factor 
> ideals.
>
> It also is about getting Project folks working together to see how their 
> parts fit (or not) in the greater constilation. Just writing a document on 
> how you could fit things together doesn't show the kinds of suffering that 
> actually integrating it into a finished whole could show.
>
> Either way though, I think a unified db-sync would go a long way to making 
> OpenStack easier to maintain as an Operator.

Yes please. Or any task that's remotely similar.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [barbican] [glance] [ironic] [neutron] [tacker] [tc] policy in code goal

[Mathieu Gagné]

On Wed, Jan 31, 2018 at 12:16 PM, Lance Bragstad  wrote:
> Hey folks,
>
> The tracking tool for the policy-and-docs-in-code goal for Queens [0]
> lists a couple projects remaining for the goal [1].  I wanted to start a
> discussion with said projects to see how we want to go about the work in
> the future, we have a couple of options.
>
> I can update the document the goal document saying the work is still
> underway for those projects. We can also set aside time at the PTG to
> finish up that work if people would like more help. This might be
> something we can leverage the baremetal/vm room for if we get enough
> interest [2].
>
> I want to get the discussion rolling if there is something we need to
> coordinate for the PTG. Thoughts?
>
> Thanks,
>
> Lance
>

As an operator, I can't wait for Glance and Neutron to complete this goal.
The policy is code is *very* useful when you do heavy work in policy.
Thanks to all working toward that goal.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][ceilometer] versioned notifications coverage

[Mathieu Gagné]

On Tue, Jan 30, 2018 at 10:19 AM, Matt Riedemann  wrote:
>
> Gibi's burndown chart is here to see what's remaining:
>
> http://burndown.peermore.com/nova-notification/
>
> And this is the list of what's already done:
>
> https://docs.openstack.org/nova/latest/reference/notifications.html#versioned-notification-samples
>

So I just discovered this documentation. It's awesome! Great job to
all the ones involved!
Now I wish we had the same in all projects. =)

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova]Nova rescue inject pasword failed

[Mathieu Gagné]

On Mon, Jan 29, 2018 at 4:57 AM, Matthew Booth  wrote:
> On 29 January 2018 at 09:27, 李杰  wrote:
>>
>>  Hi,all:
>>   I want to access to my instance under rescue state using
>> temporary password which nova rescue gave me.But this password doesn't work.
>> Can I ask how this password is injected to instance? I can't find any
>> specification how is it done.I saw the code about rescue,But it displays the
>> password has inject.
>>   I use the libvirt as the virt driver. The web said to
>> set"[libvirt]inject_password=true",but it didn't work. Is it a bug?Can you
>> give me some advice?Help in troubleshooting this issue will be appreciated.
>
>
> Ideally your rescue image will support cloud-init and you would use a config
> disk.
>
> But to reiterate, ideally your rescue image would support cloud-init and you
> would use a config disk.
>
> Matt
> --
> Matthew Booth
> Red Hat OpenStack Engineer, Compute DFG
>

Just so you know, cloud-init does not read/support the admin_pass
injected in the config-drive:
https://bugs.launchpad.net/cloud-init/+bug/1236883

Known bug for years and no fix has been approved yet for various
non-technical reasons.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][all] New Zuul Depends-On syntax

[Mathieu Gagné]

On Thu, Jan 25, 2018 at 7:08 PM, James E. Blair <cor...@inaugust.com> wrote:
> Mathieu Gagné <mga...@calavera.ca> writes:
>
>> On Thu, Jan 25, 2018 at 3:55 PM, Ben Nemec <openst...@nemebean.com> wrote:
>>>
>>>
>>> I'm curious what this means as far as best practices for inter-patch
>>> references.  In the past my understanding was the the change id was
>>> preferred, both because if gerrit changed its URL format the change id links
>>> would be updated appropriately, and also because change ids can be looked up
>>> offline in git commit messages.  Would that still be the case for everything
>>> except depends-on now?
>
> Yes, that's a down-side of URLs.  I personally think it's fine to keep
> using change-ids for anything other than Depends-On, though in many of
> those cases the commit sha may work as well.
>
>> That's my concern too. Also AFAIK, Change-Id is branch agnostic. This
>> means you can more easily cherry-pick between branches without having
>> to change the URL to match the new branch for your dependencies.
>
> Yes, there is a positive and negative aspect to this issue.
>
> On the one hand, for those times where it was convenient to say "depend
> on this change in all its forms across all branches of all projects",
> one must now add a URL for each.
>
> On the other hand, with URLs, it is now possible to indicate that a
> change specifically depends on another change targeted to one branch, or
> targeted to several branches.  Simply list each URL (or don't) as
> appropriate.  That wasn't possible before -- it wall all or none.
>
> -Jim
>

> The old syntax will continue to work for a while

I still believe Change-Id should be supported and not removed as
suggested. The use of URL assumes you have access to Gerrit to fetch
more information about the change.
This might not always be true or possible, especially when Gerrit is
kept private and only the git repository is replicated publicly and
you which to cherry-pick something (and its dependencies) from it.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][all] New Zuul Depends-On syntax

[Mathieu Gagné]

On Thu, Jan 25, 2018 at 3:55 PM, Ben Nemec  wrote:
>
>
> I'm curious what this means as far as best practices for inter-patch
> references.  In the past my understanding was the the change id was
> preferred, both because if gerrit changed its URL format the change id links
> would be updated appropriately, and also because change ids can be looked up
> offline in git commit messages.  Would that still be the case for everything
> except depends-on now?
>

That's my concern too. Also AFAIK, Change-Id is branch agnostic. This
means you can more easily cherry-pick between branches without having
to change the URL to match the new branch for your dependencies.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] heads up to users of Aggregate[Core|Ram|Disk]Filter: behavior change in >= Ocata

[Mathieu Gagné]

Hi Chris,

On Fri, Jan 19, 2018 at 10:21 AM, Chris Friesen
 wrote:
>
> The existing mechanisms to control aggregate membership will still work, so
> the remaining issue is how to control the allocation ratios.
>
> What about implementing a new HTTP API call (as a local private patch) to
> set the allocation ratios for a given host?  This would only be valid for
> your scenario where a given host is only present in a single aggregate, but
> it would allow your techs to modify the ratios.

While I agree that we can implement something in a private patch, this
is something we are trying very hard to much away from.
If Nova proposes using the placement API for such use cases, I think
it should also provides the same features as the replaced solutions
because people could have relied on those.
And suggesting a configuration management system is enough was
unfortunately a false assumption.
I still have to figure out how a workflow based on placement will be
feasible for us.
But as I said, the lack of granular ACLs is a huge concern for us and
I don't know how it can be addressed in the near future.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] heads up to users of Aggregate[Core|Ram|Disk]Filter: behavior change in >= Ocata

[Mathieu Gagné]

On Thu, Jan 18, 2018 at 5:19 PM, Jay Pipes <jaypi...@gmail.com> wrote:
> On 01/18/2018 03:54 PM, Mathieu Gagné wrote:
>>
>> Hi,
>>
>> On Tue, Jan 16, 2018 at 4:24 PM, melanie witt <melwi...@gmail.com> wrote:
>>>
>>> Hello Stackers,
>>>
>>> This is a heads up to any of you using the AggregateCoreFilter,
>>> AggregateRamFilter, and/or AggregateDiskFilter in the filter scheduler.
>>> These filters have effectively allowed operators to set overcommit ratios
>>> per aggregate rather than per compute node in <= Newton.
>>>
>>> Beginning in Ocata, there is a behavior change where aggregate-based
>>> overcommit ratios will no longer be honored during scheduling. Instead,
>>> overcommit values must be set on a per compute node basis in nova.conf.
>>>
>>> Details: as of Ocata, instead of considering all compute nodes at the
>>> start
>>> of scheduler filtering, an optimization has been added to query resource
>>> capacity from placement and prune the compute node list with the result
>>> *before* any filters are applied. Placement tracks resource capacity and
>>> usage and does *not* track aggregate metadata [1]. Because of this,
>>> placement cannot consider aggregate-based overcommit and will exclude
>>> compute nodes that do not have capacity based on per compute node
>>> overcommit.
>>>
>>> How to prepare: if you have been relying on per aggregate overcommit,
>>> during
>>> your upgrade to Ocata, you must change to using per compute node
>>> overcommit
>>> ratios in order for your scheduling behavior to stay consistent.
>>> Otherwise,
>>> you may notice increased NoValidHost scheduling failures as the
>>> aggregate-based overcommit is no longer being considered. You can safely
>>> remove the AggregateCoreFilter, AggregateRamFilter, and
>>> AggregateDiskFilter
>>> from your enabled_filters and you do not need to replace them with any
>>> other
>>> core/ram/disk filters. The placement query takes care of the
>>> core/ram/disk
>>> filtering instead, so CoreFilter, RamFilter, and DiskFilter are
>>> redundant.
>>>
>>> Thanks,
>>> -melanie
>>>
>>> [1] Placement has been a new slate for resource management and prior to
>>> placement, there were conflicts between the different methods for setting
>>> overcommit ratios that were never addressed, such as, "which value to
>>> take
>>> if a compute node has overcommit set AND the aggregate has it set? Which
>>> takes precedence?" And, "if a compute node is in more than one aggregate,
>>> which overcommit value should be taken?" So, the ambiguities were not
>>> something that was desirable to bring forward into placement.
>>
>>
>> So we are a user of this feature and I do have some questions/concerns.
>>
>> We use this feature to segregate capacity/hosts based on CPU
>> allocation ratio using aggregates.
>> This is because we have different offers/flavors based on those
>> allocation ratios. This is part of our business model.
>> A flavor extra_specs is use to schedule instances on appropriate hosts
>> using AggregateInstanceExtraSpecsFilter.
>
>
> The AggregateInstanceExtraSpecsFilter will continue to work, but this filter
> is run *after* the placement service would have already eliminated compute
> node records due to placement considering the allocation ratio set for the
> compute node provider's inventory records.

Ok. Does it mean I will have to use something else to properly filter
compute nodes based on flavor?
Is there a way for a compute node to expose some arbitrary
feature/spec instead and still use flavor extra_specs to filter?
(I still have to read on placement API)

I don't mind migrating out of aggregates but I need to find a way to
make it "self service" through the API with granular control like
aggregates used to offer.
We won't be giving access to our configuration manager to our
technicians and even less direct access to the database.
I see that you are suggesting using the placement API below, see my
comments below.


>> Our setup has a configuration management system and we use aggregates
>> exclusively when it comes to allocation ratio.
>
>
> Yes, that's going to be a problem. You will need to use your configuration
> management system to write the nova.CONF.XXX_allocation_ratio configuration
> option values appropriately for each compute node.

Yes, that's my understanding and which is a concern for us.


>> We do not re

Re: [openstack-dev] [nova] heads up to users of Aggregate[Core|Ram|Disk]Filter: behavior change in >= Ocata

[Mathieu Gagné]

Hi,

On Tue, Jan 16, 2018 at 4:24 PM, melanie witt  wrote:
> Hello Stackers,
>
> This is a heads up to any of you using the AggregateCoreFilter,
> AggregateRamFilter, and/or AggregateDiskFilter in the filter scheduler.
> These filters have effectively allowed operators to set overcommit ratios
> per aggregate rather than per compute node in <= Newton.
>
> Beginning in Ocata, there is a behavior change where aggregate-based
> overcommit ratios will no longer be honored during scheduling. Instead,
> overcommit values must be set on a per compute node basis in nova.conf.
>
> Details: as of Ocata, instead of considering all compute nodes at the start
> of scheduler filtering, an optimization has been added to query resource
> capacity from placement and prune the compute node list with the result
> *before* any filters are applied. Placement tracks resource capacity and
> usage and does *not* track aggregate metadata [1]. Because of this,
> placement cannot consider aggregate-based overcommit and will exclude
> compute nodes that do not have capacity based on per compute node
> overcommit.
>
> How to prepare: if you have been relying on per aggregate overcommit, during
> your upgrade to Ocata, you must change to using per compute node overcommit
> ratios in order for your scheduling behavior to stay consistent. Otherwise,
> you may notice increased NoValidHost scheduling failures as the
> aggregate-based overcommit is no longer being considered. You can safely
> remove the AggregateCoreFilter, AggregateRamFilter, and AggregateDiskFilter
> from your enabled_filters and you do not need to replace them with any other
> core/ram/disk filters. The placement query takes care of the core/ram/disk
> filtering instead, so CoreFilter, RamFilter, and DiskFilter are redundant.
>
> Thanks,
> -melanie
>
> [1] Placement has been a new slate for resource management and prior to
> placement, there were conflicts between the different methods for setting
> overcommit ratios that were never addressed, such as, "which value to take
> if a compute node has overcommit set AND the aggregate has it set? Which
> takes precedence?" And, "if a compute node is in more than one aggregate,
> which overcommit value should be taken?" So, the ambiguities were not
> something that was desirable to bring forward into placement.

So we are a user of this feature and I do have some questions/concerns.

We use this feature to segregate capacity/hosts based on CPU
allocation ratio using aggregates.
This is because we have different offers/flavors based on those
allocation ratios. This is part of our business model.
A flavor extra_specs is use to schedule instances on appropriate hosts
using AggregateInstanceExtraSpecsFilter.

Our setup has a configuration management system and we use aggregates
exclusively when it comes to allocation ratio.
We do not rely on cpu_allocation_ratio config in nova-scheduler or nova-compute.
One of the reasons is we do not wish to have to
update/package/redeploy our configuration management system just to
add one or multiple compute nodes to an aggregate/capacity pool.
This means anyone (likely an operator or other provisioning
technician) can perform this action without having to touch or even
know about our configuration management system.
We can also transfer capacity from one aggregate to another if there
is a need, again, using aggregate memberships. (we do "evacuate" the
node if there are instances on it)
Our capacity monitoring is based on aggregate memberships and this
offer an easy overview of the current capacity. Note that a host can
be in one and only one aggregate in our setup.

What's the migration path for us?

My understanding is that we will now be forced to have people rely on
our configuration management system (which they don't have access to)
to perform simple task we used to be able to do through the API.
I find this unfortunate and I would like to be offered an alternative
solution as the current proposed solution is not acceptable for us.
We are loosing "agility" in our operational tasks.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Boot from volume 's instance of rebuild operation

[Mathieu Gagné]

On Wed, Dec 6, 2017 at 9:13 PM, Jaze Lee  wrote:
> 2017-12-07 7:12 GMT+08:00 Matt Riedemann :
>> On 12/6/2017 2:11 AM, 李杰 wrote:
>>>
>>> Hi,all
>>>
>>> Now the boot from volume 's instance of rebuild operation has a
>>> problem.For example,after the rebuild operation,the instance 's root disk is
>>> not replace.
>>> I found the reason is that when we use the "_build_resources" function to
>>> prepare source,it obtains the block devices according to the previous
>>> instance 's uuid and attaches them to instance.So boot from volume 's
>>> instance of rebuild operation doesn't update data.
>>> To solve it,I plan to use CLI 's "metadata" option,to increase a
>>> key name "source_type".The "source_type" includes "snapshot" and "image".We
>>> can judge from "source_type".If the "source_type" is "snapshot",we can
>>> transform the given snapshot to a volume and attach this volume to
>>> instance.If the "source_type" is "image",we don't handle it.
>>> Can you give me some advice?Help in troubleshooting this issue
>>> will be appreciated.
>>>
>>
>> See: https://review.openstack.org/#/c/520660/
>>
>> We just started disallowing rebuilding a volume-backed instance where the
>> image changes during the rebuild. We don't support it in the compute
>> service, as you've found out, so we're going to make it a fast failure in
>> the API.
>
> Well, are there some reasons for you want to disallowing rebuilding a
> volume-backend instance.
> Just give the the review is not a convincing answer.
>

Rebuilding a volume-backed instance never worked. While the API
currently accepts your request, nothing will happen. The instance
won't be rebuilt as expected.

There had been a couple of proposals/changes to add support:
* https://review.openstack.org/#/c/442295/
* https://review.openstack.org/#/c/305079/

But technical challenges have been uncovered, making it harder to
implement a proper solution.

The change [1] linked by Matt Riedemann proposes failing rapidly so
the user isn't under the false impression that a rebuild is being
performed when in fact, nothing will happen.

I do agree that being able to rebuild a volume-backed instance would
be a great addition. We have been offering volume-backed instance for
more than 4 years and our users love it.
But for now, rebuild just doesn't work at all. It's better to send the
right message through the API by failing fast.

[1] https://review.openstack.org/#/c/520660/

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Super fun unshelve image_ref bugs

[Mathieu Gagné]

Hi,

On Fri, Dec 1, 2017 at 12:24 PM, Matt Riedemann  wrote:
>
> I think we can assert as follows:
>
> 2. If we're going to point the instance at an image_ref, we shouldn't delete
> that image. I don't have a good reason why besides deleting things which
> nova has a reference to in an active instance generally causes problems
> (volumes, ports, etc).
>

Not 100% related to your initial problem but...

Does it mean that you still shouldn't delete any images in Glance
until you are 100% sure that NO instances use them?
If you shouldn't ever delete an image, are there any plans to address
it in the future? Or is it a known "limitation" that people just have
to live with?
If you can delete an image used by one or more instances, how is
shelving affected or different? Should it not be affected?

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Mathieu Gagné]

Some clarifications below.

On Wed, Nov 15, 2017 at 4:52 AM, Bogdan Dobrelya  wrote:
> Thank you Mathieu for the insights!
>
>> To add details to what happened:
>> * Upgrade was never made a #1 priority. It was a one man show for far
>> too long. (myself)
>
>
> I suppose that confirms that upgrades is very nice to have in production
> deployments, eventually, maybe... (please read below to continue)
>
>> * I also happen to manage and work on other priorities.
>> * Lot of work made to prepare for multiple versions support in our
>> deployment tools. (we use Puppet)
>> * Lot of work in the packaging area to speedup packaging. (we are
>> still using deb packages but with virtualenv to stay Puppet
>> compatible)
>> * We need to forward-port private patches which upstream won't accept
>> and/or are private business logic.
>
>
> ... yet long time maintaining and landing fixes is the ops' *reality* and
> pain #1. And upgrades are only pain #2. LTS can not directly help with #2,
> but only indirectly, if the vendors' downstream teams could better cooperate
> with #1 and have more time and resources to dedicate for #2, upgrades
> stories for shipped products and distros.

We do not have a vendor. (anymore, if you consider Ubuntu
cloud-archive as a vendor)
We package and deploy ourselves.

> Let's please to not lower the real value of LTS branches and not substitute
> #1 with #2. This topic is not about bureaucracy and policies, it is about
> how could the community help vendors to cooperate over maintaining of
> commodity things, with as less bureaucracy as possible, to ease the
> operators pains in the end.
>
>> * Our developer teams didn't have enough free cycles to work right
>> away on the upgrade. (this means delays)
>> * We need to test compatibility with 3rd party systems which takes
>> some time. (and make them compatible)
>
>
> This confirms perhaps why it is vital to only run 3rd party CI jobs for LTS
> branches?

For us, 3rd party systems are internal systems outside our control or
realm of influence.
They are often in-house systems that the outside world would care very
little about.

>> * We need to update systems ever which we don't have full control.
>> This means serious delays when it comes to deployment.
>> * We need to test features/stability during some time in our dev
>> environment.
>> * We need to test features/stability during some time in our
>> staging/pre-prod environment.
>> * We need to announce and inform our users at least 2 weeks in advance
>> before performing an upgrade.
>> * We choose to upgrade one service at a time (in all regions) to avoid
>> a huge big bang upgrade. (this means more maintenance windows to plan
>> and you can't stack them too much)
>> * We need to swiftly respond to bug discovered by our users. This
>> means change of priorities and delay in other service upgrades.
>> * We will soon need to upgrade operating systems to support latest
>> OpenStack versions. (this means we have to stop OpenStack upgrades
>> until all nodes are upgraded)
>
>
> It seems that the answer to the question sounded, "Why upgrades are so
> painful and take so much time for ops?" is "as upgrades are not the
> priority. Long Time Support and maintenance are".

The cost of performing an upgrading is both time and resources
consuming which are both limited.
And you need to sync the world around you to make it happen. It's not
a one man decision/task.

When you remove all the external factors, dependencies, politics,
etc., upgrading can take an afternoon from A to Z from some projects.
We do have an internal cloud for our developers that lives in a
vacuum. Let me tell you that it's not very easy upgrade it. We are
talking about hours/days, not years.

So if I can only afford to upgrade once per year, what are my options?

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Mathieu Gagné]

On Tue, Nov 14, 2017 at 6:44 PM, John Dickinson <m...@not.mn> wrote:
>
>
> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>
>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M <kevin@pnnl.gov> wrote:
>>> The pressure for #2 comes from the inability to skip upgrades and the fact 
>>> that upgrades are hugely time consuming still.
>>>
>>> If you want to reduce the push for number #2 and help developers get their 
>>> wish of getting features into users hands sooner, the path to upgrade 
>>> really needs to be much less painful.
>>>
>>
>> +1000
>>
>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>> execute the upgrade. (and we skipped a version)
>> Scheduling all the relevant internal teams is a monumental task
>> because we don't have dedicated teams for those projects and they have
>> other priorities.
>> Upgrading affects a LOT of our systems, some we don't fully have
>> control over. And it can takes months to get new deployment on those
>> systems. (and after, we have to test compatibility, of course)
>>
>> So I guess you can understand my frustration when I'm told to upgrade
>> more often and that skipping versions is discouraged/unsupported.
>> At the current pace, I'm just falling behind. I *need* to skip
>> versions to keep up.
>>
>> So for our next upgrades, we plan on skipping even more versions if
>> the database migration allows it. (except for Nova which is a huge
>> PITA to be honest due to CellsV1)
>> I just don't see any other ways to keep up otherwise.
>
> ?!?!
>
> What does it take for this to never happen again? No operator should need to 
> plan and execute an upgrade for a whole year to upgrade one year's worth of 
> code development.
>
> We don't need new policies, new teams, more releases, fewer releases, or 
> anything like that. The goal is NOT "let's have an LTS release". The goal 
> should be "How do we make sure Mattieu and everyone else in the world can 
> actually deploy and use the software we are writing?"
>
> Can we drop the entire LTS discussion for now and focus on "make upgrades 
> take less than a year" instead? After we solve that, let's come back around 
> to LTS versions, if needed. I know there's already some work around that. 
> Let's focus there and not be distracted about the best bureaucracy for not 
> deleting two-year-old branches.

To add details to what happened:
* Upgrade was never made a #1 priority. It was a one man show for far
too long. (myself)
* I also happen to manage and work on other priorities.
* Lot of work made to prepare for multiple versions support in our
deployment tools. (we use Puppet)
* Lot of work in the packaging area to speedup packaging. (we are
still using deb packages but with virtualenv to stay Puppet
compatible)
* We need to forward-port private patches which upstream won't accept
and/or are private business logic.
* Our developer teams didn't have enough free cycles to work right
away on the upgrade. (this means delays)
* We need to test compatibility with 3rd party systems which takes
some time. (and make them compatible)
* We need to update systems ever which we don't have full control.
This means serious delays when it comes to deployment.
* We need to test features/stability during some time in our dev environment.
* We need to test features/stability during some time in our
staging/pre-prod environment.
* We need to announce and inform our users at least 2 weeks in advance
before performing an upgrade.
* We choose to upgrade one service at a time (in all regions) to avoid
a huge big bang upgrade. (this means more maintenance windows to plan
and you can't stack them too much)
* We need to swiftly respond to bug discovered by our users. This
means change of priorities and delay in other service upgrades.
* We will soon need to upgrade operating systems to support latest
OpenStack versions. (this means we have to stop OpenStack upgrades
until all nodes are upgraded)

All those details rapidly add up. We are far far away from a git pull
&& ./stack.sh

I don't want to sound like too harsh but I feel some people live in a
vacuum or an ideal world far from the reality of some operators.
The above details are just a very small glimpse into my reality. I
hope people will understand and have a different perspectives when it
comes to upgrades.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Mathieu Gagné]

On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
> The pressure for #2 comes from the inability to skip upgrades and the fact 
> that upgrades are hugely time consuming still.
>
> If you want to reduce the push for number #2 and help developers get their 
> wish of getting features into users hands sooner, the path to upgrade really 
> needs to be much less painful.
>

+1000

We are upgrading from Kilo to Mitaka. It took 1 year to plan and
execute the upgrade. (and we skipped a version)
Scheduling all the relevant internal teams is a monumental task
because we don't have dedicated teams for those projects and they have
other priorities.
Upgrading affects a LOT of our systems, some we don't fully have
control over. And it can takes months to get new deployment on those
systems. (and after, we have to test compatibility, of course)

So I guess you can understand my frustration when I'm told to upgrade
more often and that skipping versions is discouraged/unsupported.
At the current pace, I'm just falling behind. I *need* to skip
versions to keep up.

So for our next upgrades, we plan on skipping even more versions if
the database migration allows it. (except for Nova which is a huge
PITA to be honest due to CellsV1)
I just don't see any other ways to keep up otherwise.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] [Openstack-operators] replace node "tags" with node "traits"

[Mathieu Gagné]

Hi,

On Wed, Oct 25, 2017 at 10:17 AM, Loo, Ruby  wrote:
> Hello ironic'ers,
>
> A while ago, we approved a spec to add node tag support to ironic [1]. The
> feature itself did not land yet (although some of the code has). Now that
> the (nova) community has come up with traits, ironic wants to support node
> traits, and there is a spec proposing that [2]. At the ironic node level,
> this is VERY similar to the node tag support, so the thought is to drop (not
> implement) the node tagging feature, since the node traits feature could be
> used instead. There are a few differences between the tags and traits.
> "Traits" means something in OpenStack, and there are some restrictions about
> it:
>
> - max 50 per node
>
> - names must be one of those in os-traits library OR prefixed with 'CUSTOM_'
>
> For folks that wanted the node tagging feature, will this new node traits
> feature work for your use case? Should we support both tags and traits? I
> was wondering about e.g. using ironic standalone.
>
> Please feel free to comment in [2].
>
> Thanks in advance,
>
> --ruby
>
> [1]
> http://specs.openstack.org/openstack/ironic-specs/specs/approved/nodes-tagging.html
>
> [2] https://review.openstack.org/#/c/504531/
>

Are tags/traits serving a different purpose? One serves the purpose of
helping the scheduling/placement while the other is more or less aims
at grouping for the "end users"?
I understand that the code will be *very* similar but who/what will be
the consumers/users?
I fell they won't be the same and could artificially limit its use due
to technical/design "limitations". (must be in os-traits or be
prefixed by CUSTOM)

For example which I personally foresee:
* I might want to populate Ironic inventory from an external system
which would also injects the appropriate traits.
* I might also want some technical people to use/query Ironic and
allow them to tag nodes based on their own needs while not messing
with the traits part (as it's managed by an external system and will
influence the scheduling later)

Lets not assume traits/tags have the same purpose and same user.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Should we make rebuild + new image on a volume-backed instance fail fast?

[Mathieu Gagné]

> On Fri, Oct 6, 2017 at 2:02 PM, Chris Friesen
> <chris.frie...@windriver.com> wrote:
>> On 10/06/2017 11:32 AM, Mathieu Gagné wrote:
>>>
>>> Why not supporting this use case?
>>
>>
>> I don't think anyone is suggesting we support do it, but nobody has stepped
>> up to actually merge a change that implements it.
>>
>> I think what Matt is suggesting is that we make it fail fast *now*, and if
>> someone else implements it then they can remove the fast failure at the same
>> time.
>>

On Fri, Oct 6, 2017 at 2:10 PM, Mathieu Gagné <mga...@calavera.ca> wrote:
> I don't think there ever was a technical limitation to add support for it.
>
> I have nothing to back my claims but IIRC, it was more philosophical
> points against adding support for it.

So it looks like people worked on this feature in the past:
https://review.openstack.org/#/c/305079/

And there was pending technical questions. =)

Would it be worth the effort to bring that one back, maybe with a spec
so people can address technical details in there?

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Should we make rebuild + new image on a volume-backed instance fail fast?

[Mathieu Gagné]

I don't think there ever was a technical limitation to add support for it.

I have nothing to back my claims but IIRC, it was more philosophical
points against adding support for it.

--
Mathieu


On Fri, Oct 6, 2017 at 2:02 PM, Chris Friesen
<chris.frie...@windriver.com> wrote:
> On 10/06/2017 11:32 AM, Mathieu Gagné wrote:
>>
>> Why not supporting this use case?
>
>
> I don't think anyone is suggesting we support do it, but nobody has stepped
> up to actually merge a change that implements it.
>
> I think what Matt is suggesting is that we make it fail fast *now*, and if
> someone else implements it then they can remove the fast failure at the same
> time.
>
> Chris
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Should we make rebuild + new image on a volume-backed instance fail fast?

[Mathieu Gagné]

Why not supporting this use case?

Same reason as before: A user might wish to keep its IP addresses.

The use cannot do the following to bypass the limitation:
1) stop instance
2) detach root volume
3) delete root volume
4) create new volume
5) attach as root
6) boot instance

Last time I tried, operation fails at step 2. I would need to test
against latest version of Nova to confirm.

Otherwise boot-from-volume feels like a second class citizen.

--
Mathieu


On Fri, Oct 6, 2017 at 1:22 PM, Matt Riedemann  wrote:
> This came up in IRC discussion the other day, but we didn't dig into it much
> given we were all (2 of us) exhausted talking about rebuild.
>
> But we have had several bugs over the years where people expect the root
> disk to change to a newly supplied image during rebuild even if the instance
> is volume-backed.
>
> I distilled several of those bugs down to just this one and duplicated the
> rest:
>
> https://bugs.launchpad.net/nova/+bug/1482040
>
> I wanted to see if there is actually any failure on the backend when doing
> this, and there isn't - there is no instance fault or anything like that.
> It's just not what the user expects, and actually the instance image_ref is
> then shown later as the image specified during rebuild, even though that's
> not the actual image in the root disk (the volume).
>
> There have been a couple of patches proposed over time to change this:
>
> https://review.openstack.org/#/c/305079/
>
> https://review.openstack.org/#/c/201458/
>
> https://review.openstack.org/#/c/467588/
>
> And Paul Murray had a related (approved) spec at one point for detach and
> attach of root volumes:
>
> https://review.openstack.org/#/c/221732/
>
> But the blueprint was never completed.
>
> So with all of this in mind, should we at least consider, until at least
> someone owns supporting this, that the API should fail with a 400 response
> if you're trying to rebuild with a new image on a volume-backed instance?
> That way it's a fast failure in the API, similar to trying to backup a
> volume-backed instance fails fast.
>
> If we did, that would change the API response from a 202 today to a 400,
> which is something we normally don't do. I don't think a microversion would
> be necessary if we did this, however, because essentially what the user is
> asking for isn't what we're actually giving them, so it's a failure in an
> unexpected way even if there is no fault recorded, it's not what the user
> asked for. I might not be thinking of something here though, like
> interoperability for example - a cloud without this change would blissfully
> return 202 but a cloud with the change would return a 400...so that should
> be considered.
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][docs] Concerns with docs migration

[Mathieu Gagné]

On Wed, Aug 2, 2017 at 12:28 PM, Clark Boylan  wrote:
> On Wed, Aug 2, 2017, at 07:55 AM, Matt Riedemann wrote:
>> Now that Stephen Finucane is back from enjoying his youth and
>> gallivanting all over Europe, and we talked about a few things in IRC
>> this morning on the docs migration for Nova, I wanted to dump my
>> concerns here for broader consumption.
>>
>> 1. We know we have to fix a bunch of broken links by adding in redirects
>> [1] which sdague started here [2]. However, that apparently didn't catch
>> everything, e.g. [3], so I'm concerned we're missing other broken links.
>> Is there a way to find out?
>
> The infra team can generate lists of 404 urls fairly easily on the docs
> server. This won't show you everything but will show you what urls
> people are finding/using that 404.
>

Thanks for proposing this proactive solution.
I find it *very frustrating* to Google something and land on a 404 error.
Especially when the user survey keeps asking "How often do users refer
to documentation from docs.openstack.org?"
I don't often refer to the documentation but as of today, when I does,
it's only to find the page I'm looking for has moved somewhere.
What kind of experience are we giving to the users and operators? (I'm
sure it's with good intents and for the best)

I'm glad people are addressing the redirection issues.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] office hours report 2017-7-7

[Mathieu Gagné]

Resending... I found out that Gmail messed up my message with its HTML format...
Hopefully this time it will be more readable on the online archive interface.

On Tue, Jul 11, 2017 at 10:28 PM, Mathieu Gagné <mga...@calavera.ca> wrote:
> Hi,
>
> So this email is relevant to my interests as an operator. =)
>
> On Tue, Jul 11, 2017 at 9:35 PM, Lance Bragstad <lbrags...@gmail.com> wrote:
>>
>> The future of the templated catalog backend
>>
>> Some issues were uncovered, or just resurfaced, with the templated catalog
>> backend. The net of the discussion boiled down to - do we fix it or remove
>> it? The answer actually ended up being both. It was determined that instead
>> of trying to maintain and fix the existing templated backend, we should
>> deprecate it for removal [0]. Since it does provide some value, it was
>> suggested that we can start implementing a new backend based on YAML to fill
>> the purpose instead. The advantage here is that the approach is directed
>> towards a specific format (YAML). This should hopefully make things easier
>> for both developers and users.
>>
>> [0] https://review.openstack.org/#/c/482714/
>
>
> We have been exclusively using the templated catalog backend for at least 5
> years without any major issues. And it looks like we are now among the < 3%
> using templated according to the April 2017 user survey.
> ¯\_(ツ)_/¯
>
> We choose the templated catalog backend for its simplicity (especially with
> our CMS) and because it makes no sense (to me) to use and rely on an
> SQL server to serve what is essentially static content.
>
> Regarding the v3 catalog support, we do have an in-house fix we intended to
> upstream very soon (and just did right now). [1]
>
> So if the templated catalog backend gets deprecated,
> my wish would be to have access to an alternate file based
> implementation, a production grade implementation ready to be used
> before I get spammed with deprecation warnings in the keystone logs.
>
> Thanks
>
> [1] https://review.openstack.org/#/c/482766/
>
> --
> Mathieu
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] office hours report 2017-7-7

[Mathieu Gagné]

Hi,

So this email is relevant to my interests as an operator. =)

On Tue, Jul 11, 2017 at 9:35 PM, Lance Bragstad  wrote:

> *The future of the templated catalog backend*
>
> Some issues were uncovered, or just resurfaced, with the templated catalog
> backend. The net of the discussion boiled down to - do we fix it or remove
> it? The answer actually ended up being both. It was determined that instead
> of trying to maintain and fix the existing templated backend, we should
> deprecate it for removal [0]. Since it does provide some value, it was
> suggested that we can start implementing a new backend based on YAML to
> fill the purpose instead. The advantage here is that the approach is
> directed towards a specific format (YAML). This should hopefully make
> things easier for both developers and users.
>
> [0] https://review.openstack.org/#/c/482714/​
>

We have been exclusively using the templated catalog backend for at least 5
years without any major issues. And it looks like we are now among the < 3%
using templated according to the April 2017 user survey.
¯\_(ツ)_/¯

We choose the templated catalog backend for its simplicity (especially with
our CMS) and because it makes no sense (to me) to use and rely on an
SQL server to serve what is essentially static content
​.​


Regarding the v3 catalog support, we do have an in-house fix we intended to
upstream
​ very soon (and just did right now)​
. [1]


So if the templated catalog backend gets deprecated,
​my wish would be to have access to
 a
​n alternate​
​ file based
​ implementation​, a production grade implementation
 ready to be used
​ before I get spammed with deprecation warnings in the keystone logs.

Thanks

[1] https://review.openstack.org/#/c/482766/

--
Mathieu
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Documenting config drive - what do you want to see?

[Mathieu Gagné]

On Wed, May 24, 2017 at 10:39 AM, Matt Riedemann 
wrote:
>
> Rocky tipped me off to a request to document config drive which came up
at the Boston Forum, and I tracked that down to Clark's wishlist etherpad
[1] (L195) which states:
>
> "Document the config drive. The only way I have been able to figure out
how to make a config drive is by either reading nova's source code or by
reading cloud-init's source code."
>
> So naturally I have some questions, and I'm looking to flesh the idea /
request out a bit so we can start something in the in-tree nova devref.
>
> Question the first: is this existing document [2] helpful? At a high
level, that's more about 'how' rather than 'what', as in what's in the
config drive.

Thanks, I didn't know about that page. I usually read sources or boot an
instance and check by myself.

> Question the second: are people mostly looking for documentation on the
content of the config drive? I assume so, because without reading the
source code you wouldn't know, which is the terrible part.

I usually boot an instance and inspect the config drive. Usually it's for
network_data.json since (in our case) we support various network models
(flat, nic teaming, tagged vlan, etc.) and need a concrete example of each.

> Based on this, I can think of a few things we can do:
>
> 1. Start documenting the versions which come out of the metadata API
service, which regardless of whether or not you're using it, is used to
build the config drive. I'm thinking we could start with something like the
in-tree REST API version history [3]. This would basically be a change log
of each version, e.g. in 2016-06-30 you got device tags, in 2017-02-22 you
got vlan tags, etc.

+1

I'm not sure about the format as there is a lot of cases to cover.

* There a multiple supported config drive versions (2012-08-10, ...,
2017-02-22) so we need to document them all.
* How do we plan on making it easy for someone to understand which fields
will be available in each versions?
* If a field is removed, how will it be expressed in the documentation?
* Could a field change type in the future? (object vs list of objects for
example)
* The idea of a documentation similar to the REST API version history is
good. I however wouldn't have the patience to mentally "compute" the
resulting config drive. So I think we need both (history + schema/examples).
* We should document the purpose of each field and how a user can populate
or use that field. For example, I have no idea what's the purpose of the
"launch_index" field but I suspect it's related to the --max-count
parameter with nova boot command.

> 2. Start documenting the contents similar to the response tables in the
compute API reference [4]. For example, network_data.json has an example
response in this spec [5]. So have an example response and a table with an
explanation of fields in the response, so describe ethernet_mac_address and
vif_id, their type, whether or not they are optional or required, and in
which version they were added to the response, similar to how we document
microversions in the compute REST API reference.

+1

> [1] https://etherpad.openstack.org/p/openstack-user-api-improvements
> [2] https://docs.openstack.org/user-guide/cli-config-drive.html
> [3]
https://docs.openstack.org/developer/nova/api_microversion_history.html
> [4] https://developer.openstack.org/api-ref/compute/
> [5]
https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact

--
Mathieu
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Multiple default routes in instance

[Mathieu Gagné]

On Wed, May 17, 2017 at 10:03 AM, Mathieu Gagné <mga...@calavera.ca> wrote:
> Hi,
>
> When you attach multiple networks/interfaces to an instance and at
> least 2 subnets have a default route, you end up with 2 default routes
> in network_data.json.
>
> If cloud-init or any other in-guest agent is used to configure the
> network, you could end up with 2 default routes and to an extend, with
> non-working networking. (like we found out recently)
>
> How should we handle this situation where you could end up with 2
> subnets, each having a default route?
>
> * Should Nova decide which default route to expose to the instance?
> * Should the in-guest agent pick one?
>
> I'm looking for opinions and advises on how to fix this issue.
>

I would also like to draw attention to a spec I wrote which faces the same
questions and challenges related to multiple fixed-ips per port support:
https://review.openstack.org/#/c/312626/7

Quoting the commit message here:

> Each subnet can have its own routes and default route which create
challenges:
> * Should all routes be merged in the same "routes" attribute?
> * What should be the default route if more than one subnet provides a
default route? Should we provide all of them?
> * What should be the implementation logic an in-guest agent use to
determine the default route?
> * Are there use cases where one default route should be used over the
others? I'm thinking about NFV or other specialized use cases.

--
Mathieu
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Multiple default routes in instance

[Mathieu Gagné]

Hi,

When you attach multiple networks/interfaces to an instance and at
least 2 subnets have a default route, you end up with 2 default routes
in network_data.json.

If cloud-init or any other in-guest agent is used to configure the
network, you could end up with 2 default routes and to an extend, with
non-working networking. (like we found out recently)

How should we handle this situation where you could end up with 2
subnets, each having a default route?

* Should Nova decide which default route to expose to the instance?
* Should the in-guest agent pick one?

I'm looking for opinions and advises on how to fix this issue.

Thanks

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][glance][keystone] Philosophy of the service catalog (was: Who needs multiple api_servers?)

[Mathieu Gagné]

I didn't attend the LDT session but agree with the needs and
conclusions mentioned here by Mike Dorman.

It's important for us to be able to control the data path of our
internal servers and overriding the URLs is the method we are
currently using. You could do split view DNS but we prefer being
explicit (different URLs) over implicit. It ends up being much easier
for new comers to debug as they don't have to know about split view
DNS.
--
Mathieu


On Wed, May 10, 2017 at 2:29 PM, Mike Dorman  wrote:
> After discussion in the Large Deployments Team session this morning, we
> wanted to follow up on the earlier thread [1,2] about overriding endpoint
> URLs.
>
>
>
> That topic is exposing an underlying implication about the purpose of the
> service catalog.  The LDT position is that the service catalog should be for
> end user clients to do endpoint discovery.  While it can also be used for
> discovery by other OpenStack services, we desire to maintain the ability to
> override (like that which was discussed in the previous thread about
> Glance.)  In addition to the Glance to nova-compute use case, the feedback
> during the LDT session surfaced potential use cases for other services.
>
>
>
> The point to raise here from LDT is that we would like to avoid a trend
> toward services *only* supporting discovery via the service catalog, with no
> ability to override in config.  I.e., we want to maintain the
> endpoint_override (and similar) options.
>
> Thanks!
>
>
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2017-April/116028.html /
> http://lists.openstack.org/pipermail/openstack-operators/2017-April/013272.html
>
> [2]
> http://lists.openstack.org/pipermail/openstack-dev/2017-May/thread.html#116133
> /
> http://lists.openstack.org/pipermail/openstack-operators/2017-May/thread.html#13309
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Usability question for the server migrations API

[Mathieu Gagné]

The proposal looks reasonable.

This could greatly help operation team if a migration no longer
vanishes from the API once completed.
Can I assume some form of advanced filtering will be proposed to
ensure an instance doesn't end up with 1k records?
One suggestion would be filtering by date or limiting the number of
records returned.

On Fri, Apr 14, 2017 at 4:27 PM, Matt Riedemann  wrote:
> The GET /servers/{server_id}/migrations API only lists in-progress live
> migrations. This is an artifact of when it was originally introduced as the
> os-migrations API which was tightly coupled with the API operation to cancel
> a live migration.
>
> There is a spec [1] which is now approved which proposes to expand that to
> also return other types of in-progress migrations, like cold migrations,
> resizes and evacuations.
>
> What I don't like about the proposal is that it still filters out completed
> migrations from being returned. I never liked the original design where only
> in-progress live migrations would be returned. I understand why it was done
> that way, as a convenience for using those results to then cancel a live
> migration, but seriously that's something that can be filtered out properly.
>
> So what I'd propose is that in a new microversion, we'd return all migration
> records for a server, regardless of status. We could provide a status filter
> query parameter if desired to just see in-progress migrations, or completed
> migrations, etc. And the live migration cancel action API would still
> validate that the requested migration to cancel is indeed in progress first,
> else it's a 400 error.
>
> The actual migration entries in the response are quite detailed, so if
> that's a problem, we could change listing to just show some short info (id,
> status, source and target host), and then leave the actual details for the
> show API.
>
> What do operators think about this? Is this used at all? Would you like to
> get all migrations and not just in-progress migrations, with the ability to
> filter as necessary?
>
> [1] https://review.openstack.org/#/c/407237/
>
> --
>
> Thanks,
>
> Matt
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Can all non-Ironic compute drivers handle volume size extension?

[Mathieu Gagné]

On Wed, Apr 12, 2017 at 2:54 PM, Matt Riedemann  wrote:
>
> Correct, I thought about this yesterday too. And this should be a detail in
> the Cinder spec for sure, but Cinder should probably have a specific policy
> check for attempting to extend an attached volume. Having said this, I see
> that Cinder has a "volume:extend" policy rule but I don't see it actually
> checked in the code, is that a bug?
>
> But the idea is, you, as a deployer, could allow extending volumes that are
> not attached (using the existing volume:extend policy) but disable the
> ability to extend attached volumes (maybe new rule volume:extend_attached?).
> Then if you're running older computes, or not running libvirt/hyperv
> computes, etc, then you just disable the API entrypoint for the entire
> operation on the Cinder side.
>
> ^ should all be captured in the Cinder spec.
>

It has been added to Cinder spec:
https://review.openstack.org/#/c/453286/

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Can all non-Ironic compute drivers handle volume size extension?

[Mathieu Gagné]

Thanks for starting this discussion. There is a lot to cover/answer.

On Tue, Apr 11, 2017 at 6:35 PM, Matt Riedemann  wrote:
>
> This is not discoverable at the moment, for the end user or cinder, so I'm
> trying to figure out what the failure mode looks like.
>
> This all starts on the cinder side to extend the size of the attached
> volume. Cinder is going to have to see if Nova is new enough to handle this
> (via the available API versions) before accepting the request and resizing
> the volume. Then Cinder sends the event to Nova. This is where it gets
> interesting.
>
> On the Nova side, if all of the computes aren't new enough, we could just
> fail the request outright with a 409. What does Cinder do then? Rollback the
> volume resize?

This means an extend volume operation would need to check for Nova
support first.
This also means adding a new API call to fetch and discover such
capabilities per instance (from associated compute node).
If we want to catch errors in volume size extension in Nova, we will
need to find an other way, external events are async.

> But let's say the computes are new enough, but the instance is on a compute
> that does not support the operation. Then what? Do we register an instance
> fault and put the instance into ERROR state? Then the admin would need to
> intervene.
>
> Are there other ideas? Until we have capabilities (info) exposed out of the
> API we're stuck with questions like this.
>

Like TommyLike mentioned in a review, AWS introduced Live Volume
Modifications available on some instance types.
On instance types with limited support, you need to stop/start the
instance or detach/attach the volume.
On instances started before a certain date, you need to stop/start the
instance or detach/attach the volume at least once.
In all cases, the end user needs to extend the partition/filesystem in
the instance.

They have the luxury to fully control the environment and synchronize
the compute service with the volume service.
Even (speculatively) having bidirectional
orchestration/synchronization/communications or what.

I have that same luxury since I only support one volume backend and
virt driver combination.
But I now start to grasp the extend of what adding such feature
requires, especially when it implies cross-services support...

We have a matrix of compute drivers and volume backends to support
with some combinations which might never support online volume
extension.
There is the desire for OpenStack to be interoperable between clouds
so there is a strong incentive to make it work for all combinations.

I will still take the liberty to ask:

Would it be in the realm of possibilities for a deployer to have to
explicitly enable this feature?
A deployer would be able to enable such feature once all
services/components it choose to deployed fully support online volume
extension.

I know it won't address cases where a mixed of volume backends and
virt drivers are deployed.
So we would still need capabilities discoverability. This includes
volume type capabilities discoverability which I'm not sure exists
today.

Lets not start about how Horizon will discover such capabilities per
instance/volume. That's an other can of worms. =)

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] Can all non-Ironic compute drivers handle volume size extension?

[Mathieu Gagné]

On Wed, Apr 12, 2017 at 12:58 PM, Matt Riedemann  wrote:
>
> I guess we also have the instance action events. So we could avoid putting
> the instance into ERROR state but record an instance action event that the
> extend volume event failed on the compute, so at least the user/admin could
> figure out why it didn't change on the nova side.
>
> What they do after that I'm not sure. Would detaching and then re-attaching
> the now resized volume fix it?
>

There are 2 steps to the volume size extension: iscsi --rescan and
virsh blockresize.
As an admin, you could call the same API endpoint to retrigger all of
those steps.
If iscsi --rescan succeeds but virsh blockresize fails, stopping and
starting the instance will fix the issue.
This is the step we asked our customer to perform before virsh
blockresize was added to our implementation.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [Nova] Extend attached volume

[Mathieu Gagné]

Hi,

On Mon, Apr 3, 2017 at 8:40 PM, Jay S Bryant  wrote:
>
> Thank you for sharing this.  Nice to see you have a solution that looks
> agreeable to Matt.  Do you think you can get a spec pushed up and propose
> your code?
>

I will go ahead, write the spec and contribute the implementation.

Thanks

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] [Nova] Extend attached volume

[Mathieu Gagné]

On Mon, Apr 3, 2017 at 12:27 PM, Walter Boring  wrote:
> Actually, this is incorrect.
>
> The sticking point of this all was doing the coordination and initiation of
> workflow from Nova.   Cinder already has the ability to call the driver to
> do the resize of the volume.  Cinder just prevents this now, because there
> is work that has to be done on the attached side to make the new size
> actually show up.
>
> What needs to happen is:
>  A new Nova API needs to be created to initiate and coordinate this effort.
> The API would call Cinder to extend the size, then get the connection
> information from Cinder for that volume, then call os-brick to extend the
> size, then update the domain xml to tell libvirt to extend the size.   The
> end user inside the VM would have to issue the same SCSI bus rescan and
> refresh that happens inside of os-brick, to make the kernel and filesystem
> in the VM recognize the new size.
>
> os-brick does all of the heavy lifting already on the host side of things.
> The Connector API entry point:
> https://github.com/openstack/os-brick/blob/master/os_brick/initiator/initiator_connector.py#L153
>
> iSCSI example:
> https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L370
>
> os-brick's code works for single path and multipath attached volumes.
> multipath has a bunch of added complexity with resize that should already be
> taken care of here:
> https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L375
>

I would like to share our private implementation (based on Mitaka):
https://gist.github.com/mgagne/9402089c11f8c80f6d6cd49f3db76512

The implementation makes it so Cinder leverages the existing Nova
external-events endpoint to trigger the BDM update and iSCSI rescan on
the host.

As always, the guest needs to update the partition table/filesystem if
it wants to benefit from the new free space.

Let me know if this is an implementation you want me to contribute upstream.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][metadata] Is there HTTP attack issue in metadata proxy functionality offered by reference implementation?

[Mathieu Gagné]

On Wed, Nov 16, 2016 at 11:52 AM, Clint Byrum  wrote:
>
> IMO the HTTP metadata service and the way it works is one of the worst
> ideas we borrowed from EC2. Config drive (which I didn't like when I
> first saw it, but now that I've operated clouds, I love) is a simpler
> system and does not present any real surface area to the users.
>

Cannot agree more with you on that one.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][nova] Anyone interested in writing a policy generator sphinx extension?

[Mathieu Gagné]

Hi,

On Wed, Sep 21, 2016 at 10:49 AM, Matt Riedemann
 wrote:
> Nova has policy defaults in code now and we can generate the sample using
> oslopolicy-sample-generator but we'd like to get the default policy sample
> in the Nova developer documentation also, like we have for nova.conf.sample.
>
> I see we use the sphinxconfiggen extension for building the nova.conf.sample
> in our docs, but I don't see anything like that for generating docs for a
> sample policy file.
>
> Has anyone already started working on that, or is interested in working on
> that? I've never written a sphinx extension before but I'm guessing it could
> be borrowed a bit from how sphinxconfiggen was written in oslo.config.

I'm not a developer. I'm one of the operator that asked about
documentation now that policy defaults got moved in code.
I'm happy to see a proper follow up on this request, I really appreciate it. +1
I hope this thread can stay on topic so a proper documentation
solution is found.

Thanks

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] mod_wsgi: what services are people using with it?

[Mathieu Gagné]

I don't have much experience with mod_wsgi.

I just want to let you know that when I tried to run nova-api service
in mod_wsgi (with OpenStack Kilo), I had to remove the
python-librabbitmq and librabbitmq1 packages from my system or
mod_wsgi would throw Segmentation Fault (11) when a user tries to get
the VNC console URL.

I'm looking forward for more replies to your question as I'm interested too.

--
Mathieu


On Wed, Aug 17, 2016 at 4:22 PM, Nick Papadonis  wrote:
> Hi Folks,
>
> I was hacking in this area on Mitaka and enabled Glance, Cinder, Heat,
> Swift, Ironic, Horizon and Keystone under Apache mod_wsgi instead of the
> Eventlet server.Cinder, Keystone, Heat and Ironic provide Python source
> in Github to easily enable this.  It appears that Glance and Swift (despite
> the existence of
> https://github.com/openstack/swift/blob/2bf5eb775fe3ad6d3a2afddfc7572318e85d10be/doc/source/apache_deployment_guide.rst)
> provide no such Python source to call from the Apache conf file.
>
> That said, is anyone using Glance, Swift, Neutron or Nova (marked
> experimental) in production environments with mod_wsgi?  I had to put
> together code to launch a subset of these which does not appear integrated
> in Github.  Appreciate your insight.
>
> Thanks,
> Nick
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][cloud-init] Questions around 'network_data.json'

[Mathieu Gagné]

I think there is an implementation error.
The spec mentions that link type can be "vif", "phy" or (future) "bond".
Nova passes the "raw" Neutron VIF type instead.
IMO, "bridge" should be "vif" as per spec.
--
Mathieu


On Tue, May 24, 2016 at 5:20 PM, Joshua Harlow  wrote:
> Hi there all,
>
> I am working through code/refactoring in cloud-init to enable translation of
> the network_data.json file[1] provided by openstack (via nova via neutron?)
> into the equivalent sysconfig files (ubuntu files should already *mostly*
> work and systemd files are underway as well).
>
> Code for this sysconfig (WIP) is @
> https://gist.github.com/harlowja/d63a36de0b405d83be9bd3222a5454a7 (requires
> base branch
> https://code.launchpad.net/~harlowja/cloud-init/cloud-init-net-refactor
> which did some tweaks to make things easier to work with).
>
> Sadly there has been some questions around certain format conversion and
> what it means when certain formats are given, for example in the following
> example:
>
> {
>   "services": [
> {
>   "type": "dns",
>   "address": "172.19.0.12"
> }
>   ],
>   "networks": [
> {
>   "network_id": "dacd568d-5be6-4786-91fe-750c374b78b4",
>   "type": "ipv4",
>   "netmask": "255.255.252.0",
>   "link": "tap1a81968a-79",
>   "routes": [
> {
>   "netmask": "0.0.0.0",
>   "network": "0.0.0.0",
>   "gateway": "172.19.3.254"
> }
>   ],
>   "ip_address": "172.19.1.34",
>   "id": "network0"
> }
>   ],
>   "links": [
> {
>   "ethernet_mac_address": "fa:16:3e:ed:9a:59",
>   "mtu": null,
>   "type": "bridge",
>   "id": "tap1a81968a-79",
>   "vif_id": "1a81968a-797a-400f-8a80-567f997eb93f"
> }
>   ]
> }
>
> In the cloud-init code what happens is that the links get connected to the
> networks and this gets then internally parsed, but for the bridge type
> listed here it appears information is missing about exactly what to bridge
> to (eth0, ethX? something else)?
>
> Should the 'bridge' type just be ignored and it should just be treated as a
> physical device type (which will cause a different set of logic in
> cloud-init to apply); something else?
>
> Thoughts would be appreciated,
>
> [1]
> http://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/metadata-service-network-info.html#rest-api-impact
>
> -Josh
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Deprecated options in sample configs?

[Mathieu Gagné]

On Tue, May 17, 2016 at 3:51 PM, Ben Nemec  wrote:
>
> I'm +1 on removing them from sample configs.  This feels like release
> note information to me, and if someone happens to miss the release note
> then they'll get deprecation warnings in their logs.  The sample config
> entries are redundant and just clutter things up for new deployments.
>

As Matt explained, we use the sample config file to get information
about existing, deprecated and soon to be removed configuration.
I'm not reading the release notes as it's not the definitive source of
truth. Someone could very well forget to include a note about a
deprecation.
Furthermore, release notes only cover the latest version, not all of
them. This means I end up having to browse through multiple release
notes to make sure I don't miss one.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Team blogs

[Mathieu Gagné]

On Tue, May 10, 2016 at 8:05 AM, Jeremy Stanley  wrote:
> On 2016-05-09 19:46:14 -0400 (-0400), Sean Dague wrote:
>> Honestly, I'm really liking that more of them are hitting the
>> mailing list proper this time around. Discoverability is key. The
>> mailing list is a shared medium, archived forever.
>
> I feel the same (says the guy who is still in the process of
> drafting his to send to the ML, hopefully later today). I'm not sure
> what drives people to put these on random personal blogs instead,
> but the "blog" of our contributor community is the openstack-dev
> mailing list.

I find it hard to find older threads on the mailinglist.
The online HTML archives isn't great and doesn't group those kind of
threads for people to easily find them later.
Unlike mailinglist, a blog allows some form of grouping or tagging for
easy finding.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][glance] Proposal to remove `nova image-*` commands from novaclient

[Mathieu Gagné]

On Tue, Apr 5, 2016 at 11:24 PM, Monty Taylor  wrote:
> On 04/05/2016 05:07 PM, Michael Still wrote:
>
>> self.glance = glance_client.Client('2', endpoint, token=token)
>
>
> There are next to zero cases where the thing you want to do is talk to
> glance using a token and an endpoint.

I used to have a use case for that one. I'm only mentioning so you can
have a good laugh at it. =)

We have an internal development cloud where people can spawn a whole
private OpenStack infrastructure for testing and development purposes.
This means ~50 instances.
All instances communicate between them using example.org DNS, this
includes URLs found in the Keystone catalog.
Each stack has a private DNS server resolving those requests for
example.org and our internal tool configure it once the stack is
provisioned.
This means each developper has its own example.org zone which resolves
differently. They can configure their own local resolvers to use the
one found in the stack if they wish.

Now the very funny part:

Developers can decide to destroy their stack as they wish in brutal
ways. The side-effect we saw is that it left orphan volumes on our
shared block storage backend which also happens to have a maximum
number of volumes.
So to clean them up, we wrote a script that connects to each
developer's stack and try to list volumes in Cinder, compare them with
the ones found on the block storage backend and delete orphan ones.
Since everybody has the same example.org DNS in their catalog, we
needed a way to tell python-cinderclient to not use the DNS found in
the catalog but the actual IP of the developper's Cinder instance.
That's our use case where we needed the endpoint argument. =)

Good news, we found an alternative solution where we override the
Python socket resolving methods instead and override the IP from there
instead of using the endpoint argument.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The #openstack-packaging channel requires an invite?

[Mathieu Gagné]

It got renamed a couple of months/year ago to #openstack-stable:
https://bugs.launchpad.net/openstack-ci/+bug/1360324

Documentation should have been updated to reflect this change.

Channel being invite-only is probably a side-effect of the migration.

Mathieu

On 2016-03-09 12:48 PM, Matt Riedemann wrote:
> Someone showed up in the -stable channel this morning asking for help
> with packaging. I gather they were there because the IRC wiki [1] says
> the #openstack-stable channel is also for packaging discussions, given
> -stable originated from distro people.
> 
> I redirected them to https://wiki.openstack.org/wiki/Packaging which
> points to https://wiki.openstack.org/wiki/PackagerResources which says:
> 
> "The #openstack-packaging channel on Freenode is available for packaging
> collaboration and discussion."
> 
> Great!
> 
> However, you have to apparently be invited to join the elite
> #openstack-packaging channel.
> 
> What gives? Is that channel dead? Is there something else? Is there a
> secret handshake I can learn to palms to grease to get in?
> 
> [1] https://wiki.openstack.org/wiki/IRC
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] - Changing the Neutron default security group rules

[Mathieu Gagné]

On 2016-03-03 12:53 PM, Sean M. Collins wrote:
> sridhar basam wrote:
>> This doesn't sound like a neutron issue but an issue with how the
>> conntrack module for GRE changed in the kernel in 3.18.
>>
>>
>> http://comments.gmane.org/gmane.comp.security.firewalls.netfilter.general/47705
>>
>> Sri
> 
> Oooh! Wicked nice find. Thanks Sri!
> 

We had issue with GRE but unrelated to the one mentioned above.

Although security group is configured to allow GRE,
nf_conntrack_proto_gre module is not loaded by iptables/Neutron and
traffic is dropped. We had to load the module manually.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Stepping down from Puppet Core

[Mathieu Gagné]

Hi,

I would like to ask to be removed from the core reviewers team on the
Puppet for OpenStack project.

My day to day tasks and focus no longer revolve solely around Puppet and
I lack dedicated time to contribute to the project.

In the past months, I stopped actively reviewing changes compared to
what I used to at the beginning when the project was moved to
StackForge. Community code of conduct suggests I step down
considerately. [1]

I'm very proud of what the project managed to achieve in the past
months. It would be a disservice to the community to pretend I'm still
able or have time to review changes. A lot changed since and I can no
longer keep up or pretend I can review changes pedantically or
efficiently as I used to.

Today is time to formalize and face the past months reality by
announcing my wish to be removed from the core reviewers team.

I will be available to answer questions or move ownership of anything I
still have under my name.

Wishing you the best.

Mathieu

[1] http://www.openstack.org/legal/community-code-of-conduct/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] OpenStack-Announce List

[Mathieu Gagné]

On 2015-11-19 11:00 PM, Tom Fifield wrote:
>
> Personally, I no longer consider this volume "low traffic" :)
> 
> In addition, I have been recently receiving feedback that users have
> been unsubscribing from or deleting without reading the list's posts.
> 
> That isn't good news, given this is supposed to be the place where we
> can make very important announcements and have them read.
> 
> One simple suggestion might be to batch the week's client/library
> release notifications into a single email. Another might be to look at
> the audience for the list, what kind of notifications they want, and
> chose the announcements differently.
> 
> What do you think we should do to ensure the announce list remains useful?
> 

I think we should first look into the target audience.

I don't see "dev" in the list name and I suspect non-developers people
subscribe in hope to get "important announcements".

I however like being informed of projects and clients releases. They
don't happen often and they are interesting to me as an operator
(projects) and consumer of the OpenStack API (project clients).

I unfortunately don't care much about oslo libraries which are internal
projects consumed by the OpenStack developers as dependencies of the
other major projects. Receiving ~15-25 per week for such releases is too
much for me.

If people care about releases, maybe there should be a openstack-release
list where you can subscribe to tags you care about.

On a side note, I subscribed to openstack-security in hope to receive
security notices but instead they are sent to openstack-announce and
lost among all those release notices.

However, if we move security notices to openstack-security and release
notices to openstack-release, I don't know what's left to read about in
openstack-announce except the "OpenStack Community Weekly Newsletter".

Not much strong suggestion on my side. Maybe we should restrict it to
security notices and major project milestone and stable releases.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova-compute][nova][libvirt] Extending Nova-Compute for image prefetching

[Mathieu Gagné]

On 2015-10-21 10:23 AM, Markus Zoeller wrote:
> Lingxian Kong  wrote on 10/21/2015 06:44:27 AM:
> 
>> From: Lingxian Kong 
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Date: 10/21/2015 06:44 AM
>> Subject: Re: [openstack-dev] [nova-compute][nova][libvirt] Extending 
>> Nova-Compute for image prefetching
>>
>> Hi, Alberto,
>>
>> I'm really interested in your proposal, have you already submitted
>> your spec? or is there any topic for you to have a discussion with
>> Nova team in design summit?
>>
>> I wonder what if the nova compute host use shared storage backend,
>> like NFS or Ceph?
>>
>> I have another suggestion, we can consider adding this capability in
>> nova manage CLI, since it's just may be used by operator or
>> administrator(at least for create/update/delete command), right?
> 
> Could you maybe add a short explanation what the use case is? IOW,
> when and why do I (in whatever role) prefetch images? 
> 

This previous OpenStack Summit presentation at Paris is a great example
of use case:

https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/cold-start-booting-of-1000-vms-under-10-minutes

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Incompatible default parameters

[Mathieu Gagné]

This change [1] introduced the nova_catalog_info and
nova_catalog_admin_info parameters.

Martin Mágr also mentioned that default values were wrong but the change
got merged anyway.

I agree that those default values should be changed to "nova" like
initially suggested as the 2nd field is the service_name, not
service_description. See default values in Cinder. [2]

This also means default values won't work for anyone, even people with
new or existing installations.

The default values in our manifests should be changed. There is no
backward compatibility breakage since it never worked in the first place.

Furthermore, existing installations NEED to update the default values
through Hiera or other means to make it work. Those people won't see the
change in default values.

I also think those parameters (and os_privileged_user_*) are in the
wrong manifest. Those values are also used by the cinder-volume service
to perform hypervisor assisted snapshots. They need to be moved.

[1] https://review.openstack.org/#/c/219284/2
[2]
http://docs.openstack.org/kilo/config-reference/content/section_cinder.conf.html

Mathieu

On 2015-10-21 9:39 AM, Clayton O'Neill wrote:
> I think the only people that would be affected by the puppet-openstack
> module default not matching the keystone default are people that are
> trying to retrofit puppet into an already running environment.  Those
> people already have a ton of work laid out in front of them, so I’m ok
> with making it very slightly more difficult for them (if those people
> even exist).
> 
> However, for existing users of the modules, this is potentially a big
> pain, since we already have existing catalogs with the old (wrong) name.
> 
> For new users of the module, this might be slightly annoying, but again,
> they can override it.
> 
> Personally I’d prefer to leave it as is, instead of requiring all
> existing users of the modules to change.
> 
> On Wed, Oct 21, 2015 at 9:31 AM, Martin Mágr  > wrote:
> 
> Greetings,
> 
>   I'd like to discuss bug 1506061 ([1]). The main problem I'm trying
> to solve here is that using default parameters of classes
> cinder::api and nova::keystone::auth migration won't work, because
> Cinder will search for endpoint with different name ('Compute
> service' instead of 'nova'). I've submitted fix for that in first
> patchset of [2], but it was denied as backward incompatible change,
> which I agree with, and instead I just added warnings about rename
> in next cycle [3].
> 
>   So the question is if we should start following endpoint naming
> according to Keystone's default catalog or just change default value
> of $nova_catalog_.*info parameters in cinder::api to match endpoint
> created by nova::keystone::auth? I don't mind going one way or
> another, but I do think that default parameters has to create fully
> functional environment.
> 
> Thanks in advance for answers,
> Martin
> 
> [1] https://bugs.launchpad.net/puppet-nova/+bug/1506061
> [2] https://review.openstack.org/#/c/222120/1
> [3]
> 
> https://review.openstack.org/#/q/status:open+branch:master+topic:bug/1506061,n,z
> [4] https://review.openstack.org/#/c/219284/2/manifests/api.pp
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [nova] live migration in Mitaka

[Mathieu Gagné]

Hi Pavel,

On 2015-10-08 9:39 AM, Pavel Boldin wrote:
> Here you go: https://launchpad.net/~pboldin/+archive/ubuntu/libvirt-python
> 
> Use it, but please keep in mind that this is a draft reupload.
> 

Thank you for your work. I'm sure a lot of people will benefit from it.
Live migration is a major keystone to our operation excellence and
anything that can improve it is welcomed.

I see the package is built against Wily. Will it work against Trusty? I
would say yes. It depends on libvirt0 (>= 1.2.16-2ubuntu11) which is
already available in the liberty repo of UCA.

Meanwhile, I managed to package 1.2.20 for test purposes for both Trusty
and Precise (not without raging =) as I needed to move on.

On a side note, I feel libvirt is such an important piece that it would
warrant its own official PPA. (both libvirt and libvirt-python)

I added comments to the review [1] because the change doesn't work in
its current state. I tested the patch against Kilo + Trusty + libvirt
1.2.20.

My main challenge at the moment is backporting the patch further to
Icehouse+Precise so I can move instances out of precise nodes and
upgrade them to trusty. Some far, and this is very early results, my
instances are stuck in "MIGRATING" state although they were completely
migrated to the destination. (domain is running on destination, removed
from source, ping is responding and I can SSH fine)

[1] https://review.openstack.org/#/c/227278/

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [nova] live migration in Mitaka

[Mathieu Gagné]

On 2015-10-02 4:18 PM, Pavel Boldin wrote:
> 
> You have to pass device names from /dev/, e.g., if a VM has
> ephemeral disk
> attached at /dev/vdb you need to pass in 'vdb'. Format expected by
> migrate_disks is ",...".
> 
> 
> This is the format expected by the `virsh' utility and will not work in
> Python.
> 
> The libvirt-python has now support for passing lists to a parameter [1].
> 
> [1]
> http://libvirt.org/git/?p=libvirt-python.git;a=commit;h=9896626b8277e2ffba1523d2111c96b08fb799e8
>  

Thanks for the info. I was banging my head against the wall, trying to
understand why it didn't accept my list of strings.

Now the next challenge is with Ubuntu packages, only python-libvirt
1.2.15 is available in Ubuntu Willy. :-/

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live migration in Mitaka

[Mathieu Gagné]

On 2015-10-01 7:26 AM, Kashyap Chamarthy wrote:
> On Wed, Sep 30, 2015 at 11:25:12AM +, Murray, Paul (HP Cloud) wrote:
>>
>>> Please respond to this post if you have an interest in this and what
>>> you would like to see done.  Include anything you are already
>>> getting on with so we get a clear picture. 
>>
>> Thank you to those who replied to this thread. I have used the
>> contents to start an etherpad page here:
>>
>> https://etherpad.openstack.org/p/mitaka-live-migration
> 
> I added a couple of URLs for upstream libvirt work that allow for
> selective block device migration, and the in-progress generic TLS
> support work by Dan Berrange in upstream QEMU.
> 
>> I have taken the liberty of listing those that responded to the thread
>> and the authors of mentioned patches as interested people.
>  
>> From the responses and looking at the specs up for review it looks
>> like there are about five areas that could be addressed in Mitaka and
>> several others that could come later. The first five are:
>>
>>
>> - migrating instances with a mix of local disks and cinder volumes 
> 
> IIUC, this is possible with the selective block device migration work
> merged in upstream libvirt:
> 
> https://www.redhat.com/archives/libvir-list/2015-May/msg00955.html
> 

Can someone explain to me what is the actual "disk name" I have to pass
in to libvirt? I couldn't find any documentation about how to use this
feature.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [cinder] [all] The future of Cinder API v1

[Mathieu Gagné]

On 2015-09-28 11:43 PM, John Griffith wrote:
> 
> 
> On Mon, Sep 28, 2015 at 6:19 PM, Mark Voelker  > wrote:
> 
> FWIW, the most popular client libraries in the last user survey[1]
> other than OpenStack’s own clients were: libcloud (48 respondents),
> jClouds (36 respondents), Fog (34 respondents), php-opencloud (21
> respondents), DeltaCloud (which has been retired by Apache and
> hasn’t seen a commit in two years, but 17 respondents are still
> using it), pkgcloud (15 respondents), and OpenStack.NET (14
> respondents).  Of those:
> 
> * libcloud appears to support the nova-volume API but not the cinder
> API:
> 
> https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/openstack.py#L251
> 
> * jClouds appears to support only the v1 API:
> 
> https://github.com/jclouds/jclouds/tree/jclouds-1.9.1/apis/openstack-cinder/src/main/java/org/jclouds
> 
> * Fog also appears to only support the v1 API:
> https://github.com/fog/fog/blob/master/lib/fog/openstack/volume.rb#L99
> 
> * php-opencloud appears to only support the v1 API:
> https://php-opencloud.readthedocs.org/en/latest/services/volume/index.html
> 
> * DeltaCloud I honestly haven’t looked at since it’s thoroughly
> dead, but I can’t imagine it supports v2.
> 
> * pkgcloud has beta-level support for Cinder but I think it’s v1
> (may be mistaken):
> https://github.com/pkgcloud/pkgcloud/#block-storagebeta and
> 
> https://github.com/pkgcloud/pkgcloud/tree/master/lib/pkgcloud/openstack/blockstorage
> 
> * OpenStack.NET does appear to support v2:
> 
> http://www.openstacknetsdk.org/docs/html/T_net_openstack_Core_Providers_IBlockStorageProvider.htm
> 
> Now, it’s anyone’s guess as to whether or not users of those client
> libraries actually try to use them for volume operations or not
> (anecdotally I know a few clouds I help support are using client
> libraries that only support v1), and some users might well be using
> more than one library or mixing in code they wrote themselves.  But
> most of the above that support cinder do seem to rely on v1.  Some
> management tools also appear to still rely on the v1 API (such as
> RightScale:
> http://docs.rightscale.com/clouds/openstack/openstack_config_prereqs.html
> ).  From that perspective it might be useful to keep it around a
> while longer and disable it by default.  Personally I’d probably
> lean that way, especially given that folks here on the ops list are
> still reporting problems too.
> 
> That said, v1 has been deprecated since Juno, and the Juno release
> notes said it was going to be removed [2], so there’s a case to be
> made that there’s been plenty of fair warning too I suppose.
> 
> [1]
> 
> http://superuser.openstack.org/articles/openstack-application-developers-share-insights
> [2] https://wiki.openstack.org/wiki/ReleaseNotes/Juno#Upgrade_Notes_7
> 
> At Your Service,
> 
> Mark T. Voelker
> 
> 
> 
> > On Sep 28, 2015, at 7:17 PM, Sam Morrison  > wrote:
> >
> > Yeah we’re still using v1 as the clients that are packaged with
> most distros don’t support v2 easily.
> >
> > Eg. with Ubuntu Trusty they have version 1.1.1, I just updated our
> “volume” endpoint to point to v2 (we have a volumev2 endpoint too)
> and the client breaks.
> >
> > $ cinder list
> > ERROR: OpenStack Block Storage API version is set to 1 but you are
> accessing a 2 endpoint. Change its value through
> --os-volume-api-version or env[OS_VOLUME_API_VERSION].
> >
> > Sam
> >
> >
> >> On 29 Sep 2015, at 8:34 am, Matt Fischer  > wrote:
> >>
> >> Yes, people are probably still using it. Last time I tried to use
> V2 it didn't work because the clients were broken, and then it went
> back on the bottom of my to do list. Is this mess fixed?
> >>
> >>
> 
> http://lists.openstack.org/pipermail/openstack-operators/2015-February/006366.html
> >>
> >> On Mon, Sep 28, 2015 at 4:25 PM, Ivan Kolodyazhny  > wrote:
> >> Hi all,
> >>
> >> As you may know, we've got 2 APIs in Cinder: v1 and v2. Cinder v2
> API was introduced in Grizzly and v1 API is deprecated since Juno.
> >>
> >> After [1] is merged, Cinder API v1 is disabled in gates by
> default. We've got a filed bug [2] to remove Cinder v1 API at all.
> >>
> >>
> >> According to Deprecation Policy [3] looks like we are OK to
> remote it. But I would like to ask Cinder API users if any still use
> API v1.
> >> Should we remove it at all Mitaka release or just disable by
> default in the cinder.conf?
> >>
> >> AFAIR, only Rally 

Re: [openstack-dev] [nova][cinder] how to handle AZ bug 1496235?

[Mathieu Gagné]

On 2015-09-24 11:53 AM, Walter A. Boring IV wrote:
> The good thing about the Nova and Cinder clients/APIs is that
> anyone can write a quick python script to do the orchestration
> themselves, if we want to deprecate this.  I'm all for deprecating this.

I don't like this kind of reasoning which can justify close to anything.
It's easy to make those suggestions when you know Python. Please
consider non-technical/non-developers users when suggesting deprecating
features or proposing alternative solutions.

I could also say (in bad faith, I know): why have Heat when you can
write your own Python script. And yet, I don't think we would appreciate
anyone making such a controversial statement.

Our users don't know Python, use 3rd party tools (which don't often
perform/support orchestration) or the Horizon dashboard. They don't want
to have to learn Heat or Python so they can orchestrate volume creation
in place of Nova for a single instance. You don't write CloudFormation
templates on AWS just to boot an instance on volume. That's not the UX I
want to offer to my users.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] how to handle AZ bug 1496235?

[Mathieu Gagné]

Hi Matt,

On 2015-09-24 1:45 PM, Matt Riedemann wrote:
> 
> 
> On 9/24/2015 11:50 AM, Mathieu Gagné wrote:
>>
>> May I suggest the following solutions:
>>
>> 1) Add ability to disable this whole AZ concept in Cinder so it doesn't
>> fail to create volumes when Nova asks for a specific AZ. This could
>> result in the same behavior as cinder.cross_az_attach config.
> 
> That's essentially what this does:
> 
> https://review.openstack.org/#/c/217857/
> 
> It defaults to False though so you have to be aware and set it if you're
> hitting this problem.
> 
> The nova block_device code that tries to create the volume and passes
> the nova AZ should have probably been taking into account the
> cinder.cross_az_attach config option, because just blindly passing it
> was the reason why cinder added that option.  There is now a change up
> for review to consider cinder.cross_az_attach in block_device:
> 
> https://review.openstack.org/#/c/225119/
> 
> But that's still making the assumption that we should be passing the AZ
> on the volume create request and will still fail if the AZ isn't in
> cinder (and allow_availability_zone_fallback=False in cinder.conf).
> 
> In talking with Duncan this morning he's going to propose a spec for an
> attempt to clean some of this up and decouple nova from handling this
> logic.  Basically a new Cinder API where you give it an AZ and it tells
> you if that's OK.  We could then use this on the nova side before we
> ever get to the compute node and fail.

IMO, the confusion comes from what I consider a wrong usage of AZ. To
quote Sylvain Bauza from a recent review [1][2]:

"because Nova AZs and Cinder AZs are very different failure domains"

This is not the concept of AZ I learned to know from cloud providers
where an AZ is global to the region, not per-service.

Google Cloud Platform:
- Persistent disks are per-zone resources. [3]
- Resources that are specific to a zone or a region can only be used by
other resources in the same zone or region. For example, disks and
instances are both zonal resources. To attach a disk to an instance,
both resources must be in the same zone. [4]

Amazon Web Services:
- Instances and disks are per-zone resources. [5]

So now we are stuck with AZ not being consistent across services and
confusing people.


[1] https://review.openstack.org/#/c/225119/2
[2] https://review.openstack.org/#/c/225119/2/nova/virt/block_device.py
[3] https://cloud.google.com/compute/docs/disks/persistent-disks
[4] https://cloud.google.com/compute/docs/zones
[5] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/resources.html

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] how to handle AZ bug 1496235?

[Mathieu Gagné]

On 2015-09-24 3:04 AM, Duncan Thomas wrote:
> 
> I proposed a session at the Tokyo summit for a discussion of Cinder AZs,
> since there was clear confusion about what they are intended for and how
> they should be configured. Since then I've reached out to and gotten
> good feedback from, a number of operators.

Thanks for your proposition. I will make sure to attend this session.


> There are two distinct
> configurations for AZ behaviour in cinder, and both sort-of worked until
> very recently.
> 
> 1) No AZs in cinder
> This is the config where a single 'blob' of storage (most of the
> operators who responded so far are using Ceph, though that isn't
> required). The storage takes care of availability concerns, and any AZ
> info from nova should just be ignored.

Unless I'm very mistaken, I think it's the main "feature" missing from
OpenStack itself. The concept of AZ isn't global and anyone can still
make it so Nova AZ != Cinder AZ.

In my opinion, AZ should be a global concept where they are available
and the same for all services so Nova AZ == Cinder AZ. This could result
in a behavior similar to "regions within regions".

We should survey and ask how AZ are actually used by operators and
users. Some might create an AZ for each server racks, others for each
power segments in their datacenter or even business units so they can
segregate to specific physical servers. Some AZ use cases might just be
a "perverted" way of bypassing shortcomings in OpenStack itself. We
should find out those use cases and see if we should still support them
or offer them an existing or new alternatives.

(I don't run Ceph yet, only SolidFire but I guess the same could apply)

For people running Ceph (or other big clustered block storage), they
will have one big Cinder backend. For resources or business reasons,
they can't afford to create as many clusters (and Cinder AZ) as there
are AZ in Nova. So they end up with one big Cinder AZ (lets call it
az-1) in Cinder. Nova won't be able to create volumes in Cinder az-2 if
an instance is created in Nova az-2.

May I suggest the following solutions:

1) Add ability to disable this whole AZ concept in Cinder so it doesn't
fail to create volumes when Nova asks for a specific AZ. This could
result in the same behavior as cinder.cross_az_attach config.

2) Add ability for a volume backend to be in multiple AZ. Of course,
this would defeat the whole AZ concept. This could however be something
our operators/users might accept.


> 2) Cinder AZs map to Nova AZs
> In this case, some combination of storage / networking / etc couples
> storage to nova AZs. It is may be that an AZ is used as a unit of
> scaling, or it could be a real storage failure domain. Eitehr way, there
> are a number of operators who have this configuration and want to keep
> it. Storage can certainly have a failure domain, and limiting the
> scalability problem of storage to a single cmpute AZ can have definite
> advantages in failure scenarios. These people do not want cross-az attach.
> 
> My hope at the summit session was to agree these two configurations,
> discuss any scenarios not covered by these two configuration, and nail
> down the changes we need to get these to work properly. There's
> definitely been interest and activity in the operator community in
> making nova and cinder AZs interact, and every desired interaction I've
> gotten details about so far matches one of the above models.


-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] how to handle AZ bug 1496235?

[Mathieu Gagné]

On 2015-09-23 4:50 PM, Andrew Laski wrote:
> On 09/23/15 at 04:30pm, Mathieu Gagné wrote:
>> On 2015-09-23 4:12 PM, Andrew Laski wrote:
>>> On 09/23/15 at 02:55pm, Matt Riedemann wrote:
>>>>
>>>> Heh, so when I just asked in the cinder channel if we can just
>>>> deprecate nova boot from volume with source=(image|snapshot|blank)
>>>> (which automatically creates the volume and polls for it to be
>>>> available) and then add a microversion that doesn't allow it, I was
>>>> half joking, but I see we're on the same page.  This scenario seems to
>>>> introduce a lot of orchestration work that nova shouldn't necessarily
>>>> be in the business of handling.
>>>
>>> I am very much in support of this.  This has been a source of
>>> frustration for our users because it is prone to failures we can't
>>> properly expose to users and timeouts.  There are much better places to
>>> handle the orchestration of creating a volume and then booting from it
>>> than Nova.
>>>
>>
>> Unfortunately, this is a feature our users *heavily* rely on and we
>> worked very hard to make it happen. We had a private patch on our side
>> for years to optimize boot-from-volume before John Griffith came up with
>> an upstream solution for SolidFire [2] and others with a generic
>> solution [3] [4].
>>
>> Being able to "nova boot" and have everything done for you is awesome.
>> Just see what Monty Taylor mentioned in his thread about sane default
>> networking [1]. Having orchestration on the client side is just
>> something our users don't want to have to do and often complain about.
> 
> At risk of getting too offtopic I think there's an alternate solution to
> doing this in Nova or on the client side.  I think we're missing some
> sort of OpenStack API and service that can handle this.  Nova is a low
> level infrastructure API and service, it is not designed to handle these
> orchestrations.  I haven't checked in on Heat in a while but perhaps
> this is a role that it could fill.
> 
> I think that too many people consider Nova to be *the* OpenStack API
> when considering instances/volumes/networking/images and that's not
> something I would like to see continue.  Or at the very least I would
> like to see a split between the orchestration/proxy pieces and the
> "manage my VM/container/baremetal" bits.
> 

"too many people" happens to include a lot of 3rd party tools supporting
OpenStack which our users complain a lot about. Just see all the
possible way to get an external IP [5]. Introducing yet another service
would increase the pain on our users which will see their tools and
products not working even more.

Just see how EC2 is doing it [6], you won't see them suggest to use yet
another service to orchestrate what I consider a fundamental feature "I
wish to boot an instance on a volume".

The current ease to boot from volume is THE selling feature our users
want and heavily/actively use. We fought very hard to make it work and
reading about how it should be removed is frustrating.

Issues we identified shouldn't be a reason to drop this feature. Other
providers are making it work and I don't see why we couldn't. I'm
convinced we can do better.

[5]
https://github.com/openstack-infra/shade/blob/03c1556a12aabfc21de60a9fac97aea7871485a3/shade/meta.py#L106-L173
[6]
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html

Mathieu

>>
>> [1]
>> http://lists.openstack.org/pipermail/openstack-dev/2015-September/074527.html
>>
>> [2] https://review.openstack.org/#/c/142859/
>> [3] https://review.openstack.org/#/c/195795/
>> [4] https://review.openstack.org/#/c/201754/
>>
>> -- 
>> Mathieu


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][cinder] how to handle AZ bug 1496235?

[Mathieu Gagné]

On 2015-09-23 4:12 PM, Andrew Laski wrote:
> On 09/23/15 at 02:55pm, Matt Riedemann wrote:
>>
>> Heh, so when I just asked in the cinder channel if we can just
>> deprecate nova boot from volume with source=(image|snapshot|blank)
>> (which automatically creates the volume and polls for it to be
>> available) and then add a microversion that doesn't allow it, I was
>> half joking, but I see we're on the same page.  This scenario seems to
>> introduce a lot of orchestration work that nova shouldn't necessarily
>> be in the business of handling.
> 
> I am very much in support of this.  This has been a source of
> frustration for our users because it is prone to failures we can't
> properly expose to users and timeouts.  There are much better places to
> handle the orchestration of creating a volume and then booting from it
> than Nova.
> 

Unfortunately, this is a feature our users *heavily* rely on and we
worked very hard to make it happen. We had a private patch on our side
for years to optimize boot-from-volume before John Griffith came up with
an upstream solution for SolidFire [2] and others with a generic
solution [3] [4].

Being able to "nova boot" and have everything done for you is awesome.
Just see what Monty Taylor mentioned in his thread about sane default
networking [1]. Having orchestration on the client side is just
something our users don't want to have to do and often complain about.

[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-September/074527.html
[2] https://review.openstack.org/#/c/142859/
[3] https://review.openstack.org/#/c/195795/
[4] https://review.openstack.org/#/c/201754/

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Consistent support for SSL termination proxies across all API services

[Mathieu Gagné]

On 2015-09-22 1:46 PM, Sean Dague wrote:
> On 09/22/2015 12:12 PM, Mathieu Gagné wrote:
>> On 2015-09-22 7:00 AM, Sean Dague wrote:
>>>
>>> My feeling on this one is that we've got this thing in OpenStack... the
>>> Service Catalog. It definitively tells the world what the service
>>> addresses are.
>>>
>>> We should use that in the services themselves to reflect back their
>>> canonical addresses. Doing point solution rewriting of urls seems odd
>>> when we could just have Nova/Cinder/etc return documents with URLs that
>>> match what's in the service catalog for that service.
>>>
>>
>> Sorry, this won't work for us. We have a "split view" in our service
>> catalog where internal management nodes have a specific catalog and
>> public nodes (for users) have a different one.
>>
>> Implementing the secure_proxy_ssl_header config would require close to
>> little code change to all projects and accommodate our use case and
>> other ones we might not think of. For example, how do you know "from"
>> which of the following URLs (publicURL, internalURL, adminURL) the users
>> is coming? Each might be different and even not all be SSL.
>>
>> The oslo.middleware project already has the SSL middleware [1]. It would
>> only be a matter of enabling this middleware by default in the paste
>> config of all projects.
>>
>> [1]
>> https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/ssl.py
> 
> The split view definitely needs to be considered, but a big question
> here is whether we should really be doing this with multiple urls per
> catalog entry, or dedicated catalog entries for internal usage.

We are using a dedicated catalog for internal usage and override service
endpoint wherever possible in OpenStack services. We don't use
publicURL, internalURL or adminURL.


> There are a lot of things to work through to get our use of the service
> catalog consistent and useful going forward. I just don't relish another
> layer of work arounds that decide the service catalog is not a good way
> to keep track of what our service urls are, that has to be unwound later.

The oslo_middleware.ssl middleware looks to offer little overhead and
offer the maximum flexibility. I appreciate the wish to use the Keystone
catalog but I don't feel this is the right answer.

For example, if I deploy Bifrost without Keystone, I won't have a
catalog to rely on and will still have the same lack of SSL termination
proxy support.

The simplest solution is often the right one.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Consistent support for SSL termination proxies across all API services

[Mathieu Gagné]

On 2015-09-22 4:52 PM, Sean Dague wrote:
> On 09/22/2015 03:16 PM, Mathieu Gagné wrote:
>>
>> The oslo_middleware.ssl middleware looks to offer little overhead and
>> offer the maximum flexibility. I appreciate the wish to use the Keystone
>> catalog but I don't feel this is the right answer.
>>
>> For example, if I deploy Bifrost without Keystone, I won't have a
>> catalog to rely on and will still have the same lack of SSL termination
>> proxy support.
>>
>> The simplest solution is often the right one.
> 
> I do get there are specific edge cases here, but I don't think that in
> the general case we should be pushing a mode where Keystone is optional.
> It is a bedrock of our system.
> 

I understand that Keystone is "a bedrock of our system". This however
does not make it THE solution when simpler proven existing ones exist. I
fail to understand why other solutions can't be considered.

I opened a dialog with the community to express my concerns about the
lack of universal support for SSL termination proxy so we can find a
solution together which will cover ALL use cases.

I proposed using an existing solution (oslo_middleware.ssl) (that I
didn't know of until now) which takes little to no time to implement and
cover a lot of use cases and special edge cases.

Operators encounters and *HAVE* to handle a lot of edge cases. We are
trying *very hard* to open up a dialog with the developer community so
they can be heard, understood and addressed by sharing our real world
use cases.

In that specific case, my impression is that they unfortunately happens
to not be a priority when evaluating solutions and will actually make my
job harder. I'm sad to see we can't come with an agreement on that one.
I feel like I failed to properly communicate my needs as an operator and
make them understood to others.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Consistent support for SSL termination proxies across all API services

[Mathieu Gagné]

On 2015-09-22 7:00 AM, Sean Dague wrote:
> 
> My feeling on this one is that we've got this thing in OpenStack... the
> Service Catalog. It definitively tells the world what the service
> addresses are.
> 
> We should use that in the services themselves to reflect back their
> canonical addresses. Doing point solution rewriting of urls seems odd
> when we could just have Nova/Cinder/etc return documents with URLs that
> match what's in the service catalog for that service.
> 

Sorry, this won't work for us. We have a "split view" in our service
catalog where internal management nodes have a specific catalog and
public nodes (for users) have a different one.

Implementing the secure_proxy_ssl_header config would require close to
little code change to all projects and accommodate our use case and
other ones we might not think of. For example, how do you know "from"
which of the following URLs (publicURL, internalURL, adminURL) the users
is coming? Each might be different and even not all be SSL.

The oslo.middleware project already has the SSL middleware [1]. It would
only be a matter of enabling this middleware by default in the paste
config of all projects.

[1]
https://github.com/openstack/oslo.middleware/blob/master/oslo_middleware/ssl.py

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all] Consistent support for SSL termination proxies across all API services

[Mathieu Gagné]

Hi,

While debugging LP bug #1491579 [1], we identified [2] an issue where an
API sitting being a proxy performing SSL termination would not generate
the right redirection. The protocol ends up being the wrong one (http
instead of https) and this could hang your request indefinitely if
tcp/80 is not opened and a firewall drops your connection.

I suggested [3] adding support for the X-Fowarded-Proto header, thinking
Nova didn't supported it yet. In fact, someone suggested setting the
public_endpoint config instead.

So today I stumbled across this review [4] which added the
secure_proxy_ssl_header config to Nova. It allows the API to detect SSL
termination based on the (suggested) header X-Forwarded-Proto just like
previously suggested.

I also found this bug report [5] (opened in 2014) which also happens to
complain about bad URLs when API is sitting behind a proxy.

Multiple projects applied patches to try to fix the issue (based on
Launchpad comments):

* Glance added public_endpoint config
* Cinder added public_endpoint config
* Heat added secure_proxy_ssl_header config (through
heat.api.openstack:sslmiddleware_filter)
* Nova added secure_proxy_ssl_header config
* Manila added secure_proxy_ssl_header config (through
oslo_middleware.ssl:SSLMiddleware.factory)
* Ironic added public_endpoint config
* Keystone added secure_proxy_ssl_header config (LP #1370022)

As you can see, there is a lot of inconsistency between projects. (there
is more but lets start with that one)

My wish is for a common and consistent way for *ALL* OpenStack APIs to
support the same solution for this common problem. Let me tell you (and
I guess I can speak for all operators), we will be very happy to have
ONE config to remember of and set for *ALL* OpenStack services.

How can we get the ball rolling so we can fix it together once and for
all in a timely fashion?

[1] https://bugs.launchpad.net/python-novaclient/+bug/1491579
[2] https://bugs.launchpad.net/python-novaclient/+bug/1491579/comments/15
[3] https://bugs.launchpad.net/python-novaclient/+bug/1491579/comments/17
[4] https://review.openstack.org/#/c/206479/
[5] https://bugs.launchpad.net/glance/+bug/1384379

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

[Mathieu Gagné]

Hi Kevin,

On 2015-09-15 8:33 PM, Fox, Kevin M wrote:
> I am not a neutron developer, but an operator and a writer of cloud apps.

So far, I'm only an operator and heavy cloud apps user (if you can call
Vagrant an app). =)


> Yes, it is sort of a philosophical issue, and I have stated my side
> of why I think the extra complexity is worth it. Feel free to
> disagree.

I don't disagree with the general idea of NaaS or SDN. We are looking to
offer this stuff in the future so customers wishing to have more control
over their networks can have it.

I would however like for other solutions (which doesn't require
mandatory NATing, floating IPs, and routers) to be accepted and fully
supported as first class citizen.


> But either way I don't think we can ignore the complexity. There are
> three different ways to resolve it:
> 
> * No naas and naas are both supported. Ops get it easy. they pick
> which one they want, users suffer a little if they work on multiple
> clouds that differ. app developers suffer a lot since they have to
> write either two sets of software or pick the lowest common
> denominator.
> 
> Its an optimization problem. who do you shift the difficulty to?
> 
> My personal opinion again is that I'd rather suffer a little more as
> an Op and always deploy naas, rather then have to deal with the app
> developer pain of not being able to rely on it. The users/ops benefit
> the most if a strong app ecosystem can be developed on top.


So far, I'm aiming for "No naas and naas are both supported.":

- No NaaS: A public shared and routable provider network.
- NaaS: All the goodness of SDN and private networks.

While NaaS is a very nice feature for cloud apps writer, we found that
our type of users actually don't ask for it (yet) and are looking for
simplicity instead.

BTW, let me know if I got my understanding of "No naas" (very?) wrong.


As Monty Taylor said [3], we should be able to "nova boot" or "nova boot
--network public" just fine.

So lets assume I don't have NaaS yet. I only have 1 no-NaaS network
named "public" available to all tenants.

With this public shared network from no-NaaS, you should be able to boot
just fine. Your instance ends up on a public shared network with a
public IP address without NATing/Floating IPs and such. (Note that we
implemented anti-spoofing on those networks)


Now you wish to use NaaS. So you create this network named "private" or
whatever you feel naming it.

You should be fine too with "nova boot --network private" by making sure
the network name doesn't conflict with the public shared network.
Otherwise you can provide the network UUID just like before. I agree
that you loose the ability to "nova boot" without "--network". See below.


The challenge I see here is with both "no-NaaS" and "NaaS". Now you
could end up with 2 or more networks to choose from and "nova boot"
alone will get confused.


My humble suggestions are:

- Create a new client-side config to tell which network name to choose
(OS_NETWORK_NAME?) so you don't have to type it each time.
- Create a tenant specific server-side config (stored somewhere in
Nova?) to tell which network name to choose from by default.

This will restore the coolness of "nova boot" without specifying
"--network".

If you application requires a lot of networks (and complexity), I'm sure
all these "nova boot" is non-sense to you anyway and that you will
provide the actual list of networks to boot on.


Regarding the need and wish of users to keep their public IPs, you can
still use floating IPs in both cases. It's a matter of educating the
users that public IPs on the no-NaaS network aren't preserved on
destruction. I'm planning to use routes instead of NATing for the public
shared network.


So far, what I'm missing to create a truly scalable public shared
network is what is described here [2] and here [3] as you just can't
scale your L2 network infinitely. (same for private NaaS networks but
that's an other story)


Note that I'm fully aware that it creates a lot of challenges on the
Nova side related to scheduling, resizing, live migrations, evacuate,
etc. But I'm confident that those challenges aren't impossible to overcome.


Kevin, let me know if I missed a known use case you might be actively
using. I never fully used the NaaS part of Neutron so I can't tell for
sure. Or maybe I'm just stating obvious stuff and completely missing the
point of this thread. :D


[1]
http://lists.openstack.org/pipermail/openstack-dev/2015-September/074618.html
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-July/070028.html
[3] https://review.openstack.org/#/c/196812/


-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

[Mathieu Gagné]

On 2015-09-15 7:44 PM, Armando M. wrote:
> 
> 
> On 15 September 2015 at 15:11, Mathieu Gagné <mga...@internap.com
> <mailto:mga...@internap.com>> wrote:
> 
> On 2015-09-15 2:00 PM, Fox, Kevin M wrote:
> > We run several clouds where there are multiple external networks. the 
> "just run it in on THE public network" doesn't work. :/
> >
> > I also strongly recommend to users to put vms on a private network and 
> use floating ip's/load balancers. For many reasons. Such as, if you don't, 
> the ip that gets assigned to the vm helps it become a pet. you can't replace 
> the vm and get the same IP. Floating IP's and load balancers can help prevent 
> pets. It also prevents security issues with DNS and IP's. Also, for every 
> floating ip/lb I have, I usually have 3x or more the number of instances that 
> are on the private network. Sure its easy to put everything on the public 
> network, but it provides much better security if you only put what you must 
> on the public network. Consider the internet. would you want to expose every 
> device in your house directly on the internet? No. you put them in a private 
> network and poke holes just for the stuff that does. we should be encouraging 
> good security practices. If we encourage bad ones, then it will bite us later 
> when OpenStack gets a reputation for being associated with compromises.
> >
> 
> Sorry but I feel this kind of reply explains why people are still using
> nova-network over Neutron. People want simplicity and they are denied it
> at every corner because (I feel) Neutron thinks it knows better.
> 
> 
> I am sorry, but how can you associate a person's opinion to a project,
> which is a collectivity? Surely everyone is entitled to his/her opinion,
> but I don't honestly believe these are fair statements to make.

You are right, this is not fair. I apologize for that.


> The original statement by Monty Taylor is clear to me:
> 
> I wish to boot an instance that is on a public network and reachable
> without madness.
> 
> As of today, you can't unless you implement a deployer/provider specific
> solution (to scale said network). Just take a look at what actual public
> cloud providers are doing:
> 
> - Rackspace has a "magic" public network
> - GoDaddy has custom code in their nova-scheduler (AFAIK)
> - iWeb (which I work for) has custom code in front of nova-api.
> 
> We are all writing our own custom code to implement what (we feel)
> Neutron should be providing right off the bat.
> 
> 
> What is that you think Neutron should be providing right off the bat? I
> personally have never seen you publicly report usability issues that
> developers could go and look into. Let's escalate these so that the
> Neutron team can be aware.

Please understand that I'm an operator and don't have the luxury to
contribute as much as I did before. I however participate to OpenStack
Ops meetup and this is the kind of things we discuss. You can read the
use cases below to understand what I'm referring to. I don't feel the
need to add yet another version of it since there are already multiple
ones identifying my needs.

People (such as Monty) are already voicing my concerns and I didn't feel
the need to voice mine too.


> By reading the openstack-dev [1], openstack-operators [2] lists, Neutron
> specs [3] and the Large Deployment Team meeting notes [4], you will see
> that what is suggested here (a scalable public shared network) is an
> objective we wish but are struggling hard to achieve.
> 
> 
> There are many ways to skin this cat IMO, and scalable public shared
> network can really have multiple meanings, I appreciate the pointers
> nonetheless.
>  
> 
> 
> People keep asking for simplicity and Neutron looks to not be able to
> offer it due to philosophical conflicts between Neutron developers and
> actual public users/operators. We can't force our users to adhere to ONE
> networking philosophy: use NAT, floating IPs, firewall, routers, etc.
> They just don't buy it. Period. (see monty's list of public providers
> attaching VMs to public network)
> 
> 
> Public providers networking needs are not the only needs that Neutron
> tries to gather. There's a balance to be struck, and I appreciate that
> the balance may need to be adjusted, but being so dismissive is being
> myopic of the entire industry landscape.

We (my employer) also maintain private clouds and I'm fully aware of the
different between those needs. Therefore I don't think it's fair to say
that my opinion in nearsighted. Nonetheless, I would like this balance
to be adjusted and that's what I'm asking for an

Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

[Mathieu Gagné]

On 2015-09-15 6:49 PM, Doug Wiegley wrote:
> 
> 
>> On Sep 15, 2015, at 4:11 PM, Mathieu Gagné <mga...@internap.com> wrote:
>>
>>> On 2015-09-15 2:00 PM, Fox, Kevin M wrote:
>>> We run several clouds where there are multiple external networks. the "just 
>>> run it in on THE public network" doesn't work. :/
>>>
>>> I also strongly recommend to users to put vms on a private network and use 
>>> floating ip's/load balancers. For many reasons. Such as, if you don't, the 
>>> ip that gets assigned to the vm helps it become a pet. you can't replace 
>>> the vm and get the same IP. Floating IP's and load balancers can help 
>>> prevent pets. It also prevents security issues with DNS and IP's. Also, for 
>>> every floating ip/lb I have, I usually have 3x or more the number of 
>>> instances that are on the private network. Sure its easy to put everything 
>>> on the public network, but it provides much better security if you only put 
>>> what you must on the public network. Consider the internet. would you want 
>>> to expose every device in your house directly on the internet? No. you put 
>>> them in a private network and poke holes just for the stuff that does. we 
>>> should be encouraging good security practices. If we encourage bad ones, 
>>> then it will bite us later when OpenStack gets a reputation for being 
>>> associated with compromises.
>>
>> Sorry but I feel this kind of reply explains why people are still using
>> nova-network over Neutron. People want simplicity and they are denied it
>> at every corner because (I feel) Neutron thinks it knows better.
> 
> Please stop painting such generalizations.  Go to the third or fourth email 
> in this thread and you will find a spec, worked on by neutron and nova, that 
> addresses exactly this use case.
> 
> It is a valid use case, and neutron does care about it. It has wrinkles. That 
> has not stopped work on it for the common cases.
> 

I've read the neutron spec you are referring (which I mentioned in my
email) and I'm glad the subject is discussed. This was not my intention
to diminish the work done by the Neutron team to address those issues. I
wrongly associate a person's opinion to a whole project, this is not
fair, I apologize for that.

Jeremy Stanley replied to Kevin with much better words than mine.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

[Mathieu Gagné]

On 2015-09-15 8:06 PM, Doug Wiegley wrote:
> 
> “Neutron doesn’t get it and never will.”
> 
> I’m not sure how all ‘yes’ above keeps translating to this old saw, but
> is there any tiny chance we can stop living in the past and instead
> focus on the use cases that we want to solve?
> 

I apologized for my unfair statement where I very wrongly associated a
person's opinion to a whole project. I would like to move on. Thanks

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][neutron][devstack] New proposed 'default' network model

[Mathieu Gagné]

On 2015-09-15 2:00 PM, Fox, Kevin M wrote:
> We run several clouds where there are multiple external networks. the "just 
> run it in on THE public network" doesn't work. :/
> 
> I also strongly recommend to users to put vms on a private network and use 
> floating ip's/load balancers. For many reasons. Such as, if you don't, the ip 
> that gets assigned to the vm helps it become a pet. you can't replace the vm 
> and get the same IP. Floating IP's and load balancers can help prevent pets. 
> It also prevents security issues with DNS and IP's. Also, for every floating 
> ip/lb I have, I usually have 3x or more the number of instances that are on 
> the private network. Sure its easy to put everything on the public network, 
> but it provides much better security if you only put what you must on the 
> public network. Consider the internet. would you want to expose every device 
> in your house directly on the internet? No. you put them in a private network 
> and poke holes just for the stuff that does. we should be encouraging good 
> security practices. If we encourage bad ones, then it will bite us later when 
> OpenStack gets a reputation for being associated with compromises.
> 

Sorry but I feel this kind of reply explains why people are still using
nova-network over Neutron. People want simplicity and they are denied it
at every corner because (I feel) Neutron thinks it knows better.

The original statement by Monty Taylor is clear to me:

I wish to boot an instance that is on a public network and reachable
without madness.

As of today, you can't unless you implement a deployer/provider specific
solution (to scale said network). Just take a look at what actual public
cloud providers are doing:

- Rackspace has a "magic" public network
- GoDaddy has custom code in their nova-scheduler (AFAIK)
- iWeb (which I work for) has custom code in front of nova-api.

We are all writing our own custom code to implement what (we feel)
Neutron should be providing right off the bat.

By reading the openstack-dev [1], openstack-operators [2] lists, Neutron
specs [3] and the Large Deployment Team meeting notes [4], you will see
that what is suggested here (a scalable public shared network) is an
objective we wish but are struggling hard to achieve.

People keep asking for simplicity and Neutron looks to not be able to
offer it due to philosophical conflicts between Neutron developers and
actual public users/operators. We can't force our users to adhere to ONE
networking philosophy: use NAT, floating IPs, firewall, routers, etc.
They just don't buy it. Period. (see monty's list of public providers
attaching VMs to public network)

If we can accept and agree that not everyone wishes to adhere to the
"full stack of networking good practices" (TBH, I don't know how to call
this thing), it will be a good start. Otherwise I feel we won't be able
to achieve anything.

What Monty is explaining and suggesting is something we (my team) have
been struggling with for *years* and just didn't have bandwidth (we are
operators, not developers) or public charisma to change.

I'm glad Monty brought up this subject so we can officially address it.


[1] http://lists.openstack.org/pipermail/openstack-dev/2015-July/070028.html
[2]
http://lists.openstack.org/pipermail/openstack-operators/2015-August/007857.html
[3]
http://specs.openstack.org/openstack/neutron-specs/specs/liberty/get-me-a-network.html
[4]
http://lists.openstack.org/pipermail/openstack-operators/2015-June/007427.html

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] This is what disabled-by-policy should look like to the user

[Mathieu Gagné]

On 2015-09-04 12:50 PM, Monty Taylor wrote:
> On 09/04/2015 10:55 AM, Morgan Fainberg wrote:
>>
>> Obviously the translation of errors
>> would be more difficult if the enforcer is generating messages.
> 
> The type: "PolicyNotAuthorized" is a good general key. Also - even
> though the command I sent was:
> 
> neutron net-create
> 
> On the command line, the entry in the policy_file is "create_network" -
> so honestly I think that policy.json and oslo.policy should have (or be
> able to have) all of the info needed to create almost the exact same
> message. Perhaps "NeutronError" would just need to be
> "OpenStackPolicyError"?
> 
> Oh. Wait. You meant translation like i18n translation. In that case, I
> think it's easy:
> 
> message=_("Policy doesn't allow %(policy_key)s to be performed",
> policy_key="create_network")
> 
> /me waves hands
> 

I don't feel like this error message would be user-friendly:

"Policy doesn't allow os_compute_api:os-instance-actions to be performed"

Policy name aren't human readable and match nothing on the client side.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] This is what disabled-by-policy should look like to the user

[Mathieu Gagné]

On 2015-09-04 2:41 PM, Monty Taylor wrote:
> On 09/04/2015 01:42 PM, John Griffith wrote:
>>
>> ​Is no good?  You would like to see "less" in the output; like just the
>> command name itself and "Policy doesn't allow"?
>>
>> To Mathieu's point, fair statement WRT the visibility of the policy name.
> 
> Totally agree on the policy name. The one I did happened to be clear -
> that is not always the case. I'd love to see that.
> 
> But more to your question - yes, as an end user, I do't know what a
> volume_extension:volume_admin_actions:reset_status is - but I do know
> that I ran "cinder reset-state" - so getting:
> 
> 'Cloud policy does not allow you to run reset_status"
> 
> would be fairly clear to me.

Don't assume the user will run it from the (supposedly) deprecated
Cinder CLI. It could be from the new openstackclient or even an SDK
written in Ruby which might not name it "reset_status".

I would prefer a generic message over an overly specific message which
makes a lot of wrong assumption about the consumer.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] --detailed-description for OpenStack items

[Mathieu Gagné]

On 2015-08-27 1:23 PM, Tim Bell wrote:
 
 Some project such as cinder include a detailed description option where
 you can include an arbitrary string with a volume to remind the admins
 what the volume is used for.
 
 Has anyone looked at doing something similar for Nova for instances and
 Glance for images ?
 
 In many cases, the names get heavily overloaded with information.
 

I like this idea. +1

We are currently abusing the metadata fields to manage description/alias
for our custom UI instead of the hostname field since people like to put
free text. We looked into using the description field but there was no
way to update it through the API, AFAIK.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Proposing Yanis Guenane core

[Mathieu Gagné]

+1

On 2015-07-27 3:06 PM, Emilien Macchi wrote:
 Puppet group,
 
 Yanis has been working in our group for a while now.
 He has been involved in a lot of tasks, let me highlight some of them:
 
 * Many times, involved in improving consistency across our modules.
 * Strong focus on data binding, backward compatibility and flexibility.
 * Leadership on cookiebutter project [1].
 * Active participation to meetings, always with actions, and thoughts.
 * Beyond the stats, he has a good understanding of our modules, and
 quite good number of reviews, regularly.
 
 Yanis is for our group a strong asset and I would like him part of our
 core team.
 I really think his involvement, regularity and strong knowledge in
 Puppet OpenStack will really help to make our project better and stronger.
 
 I would like to open the vote to promote Yanis part of Puppet OpenStack
 core reviewers.
 
 Best regards,
 
 [1] https://github.com/openstack/puppet-openstack-cookiecutter
 
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] [ironic] Exposing provider networks in network_data.json

[Mathieu Gagné]

(adding [ironic] since baremetal use cases are involved)

On 2015-07-17 11:51 AM, Jim Rollenhagen wrote:
 On Fri, Jul 17, 2015 at 01:06:46PM +0100, John Garbutt wrote:
 On 17 July 2015 at 11:23, Sean Dague s...@dague.net wrote:
 On 07/16/2015 06:06 PM, Sean M. Collins wrote:
 On Thu, Jul 16, 2015 at 01:23:29PM PDT, Mathieu Gagné wrote:
 So it looks like there is a missing part in this feature. There should
 be a way to hide this information if the instance does not require to
 configure vlan interfaces to make network functional.

 I just commented on the review, but the provider network API extension
 is admin only, most likely for the reasons that I think someone has
 already mentioned, that it exposes details of the phyiscal network
 layout that should not be exposed to tenants.

 So, clearly, under some circumstances the network operator wants to
 expose this information, because there was the request for that feature.
 The question in my mind is what circumstances are those, and what
 additional information needs to be provided here.

 There is always a balance between the private cloud case which wants to
 enable more self service from users (and where the users are often also
 the operators), and the public cloud case where the users are outsiders
 and we want to hide as much as possible from them.

 For instance, would an additional attribute on a provider network that
 says this is cool to tell people about be an acceptable approach? Is
 there some other creative way to tell our infrastructure that these
 artifacts are meant to be exposed in this installation?

 Just kicking around ideas, because I know a pile of gate hardware for
 everyone to use is at the other side of answers to these questions. And
 given that we've been running full capacity for days now, keeping this
 ball moving forward would be great.

 Maybe we just need to add policy around who gets to see that extra
 detail, and maybe hide it by default?

 Would that deal with the concerns here?
 
 I'm not so sure. There are certain Neutron plugins that work with
 certain virt drivers (Ironic) that require this information to be passed
 to all instances built by that virt driver. However, it doesn't (and
 probably shouldn't, as to not confuse cloud-init/etc) need to be passed
 to other instances. I think the conditional for passing this as metadata
 is going to need to be some combination of operator config, Neutron
 config/driver, and virt driver.
 
 I know we don't like networking things to be conditional on the virt
 driver, but Ironic is working on feature parity with virt for
 networking, and baremetal networking is vastly different than virt
 networking. I think we're going to have to accept that.
 

How about we list the known use cases (valid or not) for baremetal so
people understand what we are referring to? We will then be able to
determine what we wish to support.

Here is my take on it:

1. Single NIC with single network in access mode
2. Single NIC with single network in trunk mode (similar to 3.)
3. Single NIC with multiple networks in trunk mode
4. Multiple NICs with 1 network/nic in access mode:
   1 NIC == 1 network in access mode
5. Multiple NICs with multiple networks in trunk mode:
   1 NIC == multiple networks in trunk mode
   (to which NIC is the network associated is left as an exercise to the
reader)
6. Multiple NICs with bonding with 1 network/bond in access mode:
   2 NICs == 1 bond == 1 network in access mode
7. Multiple NICs with bonding with multiple networks in trunk mode:
   2 NICs == 1 bond == multiple networks in trunk mode

Those examples are based on a setup with VLAN networks.

Let me know if you feel I missed use cases.

(I'm less familiar with VXLAN so excuse me if I get some stuff wrong)

For VXLAN networks, there is the question of where is the VTEP:
a. the baremetal node itself?
b. the TOR switch?

The use case a. leaves a lot of question to be answered regarding
security which I'm definitely not familiar with.

As for use case b., we have to perform VLAN translation.

We can also argue that even without VXLAN encapsulation, an operator
could wish to perform VLAN translation so the tenant isn't aware of the
actual VLAN attributed to him.

This could also allow an operator to bridge multiple L2 network segments
together while still exposing a single common VLAN to the tenant across
the whole datacenter, irregardless of the L2 network segment his
baremetal node landed in.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [horizon] Unified interface for all regions

[Mathieu Gagné]

Hi,

As anyone wondered if it could be possible/feasible to have an unified
interface where all instances (or resources) are listed in one single
page for all regions available in the catalog?

What would be the challenges to make it happen? (so you don't have to
toggle between regions)

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Exposing provider networks in network_data.json

[Mathieu Gagné]

Hi,

I stubble on this review [1] which proposes adding info about provider
networks in network_data.json.

Concerns were raised by Kevin Benton about how those information
shouldn't be exposed to virtual instances for various reasons you can
read in the review and I totally agree with those concerns.

Monty Taylor mentioned a valid use case with Ironic where the node needs
the provider networks info to properly configure itself.

While I totally agree this is clearly a valid use case, I do not believe
the proposed implementation is the right answer.

(copying and rewording my comment found in the review)

For one, it breaks the virtual instance use case as I do not understand
how cloud-init or any similar tools will now be able to consume that
information.

If you boot a virtual instance where the traffic is already decapsulated
by the hypervisor, how is cloud-init supposed to know that it doesn't
have to configure vlan network interfaces?

This important distinction between virtual and metal is not addressed in
the proposed implementation. In fact, it breaks the virtual use case and
I strongly believe it should be reverted now.

I do understand that the baremetal use case is valid but do not
understand how inexorably exposing this information to virtual instances
will not introduce confusion and errors.

So it looks like there is a missing part in this feature. There should
be a way to hide this information if the instance does not require to
configure vlan interfaces to make network functional.

Furthermore, John Garbutt mentioned Part of me worries a little about
leaking this information, but I know some guests need this info. [...]
But even in those cases its security by obscurity reasons only, and I
want to ignore those..

To that, I answer that as an public provider operator, I do not wish to
expose such information to my users. It's not up to Nova to decide for
me that exposing provider networks info is ok and those concerns be
ignored. Please do not make such decisions lightly and without asking
a second opinion from operators which are the ones who will consume your
project.

[1] https://review.openstack.org/#/c/152703/5

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Cinder] Quobyte Cinder Driver revert?

[Mathieu Gagné]

On 2015-07-03 10:31 AM, Silvan Kaiser wrote:
 Hello!
 I just found the following commit in the cinder log:
 
 commit a3f4eed52efce50c2eb1176725bc578272949d7b
 Merge: 6939b4a e896ae2
 Author: Jenkins jenk...@review.openstack.org
 mailto:jenk...@review.openstack.org
 Date:   Thu Jul 2 23:14:39 2015 +
 
 Merge Revert First version of Cinder driver for Quobyte
 
 
 Is this part of some restructuring work, etc. that i did miss?
 I could not find a gerrit review for this and had no prior information?
 I did not see any related information when i did my weekly checks of the
 cinder weekly meeting logs and am confused to find this commit.

I'm not involved in any way with the revert but found those relevant
information:

http://lists.openstack.org/pipermail/third-party-announce/2015-June/000200.html
https://review.openstack.org/#/c/191192/2


-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [packaging] [puppet] how to deal with the rename of config files in neutron on upgrade?

[Mathieu Gagné]

Adding [puppet] tag to subject.

On 2015-07-02 11:35 AM, Matt Riedemann wrote:
 This change in neutron [1] renames the linuxbridge and openvswitch
 plugin config files.  I'm familiar with the %config(noreplace) directive
 in rpm but I'm not sure if there is a special trick with rpm to rename a
 config file while not losing the changes in the config file during the
 upgrade.
 
 Is this just something that has to be handled with trickery in the %post
 macro where we merge the contents together if the old config file
 exists?  Would symbolic links help?
 
 Changes like this seem like a potential giant pain in the ass for
 packagers.
 

And people maintaining configuration manager.
Will the agent still consider/read the old configuration file?

I'm not sure how we will be able to maintain compatibility without
involving manual steps or potential misconfiguration. Furthermore, we
have to consider upgrades.

Neutron agent configuration files are already a mess in distribution
packaging. Ok, I exaggerated the situation, it's only a mess with Ubuntu
[2] where it thought it would be a great idea to read the agent config
from ml2_conf.ini instead of ovs_neutron_plugin.ini like all the other
distributions.

Now as Puppet modules authors, should we just update the path to the
configuration file and hope it's compatible with upstream packages?

 [1] https://review.openstack.org/#/c/195277/

[2] https://gist.github.com/mgagne/e2e06f5a8cb283a81cab

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] openstacklib::db::sync proposal

[Mathieu Gagné]

On 2015-06-08 2:48 AM, Yanis Guenane wrote:
 
 If we look at openstacklib::db::postgresql[1] or
 openstackib::db::mysql[2] they are simple wrapper around
 puppet resources with no extra logic, but a common resource across all
 modules.

I see a lot of resources, relationships and logic in
openstackib::db::mysql. We can clearly see added value in that one.

 Also, to move forward on this topic I will submit to a vote one of the
 three propositions during our next meeting
 
  1. Abandon this change
  2. Move everything to X::db::sync, but just run the exec there, no
 openstacklib::db::sync
  3. Move forward with current implementation
 

I vote for 2 if we can make it happen in all modules and expose the same
interface (parameters) for consistency.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] openstacklib::db::sync proposal

[Mathieu Gagné]

On 2015-06-02 12:41 PM, Yanis Guenane wrote:
 
 The openstacklib::db::sync[2] is currently only a wrapper around an exec
 that does the actual db sync, this allow to make any modification to the
 exec into a single place. The main advantage IMO is that a contributor
 is provided with the same experience as it is not the case today across
 all modules.
 

The amount of possible change to an exec resource is very limited. [1] I
don't see a value in this change which outweighs the code churn and
review load needed to put it in place. Unless we have real use cases or
outrageously genius feature to add to it, I'm not in favor of this change.

Furthermore, any change to the public interface of
openstacklib::db::sync would require changes across all our modules
anyway to benefit from this latest hypothetical feature. I think we are
starting to nitpick over as little generic code we could possibly find
to put in openstacklib.

[1] https://docs.puppetlabs.com/references/latest/type.html#exec

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Renaming the IRC channel to #openstack-puppet

[Mathieu Gagné]

Hi,

We recently asked for our IRC channel (#puppet-openstack) to be logged
by the infra team. We happen to be the only channel suffixing the word
openstack instead of prefixing it. [1]

I would like to propose renaming our IRC channnel to #openstack-puppet
to better fit the mold (convention) already in place and be more
intuitive for new comers to discover.

Jemery Stanley (fungi) explained to me that previous IRC channel renames
were done following the Ubuntu procedure. [2] Last rename I remember of
was #openstack-stable to #openstack-release and it went smoothly without
any serious problem.

What do you guys think about the idea?

[1] http://eavesdrop.openstack.org/irclogs/
[2] https://wiki.ubuntu.com/IRC/MovingChannels

Note: I already registered the channel name for safe measures.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][libvirt] Understand why we lookup libvirt domains by instance name

[Mathieu Gagné]

On 2015-05-21 3:13 AM, Daniel P. Berrange wrote:
 On Thu, May 21, 2015 at 09:34:20PM +1200, Xav Paice wrote:
 On 21/05/15 21:23, Daniel P. Berrange wrote:
 On Wed, May 20, 2015 at 03:01:50PM -0700, Michael Still wrote:
 I note that we use instance.name to lookup the libvirt domain a bunch
 in the driver. I'm wondering why we don't just use instance.uuid all
 the time -- the code for that already exists. Is there a good reason
 to not move to always using the uuid?

 I ask because instance.name is not guaranteed to be unique depending
 on how weird the nova deployment is.
 Agreed, there's no benefit to using name - internally libvirt will always
 prefer to use the UUID itself too.

 These days though, there is only a single place in nova libvirt driver
 that needs updating - the nova.virt.libvirt.host.Host class get_domain()
 method just needs to be switched to use uuid.

 Just a comment from an ops point of view - it would be miles easier when
 trying to troubleshoot, if the instance name was the uuid anyway.  I
 totally agree on using instance.uuid, just to comment that I find it a
 little painful sometimes that instance names don't match the uuid of the
 instance, but the directory structure does.  Just a bit of confusion to
 avoid at 2am when something isn't working.
 
 You do know you can configure the instance name used...
 
 eg in nova.conf
 
   instance_name_template='inst-%(uuid)s'
 

Please note that you CANNOT change instance_name_template ones instances
are created otherwise they become unmanageable by Nova. This is because
Nova tries to find the libvirt instance by its name instead of its UUID.
So since instances were spawned with the old value, it won't find
existing instances using the new value.

And no, I haven't found a way to rename an instance without
redefining/restarting it in libvirt.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][libvirt] Understand why we lookup libvirt domains by instance name

[Mathieu Gagné]

On 2015-05-21 3:38 PM, Chris Friesen wrote:
 On 05/21/2015 03:20 PM, Mathieu Gagné wrote:
 On 2015-05-21 3:13 AM, Daniel P. Berrange wrote:
 On Thu, May 21, 2015 at 09:34:20PM +1200, Xav Paice wrote:
 On 21/05/15 21:23, Daniel P. Berrange wrote:
 On Wed, May 20, 2015 at 03:01:50PM -0700, Michael Still wrote:
 I note that we use instance.name to lookup the libvirt domain a bunch
 in the driver. I'm wondering why we don't just use instance.uuid all
 the time -- the code for that already exists. Is there a good reason
 to not move to always using the uuid?

 I ask because instance.name is not guaranteed to be unique depending
 on how weird the nova deployment is.
 Agreed, there's no benefit to using name - internally libvirt will
 always
 prefer to use the UUID itself too.

 These days though, there is only a single place in nova libvirt driver
 that needs updating - the nova.virt.libvirt.host.Host class
 get_domain()
 method just needs to be switched to use uuid.

 Just a comment from an ops point of view - it would be miles easier
 when
 trying to troubleshoot, if the instance name was the uuid anyway.  I
 totally agree on using instance.uuid, just to comment that I find it a
 little painful sometimes that instance names don't match the uuid of
 the
 instance, but the directory structure does.  Just a bit of confusion to
 avoid at 2am when something isn't working.

 You do know you can configure the instance name used...

 eg in nova.conf

instance_name_template='inst-%(uuid)s'


 Please note that you CANNOT change instance_name_template ones instances
 are created otherwise they become unmanageable by Nova. This is because
 Nova tries to find the libvirt instance by its name instead of its UUID.
 So since instances were spawned with the old value, it won't find
 existing instances using the new value.

 And no, I haven't found a way to rename an instance without
 redefining/restarting it in libvirt.
 
 Is there any reason why we couldn't default to using something like
 'inst-%(uuid)s' for the instance name?
 

I guess it's for the reason I mentioned above:

To not break all OpenStack installations on Earth running with default
config values.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova][libvirt] Understand why we lookup libvirt domains by instance name

[Mathieu Gagné]

On 2015-05-21 4:18 PM, Chris Friesen wrote:


 I guess it's for the reason I mentioned above:

 To not break all OpenStack installations on Earth running with default
 config values.

 
 So that's not breaking *upgrades* of all OpenStack installations with
 default config values, which is a slightly different thing.
 

Yes it will. Try to change instance_template_name and manage existing
instances. You won't be able.

The above scenario is the same as upgrading OpenStack Nova and having
instance_template_name changes its default value. Unless someone patches
Nova to use the instance UUID instead of its name to find it in libvirt.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][keystone][heat] Are AVAILABLE_REGIONS and multi-region service catalog mutually exclusive?

[Mathieu Gagné]

On 2015-05-14 12:34 AM, David Lyle wrote:
  
 Horizon only supports authenticating to one keystone endpoint at a time,
 specifically to one of the entries in AVAILABLE_REGIONS as defined in
 settings.py. Once you have an authenticated session in Horizon, the
 region selection support is merely for filtering between regions
 registered with the keystone endpoint you authenticated to, where the
 list of regions is determined by parsing the service catalog returned to
 you with your token. 
 
 What's really unclear to me is what you are intending to ask.

I'm asking to NOT remove the feature provided by AVAILABLE_REGIONS which
is what you described: support for multiple keystone endpoint (or
OpenStack installations) in one Horizon installation.

 If you are asking about token sharing between keystones which the thread
 you linked seems to indicate. Then yes, you can have a synced token
 store. But that is an exercise left to the operator.

I'm not suggesting token sharing. I'm merely trying to explain that
AVAILABLE_REGIONS answers a different need than multi-regions in the
same keystone endpoint which Horizon already supports fine.

Those are 2 features answering different needs and AVAILABLE_REGIONS
shouldn't be removed as suggested previously: we might consider
deprecating AVAILABLE_REGIONS in Horizon.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][keystone][heat] Are AVAILABLE_REGIONS and multi-region service catalog mutually exclusive?

[Mathieu Gagné]

When using AVAILABLE_REGIONS, you get a dropdown at login time to choose
your region which is in fact your keystone endpoint.

Once logged in, you get a new dropdown at the top right to switch
between the keystone endpoints. This means you can configure an
Horizon installation to login to multiple independent OpenStack
installations.

So I don't fully understand what enhancing the multi-region support in
Keystone would mean. Would you be able to configure Horizon to login to
multiple independent OpenStack installations?

Mathieu

On 2015-05-13 5:06 PM, Geoff Arnold wrote:
 Further digging suggests that we might consider deprecating
 AVAILABLE_REGIONS in Horizon and enhancing the multi-region support in
 Keystone. It wouldn’t take a lot; the main points:
 
   * Implement the Regions API discussed back in the Havana time period
 - 
 https://etherpad.openstack.org/p/havana-availability-zone-and-region-management
  -
 but with full CRUD
   * Enhance the Endpoints API to allow filtering by region
 
 Supporting two different multi region models is problematic if we’re
 serious about things like multi-region Heat.
 
 Thoughts?
 
 Geoff
 
 On May 13, 2015, at 12:01 PM, Geoff Arnold ge...@geoffarnold.com
 mailto:ge...@geoffarnold.com wrote:

 I’m looking at implementing dynamically-configured multi-region
 support for service federation, and the prior art on multi-region
 support in Horizon is pretty sketchy. This thread:
 http://lists.openstack.org/pipermail/openstack/2014-January/004372.html
 is the only real discussion I’ve found, and it’s pretty inconclusive.

 More precisely, if I configure a single Horizon with AVAILABLE_REGIONS
 pointing at two different Keystones with region names “X” and “Y, and
 each of those Keystones returns a service catalog with multiple
 regions (“A” and “B” for one, “P”, “Q”, and “R” for the other), what’s
 Horizon going to do? Or rather, what’s it expected to do?

 Yes, I’m being lazy: I could actually configure this to see what
 happens, but hopefully it was considered during the design.

 Geoff

 PS I’ve added Heat to the subject, because from a quick read of
 https://wiki.openstack.org/wiki/Heat/Blueprints/Multi_Region_Support_for_Heat
 it looks as if Heat won’t support the AVAILABLE_REGIONS model. That
 seems like an unfortunate disconnect.



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Proposal to configure Oslo libraries

[Mathieu Gagné]

On 2015-05-07 4:19 PM, Emilien Macchi wrote:
 
 Proposals
 =
 
 #1 Creating puppet-oslo
 ... and having oslo::messaging::rabbitmq, oslo::messaging::qpid, ...,
 oslo::logging, etc.
 This module will be used only to configure actual Oslo libraries when we
 deploy OpenStack. To me, this solution is really consistent with how
 OpenStack works today and is scalable as soon we contribute Oslo
 configuration changes in this module.
 
 #2 Using puppet-openstacklib
 ... and having openstacklib::oslo::messaging::(...)
 A good thing is our modules already use openstacklib.
 But openstacklib does not configure OpenStack now, it creates some
 common defines  classes that are consumed in other modules.
 

I prefer #1 due to oslo configs being specific to OpenStack versions.

The goal of openstacklib is to (hopefully) be OpenStack version agnostic
and be used only for code common across all *our* modules.

That's why I suggest going with solution #1, unless someone comes with a
solution to support multiple OpenStack versions in openstacklib without
the use of stable branches.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][operators] How to specify Keystone v3 credentials?

[Mathieu Gagné]

On 2015-05-05 1:25 AM, Monty Taylor wrote:
 On 05/04/2015 08:47 PM, Emilien Macchi wrote:
 
 
 On 05/04/2015 10:37 PM, Rich Megginson wrote:
 On 05/04/2015 07:52 PM, Mathieu Gagné wrote:
 On 2015-05-04 9:15 PM, Rich Megginson wrote:
 On 05/04/2015 06:03 PM, Mathieu Gagné wrote:
 On 2015-05-04 7:35 PM, Rich Megginson wrote:
 The way authentication works with the Icehouse branch is
 that puppet-keystone reads the admin_token and
 admin_endpoint from /etc/keystone/keystone.conf and
 passes these to the keystone command via the
 OS_SERVICE_TOKEN env. var. and the --os-endpoint
 argument, respectively.

 This will not work on a node where Keystone is not
 installed (unless you copy /etc/keystone/keystone.conf to
 all of your nodes).

 I am assuming there are admins/operators that have
 actually deployed OpenStack using puppet on nodes where
 Keystone is not installed?
 We are provisioning keystone resources from a privileged
 keystone node which accepts the admin_token. All other
 keystone servers has the admin_token_auth middleware
 removed for obvious security reasons.


 If so, how?  How do you specify the authentication
 credentials?  Do you use environment variables?  If so,
 how are they specified?
 When provisioning resources other than Keystones ones, we
 use custom puppet resources and the credentials are passed
 as env variables to the exec command. (they are mainly
 based on exec resources)
 I'm talking about the case where you are installing an
 OpenStack service other than Keystone using puppet, and that
 puppet code for that module needs to create some sort of
 Keystone resource.

 For example, install Glance on a node other than the Keystone
 node. puppet-glance is going to call class
 glance::keystone::auth, which will call
 keystone::resource::service_identity, which will call
 keystone_user { $name }.  The openstack provider used by
 keystone_user is going to need Keystone admin credentials in
 order to create the user.
 We fixed that part by not provisioning Keystone resources
 from Glance nodes but from Keystone nodes instead.

 We do not allow our users to create users/groups/projects, only
 a user with the admin role can do it. So why would you want to
 store/use admin credentials on an unprivileged nodes such as
 Glance? IMO, the glance user shouldn't be able to
 create/edit/delete users/projects/endpoints, that's the
 keystone nodes' job.

 Ok.  You don't need the Keystone superuser admin credentials on
 the Glance node.

 Is the puppet-glance code completely separable so that you can
 call only glance::keystone::auth (or other classes that use
 Keystone resources) from the Keystone node, and all of the other
 puppet-glance code on the Glance node?  Does the same apply to
 all of the other puppet modules?


 If you do not wish to explicitly define Keystone resources for
 Glance on Keystone nodes but instead let Glance nodes manage
 their own resources, you could always use exported resources.

 You let Glance nodes export their keystone resources and then
 you ask Keystone nodes to realize them where admin credentials
 are available. (I know some people don't really like exported
 resources for various reasons)

 I'm not familiar with exported resources.  Is this a viable
 option that has less impact than just requiring Keystone
 resources to be realized on the Keystone node?
 
 I'm not in favor of having exported resources because it requires 
 PuppetDB, and a lot of people try to avoid that. For now, we've
 been able to setup all OpenStack without PuppetDB in TripleO and in
 some other installers, we might want to keep this benefit.
 
 +100
 
 We're looking at using these puppet modules in a bit, but we're also a
 few steps away from getting rid of our puppetmaster and moving to a
 completely puppet apply based workflow. I would be double-plus
 sad-panda if we were not able to use the openstack puppet modules to
 do openstack because they'd been done in such as way as to require a
 puppetmaster or puppetdb.

100% agree.

Even if you had a puppetmaster and puppetdb, you would still end up in
this eventual consistency dance of puppet runs.

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][operators] How to specify Keystone v3 credentials?

[Mathieu Gagné]

On 2015-05-04 9:15 PM, Rich Megginson wrote:
 On 05/04/2015 06:03 PM, Mathieu Gagné wrote:
 On 2015-05-04 7:35 PM, Rich Megginson wrote:
 The way authentication works with the Icehouse branch is that
 puppet-keystone reads the admin_token and admin_endpoint from
 /etc/keystone/keystone.conf and passes these to the keystone command via
 the OS_SERVICE_TOKEN env. var. and the --os-endpoint argument,
 respectively.

 This will not work on a node where Keystone is not installed (unless you
 copy /etc/keystone/keystone.conf to all of your nodes).

 I am assuming there are admins/operators that have actually deployed
 OpenStack using puppet on nodes where Keystone is not installed?
 We are provisioning keystone resources from a privileged keystone node
 which accepts the admin_token. All other keystone servers has the
 admin_token_auth middleware removed for obvious security reasons.


 If so, how?  How do you specify the authentication credentials?  Do you
 use environment variables?  If so, how are they specified?
 When provisioning resources other than Keystones ones, we use custom
 puppet resources and the credentials are passed as env variables to the
 exec command. (they are mainly based on exec resources)
 
 I'm talking about the case where you are installing an OpenStack service
 other than Keystone using puppet, and that puppet code for that module
 needs to create some sort of Keystone resource.
 
 For example, install Glance on a node other than the Keystone node.
 puppet-glance is going to call class glance::keystone::auth, which will
 call keystone::resource::service_identity, which will call keystone_user
 { $name }.  The openstack provider used by keystone_user is going to
 need Keystone admin credentials in order to create the user.

We fixed that part by not provisioning Keystone resources from Glance
nodes but from Keystone nodes instead.

We do not allow our users to create users/groups/projects, only a user
with the admin role can do it. So why would you want to store/use admin
credentials on an unprivileged nodes such as Glance? IMO, the glance
user shouldn't be able to create/edit/delete users/projects/endpoints,
that's the keystone nodes' job.

If you do not wish to explicitly define Keystone resources for Glance on
Keystone nodes but instead let Glance nodes manage their own resources,
you could always use exported resources.

You let Glance nodes export their keystone resources and then you ask
Keystone nodes to realize them where admin credentials are available. (I
know some people don't really like exported resources for various reasons)


 How are you passing those credentials?  As env. vars?  How?

As stated, we use custom Puppet resources (defined types) which are
mainly wrapper around exec. You can pass environment variable to exec
through the environment parameter. I don't like it but that's how I did
it ~2 years ago. I haven't changed it due to lack of need to change it.
This might change soon with Keystone v3.


 I'm starting to think about moving away from env variables and use a
 configuration file instead. I'm not sure yet about the implementation
 details but that's the main idea.
 
 Is there a standard openrc location?  Could openrc be extended to hold
 parameters such as the default domain to use for Keystone resources? 
 I'm not talking about OS_DOMAIN_NAME, OS_USER_DOMAIN_NAME, etc. which
 are used for _authentication_, not resource creation.

I'm not aware of any standard openrc location other than ~/.openrc
which needs to be sourced before running any OpenStack client commands.

I however understand what you mean. I do not have any idea on how I
would implement it. I'm still hoping someday to be enlightened by a
great solution.

I'm starting to think about some sort of credentials vault. You store
credentials in it and you tell your resource to use that specific
credentials. You then no longer need to pass around 6-7
variables/parameters.


 There is a similar issue when creating domain scoped resources like
 users and projects.  As opposed to editing dozens of manifests to add
 domain parameters to every user and project (and the classes that call
 keystone_user/tenant, and the classes that call those classes, etc.), is
 there some mechanism to specify a default domain to use?  If not, what
 about using the same mechanism used today to specify the Keystone
 credentials?
 I see there is support for a default domain in keystone.conf. You will
 find it defined by the identity/default_domain_id=default config value.

 Is this value not usable?
 
 It is usable, and will be used, _only on Keystone nodes_. If you are on
 a node without Keystone, where will the default id come from?

As you probably know already, Puppet can't guess those default values,
nor could Glance.

I'm suggesting to not provision keystone resources from nodes other than
keystone themselves. It solves (or avoid) a lot of problems.

I think we have to change the way we think about Keystone resources

Re: [openstack-dev] [puppet][operators] How to specify Keystone v3 credentials?

[Mathieu Gagné]

On 2015-05-04 7:35 PM, Rich Megginson wrote:
 
 The way authentication works with the Icehouse branch is that
 puppet-keystone reads the admin_token and admin_endpoint from
 /etc/keystone/keystone.conf and passes these to the keystone command via
 the OS_SERVICE_TOKEN env. var. and the --os-endpoint argument,
 respectively.
 
 This will not work on a node where Keystone is not installed (unless you
 copy /etc/keystone/keystone.conf to all of your nodes).
 
 I am assuming there are admins/operators that have actually deployed
 OpenStack using puppet on nodes where Keystone is not installed?

We are provisioning keystone resources from a privileged keystone node
which accepts the admin_token. All other keystone servers has the
admin_token_auth middleware removed for obvious security reasons.


 If so, how?  How do you specify the authentication credentials?  Do you
 use environment variables?  If so, how are they specified?

When provisioning resources other than Keystones ones, we use custom
puppet resources and the credentials are passed as env variables to the
exec command. (they are mainly based on exec resources)

I'm starting to think about moving away from env variables and use a
configuration file instead. I'm not sure yet about the implementation
details but that's the main idea.


 For Keystone v3, in order to use v3 for authentication, and in order to
 use the v3 identity api, there must be some way to specify the various
 domains to use - the domain for the user, the domain for the project, or
 the domain to get a domain scoped token.

If I understand correctly, you have to scope the user to a domain and
scope the project to a domain: user1@domain1 wishes to get a token
scoped to project1@domain2 to manage resources within the project?


 There is a similar issue when creating domain scoped resources like
 users and projects.  As opposed to editing dozens of manifests to add
 domain parameters to every user and project (and the classes that call
 keystone_user/tenant, and the classes that call those classes, etc.), is
 there some mechanism to specify a default domain to use?  If not, what
 about using the same mechanism used today to specify the Keystone
 credentials?

I see there is support for a default domain in keystone.conf. You will
find it defined by the identity/default_domain_id=default config value.

Is this value not usable? And is it reasonable to assume the domain
default will always be present?

Or is the question more related to the need to somehow override this
value in Puppet?


 The goal is that all keystone domain scoped resources will eventually
 require specifying a domain, but that will take quite a while and I
 would like to provide an incremental upgrade path.


-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] rabbit/kombu settings deprecations

[Mathieu Gagné]

On 2015-04-16 2:10 PM, Richard Raseley wrote:

 I am certainly sympathetic to your situation - having the world change
 beneath your feet is never a good place to be. =]
 
 It does seem, however, that there is a disconnect between your
 expectations (as I understand them) of what the 'master' branch
 represents, and what it has traditionally represented - the 'bleeding edge'
 
 Master is volatile - it is expected to be volatile. As the code
 progresses between releases it is expected that things can (and will)
 break.
 
 The solution, in my mind, is not to redefine what it means to be master,
 but rather to be more aggressive about back-porting changes to stable
 branches.
 
 I am very much in favor of discussions that include your proposed
 solution above (i.e. 'current-1' support), as long as it is identified
 as a transitional step; I do pretty strongly believe that the right
 long-term model is to treat master as the 'bleeding edge' and to only
 pin yourself to stable releases.
 

Should I be in the same situation as Matt Fischer (using master to
install Juno), I would use this solution:

https://github.com/michaeltchapman/puppet-openstacklib/blob/master/manifests/compat/nova.pp

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Release management and bug triage

[Mathieu Gagné]

On 2015-03-26 1:08 PM, Sebastien Badia wrote:
 
 About lp script, a short search on github (bug mgmt, merged changes):
 
  - https://github.com/openstack-infra/release-tools
  - https://github.com/markmc/openstack-lp-scripts
  - https://github.com/dolph/launchpad
 
 But we wait the publishing of Mathieu scripts :)
 

Those are great tools. I mainly invested time in the ability to
massively create/update series and milestones (which is a pain) from a
projects.yaml file.

https://github.com/mgagne/openstack-puppet-release-tools

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Release management and bug triage

[Mathieu Gagné]

On 2015-03-17 3:22 PM, Emilien Macchi wrote:
 
 A first question that comes in my mind is: should we continue to manage
 every Puppet module in a different Launchpad project? Or should we
 migrate all modules to a single project.

I prefer multiple Launchpad projects due to the fact that each project
assume you manage one project for every aspect, especially milestones
management. (which is intrinsically linked to bug management)


 So far this is what I think about both solutions, feel free to comment:
 
 Having one project per module
 Pros:
 * Really useful when having the right tools to manage Launchpad, and
 also to manage one module as a real project.
 * The solution scales to the number of modules we support.
 
 Cons:
 * I think some people don't go on Launchpad because there is so many
 projects (one per module), so they did not subscribe emails or don't
 visit the page very often.

They can subscribe to the project group instead:
https://bugs.launchpad.net/openstack-puppet-modules


 * Each time we create a module (it's not every day, I would say each
 time a new OpenStack project is started), we have to repeat the process
 for a new launchpad project.

It takes me ~2 minutes to create a project. It's not a burden at all for me.

The challenge is with release management at scale. I have a bunch of
tools which I use to create new series, milestones and release them. So
it's not that big of a deal.


 Having everything in a single project
 Pro:
 * Release management could be simpler

It's not simpler, especially if you wish to associate bugs to
milestones. You would have to assume that EVERY projects will be part of
a very synchronized release cycle. (which isn't true)

 * A single view for all the bugs in Puppet modules

It exists already:
https://bugs.launchpad.net/openstack-puppet-modules

 * Maybe a bad idea, but we can use tags to track puppet modules issues
 (ie: puppet-openstacklib whould be openstacklib)

I already tried using tags to track issues. The challenge is with
versions and releases management. You cannot associate issues to
milestones unless you make the assume that we have one single version
for ALL our modules. So far, we had occasion where a single module was
released instead of all of them.


 Con:
 * The solution does not scale much, it depends again at how we decide to
 make bug triage and release management;

If we wish to be under the big tent, I think we have to have a strong
bug triage and release management. And having only one LP project is
going to make it difficult, not easier.


-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Release management and bug triage

[Mathieu Gagné]

On 2015-03-18 12:26 PM, Emilien Macchi wrote:

 The challenge is with release management at scale. I have a bunch of
 tools which I use to create new series, milestones and release them. So
 it's not that big of a deal.
 
 Are you willing to share it?
 

Sure. I'll make it a priority to publish it before the end of the week.

It needs a bit of cleanup though. =)

-- 
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova] Request non-priority feature freeze exception for Add ability to inject routes in interfaces.template

[Mathieu Gagné]

Hi,

I am requesting the exception for the feature Add ability to inject 
routes in interfaces.template.


The potential changes in Nova are limited as the feature is opt-in.
The default interfaces.template is not changed for the reasons now 
explained in the new commit message. No comment were made so far 
requiring code refactoring, only ones about procedural technicalities. 
Furthermore, this code has been running in a production environment for 
months without any issue.


The blueprint:


https://blueprints.launchpad.net/nova/+spec/network-template-routes-injection

The change:

  https://review.openstack.org/#/c/115409/

Thanks

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev