Re: [openstack-dev] Upstream LTS Releases

[Dmitry Tantsur]

On 11/14/2017 09:01 PM, Chris Friesen wrote:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.


I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.


I would hope that the core team for upstream LTS would be the (hopefully 
experienced) people doing the downstream work that already happens within the 
various distros.


Sure, but policies may vary. People do make feature backports downsteam from 
time to time ;)




Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Dmitry Tantsur]

On 11/14/2017 11:17 PM, Doug Hellmann wrote:

Excerpts from Chris Friesen's message of 2017-11-14 15:50:08 -0600:

On 11/14/2017 02:10 PM, Doug Hellmann wrote:

Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.


I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.


I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris



Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?



For what it's worth, we often lag more than 6 months behind master and so some
of the things we backport wouldn't be allowed by the existing stable branch
support phases.  (ie aren't "critical" or security patches.)

Chris



We should include a review of some of those policies as part of
this discussion. It would seem very odd to have a fix land in master,
not make it into the stable branches, and then show up in a branch
following the LTS policy.


Actually, I'm strongly against skipping any supported (not EOL-ed) branches when 
backporting patches. This includes normal stable branches, as well as other LTS 
branches.


The main reason, if somebody wonders, is to avoid regressions for people 
upgrading from a version with a backport to a version without.




Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] removing zuul v3 legacy jobs

[Andreas Jaeger]

On 2017-11-15 01:18, Emilien Macchi wrote:
> Hi,
> 
> I'm working on migrating all TripleO CI jobs to be in-tree, I'm also
> refactoring the layout and do some cleanup.

Please don't move *all* in tree - only the legacy ones. There's a
specific set of jobs we (infra, release team) like to keep in
project-config, see
https://docs.openstack.org/infra/manual/zuulv3.html#what-not-to-convert

> It's a bit of work, that can be followed here:
> https://review.openstack.org/#/q/topic:tripleo/migrate-to-zuulv3
> 
> The only thing I ask from our team is to let me know any change in
> project-config & zuul layout, so we can update my work in tripleo-ci
> patch, otherwise it will be lost when we land the patches.

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Matt Riedemann]

On 11/14/2017 10:58 AM, Davanum Srinivas wrote:

Let's focus our energy on the etherpad please

https://etherpad.openstack.org/p/LTS-proposal

On Wed, Nov 15, 2017 at 3:35 AM, Davanum Srinivas  wrote:

Saverio,

Please see this :
https://docs.openstack.org/project-team-guide/stable-branches.html for
current policies.

On Wed, Nov 15, 2017 at 3:33 AM, Saverio Proto  wrote:

Which stable policy does that patch violate?  It's clearly a bug
because the wrong information is being logged.  I suppose it goes
against the string freeze rule? Except that we've stopped translating
log messages so maybe we don't need to worry about that in this case,
since it isn't an exception.


Well, I also would like to understand more about stable policy violations.
When I proposed such patches in the past for the release N-2 I have
always got the answer: it is not a security issue so it will not be merged.

This is a good example of how things have been working so far:

https://review.openstack.org/#/q/677eb1c4160c08cfce2900495741f0ea15f566fa

This cinder patch was merged in master. It was then merged in Mitaka.
But it was not merged in Liberty just because "only security fixes" were
allowed at that point.

You can read that in the comments:
https://review.openstack.org/#/c/306610/

Is this kind of things going to change after the discussion in Sydney ?
The discussion is not enough ? what we need to get done then ?

thank you

Saverio


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Davanum Srinivas :: https://twitter.com/dims






Heh, I'm reading this thread after approving all of those patches.

The answer as to whether it's appropriate or not, is "it depends". 
Depends on the patch, depends on the age of the branch, etc.


In this case, newton is in phase 3 so normally it's only security or 
critical fixes allowed, but in this case it's so trivial and so 
obviously wrong that I was OK with approving it just to get it in before 
we end of life the branch.


So, it depends. And because it depends, that's also why we don't 
automate the backport of every fix made on master. Because guess what, 
we also backport "fixes" that introduce regressions, and when you do 
that to n-1 (Pike at this point) then you still have a lot of time to 
detect that and fix it upstream, but regressing things on the oldest 
branch leaves very little time to (1) have it detected and (2) get it 
fixed before end of life.


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [LTS] Upstream LTS Releases

[Rochelle Grober]

Well, moving this discussion is easy.  All that takes is everyone posting 
responses to the openstack-...@lists.openstack.org mailing list instead of dev 
and ops lists.  I've cc'ed all here.  I've also added [LTS] to the subject 
(sorry to break all the threaders). So that the sig list knows what the general 
topic is.  Yeah.  It's not really a topic, but everyone is used to parsing 
those things, even if the mailserver sw isn't.

But, are the two groups willing to move this discussion to the sigs list?  If 
they are, great.  If not,  hmmm.

Anyway, here's my attempt to serve

--Rocky

> -Original Message-
> From: Erik McCormick [mailto:emccorm...@cirrusseven.com]
> Sent: Tuesday, November 14, 2017 4:25 PM
> To: Rochelle Grober 
> Cc: OpenStack Development Mailing List (not for usage questions)
> ; openstack-oper.  operat...@lists.openstack.org>
> Subject: Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases
> 
> On Tue, Nov 14, 2017 at 4:10 PM, Rochelle Grober
>  wrote:
> > Folks,
> >
> > This discussion and the people interested in it seem like a perfect
> application of the SIG process.  By turning LTS into a SIG, everyone can
> discuss the issues on the SIG mailing list and the discussion shouldn't end up
> split.  If it turns into a project, great.  If a solution is found that 
> doesn't need a
> new project, great.  Even once  there is a decision on how to move forward,
> there will still be implementation issues and enhancements, so the SIG could
> very well be long-lived.  But the important aspect of this is:  keeping the
> discussion in a place where both devs and ops can follow the whole thing and
> act on recommendations.
> >
> > Food for thought.
> >
> > --Rocky
> >
> Just to add more legs to the spider that is this thread: I think the SIG idea 
> is a
> good one. It may evolve into a project team some day, but for now it's a
> free-for-all polluting 2 mailing lists, and multiple etherpads. How do we go
> about creating one?
> 
> -Erik
> 
> >> -Original Message-
> >> From: Blair Bethwaite [mailto:blair.bethwa...@gmail.com]
> >> Sent: Tuesday, November 14, 2017 8:31 AM
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> ; openstack-oper.  >> operat...@lists.openstack.org>
> >> Subject: Re: [openstack-dev] Upstream LTS Releases
> >>
> >> Hi all - please note this conversation has been split variously
> >> across -dev and - operators.
> >>
> >> One small observation from the discussion so far is that it seems as
> >> though there are two issues being discussed under the one banner:
> >> 1) maintain old releases for longer
> >> 2) do stable releases less frequently
> >>
> >> It would be interesting to understand if the people who want longer
> >> maintenance windows would be helped by #2.
> >>
> >> On 14 November 2017 at 09:25, Doug Hellmann
> 
> >> wrote:
> >> > Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
> >> >> >> The concept, in general, is to create a new set of cores from
> >> >> >> these groups, and use 3rd party CI to validate patches. There
> >> >> >> are lots of details to be worked out yet, but our amazing UC
> >> >> >> (User
> >> >> >> Committee) will be begin working out the details.
> >> >> >
> >> >> > What is the most worrying is the exact "take over" process. Does
> >> >> > it mean that the teams will give away the +2 power to a
> >> >> > different team? Or will our (small) stable teams still be
> >> >> > responsible for landing changes? If so, will they have to learn
> >> >> > how to debug 3rd party CI
> >> jobs?
> >> >> >
> >> >> > Generally, I'm scared of both overloading the teams and losing
> >> >> > the control over quality at the same time :) Probably the final
> >> >> > proposal will
> >> clarify it..
> >> >>
> >> >> The quality of backported fixes is expected to be a direct (and
> >> >> only?) interest of those new teams of new cores, coming from users
> >> >> and operators and vendors. The more parties to establish their 3rd
> >> >> party
> >> >
> >> > We have an unhealthy focus on "3rd party" jobs in this discussion.
> >> > We should not assume that they are needed or will be present. They
> >> > may be, but we shouldn't build policy around the assumption that
> >> > they will. Why would we have third-party jobs on an old branch that
> >> > we don't have on master, for instance?
> >> >
> >> >> checking jobs, the better proposed changes communicated, which
> >> >> directly affects the quality in the end. I also suppose,
> >> >> contributors from ops world will likely be only struggling to see
> >> >> things getting fixed, and not new features adopted by legacy
> >> >> deployments they're used
> >> to maintain.
> >> >> So in theory, this works and as a mainstream developer and
> >> >> maintainer, you need no to fear of losing control over LTS code 

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[John Dickinson]

On 14 Nov 2017, at 16:08, Davanum Srinivas wrote:

> On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson  wrote:
>>
>>
>> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>>
>>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
 The pressure for #2 comes from the inability to skip upgrades and the fact 
 that upgrades are hugely time consuming still.

 If you want to reduce the push for number #2 and help developers get their 
 wish of getting features into users hands sooner, the path to upgrade 
 really needs to be much less painful.

>>>
>>> +1000
>>>
>>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>>> execute the upgrade. (and we skipped a version)
>>> Scheduling all the relevant internal teams is a monumental task
>>> because we don't have dedicated teams for those projects and they have
>>> other priorities.
>>> Upgrading affects a LOT of our systems, some we don't fully have
>>> control over. And it can takes months to get new deployment on those
>>> systems. (and after, we have to test compatibility, of course)
>>>
>>> So I guess you can understand my frustration when I'm told to upgrade
>>> more often and that skipping versions is discouraged/unsupported.
>>> At the current pace, I'm just falling behind. I *need* to skip
>>> versions to keep up.
>>>
>>> So for our next upgrades, we plan on skipping even more versions if
>>> the database migration allows it. (except for Nova which is a huge
>>> PITA to be honest due to CellsV1)
>>> I just don't see any other ways to keep up otherwise.
>>
>> ?!?!
>>
>> What does it take for this to never happen again? No operator should need to 
>> plan and execute an upgrade for a whole year to upgrade one year's worth of 
>> code development.
>>
>> We don't need new policies, new teams, more releases, fewer releases, or 
>> anything like that. The goal is NOT "let's have an LTS release". The goal 
>> should be "How do we make sure Mattieu and everyone else in the world can 
>> actually deploy and use the software we are writing?"
>>
>> Can we drop the entire LTS discussion for now and focus on "make upgrades 
>> take less than a year" instead? After we solve that, let's come back around 
>> to LTS versions, if needed. I know there's already some work around that. 
>> Let's focus there and not be distracted about the best bureaucracy for not 
>> deleting two-year-old branches.
>>
>>
>> --John
>
> John,
>
> So... Any concrete ideas on how to achieve that?
>
> Thanks,
> Dims
>

Depends on what the upgrade problems are. I'd think the project teams that 
can't currently do seamless or skip-level upgrades would know best about what's 
needed. I suspect there will be both small and large changes needed in some 
projects.

Mathieu's list of realities in a different reply seem very normal. Operators 
are responsible for more than just OpenStack projects, and they've got to 
coordinate changes in deployed OpenStack projects with other systems they are 
running. Working through that list of realities could help identify some areas 
of improvement.

Spitballing process ideas...
* use a singular tag in launchpad to track upgrade stories. better yet, report 
on the status of these across all openstack projects so anyone can see what's 
needed to get to a smooth upgrade
* redouble efforts on multi-node and rolling upgrade testing. make sure every 
project is using it
* make smooth (and skip-level) upgrades a cross-project goal and don't set 
others until that one is achieved
* add upgrade stories and tests to the interop tests
* allocate time for ops to specifically talk about upgrade stories at the PTG. 
make sure as many devs are in the room as possible.
* add your cell phone number to the project README so that any operator can 
call you as soon as they try to upgrade (perhaps not 100% serious)
* add testing infrastructure that is locked to distro-provided versions of 
dependencies (eg install on xenial with only apt or install on rhel 7 with only 
yum)
* only do one openstack release a year. keep N-2 releases around. give ops a 
chance to upgrade before we delete branches
* do an openstack release every month. severely compress the release cycle and 
force everything to work with disparate versions. this will drive good testing, 
strong, stable interfaces, and smooth upgrades


Ah, just saw Kevin's reply in a different message. I really like his idea of 
"use ops tooling for day-to-day dev work. stop using devstack".


Ultimately it will come down to typing in some code and merging it into a 
project. I do not know what's needed there. It's probably different for every 
project.



--John




>>
>>
>> /me puts on asbestos pants
>>
>>>
>>> --
>>> Mathieu
>>>
>>> ___
>>> OpenStack-operators mailing list
>>> openstack-operat...@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>> 

Re: [openstack-dev] [OpenStack-docs] [docs] Next documentation meeting

[Petr Kovar]

All,

Please consider today's meeting canceled. 

Thanks,
pk


> From: "Petr Kovar" 
> To: openstack-dev@lists.openstack.org
> Cc: "enstack.org" 
> Sent: Tuesday, November 14, 2017 11:37:39 AM
> Subject: [OpenStack-docs] [docs] Next documentation meeting
> 
> Hi docs people,
> 
> Our next documentation meeting is scheduled for Wednesday, 15th November.
> As I'm traveling this week, I'll be unable to host the meeting. Is
> anybody available to run the meeting in my stead?
> 
> If not, let's cancel it and meet in two weeks.
> 
> Thank you,
> pk
> 
> ___
> OpenStack-docs mailing list
> openstack-d...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-docs
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Fox, Kevin M]

I can think of a few ideas, though some sound painful on paper Not really 
recommending anything, just thinking out loud...

One idea is that at the root of chaos monkey. If something is hard, do it 
frequently. If upgrading is hard, we need to be doing it constantly so the pain 
gets largely eliminated. One idea would be to discourage the use of standing up 
a fresh devstack all the time by devs and have them upgrade them instead. If 
its hard, then its likely someone will chip in to make it less hard.

Another is devstack in general. the tooling used by devs and that used by ops 
are so different as to isolate the devs from ops' pain. If they used more 
opsish tooling, then they would hit the same issues and would be more likely to 
find solutions that work for both parties.

A third one is supporting multiple version upgrades in the gate. I rarely have 
a problem with a cloud has a database one version back. I have seen lots of 
issues with databases that contain data back when the cloud was instantiated 
and then upgraded multiple times.

Another option is trying to unify/detangle the upgrade procedure. upgrading 
compute kit should be one or two commands if you can live with the defaults. 
Not weeks of poring through release notes, finding correct orders from pages of 
text and testing vigorously on test systems.

How about some tool that does the: dump database to somewhere temporary, 
iterate over all the upgrade job components, and see if it will successfully 
not corrupt your database. That takes a while to do manually. Ideally it could 
even upload stacktraces back a bug tracker for attention.

Thanks,
Kevin

From: Davanum Srinivas [dava...@gmail.com]
Sent: Tuesday, November 14, 2017 4:08 PM
To: OpenStack Development Mailing List (not for usage questions)
Cc: openstack-oper.
Subject: Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson  wrote:
>
>
> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>
>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
>>> The pressure for #2 comes from the inability to skip upgrades and the fact 
>>> that upgrades are hugely time consuming still.
>>>
>>> If you want to reduce the push for number #2 and help developers get their 
>>> wish of getting features into users hands sooner, the path to upgrade 
>>> really needs to be much less painful.
>>>
>>
>> +1000
>>
>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>> execute the upgrade. (and we skipped a version)
>> Scheduling all the relevant internal teams is a monumental task
>> because we don't have dedicated teams for those projects and they have
>> other priorities.
>> Upgrading affects a LOT of our systems, some we don't fully have
>> control over. And it can takes months to get new deployment on those
>> systems. (and after, we have to test compatibility, of course)
>>
>> So I guess you can understand my frustration when I'm told to upgrade
>> more often and that skipping versions is discouraged/unsupported.
>> At the current pace, I'm just falling behind. I *need* to skip
>> versions to keep up.
>>
>> So for our next upgrades, we plan on skipping even more versions if
>> the database migration allows it. (except for Nova which is a huge
>> PITA to be honest due to CellsV1)
>> I just don't see any other ways to keep up otherwise.
>
> ?!?!
>
> What does it take for this to never happen again? No operator should need to 
> plan and execute an upgrade for a whole year to upgrade one year's worth of 
> code development.
>
> We don't need new policies, new teams, more releases, fewer releases, or 
> anything like that. The goal is NOT "let's have an LTS release". The goal 
> should be "How do we make sure Mattieu and everyone else in the world can 
> actually deploy and use the software we are writing?"
>
> Can we drop the entire LTS discussion for now and focus on "make upgrades 
> take less than a year" instead? After we solve that, let's come back around 
> to LTS versions, if needed. I know there's already some work around that. 
> Let's focus there and not be distracted about the best bureaucracy for not 
> deleting two-year-old branches.
>
>
> --John

John,

So... Any concrete ideas on how to achieve that?

Thanks,
Dims

>
>
> /me puts on asbestos pants
>
>>
>> --
>> Mathieu
>>
>> ___
>> OpenStack-operators mailing list
>> openstack-operat...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



--
Davanum Srinivas :: https://twitter.com/dims

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Erik McCormick]

On Tue, Nov 14, 2017 at 4:10 PM, Rochelle Grober
 wrote:
> Folks,
>
> This discussion and the people interested in it seem like a perfect 
> application of the SIG process.  By turning LTS into a SIG, everyone can 
> discuss the issues on the SIG mailing list and the discussion shouldn't end 
> up split.  If it turns into a project, great.  If a solution is found that 
> doesn't need a new project, great.  Even once  there is a decision on how to 
> move forward, there will still be implementation issues and enhancements, so 
> the SIG could very well be long-lived.  But the important aspect of this is:  
> keeping the discussion in a place where both devs and ops can follow the 
> whole thing and act on recommendations.
>
> Food for thought.
>
> --Rocky
>
Just to add more legs to the spider that is this thread: I think the
SIG idea is a good one. It may evolve into a project team some day,
but for now it's a free-for-all polluting 2 mailing lists, and
multiple etherpads. How do we go about creating one?

-Erik

>> -Original Message-
>> From: Blair Bethwaite [mailto:blair.bethwa...@gmail.com]
>> Sent: Tuesday, November 14, 2017 8:31 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> ; openstack-oper. > operat...@lists.openstack.org>
>> Subject: Re: [openstack-dev] Upstream LTS Releases
>>
>> Hi all - please note this conversation has been split variously across -dev 
>> and -
>> operators.
>>
>> One small observation from the discussion so far is that it seems as though
>> there are two issues being discussed under the one banner:
>> 1) maintain old releases for longer
>> 2) do stable releases less frequently
>>
>> It would be interesting to understand if the people who want longer
>> maintenance windows would be helped by #2.
>>
>> On 14 November 2017 at 09:25, Doug Hellmann 
>> wrote:
>> > Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
>> >> >> The concept, in general, is to create a new set of cores from
>> >> >> these groups, and use 3rd party CI to validate patches. There are
>> >> >> lots of details to be worked out yet, but our amazing UC (User
>> >> >> Committee) will be begin working out the details.
>> >> >
>> >> > What is the most worrying is the exact "take over" process. Does it
>> >> > mean that the teams will give away the +2 power to a different
>> >> > team? Or will our (small) stable teams still be responsible for
>> >> > landing changes? If so, will they have to learn how to debug 3rd party 
>> >> > CI
>> jobs?
>> >> >
>> >> > Generally, I'm scared of both overloading the teams and losing the
>> >> > control over quality at the same time :) Probably the final proposal 
>> >> > will
>> clarify it..
>> >>
>> >> The quality of backported fixes is expected to be a direct (and
>> >> only?) interest of those new teams of new cores, coming from users
>> >> and operators and vendors. The more parties to establish their 3rd
>> >> party
>> >
>> > We have an unhealthy focus on "3rd party" jobs in this discussion. We
>> > should not assume that they are needed or will be present. They may
>> > be, but we shouldn't build policy around the assumption that they
>> > will. Why would we have third-party jobs on an old branch that we
>> > don't have on master, for instance?
>> >
>> >> checking jobs, the better proposed changes communicated, which
>> >> directly affects the quality in the end. I also suppose, contributors
>> >> from ops world will likely be only struggling to see things getting
>> >> fixed, and not new features adopted by legacy deployments they're used
>> to maintain.
>> >> So in theory, this works and as a mainstream developer and
>> >> maintainer, you need no to fear of losing control over LTS code :)
>> >>
>> >> Another question is how to not block all on each over, and not push
>> >> contributors away when things are getting awry, jobs failing and
>> >> merging is blocked for a long time, or there is no consensus reached
>> >> in a code review. I propose the LTS policy to enforce CI jobs be
>> >> non-voting, as a first step on that way, and giving every LTS team
>> >> member a core rights maybe? Not sure if that works though.
>> >
>> > I'm not sure what change you're proposing for CI jobs and their voting
>> > status. Do you mean we should make the jobs non-voting as soon as the
>> > branch passes out of the stable support period?
>> >
>> > Regarding the review team, anyone on the review team for a branch that
>> > goes out of stable support will need to have +2 rights in that branch.
>> > Otherwise there's no point in saying that they're maintaining the
>> > branch.
>> >
>> > Doug
>> >
>> >
>> __
>> 
>> >  OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > 

[openstack-dev] [tripleo] removing zuul v3 legacy jobs

[Emilien Macchi]

Hi,

I'm working on migrating all TripleO CI jobs to be in-tree, I'm also
refactoring the layout and do some cleanup.
It's a bit of work, that can be followed here:
https://review.openstack.org/#/q/topic:tripleo/migrate-to-zuulv3

The only thing I ask from our team is to let me know any change in
project-config & zuul layout, so we can update my work in tripleo-ci
patch, otherwise it will be lost when we land the patches.

Thanks,
-- 
Emilien Macchi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Erik McCormick]

On Tue, Nov 14, 2017 at 6:44 PM, John Dickinson  wrote:
>
>
> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>
>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
>>> The pressure for #2 comes from the inability to skip upgrades and the fact 
>>> that upgrades are hugely time consuming still.
>>>
>>> If you want to reduce the push for number #2 and help developers get their 
>>> wish of getting features into users hands sooner, the path to upgrade 
>>> really needs to be much less painful.
>>>
>>
>> +1000
>>
>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>> execute the upgrade. (and we skipped a version)
>> Scheduling all the relevant internal teams is a monumental task
>> because we don't have dedicated teams for those projects and they have
>> other priorities.
>> Upgrading affects a LOT of our systems, some we don't fully have
>> control over. And it can takes months to get new deployment on those
>> systems. (and after, we have to test compatibility, of course)
>>
>> So I guess you can understand my frustration when I'm told to upgrade
>> more often and that skipping versions is discouraged/unsupported.
>> At the current pace, I'm just falling behind. I *need* to skip
>> versions to keep up.
>>
>> So for our next upgrades, we plan on skipping even more versions if
>> the database migration allows it. (except for Nova which is a huge
>> PITA to be honest due to CellsV1)
>> I just don't see any other ways to keep up otherwise.
>
> ?!?!
>
> What does it take for this to never happen again? No operator should need to 
> plan and execute an upgrade for a whole year to upgrade one year's worth of 
> code development.
>
> We don't need new policies, new teams, more releases, fewer releases, or 
> anything like that. The goal is NOT "let's have an LTS release". The goal 
> should be "How do we make sure Mattieu and everyone else in the world can 
> actually deploy and use the software we are writing?"
>
> Can we drop the entire LTS discussion for now and focus on "make upgrades 
> take less than a year" instead? After we solve that, let's come back around 
> to LTS versions, if needed. I know there's already some work around that. 
> Let's focus there and not be distracted about the best bureaucracy for not 
> deleting two-year-old branches.
>
>
> --John
>
>
>
> /me puts on asbestos pants
>

OK, let's tone down the flamethrower there a bit Mr. Asbestos Pants
;). The LTS push is not lieu of the quest for simpler upgrades. There
is also an effort to enable fast-forward upgrades going on. However,
this is a non-trivial task that will take many cycles to get to a
point where it's truly what you're looking for. The long term desire
of having LTS releases encompasses being able to hop from one LTS to
the next without stopping over. We just aren't there yet.

However, what we *can* do is make it so when mgagne finally gets to
Newton (or Ocata or wherever) on his next run, the code isn't
completely EOL and it can still receive some important patches. This
can be accomplished in the very near term, and that is what a certain
subset of us are focused on.

We still desire to skip versions. We still desire to have upgrades be
non-disruptive and non-destructive. This is just one step on the way
to that. This discussion has been going on for cycle after cycle with
little more than angst between ops and devs to show for it. This is
the first time we've had progress on this ball of goo that really
matters. Let's all be proactive contributors to the solution.

Those interested in having a say in the policy, put your $0.02 here:
https://etherpad.openstack.org/p/LTS-proposal

Peace, Love, and International Grooviness,
Erik

>>
>> --
>> Mathieu
>>
>> ___
>> OpenStack-operators mailing list
>> openstack-operat...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Davanum Srinivas]

On Wed, Nov 15, 2017 at 10:44 AM, John Dickinson  wrote:
>
>
> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>
>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
>>> The pressure for #2 comes from the inability to skip upgrades and the fact 
>>> that upgrades are hugely time consuming still.
>>>
>>> If you want to reduce the push for number #2 and help developers get their 
>>> wish of getting features into users hands sooner, the path to upgrade 
>>> really needs to be much less painful.
>>>
>>
>> +1000
>>
>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>> execute the upgrade. (and we skipped a version)
>> Scheduling all the relevant internal teams is a monumental task
>> because we don't have dedicated teams for those projects and they have
>> other priorities.
>> Upgrading affects a LOT of our systems, some we don't fully have
>> control over. And it can takes months to get new deployment on those
>> systems. (and after, we have to test compatibility, of course)
>>
>> So I guess you can understand my frustration when I'm told to upgrade
>> more often and that skipping versions is discouraged/unsupported.
>> At the current pace, I'm just falling behind. I *need* to skip
>> versions to keep up.
>>
>> So for our next upgrades, we plan on skipping even more versions if
>> the database migration allows it. (except for Nova which is a huge
>> PITA to be honest due to CellsV1)
>> I just don't see any other ways to keep up otherwise.
>
> ?!?!
>
> What does it take for this to never happen again? No operator should need to 
> plan and execute an upgrade for a whole year to upgrade one year's worth of 
> code development.
>
> We don't need new policies, new teams, more releases, fewer releases, or 
> anything like that. The goal is NOT "let's have an LTS release". The goal 
> should be "How do we make sure Mattieu and everyone else in the world can 
> actually deploy and use the software we are writing?"
>
> Can we drop the entire LTS discussion for now and focus on "make upgrades 
> take less than a year" instead? After we solve that, let's come back around 
> to LTS versions, if needed. I know there's already some work around that. 
> Let's focus there and not be distracted about the best bureaucracy for not 
> deleting two-year-old branches.
>
>
> --John

John,

So... Any concrete ideas on how to achieve that?

Thanks,
Dims

>
>
> /me puts on asbestos pants
>
>>
>> --
>> Mathieu
>>
>> ___
>> OpenStack-operators mailing list
>> openstack-operat...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Mathieu Gagné]

On Tue, Nov 14, 2017 at 6:44 PM, John Dickinson  wrote:
>
>
> On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:
>
>> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
>>> The pressure for #2 comes from the inability to skip upgrades and the fact 
>>> that upgrades are hugely time consuming still.
>>>
>>> If you want to reduce the push for number #2 and help developers get their 
>>> wish of getting features into users hands sooner, the path to upgrade 
>>> really needs to be much less painful.
>>>
>>
>> +1000
>>
>> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
>> execute the upgrade. (and we skipped a version)
>> Scheduling all the relevant internal teams is a monumental task
>> because we don't have dedicated teams for those projects and they have
>> other priorities.
>> Upgrading affects a LOT of our systems, some we don't fully have
>> control over. And it can takes months to get new deployment on those
>> systems. (and after, we have to test compatibility, of course)
>>
>> So I guess you can understand my frustration when I'm told to upgrade
>> more often and that skipping versions is discouraged/unsupported.
>> At the current pace, I'm just falling behind. I *need* to skip
>> versions to keep up.
>>
>> So for our next upgrades, we plan on skipping even more versions if
>> the database migration allows it. (except for Nova which is a huge
>> PITA to be honest due to CellsV1)
>> I just don't see any other ways to keep up otherwise.
>
> ?!?!
>
> What does it take for this to never happen again? No operator should need to 
> plan and execute an upgrade for a whole year to upgrade one year's worth of 
> code development.
>
> We don't need new policies, new teams, more releases, fewer releases, or 
> anything like that. The goal is NOT "let's have an LTS release". The goal 
> should be "How do we make sure Mattieu and everyone else in the world can 
> actually deploy and use the software we are writing?"
>
> Can we drop the entire LTS discussion for now and focus on "make upgrades 
> take less than a year" instead? After we solve that, let's come back around 
> to LTS versions, if needed. I know there's already some work around that. 
> Let's focus there and not be distracted about the best bureaucracy for not 
> deleting two-year-old branches.

To add details to what happened:
* Upgrade was never made a #1 priority. It was a one man show for far
too long. (myself)
* I also happen to manage and work on other priorities.
* Lot of work made to prepare for multiple versions support in our
deployment tools. (we use Puppet)
* Lot of work in the packaging area to speedup packaging. (we are
still using deb packages but with virtualenv to stay Puppet
compatible)
* We need to forward-port private patches which upstream won't accept
and/or are private business logic.
* Our developer teams didn't have enough free cycles to work right
away on the upgrade. (this means delays)
* We need to test compatibility with 3rd party systems which takes
some time. (and make them compatible)
* We need to update systems ever which we don't have full control.
This means serious delays when it comes to deployment.
* We need to test features/stability during some time in our dev environment.
* We need to test features/stability during some time in our
staging/pre-prod environment.
* We need to announce and inform our users at least 2 weeks in advance
before performing an upgrade.
* We choose to upgrade one service at a time (in all regions) to avoid
a huge big bang upgrade. (this means more maintenance windows to plan
and you can't stack them too much)
* We need to swiftly respond to bug discovered by our users. This
means change of priorities and delay in other service upgrades.
* We will soon need to upgrade operating systems to support latest
OpenStack versions. (this means we have to stop OpenStack upgrades
until all nodes are upgraded)

All those details rapidly add up. We are far far away from a git pull
&& ./stack.sh

I don't want to sound like too harsh but I feel some people live in a
vacuum or an ideal world far from the reality of some operators.
The above details are just a very small glimpse into my reality. I
hope people will understand and have a different perspectives when it
comes to upgrades.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Mike Smith]

For those wondering why operators can’t always upgrade sooner, I can add a 
little bit of color:  In our clouds, we have a couple vendors (one network 
plugin, one cinder driver) and those vendors typically are 1-3 releases behind 
‘cutting edge’.  By the time they support the version we want to go to, that 
version is almost end-of-life, which can make things interesting.  On the 
bright side, by then there are usually some helpful articles out there about 
the issues upgrading from A to B.

As for the planning time required - for us, it mostly boils down to testing or 
doing it at a time when some amount of disruption is at least somewhat 
tolerable.  For example, for online retail folks like me, upgrading between 
October and December would be out of the question due to the busy shopping 
season that is almost upon us.

I will say that I was very impressed with some of the containerized demos that 
were given at the Summit last week.  I plan to look into some containerized 
options next year which hopefully could ease the upgrade process for us.  
Still, there is a lot of testing involved, coordination with 3rd parties, and 
other stars that would still have to align.

At Overstock we have also started maintaining two completely separate 
production clouds and have orchestration to build/rebuild VMs on either one as 
needed.  Most the time we spread all our apps across both clouds.  So next year 
when we get the chance to upgrade cloud A, we can either rebuild things on B, 
or just shut them down while we rebuild A.  Then we would repeat on cloud B.  
Hopefully this eases our upgrade process…at least that’s what we are hoping!

My 2 cents.  Thanks



On Nov 14, 2017, at 4:44 PM, John Dickinson > 
wrote:



On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:

On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M 
> wrote:
The pressure for #2 comes from the inability to skip upgrades and the fact that 
upgrades are hugely time consuming still.

If you want to reduce the push for number #2 and help developers get their wish 
of getting features into users hands sooner, the path to upgrade really needs 
to be much less painful.


+1000

We are upgrading from Kilo to Mitaka. It took 1 year to plan and
execute the upgrade. (and we skipped a version)
Scheduling all the relevant internal teams is a monumental task
because we don't have dedicated teams for those projects and they have
other priorities.
Upgrading affects a LOT of our systems, some we don't fully have
control over. And it can takes months to get new deployment on those
systems. (and after, we have to test compatibility, of course)

So I guess you can understand my frustration when I'm told to upgrade
more often and that skipping versions is discouraged/unsupported.
At the current pace, I'm just falling behind. I *need* to skip
versions to keep up.

So for our next upgrades, we plan on skipping even more versions if
the database migration allows it. (except for Nova which is a huge
PITA to be honest due to CellsV1)
I just don't see any other ways to keep up otherwise.

?!?!

What does it take for this to never happen again? No operator should need to 
plan and execute an upgrade for a whole year to upgrade one year's worth of 
code development.

We don't need new policies, new teams, more releases, fewer releases, or 
anything like that. The goal is NOT "let's have an LTS release". The goal 
should be "How do we make sure Mattieu and everyone else in the world can 
actually deploy and use the software we are writing?"

Can we drop the entire LTS discussion for now and focus on "make upgrades take 
less than a year" instead? After we solve that, let's come back around to LTS 
versions, if needed. I know there's already some work around that. Let's focus 
there and not be distracted about the best bureaucracy for not deleting 
two-year-old branches.


--John



/me puts on asbestos pants


--
Mathieu

___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators




CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you 

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[John Dickinson]

On 14 Nov 2017, at 15:18, Mathieu Gagné wrote:

> On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
>> The pressure for #2 comes from the inability to skip upgrades and the fact 
>> that upgrades are hugely time consuming still.
>>
>> If you want to reduce the push for number #2 and help developers get their 
>> wish of getting features into users hands sooner, the path to upgrade really 
>> needs to be much less painful.
>>
>
> +1000
>
> We are upgrading from Kilo to Mitaka. It took 1 year to plan and
> execute the upgrade. (and we skipped a version)
> Scheduling all the relevant internal teams is a monumental task
> because we don't have dedicated teams for those projects and they have
> other priorities.
> Upgrading affects a LOT of our systems, some we don't fully have
> control over. And it can takes months to get new deployment on those
> systems. (and after, we have to test compatibility, of course)
>
> So I guess you can understand my frustration when I'm told to upgrade
> more often and that skipping versions is discouraged/unsupported.
> At the current pace, I'm just falling behind. I *need* to skip
> versions to keep up.
>
> So for our next upgrades, we plan on skipping even more versions if
> the database migration allows it. (except for Nova which is a huge
> PITA to be honest due to CellsV1)
> I just don't see any other ways to keep up otherwise.

?!?!

What does it take for this to never happen again? No operator should need to 
plan and execute an upgrade for a whole year to upgrade one year's worth of 
code development.

We don't need new policies, new teams, more releases, fewer releases, or 
anything like that. The goal is NOT "let's have an LTS release". The goal 
should be "How do we make sure Mattieu and everyone else in the world can 
actually deploy and use the software we are writing?"

Can we drop the entire LTS discussion for now and focus on "make upgrades take 
less than a year" instead? After we solve that, let's come back around to LTS 
versions, if needed. I know there's already some work around that. Let's focus 
there and not be distracted about the best bureaucracy for not deleting 
two-year-old branches.


--John



/me puts on asbestos pants

>
> --
> Mathieu
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Mathieu Gagné]

On Tue, Nov 14, 2017 at 6:00 PM, Fox, Kevin M  wrote:
> The pressure for #2 comes from the inability to skip upgrades and the fact 
> that upgrades are hugely time consuming still.
>
> If you want to reduce the push for number #2 and help developers get their 
> wish of getting features into users hands sooner, the path to upgrade really 
> needs to be much less painful.
>

+1000

We are upgrading from Kilo to Mitaka. It took 1 year to plan and
execute the upgrade. (and we skipped a version)
Scheduling all the relevant internal teams is a monumental task
because we don't have dedicated teams for those projects and they have
other priorities.
Upgrading affects a LOT of our systems, some we don't fully have
control over. And it can takes months to get new deployment on those
systems. (and after, we have to test compatibility, of course)

So I guess you can understand my frustration when I'm told to upgrade
more often and that skipping versions is discouraged/unsupported.
At the current pace, I'm just falling behind. I *need* to skip
versions to keep up.

So for our next upgrades, we plan on skipping even more versions if
the database migration allows it. (except for Nova which is a huge
PITA to be honest due to CellsV1)
I just don't see any other ways to keep up otherwise.

--
Mathieu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Fox, Kevin M]

The pressure for #2 comes from the inability to skip upgrades and the fact that 
upgrades are hugely time consuming still.

If you want to reduce the push for number #2 and help developers get their wish 
of getting features into users hands sooner, the path to upgrade really needs 
to be much less painful.

Thanks,
Kevin

From: Erik McCormick [emccorm...@cirrusseven.com]
Sent: Tuesday, November 14, 2017 9:21 AM
To: Blair Bethwaite
Cc: OpenStack Development Mailing List (not for usage questions); 
openstack-oper.
Subject: Re: [openstack-dev] [Openstack-operators]  Upstream LTS Releases

On Tue, Nov 14, 2017 at 11:30 AM, Blair Bethwaite
 wrote:
> Hi all - please note this conversation has been split variously across
> -dev and -operators.
>
> One small observation from the discussion so far is that it seems as
> though there are two issues being discussed under the one banner:
> 1) maintain old releases for longer
> 2) do stable releases less frequently
>
> It would be interesting to understand if the people who want longer
> maintenance windows would be helped by #2.
>

I would like to hear from people who do *not* want #2 and why not.
What are the benefits of 6 months vs. 1 year? I have heard objections
in the hallway track, but I have struggled to retain the rationale for
more than 10 seconds. I think this may be more of a religious
discussion that could take a while though.

#1 is something we can act on right now with the eventual goal of
being able to skip releases entirely. We are addressing the
maintenance of the old issue right now. As we get farther down the
road of fast-forward upgrade tooling, then we will be able to please
those wishing for a slower upgrade cadence, and those that want to
stay on the bleeding edge simultaneously.

-Erik

> On 14 November 2017 at 09:25, Doug Hellmann  wrote:
>> Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
>>> >> The concept, in general, is to create a new set of cores from these
>>> >> groups, and use 3rd party CI to validate patches. There are lots of
>>> >> details to be worked out yet, but our amazing UC (User Committee) will
>>> >> be begin working out the details.
>>> >
>>> > What is the most worrying is the exact "take over" process. Does it mean 
>>> > that
>>> > the teams will give away the +2 power to a different team? Or will our 
>>> > (small)
>>> > stable teams still be responsible for landing changes? If so, will they 
>>> > have to
>>> > learn how to debug 3rd party CI jobs?
>>> >
>>> > Generally, I'm scared of both overloading the teams and losing the 
>>> > control over
>>> > quality at the same time :) Probably the final proposal will clarify it..
>>>
>>> The quality of backported fixes is expected to be a direct (and only?)
>>> interest of those new teams of new cores, coming from users and
>>> operators and vendors. The more parties to establish their 3rd party
>>
>> We have an unhealthy focus on "3rd party" jobs in this discussion. We
>> should not assume that they are needed or will be present. They may be,
>> but we shouldn't build policy around the assumption that they will. Why
>> would we have third-party jobs on an old branch that we don't have on
>> master, for instance?
>>
>>> checking jobs, the better proposed changes communicated, which directly
>>> affects the quality in the end. I also suppose, contributors from ops
>>> world will likely be only struggling to see things getting fixed, and
>>> not new features adopted by legacy deployments they're used to maintain.
>>> So in theory, this works and as a mainstream developer and maintainer,
>>> you need no to fear of losing control over LTS code :)
>>>
>>> Another question is how to not block all on each over, and not push
>>> contributors away when things are getting awry, jobs failing and merging
>>> is blocked for a long time, or there is no consensus reached in a code
>>> review. I propose the LTS policy to enforce CI jobs be non-voting, as a
>>> first step on that way, and giving every LTS team member a core rights
>>> maybe? Not sure if that works though.
>>
>> I'm not sure what change you're proposing for CI jobs and their voting
>> status. Do you mean we should make the jobs non-voting as soon as the
>> branch passes out of the stable support period?
>>
>> Regarding the review team, anyone on the review team for a branch
>> that goes out of stable support will need to have +2 rights in that
>> branch. Otherwise there's no point in saying that they're maintaining
>> the branch.
>>
>> Doug
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Cheers,
> ~Blairo
>
> ___
> 

Re: [openstack-dev] Upstream LTS Releases

[Doug Hellmann]

Excerpts from Chris Friesen's message of 2017-11-14 15:50:08 -0600:
> On 11/14/2017 02:10 PM, Doug Hellmann wrote:
> > Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:
> >> On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:
> >>
>  The quality of backported fixes is expected to be a direct (and only?)
>  interest of those new teams of new cores, coming from users and 
>  operators and
>  vendors.
> >>>
> >>> I'm not assuming bad intentions, not at all. But there is a lot of 
> >>> involved in a
> >>> decision whether to make a backport or not. Will these people be able to
> >>> evaluate a risk of each patch? Do they have enough context on how that 
> >>> release
> >>> was implemented and what can break? Do they understand why feature 
> >>> backports are
> >>> bad? Why they should not skip (supported) releases when backporting?
> >>>
> >>> I know a lot of very reasonable people who do not understand the things 
> >>> above
> >>> really well.
> >>
> >> I would hope that the core team for upstream LTS would be the (hopefully
> >> experienced) people doing the downstream work that already happens within 
> >> the
> >> various distros.
> >>
> >> Chris
> >>
> >
> > Presumably those are the same people we've been trying to convince
> > to work on the existing stable branches for the last 5 years. What
> > makes these extended branches more appealing to those people than
> > the existing branches? Is it the reduced requirements on maintaining
> > test jobs? Or maybe some other policy change that could be applied
> > to the stable branches?
> 
> 
> For what it's worth, we often lag more than 6 months behind master and so 
> some 
> of the things we backport wouldn't be allowed by the existing stable branch 
> support phases.  (ie aren't "critical" or security patches.)
> 
> Chris
> 

We should include a review of some of those policies as part of
this discussion. It would seem very odd to have a fix land in master,
not make it into the stable branches, and then show up in a branch
following the LTS policy.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] automatic migration from classic drivers to hardware types?

[Alex Schultz]

On Tue, Nov 14, 2017 at 8:10 AM, Dmitry Tantsur  wrote:
> Hi folks!
>
> This was raised several times, now I want to bring it to the wider audience.
> We're planning [1] to deprecate classic drivers in Queens and remove them in
> Rocky. It was pointed at the Forum that we'd better provide an automatic
> migration.
>
> I'd like to hear your opinion on the options:
>
> (1) Migration as part of 'ironic-dbsync upgrade'
>
> Pros:
> * nothing new to do for the operators
>
> Cons:
> * upgrade will fail completely, if for some nodes the matching hardware
> types and/or interfaces are not enabled in ironic.conf
>
> (2) A separate script for migration
>
> Pros:
> * can be done in advance (even while still on Pike)
> * a failure won't fail the whole upgrade
> * will rely on drivers enabled in actually running conductors, not on
> ironic.conf
>
> Cons:
> * a new upgrade action before Rocky
> * won't be available in packaging
> * unclear how to update nodes that are in some process (e.g. cleaning), will
> probably have to be run several times
>
> (3) Migration as part of 'ironic-dbsync online_data_migration'
>
> Pros:
> * nothing new to do for the operators, similar to (1)
> * probably a more natural place to do this than (1)
> * can rely on drivers enabled in actually running conductors, not on
> ironic.conf
>
> Cons:
> * data migration will fail, if for some nodes the matching hardware types
> and/or interfaces are not enabled in ironic.conf
>

Rather than fail in various ways, why not do like what nova has with a
pre-upgrade status check[0] and then just handle it in ironic-dbsync
upgrade?   This would allow operators to check prior to running the
upgrade to understand what might need to be changed.  Additionally the
upgrade command itself could leverage the status check to fail nicely.


> (4) Do nothing, let operators handle the migration.
>

Please no.

>
> The most reasonable option for me seems (3), then (4). What do you think?
>

So this was chatted about in relation to some environment tooling we
have where we currently have where older 'pxe_ipmitool' defined and
this will need to switch to be 'ipmi'[1]. The issue with the hard
cutover on this one is any tooling which may have been written that
currently works with multiple openstack releases to generate the
required json for ironic will now have to take that into account.  I
know in our case we'll be needing to support newton for longer so
making the tooling openstack aware around this is just further
tech-debt that we'll be creating. Is there a better solution that
could be done either in ironic client or in the API to gracefully
handle this transition for a longer period of time?  I think this may
be one of those decisions that has a far reaching impact on
deployers/operators due changes they will have to make to support
multiple versions or as they upgrade between versions and they aren't
fully aware of yet as many may not be on Ocata.  This change seems
like it has a high UX impact and IMHO should be done very carefully.

Thanks,
-Alex

[0] https://docs.openstack.org/nova/pike/cli/nova-status.html
[1] 
http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2017-11-14.log.html#t2017-11-14T15:36:45


> Dmitry
>
> [1]
> http://specs.openstack.org/openstack/ironic-specs/specs/approved/classic-drivers-future.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Chris Friesen]

On 11/14/2017 02:10 PM, Doug Hellmann wrote:

Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.


I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.


I would hope that the core team for upstream LTS would be the (hopefully
experienced) people doing the downstream work that already happens within the
various distros.

Chris



Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?



For what it's worth, we often lag more than 6 months behind master and so some 
of the things we backport wouldn't be allowed by the existing stable branch 
support phases.  (ie aren't "critical" or security patches.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] Sydney Summit Recap

[Lance Bragstad]

Hey all,

I just finished dumping everything from the summit into a blog post [0].
If you have a summary or feedback - we can use this thread to advertise
it and discuss.

Thanks,

Lance

[0] https://www.lbragstad.com/blog/openstack-summit-sydney-recap





signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] multiple agents with segment access

[Ihar Hrachyshka]

In general, you should be able to run both regular l2 agent (ovs) and
sriov agent. If you have problems with it, we should probably assume
it's a bug. Please report.

On Tue, Nov 14, 2017 at 8:02 AM, Legacy, Allain
 wrote:
> Is it a known limitation that the ML2 plugin and associated drivers do not 
> properly handle cases where there are multiple agents running on the same 
> host?   That is, two or more agents that reside on the same host and both 
> report device/interface mappings  for physical networks?One such setup is 
> a set of nodes that are both running an L2 agent and a NIC SRIOV agent.
>
> We have found two problems specific to this configuration and are wondering 
> if these are known limitations, or simply not supported.
>
> For example, in a configuration where a host has L2, DHCP, and SRIOV agents 
> running on it, then:
>
> 1)   The DHCP scheduler can schedule a network to be hosted by that DHCP 
> agent even if the physical network is only supported by the SRIOV agent and 
> not by the L2 agent.  This can end up isolating the DHCP server as it will 
> not have access to the underlying physical segment.   This is happening 
> because "filter_hosts_with_network_access" in the Ml2Plugin will include the 
> host because the SRIOV mech driver reports that it has access to that segment.
>
> 2)   The Segment Plugin "segmenthostmappings" table gets overwritten with 
> tuples for whichever agent reports in last.  For example, if the L2 agent has 
> physical network mappings for 2 different physical networks (physnet0, 
> physnet1), but the SRIOV NIC agent only has 1 physical network reported 
> (physnet2), the segmenthostmappings table will have data for only physnet2 if 
> the SRIOV NIC agent reported last, or data for only both physnet0 and 
> physnet1 if the L2 agent reported last.  Data for physical networks belonging 
> to the other agent will be erroneously removed as stale entries.
>
> If it has been discussed before is there a plan to address this type of 
> configuration?
>
> Regards.
> Allain
>
>
>
> Allain Legacy, Software Developer, Wind River
> direct 613.270.2279  fax 613.492.7870 skype allain.legacy
> 350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][release][trove][infra] Pre-release of openstack/trove failed

[Clark Boylan]

On Tue, Nov 14, 2017, at 12:15 PM, Doug Hellmann wrote:
> Excerpts from Clark Boylan's message of 2017-11-14 09:04:07 -0800:
> > 
> > On Tue, Nov 14, 2017, at 09:01 AM, Doug Hellmann wrote:
> > > Excerpts from zuul's message of 2017-11-14 16:24:33 +:
> > > > Build failed.
> > > > 
> > > > - release-openstack-python-without-pypi 
> > > > http://logs.openstack.org/17/175bd53e7b18a9dc4d42e60fe9225a5748eded34/pre-release/release-openstack-python-without-pypi/7a9474c/
> > > >  : POST_FAILURE in 14m 47s
> > > > - announce-release announce-release : SKIPPED
> > > > 
> > > 
> > > I can't find the error message in the job output log for this job. Maybe
> > > someone more familiar with the new log format can help?
> > 
> > If you look in ara you get http://paste.openstack.org/show/626285/
> > (pasted here because ara deeplinking not working with firefox, fix for
> > that should be out soon I think). Ansible is failing to update
> > permissions on a file. I think ansible does this to ensure that rsync
> > can read the files when it copies them.
> > 
> > Clark
> > 
> 
> Thanks. The ara page wasn't loading for me, but maybe I just wasn't
> patient enough.

Responses were slow, something was pulling significant amounts of data
out of the server earlier today to the detriment of other users (things
would eventually load though so wasn't a complete outage/Dos).

> 
> Is there any way to tell if this is a custom role or playbook? Or
> if it's part of the standard job?

Ara is again the magic here. I've pulled out chrome so that I can deep
link,
http://logs.openstack.org/17/175bd53e7b18a9dc4d42e60fe9225a5748eded34/pre-release/release-openstack-python-without-pypi/7a9474c/ara/file/18fc1bcc-3d6f-45e6-833e-c9fccefc7e72/#line-1
but that shows you exactly where the task comes from
(git.openstack.org/openstack-infra/zuul-jobs/roles/publish-artifacts-to-fileserver/tasks/main.yaml).

> 
> There is no "images/README" file in the trove repository. Can someone
> on the trove team tell us if that file comes from building trove?
> 
> Doug


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Rochelle Grober]

Folks,

This discussion and the people interested in it seem like a perfect application 
of the SIG process.  By turning LTS into a SIG, everyone can discuss the issues 
on the SIG mailing list and the discussion shouldn't end up split.  If it turns 
into a project, great.  If a solution is found that doesn't need a new project, 
great.  Even once  there is a decision on how to move forward, there will still 
be implementation issues and enhancements, so the SIG could very well be 
long-lived.  But the important aspect of this is:  keeping the discussion in a 
place where both devs and ops can follow the whole thing and act on 
recommendations.

Food for thought.

--Rocky

> -Original Message-
> From: Blair Bethwaite [mailto:blair.bethwa...@gmail.com]
> Sent: Tuesday, November 14, 2017 8:31 AM
> To: OpenStack Development Mailing List (not for usage questions)
> ; openstack-oper.  operat...@lists.openstack.org>
> Subject: Re: [openstack-dev] Upstream LTS Releases
> 
> Hi all - please note this conversation has been split variously across -dev 
> and -
> operators.
> 
> One small observation from the discussion so far is that it seems as though
> there are two issues being discussed under the one banner:
> 1) maintain old releases for longer
> 2) do stable releases less frequently
> 
> It would be interesting to understand if the people who want longer
> maintenance windows would be helped by #2.
> 
> On 14 November 2017 at 09:25, Doug Hellmann 
> wrote:
> > Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
> >> >> The concept, in general, is to create a new set of cores from
> >> >> these groups, and use 3rd party CI to validate patches. There are
> >> >> lots of details to be worked out yet, but our amazing UC (User
> >> >> Committee) will be begin working out the details.
> >> >
> >> > What is the most worrying is the exact "take over" process. Does it
> >> > mean that the teams will give away the +2 power to a different
> >> > team? Or will our (small) stable teams still be responsible for
> >> > landing changes? If so, will they have to learn how to debug 3rd party CI
> jobs?
> >> >
> >> > Generally, I'm scared of both overloading the teams and losing the
> >> > control over quality at the same time :) Probably the final proposal will
> clarify it..
> >>
> >> The quality of backported fixes is expected to be a direct (and
> >> only?) interest of those new teams of new cores, coming from users
> >> and operators and vendors. The more parties to establish their 3rd
> >> party
> >
> > We have an unhealthy focus on "3rd party" jobs in this discussion. We
> > should not assume that they are needed or will be present. They may
> > be, but we shouldn't build policy around the assumption that they
> > will. Why would we have third-party jobs on an old branch that we
> > don't have on master, for instance?
> >
> >> checking jobs, the better proposed changes communicated, which
> >> directly affects the quality in the end. I also suppose, contributors
> >> from ops world will likely be only struggling to see things getting
> >> fixed, and not new features adopted by legacy deployments they're used
> to maintain.
> >> So in theory, this works and as a mainstream developer and
> >> maintainer, you need no to fear of losing control over LTS code :)
> >>
> >> Another question is how to not block all on each over, and not push
> >> contributors away when things are getting awry, jobs failing and
> >> merging is blocked for a long time, or there is no consensus reached
> >> in a code review. I propose the LTS policy to enforce CI jobs be
> >> non-voting, as a first step on that way, and giving every LTS team
> >> member a core rights maybe? Not sure if that works though.
> >
> > I'm not sure what change you're proposing for CI jobs and their voting
> > status. Do you mean we should make the jobs non-voting as soon as the
> > branch passes out of the stable support period?
> >
> > Regarding the review team, anyone on the review team for a branch that
> > goes out of stable support will need to have +2 rights in that branch.
> > Otherwise there's no point in saying that they're maintaining the
> > branch.
> >
> > Doug
> >
> >
> __
> 
> >  OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> --
> Cheers,
> ~Blairo
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development 

Re: [openstack-dev] [Release-job-failures][neutron] Tag of openstack/networking-midonet failed

[Doug Hellmann]

Excerpts from zuul's message of 2017-11-14 20:13:45 +:
> Unable to freeze job graph: Unable to modify final job  publish-openstack-releasenotes branches: None source: 
> openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
> required_projects={'openstack/neutron':  0x7ff89c7ab550>} with variant  None source: 
> openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#318>
> 

This looks like the same failure that we had with trove-dashboard, but
probably because the tox environment tries to install neutron instead of
horizon.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][release][trove][infra] Pre-release of openstack/trove failed

[Doug Hellmann]

Excerpts from Clark Boylan's message of 2017-11-14 09:04:07 -0800:
> 
> On Tue, Nov 14, 2017, at 09:01 AM, Doug Hellmann wrote:
> > Excerpts from zuul's message of 2017-11-14 16:24:33 +:
> > > Build failed.
> > > 
> > > - release-openstack-python-without-pypi 
> > > http://logs.openstack.org/17/175bd53e7b18a9dc4d42e60fe9225a5748eded34/pre-release/release-openstack-python-without-pypi/7a9474c/
> > >  : POST_FAILURE in 14m 47s
> > > - announce-release announce-release : SKIPPED
> > > 
> > 
> > I can't find the error message in the job output log for this job. Maybe
> > someone more familiar with the new log format can help?
> 
> If you look in ara you get http://paste.openstack.org/show/626285/
> (pasted here because ara deeplinking not working with firefox, fix for
> that should be out soon I think). Ansible is failing to update
> permissions on a file. I think ansible does this to ensure that rsync
> can read the files when it copies them.
> 
> Clark
> 

Thanks. The ara page wasn't loading for me, but maybe I just wasn't
patient enough.

Is there any way to tell if this is a custom role or playbook? Or
if it's part of the standard job?

There is no "images/README" file in the trove repository. Can someone
on the trove team tell us if that file comes from building trove?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Doug Hellmann]

Excerpts from Chris Friesen's message of 2017-11-14 14:01:58 -0600:
> On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:
> 
> >> The quality of backported fixes is expected to be a direct (and only?)
> >> interest of those new teams of new cores, coming from users and operators 
> >> and
> >> vendors.
> >
> > I'm not assuming bad intentions, not at all. But there is a lot of involved 
> > in a
> > decision whether to make a backport or not. Will these people be able to
> > evaluate a risk of each patch? Do they have enough context on how that 
> > release
> > was implemented and what can break? Do they understand why feature 
> > backports are
> > bad? Why they should not skip (supported) releases when backporting?
> >
> > I know a lot of very reasonable people who do not understand the things 
> > above
> > really well.
> 
> I would hope that the core team for upstream LTS would be the (hopefully 
> experienced) people doing the downstream work that already happens within the 
> various distros.
> 
> Chris
> 

Presumably those are the same people we've been trying to convince
to work on the existing stable branches for the last 5 years. What
makes these extended branches more appealing to those people than
the existing branches? Is it the reduced requirements on maintaining
test jobs? Or maybe some other policy change that could be applied
to the stable branches?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Davanum Srinivas]

On Wed, Nov 15, 2017 at 7:01 AM, Chris Friesen
 wrote:
> On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:
>
>>> The quality of backported fixes is expected to be a direct (and only?)
>>> interest of those new teams of new cores, coming from users and operators
>>> and
>>> vendors.
>>
>>
>> I'm not assuming bad intentions, not at all. But there is a lot of
>> involved in a
>> decision whether to make a backport or not. Will these people be able to
>> evaluate a risk of each patch? Do they have enough context on how that
>> release
>> was implemented and what can break? Do they understand why feature
>> backports are
>> bad? Why they should not skip (supported) releases when backporting?
>>
>> I know a lot of very reasonable people who do not understand the things
>> above
>> really well.
>
>
> I would hope that the core team for upstream LTS would be the (hopefully
> experienced) people doing the downstream work that already happens within
> the various distros.

+1 Chris!

>
> Chris
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Paul Belanger]

On Tue, Nov 14, 2017 at 11:25:03AM -0500, Doug Hellmann wrote:
> Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
> > >> The concept, in general, is to create a new set of cores from these
> > >> groups, and use 3rd party CI to validate patches. There are lots of
> > >> details to be worked out yet, but our amazing UC (User Committee) will
> > >> be begin working out the details.
> > > 
> > > What is the most worrying is the exact "take over" process. Does it mean 
> > > that 
> > > the teams will give away the +2 power to a different team? Or will our 
> > > (small) 
> > > stable teams still be responsible for landing changes? If so, will they 
> > > have to 
> > > learn how to debug 3rd party CI jobs?
> > > 
> > > Generally, I'm scared of both overloading the teams and losing the 
> > > control over 
> > > quality at the same time :) Probably the final proposal will clarify it..
> > 
> > The quality of backported fixes is expected to be a direct (and only?) 
> > interest of those new teams of new cores, coming from users and 
> > operators and vendors. The more parties to establish their 3rd party 
> 
> We have an unhealthy focus on "3rd party" jobs in this discussion. We
> should not assume that they are needed or will be present. They may be,
> but we shouldn't build policy around the assumption that they will. Why
> would we have third-party jobs on an old branch that we don't have on
> master, for instance?
> 
I get the feeling more people are comfortable contributing to their own 3rd
party CI then upstream openstack CI systems.  Either because they don't have the
time or don't understand how it works. I agree with Doug, for this to work, I
think we need to have a health amount of people helping keep the CI systems
running upstream, then depending on 3rd party CI downstream.

As a comment, I think getting involved in the openstack-stablemaint team will be
a good first step towards the goals people are interested in here. I'm happy to
help work with others, and I'm taking tonyb up on his offer to start helping too
:)

> > checking jobs, the better proposed changes communicated, which directly 
> > affects the quality in the end. I also suppose, contributors from ops 
> > world will likely be only struggling to see things getting fixed, and 
> > not new features adopted by legacy deployments they're used to maintain. 
> > So in theory, this works and as a mainstream developer and maintainer, 
> > you need no to fear of losing control over LTS code :)
> > 
> > Another question is how to not block all on each over, and not push 
> > contributors away when things are getting awry, jobs failing and merging 
> > is blocked for a long time, or there is no consensus reached in a code 
> > review. I propose the LTS policy to enforce CI jobs be non-voting, as a 
> > first step on that way, and giving every LTS team member a core rights 
> > maybe? Not sure if that works though.
> 
> I'm not sure what change you're proposing for CI jobs and their voting
> status. Do you mean we should make the jobs non-voting as soon as the
> branch passes out of the stable support period?
> 
> Regarding the review team, anyone on the review team for a branch
> that goes out of stable support will need to have +2 rights in that
> branch. Otherwise there's no point in saying that they're maintaining
> the branch.
> 
> Doug
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Chris Friesen]

On 11/14/2017 01:28 PM, Dmitry Tantsur wrote:


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and operators and
vendors.


I'm not assuming bad intentions, not at all. But there is a lot of involved in a
decision whether to make a backport or not. Will these people be able to
evaluate a risk of each patch? Do they have enough context on how that release
was implemented and what can break? Do they understand why feature backports are
bad? Why they should not skip (supported) releases when backporting?

I know a lot of very reasonable people who do not understand the things above
really well.


I would hope that the core team for upstream LTS would be the (hopefully 
experienced) people doing the downstream work that already happens within the 
various distros.


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Samuel Cassiba]

On Tue, Nov 14, 2017 at 11:28 AM, Dmitry Tantsur  wrote:
> On 11/14/2017 05:08 PM, Bogdan Dobrelya wrote:

 The concept, in general, is to create a new set of cores from these
 groups, and use 3rd party CI to validate patches. There are lots of
 details to be worked out yet, but our amazing UC (User Committee) will
 be begin working out the details.
>>>
>>>
>>> What is the most worrying is the exact "take over" process. Does it mean
>>> that the teams will give away the +2 power to a different team? Or will our
>>> (small) stable teams still be responsible for landing changes? If so, will
>>> they have to learn how to debug 3rd party CI jobs?
>>>
>>> Generally, I'm scared of both overloading the teams and losing the
>>> control over quality at the same time :) Probably the final proposal will
>>> clarify it..
>>
>>
>> The quality of backported fixes is expected to be a direct (and only?)
>> interest of those new teams of new cores, coming from users and operators
>> and vendors.
>
>
> I'm not assuming bad intentions, not at all. But there is a lot of involved
> in a decision whether to make a backport or not. Will these people be able
> to evaluate a risk of each patch? Do they have enough context on how that
> release was implemented and what can break? Do they understand why feature
> backports are bad? Why they should not skip (supported) releases when
> backporting?
>
> I know a lot of very reasonable people who do not understand the things
> above really well.
>

I think there is more of a general "yes, but..." feel and not so much
a misunderstanding or lack of understanding entirely. With my operator
and PTL hats on, I'm in favor of a release cadence that is favorable
for the *people* involved. It's already proven that the current model
is broken or lacking in some way, simply by having these
conversations. With the status quo, it's almost a death march from one
release to the next, but nobody really wants to prolong that pain
because this topic comes up again and again.

Ideally, contributors are empowered enough to pick up the reins and
deliver the changes themselves, and some are, but it's pretty damned
daunting from the outside. The new contributors who want to contribute
but don't see the way in, probably because we haven't said mellon, are
left scratching their heads and eventually deem OpenStack as Not
Ready. It's almost like a perception exists that being able to even
submit a one-line patch is a gate to admittance. Unfortunately, less
and less are willing to pay that toll, no matter how nice the project
is on the other side.

-- 
Best,
Samuel Cassiba

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] inclusion of openstack/networking-generic-switch project under OpenStack baremetal program

[Villalovos, John L]

Thanks for sending this out.

I would vote for Option 1.

Thanks,
John

From: Pavlo Shchelokovskyy [mailto:pshchelokovs...@mirantis.com]
Sent: Tuesday, November 14, 2017 8:16 AM
To: OpenStack Development Mailing List (not for usage questions) 

Subject: [openstack-dev] [ironic] inclusion of 
openstack/networking-generic-switch project under OpenStack baremetal program

Hi all,

as this topic it was recently brought up in ironic IRC meeting, I'd like to 
start a discussion on the subject.

A quick recap - networking-generic-switch project (n-g-s) was born out of 
necessity to do two things:

-  test the "network isolation for baremetal nodes" (a.k.a. multi-tenancy) 
feature of ironic on upstream gates in virtualized environment and
- do the same on cheap/simple/dumb hardware switches that are not supported by 
other various openstack/networking-* projects.

Back when it was created AFAIR neutron governance (neutron stadium) was under 
some changes, so in the end n-g-s ended up not belonging to any official 
program.

Over time n-g-s grew to be an essential part of ironic gate testing (similar to 
virtualbmc). What's more, we have reports that it is already being used in 
production.

Currently the core reviewers team of n-g-s consists of 4 people (2 of those are 
currently core reviewers in ironic too), all of them are working for the same 
company (Mirantis). This poses some risk as companies and people come and go, 
plus since some voting ironic gate jobs depend on n-g-s stability, a more 
diverse group of core reviewers from baremetal program might be beneficial to 
be able to land patches in case of severe gate troubles.

Currently I know of 3 proposed ways to change the current situation:

1) include n-g-s under ironic (OpenStack Baremetal program) governance, 
effectively including ironic-core team to the core team of  n-g-s similar to 
how ironic-inspector currently governed (keeping an extended sub-core team). 
Reasoning for addition is the same as with virtualbmc/sushy projects, with the 
debatable difference that the actual scope of n-g-s is quite bigger and 
apparently includes production use-cases;

2) keep things as they are now, just add ironic-stable-maint team to the n-g-s 
core reviewers to decrease low diversity risks;

3) merge the code from n-g-s into networking-baremetal project which is already 
under ironic governance.

As a core in n-g-s myself I'm happy with either 1) or 2), but not really fond 
of 3) as it kind of stretches the networking-baremetal scope too much IMHO.

Eager to hear your comments and proposals.

Cheers,
--
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Dmitry Tantsur]

On 11/14/2017 06:21 PM, Erik McCormick wrote:

On Tue, Nov 14, 2017 at 11:30 AM, Blair Bethwaite
 wrote:

Hi all - please note this conversation has been split variously across
-dev and -operators.

One small observation from the discussion so far is that it seems as
though there are two issues being discussed under the one banner:
1) maintain old releases for longer
2) do stable releases less frequently

It would be interesting to understand if the people who want longer
maintenance windows would be helped by #2.



I would like to hear from people who do *not* want #2 and why not.
What are the benefits of 6 months vs. 1 year? I have heard objections
in the hallway track, but I have struggled to retain the rationale for
more than 10 seconds. I think this may be more of a religious
discussion that could take a while though.


One point is maintenance burden. Everything that has to be deprecated and 
removed will have to be kept for twice more time in the worst case.


The second point is that contributors, from my experience, don't like waiting 
many months for their shiny feature to get released. That will increase pressure 
on the teams in the end of every release to get everything in - or it will have 
to wait 1 year.


Note that both points apply even if you do "less-stable" releases between stable 
ones.




#1 is something we can act on right now with the eventual goal of
being able to skip releases entirely. We are addressing the
maintenance of the old issue right now. As we get farther down the
road of fast-forward upgrade tooling, then we will be able to please
those wishing for a slower upgrade cadence, and those that want to
stay on the bleeding edge simultaneously.

-Erik


On 14 November 2017 at 09:25, Doug Hellmann  wrote:

Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.


What is the most worrying is the exact "take over" process. Does it mean that
the teams will give away the +2 power to a different team? Or will our (small)
stable teams still be responsible for landing changes? If so, will they have to
learn how to debug 3rd party CI jobs?

Generally, I'm scared of both overloading the teams and losing the control over
quality at the same time :) Probably the final proposal will clarify it..


The quality of backported fixes is expected to be a direct (and only?)
interest of those new teams of new cores, coming from users and
operators and vendors. The more parties to establish their 3rd party


We have an unhealthy focus on "3rd party" jobs in this discussion. We
should not assume that they are needed or will be present. They may be,
but we shouldn't build policy around the assumption that they will. Why
would we have third-party jobs on an old branch that we don't have on
master, for instance?


checking jobs, the better proposed changes communicated, which directly
affects the quality in the end. I also suppose, contributors from ops
world will likely be only struggling to see things getting fixed, and
not new features adopted by legacy deployments they're used to maintain.
So in theory, this works and as a mainstream developer and maintainer,
you need no to fear of losing control over LTS code :)

Another question is how to not block all on each over, and not push
contributors away when things are getting awry, jobs failing and merging
is blocked for a long time, or there is no consensus reached in a code
review. I propose the LTS policy to enforce CI jobs be non-voting, as a
first step on that way, and giving every LTS team member a core rights
maybe? Not sure if that works though.


I'm not sure what change you're proposing for CI jobs and their voting
status. Do you mean we should make the jobs non-voting as soon as the
branch passes out of the stable support period?

Regarding the review team, anyone on the review team for a branch
that goes out of stable support will need to have +2 rights in that
branch. Otherwise there's no point in saying that they're maintaining
the branch.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Cheers,
~Blairo

___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
openstack-operat...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [openstack-dev] Upstream LTS Releases

[Dmitry Tantsur]

On 11/14/2017 05:08 PM, Bogdan Dobrelya wrote:

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.


What is the most worrying is the exact "take over" process. Does it mean that 
the teams will give away the +2 power to a different team? Or will our (small) 
stable teams still be responsible for landing changes? If so, will they have 
to learn how to debug 3rd party CI jobs?


Generally, I'm scared of both overloading the teams and losing the control 
over quality at the same time :) Probably the final proposal will clarify it..


The quality of backported fixes is expected to be a direct (and only?) interest 
of those new teams of new cores, coming from users and operators and vendors. 


I'm not assuming bad intentions, not at all. But there is a lot of involved in a 
decision whether to make a backport or not. Will these people be able to 
evaluate a risk of each patch? Do they have enough context on how that release 
was implemented and what can break? Do they understand why feature backports are 
bad? Why they should not skip (supported) releases when backporting?


I know a lot of very reasonable people who do not understand the things above 
really well.


The more parties to establish their 3rd party checking jobs, the better proposed 
changes communicated, which directly affects the quality in the end. I also 
suppose, contributors from ops world will likely be only struggling to see 
things getting fixed, and not new features adopted by legacy deployments they're 
used to maintain. So in theory, this works and as a mainstream developer and 
maintainer, you need no to fear of losing control over LTS code :)


Another question is how to not block all on each over, and not push contributors 
away when things are getting awry, jobs failing and merging is blocked for a 
long time, or there is no consensus reached in a code review. I propose the LTS 
policy to enforce CI jobs be non-voting, as a first step on that way, and giving 
every LTS team member a core rights maybe? Not sure if that works though.





__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Chris Friesen]

On 11/14/2017 10:25 AM, Doug Hellmann wrote:

Why
would we have third-party jobs on an old branch that we don't have on
master, for instance?


One possible reason is to test the stable version of OpenStack against a stable 
version of the underlying OS distro.   (Where that distro may not meet the 
package version requirements for running master.)


Chris

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Mohammed Naser]

Jens,

That's quite an interesting catch.  I'm reaching out to the author of
this change to get some more information.

Thanks,
Mohammed

On Tue, Nov 14, 2017 at 2:02 PM, Jens Harbott  wrote:
> 2017-11-14 16:29 GMT+00:00 Mohammed Naser :
>> Hi everyone,
>>
>> Thank you so much for the work on this, I'm sure we can progress with
>> this together.  I have noticed that this only occurs in master and
>> never in the stable branches.  Also, it only occurs under Ubuntu (so
>> maybe something related to mod_wsgi version?)
>>
>> Given that we don't have any "master" built packages for Ubuntu, we
>> test against the latest release which is the pike release.
>>
>> https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L6-L10
>>
>> I've noticed the issue is not as present in older branches but much
>> more visible in master.
>
> So does the issue not happen at all for stable/pike or less often?
> Anyway that would seem to indicate not an issue with the Ubuntu
> packages, but with the way they are deployed.
>
> If you look at [1] you can see that for pike you setup nova-api wsgi
> with workers=1 and threads=$::os_workers, which in master was swapped,
> see [2]. I'd suggest testing a revert of that change.
>
> [1] 
> https://github.com/openstack/puppet-nova/blob/stable/pike/manifests/wsgi/apache_api.pp
> [2] 
> https://github.com/openstack/puppet-nova/commit/df638e2526d2d957318519dfcfb9098cb7726095
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Jens Harbott]

2017-11-14 16:29 GMT+00:00 Mohammed Naser :
> Hi everyone,
>
> Thank you so much for the work on this, I'm sure we can progress with
> this together.  I have noticed that this only occurs in master and
> never in the stable branches.  Also, it only occurs under Ubuntu (so
> maybe something related to mod_wsgi version?)
>
> Given that we don't have any "master" built packages for Ubuntu, we
> test against the latest release which is the pike release.
>
> https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L6-L10
>
> I've noticed the issue is not as present in older branches but much
> more visible in master.

So does the issue not happen at all for stable/pike or less often?
Anyway that would seem to indicate not an issue with the Ubuntu
packages, but with the way they are deployed.

If you look at [1] you can see that for pike you setup nova-api wsgi
with workers=1 and threads=$::os_workers, which in master was swapped,
see [2]. I'd suggest testing a revert of that change.

[1] 
https://github.com/openstack/puppet-nova/blob/stable/pike/manifests/wsgi/apache_api.pp
[2] 
https://github.com/openstack/puppet-nova/commit/df638e2526d2d957318519dfcfb9098cb7726095

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] inclusion of openstack/networking-generic-switch project under OpenStack baremetal program

[Dmitry Tantsur]

Hi!

Thanks for raising this.

On 11/14/2017 05:16 PM, Pavlo Shchelokovskyy wrote:

Hi all,

as this topic it was recently brought up in ironic IRC meeting, I'd like to 
start a discussion on the subject.


A quick recap - networking-generic-switch project (n-g-s) was born out of 
necessity to do two things:


-  test the "network isolation for baremetal nodes" (a.k.a. multi-tenancy) 
feature of ironic on upstream gates in virtualized environment and
- do the same on cheap/simple/dumb hardware switches that are not supported by 
other various openstack/networking-* projects.


Back when it was created AFAIR neutron governance (neutron stadium) was under 
some changes, so in the end n-g-s ended up not belonging to any official program.


Over time n-g-s grew to be an essential part of ironic gate testing (similar to 
virtualbmc). What's more, we have reports that it is already being used in 
production.


Currently the core reviewers team of n-g-s consists of 4 people (2 of those are 
currently core reviewers in ironic too), all of them are working for the same 
company (Mirantis). This poses some risk as companies and people come and go, 
plus since some voting ironic gate jobs depend on n-g-s stability, a more 
diverse group of core reviewers from baremetal program might be beneficial to be 
able to land patches in case of severe gate troubles.






Currently I know of 3 proposed ways to change the current situation:

1) include n-g-s under ironic (OpenStack Baremetal program) governance, 
effectively including ironic-core team to the core team of  n-g-s similar to how 
ironic-inspector currently governed (keeping an extended sub-core team). 
Reasoning for addition is the same as with virtualbmc/sushy projects, with the 
debatable difference that the actual scope of n-g-s is quite bigger and 
apparently includes production use-cases;


Some time ago I would go for option (2), now I guess I'm more with option (1). I 
think we have to recognize that certain things we don't target for production 
(yet or at all) end up in production. We cannot hide from this fact.


Furthermore, not all productions are the same. While networking-generic-switch 
certainly has (probably addressable) problems, it may be just enough for many 
cases, especially for people just starting with ironic. This is the key point 
that makes me prefer this option - we always need to work on an easier start.




2) keep things as they are now, just add ironic-stable-maint team to the n-g-s 
core reviewers to decrease low diversity risks;


This would work, but I think it hides the problem instead of fixing it - see 
above.



3) merge the code from n-g-s into networking-baremetal project which is already 
under ironic governance.


Well.. projects are cheap, let's have more of them :) Seriously though, I 
sometimes feel that networking-baremetal *already* does too many unrelated 
things. Let's not add to it.




As a core in n-g-s myself I'm happy with either 1) or 2), but not really fond of 
3) as it kind of stretches the networking-baremetal scope too much IMHO.


TL;DR I vote for (1) with properly documenting the scope and limitations.

Dmitry



Eager to hear your comments and proposals.

Cheers,
--
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][inspector] HA Demo

[milanisko k]

Folks,

I'd like to share the progress of the HA Inspector effort with you.
TL;DR I'm able to run 2 instances of ironic inspector in an active-active
fashion in a modified devstack deployment.

I've uploaded a Github repo containing my config[1] to take to
reproduce as well as a video[2] of me doing so.

Please, let me know WDYT or just tell me I'm crazy on the #openstack-ironic
channel ;)

Cheers,
milan

[1] https://github.com/dparalen/ironic-inspector-ha-demo
[2] https://www.youtube.com/watch?v=I5wLmBfbhds
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] Upstream LTS Releases

[Erik McCormick]

On Tue, Nov 14, 2017 at 11:30 AM, Blair Bethwaite
 wrote:
> Hi all - please note this conversation has been split variously across
> -dev and -operators.
>
> One small observation from the discussion so far is that it seems as
> though there are two issues being discussed under the one banner:
> 1) maintain old releases for longer
> 2) do stable releases less frequently
>
> It would be interesting to understand if the people who want longer
> maintenance windows would be helped by #2.
>

I would like to hear from people who do *not* want #2 and why not.
What are the benefits of 6 months vs. 1 year? I have heard objections
in the hallway track, but I have struggled to retain the rationale for
more than 10 seconds. I think this may be more of a religious
discussion that could take a while though.

#1 is something we can act on right now with the eventual goal of
being able to skip releases entirely. We are addressing the
maintenance of the old issue right now. As we get farther down the
road of fast-forward upgrade tooling, then we will be able to please
those wishing for a slower upgrade cadence, and those that want to
stay on the bleeding edge simultaneously.

-Erik

> On 14 November 2017 at 09:25, Doug Hellmann  wrote:
>> Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
>>> >> The concept, in general, is to create a new set of cores from these
>>> >> groups, and use 3rd party CI to validate patches. There are lots of
>>> >> details to be worked out yet, but our amazing UC (User Committee) will
>>> >> be begin working out the details.
>>> >
>>> > What is the most worrying is the exact "take over" process. Does it mean 
>>> > that
>>> > the teams will give away the +2 power to a different team? Or will our 
>>> > (small)
>>> > stable teams still be responsible for landing changes? If so, will they 
>>> > have to
>>> > learn how to debug 3rd party CI jobs?
>>> >
>>> > Generally, I'm scared of both overloading the teams and losing the 
>>> > control over
>>> > quality at the same time :) Probably the final proposal will clarify it..
>>>
>>> The quality of backported fixes is expected to be a direct (and only?)
>>> interest of those new teams of new cores, coming from users and
>>> operators and vendors. The more parties to establish their 3rd party
>>
>> We have an unhealthy focus on "3rd party" jobs in this discussion. We
>> should not assume that they are needed or will be present. They may be,
>> but we shouldn't build policy around the assumption that they will. Why
>> would we have third-party jobs on an old branch that we don't have on
>> master, for instance?
>>
>>> checking jobs, the better proposed changes communicated, which directly
>>> affects the quality in the end. I also suppose, contributors from ops
>>> world will likely be only struggling to see things getting fixed, and
>>> not new features adopted by legacy deployments they're used to maintain.
>>> So in theory, this works and as a mainstream developer and maintainer,
>>> you need no to fear of losing control over LTS code :)
>>>
>>> Another question is how to not block all on each over, and not push
>>> contributors away when things are getting awry, jobs failing and merging
>>> is blocked for a long time, or there is no consensus reached in a code
>>> review. I propose the LTS policy to enforce CI jobs be non-voting, as a
>>> first step on that way, and giving every LTS team member a core rights
>>> maybe? Not sure if that works though.
>>
>> I'm not sure what change you're proposing for CI jobs and their voting
>> status. Do you mean we should make the jobs non-voting as soon as the
>> branch passes out of the stable support period?
>>
>> Regarding the review team, anyone on the review team for a branch
>> that goes out of stable support will need to have +2 rights in that
>> branch. Otherwise there's no point in saying that they're maintaining
>> the branch.
>>
>> Doug
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Cheers,
> ~Blairo
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][release][trove][infra] Pre-release of openstack/trove failed

[Clark Boylan]

On Tue, Nov 14, 2017, at 09:01 AM, Doug Hellmann wrote:
> Excerpts from zuul's message of 2017-11-14 16:24:33 +:
> > Build failed.
> > 
> > - release-openstack-python-without-pypi 
> > http://logs.openstack.org/17/175bd53e7b18a9dc4d42e60fe9225a5748eded34/pre-release/release-openstack-python-without-pypi/7a9474c/
> >  : POST_FAILURE in 14m 47s
> > - announce-release announce-release : SKIPPED
> > 
> 
> I can't find the error message in the job output log for this job. Maybe
> someone more familiar with the new log format can help?

If you look in ara you get http://paste.openstack.org/show/626285/
(pasted here because ara deeplinking not working with firefox, fix for
that should be out soon I think). Ansible is failing to update
permissions on a file. I think ansible does this to ensure that rsync
can read the files when it copies them.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][release][trove][infra] Pre-release of openstack/trove failed

[Doug Hellmann]

Excerpts from zuul's message of 2017-11-14 16:24:33 +:
> Build failed.
> 
> - release-openstack-python-without-pypi 
> http://logs.openstack.org/17/175bd53e7b18a9dc4d42e60fe9225a5748eded34/pre-release/release-openstack-python-without-pypi/7a9474c/
>  : POST_FAILURE in 14m 47s
> - announce-release announce-release : SKIPPED
> 

I can't find the error message in the job output log for this job. Maybe
someone more familiar with the new log format can help?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Davanum Srinivas]

Blair,

Please add #2 as a line proposal in:
https://etherpad.openstack.org/p/LTS-proposal

So far it's focused on #1

Thanks,
Dims

On Wed, Nov 15, 2017 at 3:30 AM, Blair Bethwaite
 wrote:
> Hi all - please note this conversation has been split variously across
> -dev and -operators.
>
> One small observation from the discussion so far is that it seems as
> though there are two issues being discussed under the one banner:
> 1) maintain old releases for longer
> 2) do stable releases less frequently
>
> It would be interesting to understand if the people who want longer
> maintenance windows would be helped by #2.
>
> On 14 November 2017 at 09:25, Doug Hellmann  wrote:
>> Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
>>> >> The concept, in general, is to create a new set of cores from these
>>> >> groups, and use 3rd party CI to validate patches. There are lots of
>>> >> details to be worked out yet, but our amazing UC (User Committee) will
>>> >> be begin working out the details.
>>> >
>>> > What is the most worrying is the exact "take over" process. Does it mean 
>>> > that
>>> > the teams will give away the +2 power to a different team? Or will our 
>>> > (small)
>>> > stable teams still be responsible for landing changes? If so, will they 
>>> > have to
>>> > learn how to debug 3rd party CI jobs?
>>> >
>>> > Generally, I'm scared of both overloading the teams and losing the 
>>> > control over
>>> > quality at the same time :) Probably the final proposal will clarify it..
>>>
>>> The quality of backported fixes is expected to be a direct (and only?)
>>> interest of those new teams of new cores, coming from users and
>>> operators and vendors. The more parties to establish their 3rd party
>>
>> We have an unhealthy focus on "3rd party" jobs in this discussion. We
>> should not assume that they are needed or will be present. They may be,
>> but we shouldn't build policy around the assumption that they will. Why
>> would we have third-party jobs on an old branch that we don't have on
>> master, for instance?
>>
>>> checking jobs, the better proposed changes communicated, which directly
>>> affects the quality in the end. I also suppose, contributors from ops
>>> world will likely be only struggling to see things getting fixed, and
>>> not new features adopted by legacy deployments they're used to maintain.
>>> So in theory, this works and as a mainstream developer and maintainer,
>>> you need no to fear of losing control over LTS code :)
>>>
>>> Another question is how to not block all on each over, and not push
>>> contributors away when things are getting awry, jobs failing and merging
>>> is blocked for a long time, or there is no consensus reached in a code
>>> review. I propose the LTS policy to enforce CI jobs be non-voting, as a
>>> first step on that way, and giving every LTS team member a core rights
>>> maybe? Not sure if that works though.
>>
>> I'm not sure what change you're proposing for CI jobs and their voting
>> status. Do you mean we should make the jobs non-voting as soon as the
>> branch passes out of the stable support period?
>>
>> Regarding the review team, anyone on the review team for a branch
>> that goes out of stable support will need to have +2 rights in that
>> branch. Otherwise there's no point in saying that they're maintaining
>> the branch.
>>
>> Doug
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Cheers,
> ~Blairo
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Davanum Srinivas]

Let's focus our energy on the etherpad please

https://etherpad.openstack.org/p/LTS-proposal

On Wed, Nov 15, 2017 at 3:35 AM, Davanum Srinivas  wrote:
> Saverio,
>
> Please see this :
> https://docs.openstack.org/project-team-guide/stable-branches.html for
> current policies.
>
> On Wed, Nov 15, 2017 at 3:33 AM, Saverio Proto  
> wrote:
>>> Which stable policy does that patch violate?  It's clearly a bug
>>> because the wrong information is being logged.  I suppose it goes
>>> against the string freeze rule? Except that we've stopped translating
>>> log messages so maybe we don't need to worry about that in this case,
>>> since it isn't an exception.
>>
>> Well, I also would like to understand more about stable policy violations.
>> When I proposed such patches in the past for the release N-2 I have
>> always got the answer: it is not a security issue so it will not be merged.
>>
>> This is a good example of how things have been working so far:
>>
>> https://review.openstack.org/#/q/677eb1c4160c08cfce2900495741f0ea15f566fa
>>
>> This cinder patch was merged in master. It was then merged in Mitaka.
>> But it was not merged in Liberty just because "only security fixes" were
>> allowed at that point.
>>
>> You can read that in the comments:
>> https://review.openstack.org/#/c/306610/
>>
>> Is this kind of things going to change after the discussion in Sydney ?
>> The discussion is not enough ? what we need to get done then ?
>>
>> thank you
>>
>> Saverio
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Davanum Srinivas :: https://twitter.com/dims



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [octavia] amphora fails to send request to members

[Michael Johnson]

Yipei,

Yeah, we have clearly identified the problem.  Those two default route
lines should not be different.  See my devstack:
sudo: unable to resolve host amphora-20a717b4-eb97-4b5c-a11a-0633fe61f135
default via 10.0.0.1 dev eth1  table 1 onlink
default via 10.0.0.1 dev eth1 onlink

So the issue here is the gateway neutron gave us for the subnet is
different than the one the amphora is receiving via DHCP.

The "default via 10.0.1.1 dev eth1  table 1 onlink " would not be
present if neutron didn't give us a gateway address and there are not
host routes on the subnet.

Did the gateway for the subnet changed after the amphora was booted?
Was the amphora live-migrated? Was there a host route on that subnet?

Michael

On Tue, Nov 14, 2017 at 12:29 AM, Yipei Niu  wrote:
> Hi, Michael,
>
> Please ignore my last two mails. Sorry about that.
>
> The results of the two commands are as follows.
>
> ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
> amphora-haproxy ip route show table all
> sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
> default via 10.0.1.1 dev eth1  table 1 onlink
> default via 10.0.1.10 dev eth1
> 10.0.1.0/24 dev eth1  proto kernel  scope link  src 10.0.1.8
> broadcast 10.0.1.0 dev eth1  table local  proto kernel  scope link  src
> 10.0.1.8
> local 10.0.1.4 dev eth1  table local  proto kernel  scope host  src 10.0.1.8
> local 10.0.1.8 dev eth1  table local  proto kernel  scope host  src 10.0.1.8
> broadcast 10.0.1.255 dev eth1  table local  proto kernel  scope link  src
> 10.0.1.8
> fe80::/64 dev eth1  proto kernel  metric 256  pref medium
> unreachable default dev lo  table unspec  proto kernel  metric 4294967295
> error -101 pref medium
> local fe80::f816:3eff:febe:5ad5 dev lo  table local  proto none  metric 0
> pref medium
> ff00::/8 dev eth1  table local  metric 256  pref medium
> unreachable default dev lo  table unspec  proto kernel  metric 4294967295
> error -101 pref medium
>
> ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
> amphora-haproxy ip rule show
> sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
> 0: from all lookup local
> 100: from 10.0.1.4 lookup 1
> 32766: from all lookup main
> 32767: from all lookup default
>
> I think I know the source. When haproxy receives packets sent by curl, it
> responds with taking VIP as source ip for 3-way handshake. Before adding
> “100: from 10.0.1.9 lookup 1”, the datagrams are routed based on main table.
> After adding "100: from 10.0.1.9 lookup 1", the haproxy tries to find the
> gateway based on "default via 10.0.1.1 dev eth1  table 1 onlink". However,
> if there is no router, the gateway ip is missing, making haproxy fails to
> build tcp connection.
>
> Best regards,
> Yipei
>
>
> On Tue, Nov 14, 2017 at 11:04 AM, Yipei Niu  wrote:
>>
>> Hi, Michael,
>>
>> Sorry about the typo in the last mail. Please just ignore the last mail.
>>
>> In the environment where octavia and tricircle are installed together, I
>> created a router and attached subnet1 to it. Then I bind the mac address of
>> 10.0.1.10 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
>> manually making amphora knows the mac address of 10.0.1.1 (actually it is
>> the mac of 10.0.1.10, since 10.0.1.1 does not exist), it works.
>>
>> I also run the commands in this environment.
>>
>> ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
>> amphora-haproxy ip route show table all
>> sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
>> default via 10.0.1.1 dev eth1  table 1 onlink
>> default via 10.0.1.10 dev eth1
>> 10.0.1.0/24 dev eth1  proto kernel  scope link  src 10.0.1.8
>> broadcast 10.0.1.0 dev eth1  table local  proto kernel  scope link  src
>> 10.0.1.8
>> local 10.0.1.4 dev eth1  table local  proto kernel  scope host  src
>> 10.0.1.8
>> local 10.0.1.8 dev eth1  table local  proto kernel  scope host  src
>> 10.0.1.8
>> broadcast 10.0.1.255 dev eth1  table local  proto kernel  scope link  src
>> 10.0.1.8
>> fe80::/64 dev eth1  proto kernel  metric 256  pref medium
>> unreachable default dev lo  table unspec  proto kernel  metric 4294967295
>> error -101 pref medium
>> local fe80::f816:3eff:febe:5ad5 dev lo  table local  proto none  metric 0
>> pref medium
>> ff00::/8 dev eth1  table local  metric 256  pref medium
>> unreachable default dev lo  table unspec  proto kernel  metric 4294967295
>> error -101 pref medium
>> ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
>> amphora-haproxy ip rule show
>> sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
>> 0: from all lookup local
>> 100: from 10.0.1.4 lookup 1
>> 32766: from all lookup main
>> 32767: from all lookup default
>>
>>
>> To make the situation clear, I run the commands in the environment
>> installed octavia alone. Please note that in this environment, there is no
>> router. 

Re: [openstack-dev] [Release-job-failures][zuul][infra][trove] Tag of openstack/trove-dashboard failed

[Doug Hellmann]

Excerpts from Clark Boylan's message of 2017-11-14 08:31:15 -0800:
> On Tue, Nov 14, 2017, at 08:09 AM, Doug Hellmann wrote:
> > Excerpts from zuul's message of 2017-11-14 16:01:20 +:
> > > Unable to freeze job graph: Unable to modify final job  > > publish-openstack-releasenotes branches: None source: 
> > > openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
> > > required_projects={'openstack/horizon':  > > 0x7ff8f8b0a710>} with variant  > > branches: None source: 
> > > openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#291>
> > > 
> > 
> > Is there some way to detect this type of error before we approve a
> > release?
> 
> My understanding is Zuul won't do complete pre merge testing of these
> jobs because they run in a post merge context and have access to secrets
> for stuff like AFS publishing. Zuul does do syntax checking on these
> jobs pre merge though so if we could get Zuul to check "final" state
> without building an entire job graph that may solve the problem. I'm not
> familiar enough with Zuul's config compiler to know if that is
> reasonable though.
> 
> It is possible that Zuul would notice the failure post merge when
> attempting to run any jobs against the repo when it is in this state.
> Though my hunch is we didn't notice until the release jobs ran because
> hitting the error currently requires you to attempt to queue up the
> specific broken job.
> 
> Likely any fix for this will have to happen in Zuul's config handler(s)
> to notice this error early rather than late on job execution.
> 
> Clark
> 

OK. I'd be happy to run an extra validation step from a job attached to
the release request so we can know "the release notes build is going to
fail if you approve this release" or whatever.

I'm assuming the job is always failing, but no one has noticed until we
got the email as part of running the job in the release queue.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic] inclusion of openstack/networking-generic-switch project under OpenStack baremetal program

[Julia Kreger]

On Tue, Nov 14, 2017 at 9:16 AM, Pavlo Shchelokovskyy
 wrote:
> As a core in n-g-s myself I'm happy with either 1) or 2), but not really
> fond of 3) as it kind of stretches the networking-baremetal scope too much
> IMHO.

Personally, I'm happy with 1 or 2. I personally think we might as well
lean towards 1 from a standpoint that we've had operators bring up
using n-g-s, and while we as a community predicate that it is not
intended for production use, and they are aware of that, they need
something that works for their baremetal deployment scenario. Given
the common usage, it just seems logical to me that we go for option 1.
Especially since we already have a fairly good practice in Ironic of
not approving things in sub-project that we do not understand.

-Julia

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Davanum Srinivas]

Saverio,

Please see this :
https://docs.openstack.org/project-team-guide/stable-branches.html for
current policies.

On Wed, Nov 15, 2017 at 3:33 AM, Saverio Proto  wrote:
>> Which stable policy does that patch violate?  It's clearly a bug
>> because the wrong information is being logged.  I suppose it goes
>> against the string freeze rule? Except that we've stopped translating
>> log messages so maybe we don't need to worry about that in this case,
>> since it isn't an exception.
>
> Well, I also would like to understand more about stable policy violations.
> When I proposed such patches in the past for the release N-2 I have
> always got the answer: it is not a security issue so it will not be merged.
>
> This is a good example of how things have been working so far:
>
> https://review.openstack.org/#/q/677eb1c4160c08cfce2900495741f0ea15f566fa
>
> This cinder patch was merged in master. It was then merged in Mitaka.
> But it was not merged in Liberty just because "only security fixes" were
> allowed at that point.
>
> You can read that in the comments:
> https://review.openstack.org/#/c/306610/
>
> Is this kind of things going to change after the discussion in Sydney ?
> The discussion is not enough ? what we need to get done then ?
>
> thank you
>
> Saverio
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Need help with Openstack Swift Installation and configuration with S3

[John Dickinson]

To do this you want to use the `swift3` middleware, available at 
https://github.com/openstack/swift3. Its `README.md` file has installation 
instructions.

Also note that the Swift community is currently integrating the `swift3` 
middleware into Swift's code repo. In the future, you will not need to install 
any external components (ie `swift3`); you'll just have it available out of the 
box.

--John




On 14 Nov 2017, at 4:01, Shalabh Aggarwal wrote:

> Hi,
>
> We just started with Openstack Swift which we intend to use as a
> replacement for an API which was written to work with AWS S3.
> We know that Swift is S3 compatible and our API should work out of the box
> with it.
>
> We have not been able to get Swift running with S3 plugins.
> I was wondering if there is a go to documentation which covers the
> installation of Openstack Swift along with configuration with S3.
>
> Any help would be highly appreciated.
>
> Thanks!
>
> -- 
> Regards,
>
> Shalabh Aggarwal
> t: +91 9990960159
> w: www.shalabhaggarwal.com


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Saverio Proto]

> Which stable policy does that patch violate?  It's clearly a bug
> because the wrong information is being logged.  I suppose it goes
> against the string freeze rule? Except that we've stopped translating
> log messages so maybe we don't need to worry about that in this case,
> since it isn't an exception.

Well, I also would like to understand more about stable policy violations.
When I proposed such patches in the past for the release N-2 I have
always got the answer: it is not a security issue so it will not be merged.

This is a good example of how things have been working so far:

https://review.openstack.org/#/q/677eb1c4160c08cfce2900495741f0ea15f566fa

This cinder patch was merged in master. It was then merged in Mitaka.
But it was not merged in Liberty just because "only security fixes" were
allowed at that point.

You can read that in the comments:
https://review.openstack.org/#/c/306610/

Is this kind of things going to change after the discussion in Sydney ?
The discussion is not enough ? what we need to get done then ?

thank you

Saverio


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][zuul][infra][trove] Tag of openstack/trove-dashboard failed

[Clark Boylan]

On Tue, Nov 14, 2017, at 08:09 AM, Doug Hellmann wrote:
> Excerpts from zuul's message of 2017-11-14 16:01:20 +:
> > Unable to freeze job graph: Unable to modify final job  > publish-openstack-releasenotes branches: None source: 
> > openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
> > required_projects={'openstack/horizon':  > 0x7ff8f8b0a710>} with variant  > None source: 
> > openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#291>
> > 
> 
> Is there some way to detect this type of error before we approve a
> release?

My understanding is Zuul won't do complete pre merge testing of these
jobs because they run in a post merge context and have access to secrets
for stuff like AFS publishing. Zuul does do syntax checking on these
jobs pre merge though so if we could get Zuul to check "final" state
without building an entire job graph that may solve the problem. I'm not
familiar enough with Zuul's config compiler to know if that is
reasonable though.

It is possible that Zuul would notice the failure post merge when
attempting to run any jobs against the repo when it is in this state.
Though my hunch is we didn't notice until the release jobs ran because
hitting the error currently requires you to attempt to queue up the
specific broken job.

Likely any fix for this will have to happen in Zuul's config handler(s)
to notice this error early rather than late on job execution.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Blair Bethwaite]

Hi all - please note this conversation has been split variously across
-dev and -operators.

One small observation from the discussion so far is that it seems as
though there are two issues being discussed under the one banner:
1) maintain old releases for longer
2) do stable releases less frequently

It would be interesting to understand if the people who want longer
maintenance windows would be helped by #2.

On 14 November 2017 at 09:25, Doug Hellmann  wrote:
> Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
>> >> The concept, in general, is to create a new set of cores from these
>> >> groups, and use 3rd party CI to validate patches. There are lots of
>> >> details to be worked out yet, but our amazing UC (User Committee) will
>> >> be begin working out the details.
>> >
>> > What is the most worrying is the exact "take over" process. Does it mean 
>> > that
>> > the teams will give away the +2 power to a different team? Or will our 
>> > (small)
>> > stable teams still be responsible for landing changes? If so, will they 
>> > have to
>> > learn how to debug 3rd party CI jobs?
>> >
>> > Generally, I'm scared of both overloading the teams and losing the control 
>> > over
>> > quality at the same time :) Probably the final proposal will clarify it..
>>
>> The quality of backported fixes is expected to be a direct (and only?)
>> interest of those new teams of new cores, coming from users and
>> operators and vendors. The more parties to establish their 3rd party
>
> We have an unhealthy focus on "3rd party" jobs in this discussion. We
> should not assume that they are needed or will be present. They may be,
> but we shouldn't build policy around the assumption that they will. Why
> would we have third-party jobs on an old branch that we don't have on
> master, for instance?
>
>> checking jobs, the better proposed changes communicated, which directly
>> affects the quality in the end. I also suppose, contributors from ops
>> world will likely be only struggling to see things getting fixed, and
>> not new features adopted by legacy deployments they're used to maintain.
>> So in theory, this works and as a mainstream developer and maintainer,
>> you need no to fear of losing control over LTS code :)
>>
>> Another question is how to not block all on each over, and not push
>> contributors away when things are getting awry, jobs failing and merging
>> is blocked for a long time, or there is no consensus reached in a code
>> review. I propose the LTS policy to enforce CI jobs be non-voting, as a
>> first step on that way, and giving every LTS team member a core rights
>> maybe? Not sure if that works though.
>
> I'm not sure what change you're proposing for CI jobs and their voting
> status. Do you mean we should make the jobs non-voting as soon as the
> branch passes out of the stable support period?
>
> Regarding the review team, anyone on the review team for a branch
> that goes out of stable support will need to have +2 rights in that
> branch. Otherwise there's no point in saying that they're maintaining
> the branch.
>
> Doug
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Cheers,
~Blairo

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Mohammed Naser]

Hi everyone,

Thank you so much for the work on this, I'm sure we can progress with
this together.  I have noticed that this only occurs in master and
never in the stable branches.  Also, it only occurs under Ubuntu (so
maybe something related to mod_wsgi version?)

Given that we don't have any "master" built packages for Ubuntu, we
test against the latest release which is the pike release.

https://github.com/openstack/puppet-openstack-integration/blob/master/manifests/repos.pp#L6-L10

I've noticed the issue is not as present in older branches but much
more visible in master.

Thanks,
Mohammed

On Tue, Nov 14, 2017 at 6:21 AM, Tobias Urdin  wrote:
> Yea, I've been scavenging the logs for any kind of indicator on what
> might have gone wrong but I can't see anything
> related to a deadlock even though I'm very certain that's the issue but
> don't know what's causing it.
>
> Perhaps we will need to manually recreate this issue and then
> troubleshoot it manually.
> The apache2 mod_wsgi config should be OK according to the docs [1].
>
> Best regards
>
> [1]
> http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html
>
> On 11/14/2017 11:12 AM, Jens Harbott wrote:
>> 2017-11-14 8:24 GMT+00:00 Tobias Urdin :
>>> Trying to trace this, tempest calls the POST /servers//action
>>> API endpoint for the nova compute api.
>>>
>>> https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82
>>>
>>> Nova then takes the requests and tries to do this floating ip association
>>> using the neutron server api.
>>>
>>> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz
>>>
>>> 2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips
>>> [req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647
>>> 2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate
>>> floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance
>>> d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to
>>> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
>>> timed out: ConnectTimeout: Request to
>>> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
>>> timed out
>>>
>>> Checking that timestamp in the neutron-server logs:
>>> http://paste.openstack.org/show/626240/
>>>
>>> We can see that during this timestamp right before at 23:12:30.377 and then
>>> after 23:12:35.611 everything seems to be doing fine.
>>> So there is some connectivity issues to the neutron API from where the Nova
>>> API is running causing a timeout.
>>>
>>> Now some more questions would be:
>>>
>>> * Why is the return code 400? Are we being fooled or is it actually a
>>> connection timeout.
>>> * Is the Neutron API stuck causing the failed connection? All talk are done
>>> over loopback so chance of a problem there is very low.
>>> * Any firewall catching this? Not likely since the agent processes requests
>>> right before and after.
>>>
>>> I can't find anything interesting in the overall other system logs that
>>> could explain that.
>>> Back to the logs!
>> I'm pretty certain that this is a deadlock between nova and neutron,
>> though I cannot put my finger on the exact spot yet. But looking at
>> the neutron log that you extracted you can see that neutron indeed
>> tries to give a successful answer to the fip request just after nova
>> has given up waiting for it (seems the timeout is 30s here):
>>
>> 2017-10-29 23:12:35.932 18958 INFO neutron.wsgi
>> [req-e737b7dd-ed9c-46a7-911b-eb77efe11aa8
>> 42526a28b1a14c629b83908b2d75c647 2493426e6a3c4253a60c0b7eb35cfe19 -
>> default default] 127.0.0.1 "PUT
>> /v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c HTTP/1.1"
>> status: 200  len: 746 time: 30.4427412
>>
>> Also, looking at
>> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/apache_config/10-nova_api_wsgi.conf.txt.gz
>> is seems that nova-api is started with two processes and one thread,
>> not sure if that means two processes with one thread each or only one
>> thread total, anyway nova-api might be getting stuck there.
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack 

Re: [openstack-dev] Upstream LTS Releases

[Doug Hellmann]

Excerpts from Bogdan Dobrelya's message of 2017-11-14 17:08:31 +0100:
> >> The concept, in general, is to create a new set of cores from these
> >> groups, and use 3rd party CI to validate patches. There are lots of
> >> details to be worked out yet, but our amazing UC (User Committee) will
> >> be begin working out the details.
> > 
> > What is the most worrying is the exact "take over" process. Does it mean 
> > that 
> > the teams will give away the +2 power to a different team? Or will our 
> > (small) 
> > stable teams still be responsible for landing changes? If so, will they 
> > have to 
> > learn how to debug 3rd party CI jobs?
> > 
> > Generally, I'm scared of both overloading the teams and losing the control 
> > over 
> > quality at the same time :) Probably the final proposal will clarify it..
> 
> The quality of backported fixes is expected to be a direct (and only?) 
> interest of those new teams of new cores, coming from users and 
> operators and vendors. The more parties to establish their 3rd party 

We have an unhealthy focus on "3rd party" jobs in this discussion. We
should not assume that they are needed or will be present. They may be,
but we shouldn't build policy around the assumption that they will. Why
would we have third-party jobs on an old branch that we don't have on
master, for instance?

> checking jobs, the better proposed changes communicated, which directly 
> affects the quality in the end. I also suppose, contributors from ops 
> world will likely be only struggling to see things getting fixed, and 
> not new features adopted by legacy deployments they're used to maintain. 
> So in theory, this works and as a mainstream developer and maintainer, 
> you need no to fear of losing control over LTS code :)
> 
> Another question is how to not block all on each over, and not push 
> contributors away when things are getting awry, jobs failing and merging 
> is blocked for a long time, or there is no consensus reached in a code 
> review. I propose the LTS policy to enforce CI jobs be non-voting, as a 
> first step on that way, and giving every LTS team member a core rights 
> maybe? Not sure if that works though.

I'm not sure what change you're proposing for CI jobs and their voting
status. Do you mean we should make the jobs non-voting as soon as the
branch passes out of the stable support period?

Regarding the review team, anyone on the review team for a branch
that goes out of stable support will need to have +2 rights in that
branch. Otherwise there's no point in saying that they're maintaining
the branch.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Davanum Srinivas]

Bogdan, Team,

So i got this etherpad started. Please add policy ideas at the top and
volunteer for the team too.,

https://etherpad.openstack.org/p/LTS-proposal

Thanks,
Dims

On Wed, Nov 15, 2017 at 3:08 AM, Bogdan Dobrelya  wrote:
>>> The concept, in general, is to create a new set of cores from these
>>> groups, and use 3rd party CI to validate patches. There are lots of
>>> details to be worked out yet, but our amazing UC (User Committee) will
>>> be begin working out the details.
>>
>>
>> What is the most worrying is the exact "take over" process. Does it mean
>> that the teams will give away the +2 power to a different team? Or will our
>> (small) stable teams still be responsible for landing changes? If so, will
>> they have to learn how to debug 3rd party CI jobs?
>>
>> Generally, I'm scared of both overloading the teams and losing the control
>> over quality at the same time :) Probably the final proposal will clarify
>> it..
>
>
> The quality of backported fixes is expected to be a direct (and only?)
> interest of those new teams of new cores, coming from users and operators
> and vendors. The more parties to establish their 3rd party checking jobs,
> the better proposed changes communicated, which directly affects the quality
> in the end. I also suppose, contributors from ops world will likely be only
> struggling to see things getting fixed, and not new features adopted by
> legacy deployments they're used to maintain. So in theory, this works and as
> a mainstream developer and maintainer, you need no to fear of losing control
> over LTS code :)
>
> Another question is how to not block all on each over, and not push
> contributors away when things are getting awry, jobs failing and merging is
> blocked for a long time, or there is no consensus reached in a code review.
> I propose the LTS policy to enforce CI jobs be non-voting, as a first step
> on that way, and giving every LTS team member a core rights maybe? Not sure
> if that works though.
>
> --
> Best regards,
> Bogdan Dobrelya,
> Irc #bogdando
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Doug Hellmann]

Excerpts from Davanum Srinivas (dims)'s message of 2017-11-15 03:04:38 +1100:
> Flavio, Saverio,
> 
> Agree, that review may be a good example of what could be done. More info 
> below.
> 
> Saverio said - "with the old Stable Release thinking this patch would
> not be accepted on old stable branches."
> My response - "Those branches are still under stable policy. That has
> not changed just because of an email thread or a discussion in Forum"

Which stable policy does that patch violate?  It's clearly a bug
because the wrong information is being logged.  I suppose it goes
against the string freeze rule? Except that we've stopped translating
log messages so maybe we don't need to worry about that in this case,
since it isn't an exception.

> Saverio said - "Let's see if this gets accepted back to stable/newton"
> My response - "The branch the review is against is still under stable
> policy. So things that will or will not be backported will not change"
> 
> Saverio said - "Please note that a developers/operators that make the
> effort of fixing this in master, should do also all the cherry-pickes
> back. We dont have any automatic procudure for this."
> My response - "How the cherry-picks are done for stable branches will
> not change. This is a stable branch, so there is no automatic
> procedure for backporting"

Do we really not have any automation for proposing a cherry-pick
back through multiple branches? I know it can be done with a bunch
of clicks in gerrit, but it seems like it should be possible to
automate. Can someone write that script and put it into the
release-tools repo?

> I really want folks to help with stable first, learn how things are
> done and then propose changes to stable branch policies and help
> execute them
> 
> If folks want to chase LTS, then we are outlining a procedure/process
> that is a first step towards LTS eventually.
> 
> Thanks,
> Dims
> 
> On Wed, Nov 15, 2017 at 2:46 AM, Flavio Percoco  wrote:
> > On 14/11/17 22:33 +1100, Davanum Srinivas wrote:
> >>
> >> Saverio,
> >>
> >> This is still under the stable team reviews... NOT LTS.
> >>
> >> Your contacts for the Nova Stable team is ...
> >> https://review.openstack.org/#/admin/groups/540,members
> >>
> >> Let's please be clear, we need new people to help with LTS plans.
> >> Current teams can't scale, they should not have to and it's totally
> >> unfair to expect them to do so.
> >
> >
> > I think you may have misunderstood Saverio's email. IIUC, what he was trying
> > to
> > do was provide an example in favor of the LTS branches as discussed in
> > Sydney,
> > rather than requesting for reviews or suggesting the stable team should do
> > LTS.
> >
> > Flavio
> >
> >> On Tue, Nov 14, 2017 at 8:02 PM, Saverio Proto  wrote:
> >>>
> >>> Hello,
> >>>
> >>> here an example of a trivial patch that is important for people that
> >>> do operations, and they have to troubleshoot stuff.
> >>>
> >>> with the old Stable Release thinking this patch would not be accepted
> >>> on old stable branches.
> >>>
> >>> Let's see if this gets accepted back to stable/newton
> >>>
> >>>
> >>> https://review.openstack.org/#/q/If525313c63c4553abe8bea6f2bfaf75431ed18ea
> >>>
> >>> Please note that a developers/operators that make the effort of fixing
> >>> this in master, should do also all the cherry-pickes back. We dont
> >>> have any automatic procudure for this.
> >>>
> >>> thank you
> >>>
> >>> Saverio
> >
> >
> > --
> > @flaper87
> > Flavio Percoco
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic] Summary of ironic sessions from Sydney

[Julia Kreger]

Greetings ironic folk!

Like many other teams, we had very few ironic contributors make it to
Sydney. As such, I wanted to go ahead and write up a summary that
covers takeaways, questions, and obvious action items for the
community that were raised by operators and users present during the
sessions, so that we can use this as feedback to help guide our next
steps and feature planning.

Much of this is from my memory combined with notes on the various
etherpads. I would like to explicitly thank NobodyCam for reading
through this in advance to see if I was missing anything at a high
level since he was present in the vast majority of these sessions, and
dtantsur for sanity checking the content and asking for some
elaboration in some cases.

-Julia



Ironic Project Update
=

Questions largely arose around use of boot from volume, including some
scenarios we anticipated that would arise, as well as new scenarios
that we had not considered.

Boot nodes booting from the same volume
---

From a technical standpoint, when BFV is used with iPXE chain loading,
the chain loader reads the boot loader and related data from the
cinder (or, realistically, any iSCSI volume). This means that a
skilled operator is able to craft a specific volume that may just turn
around and unpack a ramdisk and operate the machine solely from RAM,
or that utilize an NFS root.

This sort of technical configuration would not be something an average
user would make use of, but there are actual use cases that some large
scale deployment operators would make use of and that would provide
them value.

Additionally, this topic and the desire for this capability also come
up during the “Building a bare metal cloud is hard” talk Q

Action Item: Check the data model to see if we prohibit, and consider
removing the prohibition against using the same volume across nodes,
if any.

Cinder-less BFV support
---

Some operators are curious about booting Ironic managed nodes without
cinder in a BFV context. This is something we anticipated and built
the API and CLI interfaces to support this. Realistically, we just
need to offer the ability for the data to be read and utilized.

Action Item: Review code and ensure that we have a some sort of no-op
driver or method that allows cinder-less node booting. For existing
drivers, it would be the shipment of the information to the BMC or the
write-out of iPXE templates as necessary.

Boot IPA from a cinder volume
-

With larger IPA images, specifically in cases where the image contains
a substantial amount of utilized or tooling to perform cleaning,
providing a mechanism to point the deployment Ramdisk to a cinder
volume would allow more efficient IO access.

Action Item: Discuss further - Specifically how we could support as we
would need to better understand how some of the operators might use
such functionality.

Dedicated Storage Fabric support


A question of dedicated storage fabric/networking support arose. For
users of FibreChannel, they generally have a dedicated storage fabric
by the very nature of separate infrasturcture. However, with ethernet
networking where iSCSI software initiators are used, or even possibly
converged network adapters, things get a little more complex.

Presently, with the iPXE boot from volume support, we boot using the
same interface details for the neutron VIF that the node is attached
with.

Moving forward, with BFV, the concept was to support the use of
explicitly defined interfaces as storage interfaces, which could be
denoted as "volume connectors" in ironic by type defined as "mac". In
theory, we begin to get functionality along these lines once
https://review.openstack.org/#/c/468353/ lands, as the user could
define two networks, and the storage network should then fall to the
explicit volume connector interface(s). The operator would just need
to ensure that the settings being used on that storage network are
such that the node can boot and reach the iSCSI endpoint, and that a
default route is not provided.

The question then may be, does Ironic do this quietly for the user
requesting the VM or not, and how do we document the use such that
operators can conceptualize it. How do we make this work at a larger
scale? How could this fit or not fit into multi-site deployments?

In order to determine if there is more to do, we need to have more
discussions with operators.

Action items:

* Determine overall needs for operators, since this is implementation
architecture centric.
* Plan forward path form there, if it makes sense.

Note: This may require more information to be stored or leveraged in
terms of structural or location based data.

Migration questions from classic drivers to Hardware types
--

One explicit question from the operator community was if we intended
to perform a 

[openstack-dev] [ironic] inclusion of openstack/networking-generic-switch project under OpenStack baremetal program

[Pavlo Shchelokovskyy]

Hi all,

as this topic it was recently brought up in ironic IRC meeting, I'd like to
start a discussion on the subject.

A quick recap - networking-generic-switch project (n-g-s) was born out of
necessity to do two things:

-  test the "network isolation for baremetal nodes" (a.k.a. multi-tenancy)
feature of ironic on upstream gates in virtualized environment and
- do the same on cheap/simple/dumb hardware switches that are not supported
by other various openstack/networking-* projects.

Back when it was created AFAIR neutron governance (neutron stadium) was
under some changes, so in the end n-g-s ended up not belonging to any
official program.

Over time n-g-s grew to be an essential part of ironic gate testing
(similar to virtualbmc). What's more, we have reports that it is already
being used in production.

Currently the core reviewers team of n-g-s consists of 4 people (2 of those
are currently core reviewers in ironic too), all of them are working for
the same company (Mirantis). This poses some risk as companies and people
come and go, plus since some voting ironic gate jobs depend on n-g-s
stability, a more diverse group of core reviewers from baremetal program
might be beneficial to be able to land patches in case of severe gate
troubles.

Currently I know of 3 proposed ways to change the current situation:

1) include n-g-s under ironic (OpenStack Baremetal program) governance,
effectively including ironic-core team to the core team of  n-g-s similar
to how ironic-inspector currently governed (keeping an extended sub-core
team). Reasoning for addition is the same as with virtualbmc/sushy
projects, with the debatable difference that the actual scope of n-g-s is
quite bigger and apparently includes production use-cases;

2) keep things as they are now, just add ironic-stable-maint team to the
n-g-s core reviewers to decrease low diversity risks;

3) merge the code from n-g-s into networking-baremetal project which is
already under ironic governance.

As a core in n-g-s myself I'm happy with either 1) or 2), but not really
fond of 3) as it kind of stretches the networking-baremetal scope too much
IMHO.

Eager to hear your comments and proposals.

Cheers,
-- 
Dr. Pavlo Shchelokovskyy
Senior Software Engineer
Mirantis Inc
www.mirantis.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][zuul][infra][trove] Tag of openstack/trove-dashboard failed

[Doug Hellmann]

Excerpts from zuul's message of 2017-11-14 16:01:20 +:
> Unable to freeze job graph: Unable to modify final job  publish-openstack-releasenotes branches: None source: 
> openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
> required_projects={'openstack/horizon':  0x7ff8f8b0a710>} with variant  None source: 
> openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#291>
> 

Is there some way to detect this type of error before we approve a
release?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Upstream LTS Releases

[Bogdan Dobrelya]

The concept, in general, is to create a new set of cores from these
groups, and use 3rd party CI to validate patches. There are lots of
details to be worked out yet, but our amazing UC (User Committee) will
be begin working out the details.


What is the most worrying is the exact "take over" process. Does it mean that 
the teams will give away the +2 power to a different team? Or will our (small) 
stable teams still be responsible for landing changes? If so, will they have to 
learn how to debug 3rd party CI jobs?


Generally, I'm scared of both overloading the teams and losing the control over 
quality at the same time :) Probably the final proposal will clarify it..


The quality of backported fixes is expected to be a direct (and only?) 
interest of those new teams of new cores, coming from users and 
operators and vendors. The more parties to establish their 3rd party 
checking jobs, the better proposed changes communicated, which directly 
affects the quality in the end. I also suppose, contributors from ops 
world will likely be only struggling to see things getting fixed, and 
not new features adopted by legacy deployments they're used to maintain. 
So in theory, this works and as a mainstream developer and maintainer, 
you need no to fear of losing control over LTS code :)


Another question is how to not block all on each over, and not push 
contributors away when things are getting awry, jobs failing and merging 
is blocked for a long time, or there is no consensus reached in a code 
review. I propose the LTS policy to enforce CI jobs be non-voting, as a 
first step on that way, and giving every LTS team member a core rights 
maybe? Not sure if that works though.


--
Best regards,
Bogdan Dobrelya,
Irc #bogdando

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [api] APIs schema consumption discussion

[Doug Hellmann]

Excerpts from Gilles Dubreuil's message of 2017-11-14 10:15:02 +1100:
> Hi,
> 
> Follow-up conversation from our last "API SIG feedback and discussion 
> session" at Sydney Summit [1], about APIs schema consumption.
> 
> Let's summarize the current situation.
> 
> Each OpenStack project has an "API-source" folder containing RST files 
> describing its API structure ([2] for example). Those files are in turn 
> consumed by the Sphinx library to generate each project's API reference 
> manual which are then available in the API guide documentation [3]. Such 
> effort has made the APIs harmoniously consistent across all OpenStack 
> projects and has also created a "de-facto" API schema.
> 
> While the RST files are used by the documentation, they are not readily 
> consumable by SDKs. Of course the APIs schema can be extracted by web 
> crawling the Reference guides, which in turn can be used by any 
> language. Such approach works [4] and help the Misty project [5] (Ruby 
> SDK) started with more languages to exploit the same approach.
> 
> Therefore to allow better automation, the next step would be to have the 
> APIs schema stored directly into each project's repo so the SDKs could 
> consume them straight from the source. This is why we've started 
> discussing how to have the schema either extracted from the RST files or 
> alternatively having the API described directly into its own file. The 
> latter would provide a different work flow: "Yaml -> RST -> Reference 
> doco" instead of "RST -> Reference doco -> Yaml".
> 
> So the question at this stage is: "Which of the work flow to choose from?".
> 
> To clarify the needs, it's important to note that we found out that none 
> of the SDKs project, besides OSC (OpenStack Client, thanks to Dean), 
> have full time working teams to maintain each SDK, which besides the 
> natural structural complexity inherent to the cloud context, makes the 
> task of keeping a SDK up to date very difficult. Especially as projects 
> moves forward. Automatically managing Openstack APIs is inevitable for 
> consumers. Another example/feedback was provided by the presenters of 
> "AT’s Strategy for Implementing a Next Generation OpenStack Cloud" 
> session during Sydney Keynotes, as they don't handle the Openstack API 
> manually!
> 
> APIs consumers and providers, any thoughts?
> 
> [1] 
> https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20442/api-sig-feedback-and-discussion-session
> [2] https://github.com/openstack/nova/tree/master/api-guide/source
> [3] https://developer.openstack.org/api-guide/quick-start/index.html
> [4] https://github.com/flystack/openstack-APIs
> [5] https://github.com/flystack/misty
> 
> Regards,
> Gilles

Please do not build something that looks like SOAP based on parsing RST
files. Surely we can at least work directly from JSONSchema inputs?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][neutron][infra] Tag of openstack/neutron-fwaas-dashboard failed

[Andreas Jaeger]

On 2017-11-14 17:03, Doug Hellmann wrote:
> Excerpts from Andreas Jaeger's message of 2017-11-14 09:31:48 +0100:
>> On 2017-11-13 22:09, Doug Hellmann wrote:
>>> Excerpts from zuul's message of 2017-11-13 20:37:18 +:
 Unable to freeze job graph: Unable to modify final job >>> publish-openstack-releasenotes branches: None source: 
 openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
 required_projects={'openstack/horizon': >>> 0x7ff848d06b70>} with variant >>> branches: None source: 
 openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#285>

>>>
>>> It looks like there is a configuration issue with
>>> neutron-fwaas-dashboard.
>>
>> Yes, we marked  publish-openstack-releasenotes as final - and then the
>> job added requirements to it.
>>
>> I see at least these two different fixes:
>> - remove the final: true from the job
>> - add neutron and horizon to the job like we done for release job. But
>> there are other projects that have even more requirements.
>>
>> Infra team, what's the best approach here?
>>
>> Andreas
> 
> No projects should even need to install *themselves* much less their
> other dependencies to build release notes. It should be possible to
> build release notes just with sphinx and reno (and their dependencies).

It should - but that's not how those projects seem to be set up ;(

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Davanum Srinivas]

Flavio, Saverio,

Agree, that review may be a good example of what could be done. More info below.

Saverio said - "with the old Stable Release thinking this patch would
not be accepted on old stable branches."
My response - "Those branches are still under stable policy. That has
not changed just because of an email thread or a discussion in Forum"

Saverio said - "Let's see if this gets accepted back to stable/newton"
My response - "The branch the review is against is still under stable
policy. So things that will or will not be backported will not change"

Saverio said - "Please note that a developers/operators that make the
effort of fixing this in master, should do also all the cherry-pickes
back. We dont have any automatic procudure for this."
My response - "How the cherry-picks are done for stable branches will
not change. This is a stable branch, so there is no automatic
procedure for backporting"

I really want folks to help with stable first, learn how things are
done and then propose changes to stable branch policies and help
execute them

If folks want to chase LTS, then we are outlining a procedure/process
that is a first step towards LTS eventually.

Thanks,
Dims

On Wed, Nov 15, 2017 at 2:46 AM, Flavio Percoco  wrote:
> On 14/11/17 22:33 +1100, Davanum Srinivas wrote:
>>
>> Saverio,
>>
>> This is still under the stable team reviews... NOT LTS.
>>
>> Your contacts for the Nova Stable team is ...
>> https://review.openstack.org/#/admin/groups/540,members
>>
>> Let's please be clear, we need new people to help with LTS plans.
>> Current teams can't scale, they should not have to and it's totally
>> unfair to expect them to do so.
>
>
> I think you may have misunderstood Saverio's email. IIUC, what he was trying
> to
> do was provide an example in favor of the LTS branches as discussed in
> Sydney,
> rather than requesting for reviews or suggesting the stable team should do
> LTS.
>
> Flavio
>
>> On Tue, Nov 14, 2017 at 8:02 PM, Saverio Proto  wrote:
>>>
>>> Hello,
>>>
>>> here an example of a trivial patch that is important for people that
>>> do operations, and they have to troubleshoot stuff.
>>>
>>> with the old Stable Release thinking this patch would not be accepted
>>> on old stable branches.
>>>
>>> Let's see if this gets accepted back to stable/newton
>>>
>>>
>>> https://review.openstack.org/#/q/If525313c63c4553abe8bea6f2bfaf75431ed18ea
>>>
>>> Please note that a developers/operators that make the effort of fixing
>>> this in master, should do also all the cherry-pickes back. We dont
>>> have any automatic procudure for this.
>>>
>>> thank you
>>>
>>> Saverio
>
>
> --
> @flaper87
> Flavio Percoco



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][neutron][infra] Tag of openstack/neutron-fwaas-dashboard failed

[Doug Hellmann]

Excerpts from Andreas Jaeger's message of 2017-11-14 09:31:48 +0100:
> On 2017-11-13 22:09, Doug Hellmann wrote:
> > Excerpts from zuul's message of 2017-11-13 20:37:18 +:
> >> Unable to freeze job graph: Unable to modify final job  >> publish-openstack-releasenotes branches: None source: 
> >> openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
> >> required_projects={'openstack/horizon':  >> 0x7ff848d06b70>} with variant  >> branches: None source: 
> >> openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#285>
> >>
> > 
> > It looks like there is a configuration issue with
> > neutron-fwaas-dashboard.
> 
> Yes, we marked  publish-openstack-releasenotes as final - and then the
> job added requirements to it.
> 
> I see at least these two different fixes:
> - remove the final: true from the job
> - add neutron and horizon to the job like we done for release job. But
> there are other projects that have even more requirements.
> 
> Infra team, what's the best approach here?
> 
> Andreas

No projects should even need to install *themselves* much less their
other dependencies to build release notes. It should be possible to
build release notes just with sphinx and reno (and their dependencies).

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] multiple agents with segment access

[Legacy, Allain]

Is it a known limitation that the ML2 plugin and associated drivers do not 
properly handle cases where there are multiple agents running on the same host? 
  That is, two or more agents that reside on the same host and both report 
device/interface mappings  for physical networks?One such setup is a set of 
nodes that are both running an L2 agent and a NIC SRIOV agent.   

We have found two problems specific to this configuration and are wondering if 
these are known limitations, or simply not supported.   

For example, in a configuration where a host has L2, DHCP, and SRIOV agents 
running on it, then:

1)   The DHCP scheduler can schedule a network to be hosted by that DHCP agent 
even if the physical network is only supported by the SRIOV agent and not by 
the L2 agent.  This can end up isolating the DHCP server as it will not have 
access to the underlying physical segment.   This is happening because 
"filter_hosts_with_network_access" in the Ml2Plugin will include the host 
because the SRIOV mech driver reports that it has access to that segment.   

2)   The Segment Plugin "segmenthostmappings" table gets overwritten with 
tuples for whichever agent reports in last.  For example, if the L2 agent has 
physical network mappings for 2 different physical networks (physnet0, 
physnet1), but the SRIOV NIC agent only has 1 physical network reported 
(physnet2), the segmenthostmappings table will have data for only physnet2 if 
the SRIOV NIC agent reported last, or data for only both physnet0 and physnet1 
if the L2 agent reported last.  Data for physical networks belonging to the 
other agent will be erroneously removed as stale entries.

If it has been discussed before is there a plan to address this type of 
configuration?  

Regards.
Allain



Allain Legacy, Software Developer, Wind River
direct 613.270.2279  fax 613.492.7870 skype allain.legacy
350 Terry Fox Drive, Suite 200, Ottawa, Ontario, K2K 2W5

 



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] LTS pragmatic example

[Flavio Percoco]

On 14/11/17 22:33 +1100, Davanum Srinivas wrote:

Saverio,

This is still under the stable team reviews... NOT LTS.

Your contacts for the Nova Stable team is ...
https://review.openstack.org/#/admin/groups/540,members

Let's please be clear, we need new people to help with LTS plans.
Current teams can't scale, they should not have to and it's totally
unfair to expect them to do so.


I think you may have misunderstood Saverio's email. IIUC, what he was trying to
do was provide an example in favor of the LTS branches as discussed in Sydney,
rather than requesting for reviews or suggesting the stable team should do LTS.

Flavio


On Tue, Nov 14, 2017 at 8:02 PM, Saverio Proto  wrote:

Hello,

here an example of a trivial patch that is important for people that
do operations, and they have to troubleshoot stuff.

with the old Stable Release thinking this patch would not be accepted
on old stable branches.

Let's see if this gets accepted back to stable/newton

https://review.openstack.org/#/q/If525313c63c4553abe8bea6f2bfaf75431ed18ea

Please note that a developers/operators that make the effort of fixing
this in master, should do also all the cherry-pickes back. We dont
have any automatic procudure for this.

thank you

Saverio


--
@flaper87
Flavio Percoco


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cinder] Weekly Meeting Back-on 11/15/2017 ...

[Jay S Bryant]

Team,

Just a reminder that we will now return to our regularly scheduled 
weekly Cinder meetings starting 11/15/2017.


I hope you all enjoyed a little break and thank you to whoever updated 
the etherpad for 11/8.  :-)


An important reminder to everyone in the US (and possibly elsewhere) 
that the meeting is now an hour earlier due to Daylight Savings Time 
kicking in.  So, please check your calendars and make sure to join at 
the right time.


Please update the etherpad with your topics! [1]

Thanks!

Jay

[1] https://etherpad.openstack.org/p/cinder-queens-meeting-agendas



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic] automatic migration from classic drivers to hardware types?

[Dmitry Tantsur]

Hi folks!

This was raised several times, now I want to bring it to the wider audience. 
We're planning [1] to deprecate classic drivers in Queens and remove them in 
Rocky. It was pointed at the Forum that we'd better provide an automatic migration.


I'd like to hear your opinion on the options:

(1) Migration as part of 'ironic-dbsync upgrade'

Pros:
* nothing new to do for the operators

Cons:
* upgrade will fail completely, if for some nodes the matching hardware types 
and/or interfaces are not enabled in ironic.conf


(2) A separate script for migration

Pros:
* can be done in advance (even while still on Pike)
* a failure won't fail the whole upgrade
* will rely on drivers enabled in actually running conductors, not on 
ironic.conf

Cons:
* a new upgrade action before Rocky
* won't be available in packaging
* unclear how to update nodes that are in some process (e.g. cleaning), will 
probably have to be run several times


(3) Migration as part of 'ironic-dbsync online_data_migration'

Pros:
* nothing new to do for the operators, similar to (1)
* probably a more natural place to do this than (1)
* can rely on drivers enabled in actually running conductors, not on ironic.conf

Cons:
* data migration will fail, if for some nodes the matching hardware types and/or 
interfaces are not enabled in ironic.conf


(4) Do nothing, let operators handle the migration.


The most reasonable option for me seems (3), then (4). What do you think?

Dmitry

[1] 
http://specs.openstack.org/openstack/ironic-specs/specs/approved/classic-drivers-future.html


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][ci][infra] PTI jobs for Puppet modules

[Jeremy Stanley]

On 2017-11-13 14:26:38 -0500 (-0500), Mohammed Naser wrote:
[...]
> Given that Puppet OpenStack has a large number of modules, in addition
> to Infra's puppet modules, it would be good to have a project-template
> specific for build and release of Puppet modules.
[...]

Coming up with something consistent across puppet-openstack and
openstack-infra modules could be tough. Right now the Infra projects
do integration testing with Beaker, still run some jobs on Ubuntu
Trusty (because we still have some production servers running on it)
and aren't released on Puppet Forge. We also support different
versions of Puppet than the Puppet OpenStack deliverables last time
I checked.
-- 
Jeremy Stanley


signature.asc
Description: Digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Need help with Openstack Swift Installation and configuration with S3

[Shalabh Aggarwal]

Hi,

We just started with Openstack Swift which we intend to use as a
replacement for an API which was written to work with AWS S3.
We know that Swift is S3 compatible and our API should work out of the box
with it.

We have not been able to get Swift running with S3 plugins.
I was wondering if there is a go to documentation which covers the
installation of Openstack Swift along with configuration with S3.

Any help would be highly appreciated.

Thanks!

-- 
Regards,

Shalabh Aggarwal
t: +91 9990960159
w: www.shalabhaggarwal.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] LTS pragmatic example

[Davanum Srinivas]

Saverio,

This is still under the stable team reviews... NOT LTS.

Your contacts for the Nova Stable team is ...
https://review.openstack.org/#/admin/groups/540,members

Let's please be clear, we need new people to help with LTS plans.
Current teams can't scale, they should not have to and it's totally
unfair to expect them to do so.

Thanks,
Dims

On Tue, Nov 14, 2017 at 8:02 PM, Saverio Proto  wrote:
> Hello,
>
> here an example of a trivial patch that is important for people that
> do operations, and they have to troubleshoot stuff.
>
> with the old Stable Release thinking this patch would not be accepted
> on old stable branches.
>
> Let's see if this gets accepted back to stable/newton
>
> https://review.openstack.org/#/q/If525313c63c4553abe8bea6f2bfaf75431ed18ea
>
> Please note that a developers/operators that make the effort of fixing
> this in master, should do also all the cherry-pickes back. We dont
> have any automatic procudure for this.
>
> thank you
>
> Saverio
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

Yea, I've been scavenging the logs for any kind of indicator on what
might have gone wrong but I can't see anything
related to a deadlock even though I'm very certain that's the issue but
don't know what's causing it.

Perhaps we will need to manually recreate this issue and then
troubleshoot it manually.
The apache2 mod_wsgi config should be OK according to the docs [1].

Best regards

[1]
http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html

On 11/14/2017 11:12 AM, Jens Harbott wrote:
> 2017-11-14 8:24 GMT+00:00 Tobias Urdin :
>> Trying to trace this, tempest calls the POST /servers//action
>> API endpoint for the nova compute api.
>>
>> https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82
>>
>> Nova then takes the requests and tries to do this floating ip association
>> using the neutron server api.
>>
>> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz
>>
>> 2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips
>> [req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647
>> 2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate
>> floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance
>> d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to
>> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
>> timed out: ConnectTimeout: Request to
>> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
>> timed out
>>
>> Checking that timestamp in the neutron-server logs:
>> http://paste.openstack.org/show/626240/
>>
>> We can see that during this timestamp right before at 23:12:30.377 and then
>> after 23:12:35.611 everything seems to be doing fine.
>> So there is some connectivity issues to the neutron API from where the Nova
>> API is running causing a timeout.
>>
>> Now some more questions would be:
>>
>> * Why is the return code 400? Are we being fooled or is it actually a
>> connection timeout.
>> * Is the Neutron API stuck causing the failed connection? All talk are done
>> over loopback so chance of a problem there is very low.
>> * Any firewall catching this? Not likely since the agent processes requests
>> right before and after.
>>
>> I can't find anything interesting in the overall other system logs that
>> could explain that.
>> Back to the logs!
> I'm pretty certain that this is a deadlock between nova and neutron,
> though I cannot put my finger on the exact spot yet. But looking at
> the neutron log that you extracted you can see that neutron indeed
> tries to give a successful answer to the fip request just after nova
> has given up waiting for it (seems the timeout is 30s here):
>
> 2017-10-29 23:12:35.932 18958 INFO neutron.wsgi
> [req-e737b7dd-ed9c-46a7-911b-eb77efe11aa8
> 42526a28b1a14c629b83908b2d75c647 2493426e6a3c4253a60c0b7eb35cfe19 -
> default default] 127.0.0.1 "PUT
> /v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c HTTP/1.1"
> status: 200  len: 746 time: 30.4427412
>
> Also, looking at
> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/apache_config/10-nova_api_wsgi.conf.txt.gz
> is seems that nova-api is started with two processes and one thread,
> not sure if that means two processes with one thread each or only one
> thread total, anyway nova-api might be getting stuck there.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Next steps for pre-deployment workflows (e.g derive parameters)

[Saravanan KR]

As discussed in IRC, I have collated all the important discussions to
the etherpad (gdoc was not publicly shareable).

https://etherpad.openstack.org/p/tripleo-derive-parameters-v2

Lets continue discussion on the etherpad to finalize.

Regards,
Saravanan KR

On Thu, Nov 9, 2017 at 11:05 AM, Saravanan KR  wrote:
> On Thu, Nov 9, 2017 at 2:57 AM, Jiri Tomasek  wrote:
>>
>>
>> On Wed, Nov 8, 2017 at 6:09 AM, Steven Hardy  wrote:
>>>
>>> Hi all,
>>>
>>> Today I had a productive hallway discussion with jtomasek and
>>> stevebaker re $subject, so I wanted to elaborate here for the benefit
>>> of those folks not present.  Hopefully we can get feedback on the
>>> ideas and see if it makes sense to continue and work on some patches:
>>>
>>> The problem under discussion is how do we run pre-deployment workflows
>>> (such as those integrated recently to calculate derived parameters,
>>> and in future perhaps also those which download container images etc),
>>> and in particular how do we make these discoverable via the UI
>>> (including any input parameters).
>>>
>>> The idea we came up with has two parts:
>>>
>>> 1. Add a new optional section to roles_data for services that require
>>> pre-deploy workflows
>>>
>>> E.g something like this:
>>>
>>>  pre_deploy_workflows:
>>> - derive_params:
>>>   workflow_name:
>>> tripleo.derive_params_formulas.v1.dpdk_derive_params
>>>   inputs:
>>>   ...
>>>
>>> This would allow us to associate a specific mistral workflow with a
>>> given service template, and also work around the fact that currently
>>> mistral inputs don't have any schema (only key/value input) as we
>>> could encode the required type and any constraints in the inputs block
>>> (clearly this could be removed in future should typed parameters
>>> become available in mistral).
>>>
>>> 2. Add a new workflow that calculates the enabled services and returns
>>> all pre_deploy_workflows
>>>
>>> This would take all enabled environments, then use heat to validate
>>> the configuration and return the merged resource registry (which will
>>> require https://review.openstack.org/#/c/509760/), then we would
>>> iterate over all enabled services in the registry and extract a given
>>> roles_data key (e.g pre_deploy_workflows)
>>>
>>> The result of the workflow would be a list of all pre_deploy_workflows
>>> for all enabled services, which the UI could then use to run the
>>> workflows as part of the pre-deploy process.
>>
>>
>> As I think about this more, we may find out that matching a service to
>> workflow is not enough as workflow may require several services (together
>> defining a feature) So maybe doing it in separate file would help. E.g.
>>
>> pre-deploy-workflows.yaml
>> - name: my.workflow
>>   services: a, b, c, d
>>
>> Maybe there is a better way, maybe this is not even needed. I am not sure.
>> What do you think?
>
> Currently, HCI derive parameters workflow is invoked if a role has
> both NovaCompute and CephOSD services enabled.
>
>>
>>
>> What I really like about this proposal is that it provides a standard way to
>> configure deployment features and provides clear means to add additional
>> such configurations.
>>
>> The resulting deployment configuration steps in GUI would look following:
>>
>> 1/ Hardware (reg. nodes, introspect etc)
>>
>> 2/ High level deployment configuration (basically selecting additional
>> environment files)
>>
>> 3/ Roles management (Roles selection, roles -> nodes assignment, roles
>> configuration - setting roles_data properties)
>>
>> 4/ Network configuration -  network configuration wizard: (I'll describe
>> this in separate email)
>>
>> 5/ Deployment Features configuration (This proposal) - a list of features to
>> configure, the list is nicely generated from information provided in
>> previous steps, user has all the information to configure those features at
>> hand and can go through these step by step.
>
> Agreed on the UI workflow.
>
> For DPDK and SR-IOV, there are common host specific parameters to be
> derived. It has been added as a separate host-specific parameters
> workflow. And both DPDK and SR-IOV workflow execution should follow
> host-specific workflow.
> In case of DPDK and HCI in same role, it is expected that DPDK
> workflow is executed before HCI. And the service configuration should
> provide this order to UI.
> I am not able to realize how this information will be provided and
> processed in UI with user. Do you have a UI wire frame for this
> workflow?
>
>>
>> 6/ Advanced deployment config - a view providing a way to review
>> Environment/Roles/Services parameters, search and tweak them if needed.
>>
>> 7/ Deploy.
>>
>> I believe these steps should cover anything we should need to do for
>> deployment configuration.
>>
>> -- Jirka
>>
>>
>>>
>>>
>>> If this makes sense I can go ahead and push some patches so we can
>>> iterate on the 

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Jens Harbott]

2017-11-14 8:24 GMT+00:00 Tobias Urdin :
> Trying to trace this, tempest calls the POST /servers//action
> API endpoint for the nova compute api.
>
> https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82
>
> Nova then takes the requests and tries to do this floating ip association
> using the neutron server api.
>
> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz
>
> 2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips
> [req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647
> 2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate
> floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance
> d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to
> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
> timed out: ConnectTimeout: Request to
> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
> timed out
>
> Checking that timestamp in the neutron-server logs:
> http://paste.openstack.org/show/626240/
>
> We can see that during this timestamp right before at 23:12:30.377 and then
> after 23:12:35.611 everything seems to be doing fine.
> So there is some connectivity issues to the neutron API from where the Nova
> API is running causing a timeout.
>
> Now some more questions would be:
>
> * Why is the return code 400? Are we being fooled or is it actually a
> connection timeout.
> * Is the Neutron API stuck causing the failed connection? All talk are done
> over loopback so chance of a problem there is very low.
> * Any firewall catching this? Not likely since the agent processes requests
> right before and after.
>
> I can't find anything interesting in the overall other system logs that
> could explain that.
> Back to the logs!

I'm pretty certain that this is a deadlock between nova and neutron,
though I cannot put my finger on the exact spot yet. But looking at
the neutron log that you extracted you can see that neutron indeed
tries to give a successful answer to the fip request just after nova
has given up waiting for it (seems the timeout is 30s here):

2017-10-29 23:12:35.932 18958 INFO neutron.wsgi
[req-e737b7dd-ed9c-46a7-911b-eb77efe11aa8
42526a28b1a14c629b83908b2d75c647 2493426e6a3c4253a60c0b7eb35cfe19 -
default default] 127.0.0.1 "PUT
/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c HTTP/1.1"
status: 200  len: 746 time: 30.4427412

Also, looking at
http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/apache_config/10-nova_api_wsgi.conf.txt.gz
is seems that nova-api is started with two processes and one thread,
not sure if that means two processes with one thread each or only one
thread total, anyway nova-api might be getting stuck there.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron][classifier] CCF Meeting

[Shaughnessy, David]

Hi everyone.
Reminder that the Common Classification Framework meeting is at 14:00 UTC.
The Agenda can be found here: 
https://wiki.openstack.org/wiki/Neutron/CommonClassificationFramework#All_Meetings.27_Agendas
Regards.
David.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

Am I actually hallucinating or is it the nova API that cannot communicate with 
Keystone?
Cannot substantiate this with any logs for keystone.

2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
[req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647 
2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate 
floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance 
d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out: ConnectTimeout: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
Traceback (most recent call last):
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File 
"/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/floating_ips.py", 
line 267, in _add_floating_ip
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
fixed_address=fixed_address)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 83, in 
wrapper
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
res = f(self, context, *args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 
1759, in associate_floating_ip
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
client.update_floatingip(fip['id'], {'floatingip': param})
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 99, 
in wrapper
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
ret = obj(*args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 935, 
in update_floatingip
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
return self.put(self.floatingip_path % (floatingip), body=body)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 99, 
in wrapper
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
ret = obj(*args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 361, 
in put
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
headers=headers, params=params)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 99, 
in wrapper
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
ret = obj(*args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 329, 
in retry_request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
headers=headers, params=params)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 99, 
in wrapper
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
ret = obj(*args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 280, 
in do_request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
resp, replybody = self.httpclient.do_request(action, method, body=body)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 342, in 
do_request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
return self.request(url, method, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 330, in 
request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
resp = super(SessionClient, self).request(*args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 192, in 
request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
return self.session.request(url, method, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR 

Re: [openstack-dev] [Neutron] Infiniband & Melanox OFED 4.2 eIPoIB support

[Moshe Levi]

This more question to mellanox then to neutron community. 
I suggest that you will send me private mail regarding this issue and we can 
take it with the relevant people in Mellanox.
My email is mosh...@mellanox.com.  

> -Original Message-
> From: Massimo Benini [mailto:ben...@cscs.ch]
> Sent: Tuesday, November 14, 2017 11:39 AM
> To: openstack-dev@lists.openstack.org
> Subject: [openstack-dev] [Neutron] Infiniband & Melanox OFED 4.2 eIPoIB
> support
> 
> Dear Neutron developers,
> 
> we are building up an OpenStack (Pike) cluster based on CentOS 7.4 and
> Mellanox Ofed 4.2.
> As far as we understand Neutron rely on eIPoIB in order to manage
> Infiniband networks but unfortunately Mellanox dropped this feature for the
> new OFED versions.
> Here below the Mellanox support answer about that:
> 
> "Please note that as of Mellanox OFED 4.0 eIPoIB is no longer supported in
> Mellanox OFED.If you wish to use it we recommend using the 3.4.X Drivers
> which still supported this feature."
> 
> Obviously we can not use the 3.X version of OFED because is not supported
> by RHEL 7.4 and also because we don't want to stick with an old software
> version.
> Do you have any plan to overcome this issue?
> 
> Thank you, regards
> 
> Massimo Benini
> 
> 
> __
> 
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.
> openstack.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fopenstack-
> dev=02%7C01%7Cmoshele%40mellanox.com%7C6de1b0a14e4540045a
> 4108d52b434d35%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636
> 462490423586383=oEdi3vFeR4CEBom7rZNHajd%2BmujXhuHrT49yTx7
> Q5Bk%3D=0
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Infiniband & Melanox OFED 4.2 eIPoIB support

[Massimo Benini]

Dear Neutron developers,

we are building up an OpenStack (Pike) cluster based on CentOS 7.4 and 
Mellanox Ofed 4.2.
As far as we understand Neutron rely on eIPoIB in order to manage 
Infiniband networks but unfortunately Mellanox dropped this feature for 
the new OFED versions.

Here below the Mellanox support answer about that:

"Please note that as of Mellanox OFED 4.0 eIPoIB is no longer supported 
in Mellanox OFED.If you wish to use it we recommend using the 3.4.X 
Drivers which still supported this feature."


Obviously we can not use the 3.X version of OFED because is not 
supported by RHEL 7.4 and also because we don't want to stick with an 
old software version.

Do you have any plan to overcome this issue?

Thank you, regards

Massimo Benini


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] LTS pragmatic example

[Saverio Proto]

Hello,

here an example of a trivial patch that is important for people that
do operations, and they have to troubleshoot stuff.

with the old Stable Release thinking this patch would not be accepted
on old stable branches.

Let's see if this gets accepted back to stable/newton

https://review.openstack.org/#/q/If525313c63c4553abe8bea6f2bfaf75431ed18ea

Please note that a developers/operators that make the effort of fixing
this in master, should do also all the cherry-pickes back. We dont
have any automatic procudure for this.

thank you

Saverio

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [openstack-meteos] Meteos Project Weekly Meeting

[Digambar Patil]

Hi All,

I am starting meteos project meeting from coming thursday.
Meteos meeting will be held every thursday 4:30 am UTC.

Please go through the meeting plan:
https://wiki.openstack.org/wiki/Meetings/meteos


Thanks,
Digambar
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] support graphviz in document publishing

[Yujun Zhang (ZTE)]

On Tue, Nov 14, 2017 at 5:05 AM Antoine Musso  wrote:

> Hello,
>
> I had some success with http://blockdiag.com/ which, IIRC, is pure
> python. It supports various kind of graphs (block, sequences, network ...).
>
> The few I did were for Zuul:
> https://docs.openstack.org/infra/zuul/user/gating.html
>
> In sphinx that looks like:
>
> .. blockdiag::
>
>   blockdiag foo {
> node_width = 40;
> span_width = 40;
> A <- B <- C <- D <- E;
>   }
>
>
> sphinxcontrib-blockdiag
> https://pypi.python.org/pypi/sphinxcontrib-blockdiag
>
> Might be worth looking at if you want something simpler than graphviz
> and to not depend on it.
>

Really nice diagram and syntax. I will surely consider it for some workflow
in the specification.

Thank you for the recommendation, Antoine

-- 
Yujun Zhang
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Release-job-failures][neutron][infra] Tag of openstack/neutron-fwaas-dashboard failed

[Andreas Jaeger]

On 2017-11-13 22:09, Doug Hellmann wrote:
> Excerpts from zuul's message of 2017-11-13 20:37:18 +:
>> Unable to freeze job graph: Unable to modify final job > publish-openstack-releasenotes branches: None source: 
>> openstack-infra/project-config/zuul.d/jobs.yaml@master#26> attribute 
>> required_projects={'openstack/horizon': > 0x7ff848d06b70>} with variant > None source: 
>> openstack-infra/openstack-zuul-jobs/zuul.d/project-templates.yaml@master#285>
>>
> 
> It looks like there is a configuration issue with
> neutron-fwaas-dashboard.

Yes, we marked  publish-openstack-releasenotes as final - and then the
job added requirements to it.

I see at least these two different fixes:
- remove the final: true from the job
- add neutron and horizon to the job like we done for release job. But
there are other projects that have even more requirements.

Infra team, what's the best approach here?

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] support graphviz in document publishing

[Yujun Zhang (ZTE)]

On Mon, Nov 13, 2017 at 5:41 PM Andreas Jaeger  wrote:

> You can easily include them for any repository - add the python
> requirements to test-requirments and add the binary requirements to
> bindep.txt - as explained in
> https://docs.openstack.org/infra/manual/drivers.html#package-requirements


Thanks Andreas.

Besides the dependency you mentioned, I just realized that we need also to
add the extension to sphinx conf[1] to make it effective.

https://review.openstack.org/#/c/518196/8/doc/source/conf.py
-- 
Yujun Zhang
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [octavia] amphora fails to send request to members

[Yipei Niu]

Hi, Michael,

Please ignore my last two mails. Sorry about that.

The results of the two commands are as follows.

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip route show table all
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
default via 10.0.1.1 dev eth1  table 1 onlink
default via 10.0.1.10 dev eth1
10.0.1.0/24 dev eth1  proto kernel  scope link  src 10.0.1.8
broadcast 10.0.1.0 dev eth1  table local  proto kernel  scope link  src
10.0.1.8
local 10.0.1.4 dev eth1  table local  proto kernel  scope host  src
10.0.1.8
local 10.0.1.8 dev eth1  table local  proto kernel  scope host  src
10.0.1.8
broadcast 10.0.1.255 dev eth1  table local  proto kernel  scope link  src
10.0.1.8
fe80::/64 dev eth1  proto kernel  metric 256  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295
<0429%20496%207295>  error -101 pref medium
local fe80::f816:3eff:febe:5ad5 dev lo  table local  proto none  metric 0
pref medium
ff00::/8 dev eth1  table local  metric 256  pref medium
unreachable default dev lo  table unspec  proto kernel  metric 4294967295
<0429%20496%207295>  error -101 pref medium

ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
amphora-haproxy ip rule show
sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
0: from all lookup local
100: from 10.0.1.4 lookup 1
32766: from all lookup main
32767: from all lookup default

I think I know the source. When haproxy receives packets sent by curl, it
responds with taking VIP as source ip for 3-way handshake. Before adding “
100: from 10.0.1.9 lookup 1”, the datagrams are routed based on main table.
After adding "100: from 10.0.1.9 lookup 1", the haproxy tries to find the
gateway based on "default via 10.0.1.1 dev eth1  table 1 onlink". However,
if there is no router, the gateway ip is missing, making haproxy fails to
build tcp connection.

Best regards,
Yipei


On Tue, Nov 14, 2017 at 11:04 AM, Yipei Niu  wrote:

> Hi, Michael,
>
> Sorry about the typo in the last mail. Please just ignore the last mail.
>
> In the environment where octavia and tricircle are installed together, I
> created a router and attached subnet1 to it. Then I bind the mac address of
> 10.0.1.10 (real gateway) to ip of 10.0.1.1 in the amphora arp cache,
> manually making amphora knows the mac address of 10.0.1.1 (actually it is
> the mac of 10.0.1.10, since 10.0.1.1 does not exist), it works.
>
> I also run the commands in this environment.
>
> ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
> amphora-haproxy ip route show table all
> sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
> default via 10.0.1.1 dev eth1  table 1 onlink
> default via 10.0.1.10 dev eth1
> 10.0.1.0/24 dev eth1  proto kernel  scope link  src 10.0.1.8
> broadcast 10.0.1.0 dev eth1  table local  proto kernel  scope link  src
> 10.0.1.8
> local 10.0.1.4 dev eth1  table local  proto kernel  scope host  src
> 10.0.1.8
> local 10.0.1.8 dev eth1  table local  proto kernel  scope host  src
> 10.0.1.8
> broadcast 10.0.1.255 dev eth1  table local  proto kernel  scope link  src
> 10.0.1.8
> fe80::/64 dev eth1  proto kernel  metric 256  pref medium
> unreachable default dev lo  table unspec  proto kernel  metric 4294967295
> <0429%20496%207295>  error -101 pref medium
> local fe80::f816:3eff:febe:5ad5 dev lo  table local  proto none  metric 0
> pref medium
> ff00::/8 dev eth1  table local  metric 256  pref medium
> unreachable default dev lo  table unspec  proto kernel  metric 4294967295
> <0429%20496%207295>  error -101 pref medium
> ubuntu@amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec
> amphora-haproxy ip rule show
> sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4ee-05b695e2b71f
> 0: from all lookup local
> 100: from 10.0.1.4 lookup 1
> 32766: from all lookup main
> 32767: from all lookup default
>
>
> To make the situation clear, I run the commands in the environment
> installed octavia alone. Please note that in this environment, there is no
> router. The results are as follows.
>
> stack@stack-VirtualBox:~$ neutron lbaas-loadbalancer-list
> neutron CLI is deprecated and will be removed in the future. Use openstack
> CLI instead.
> +--+--+-
> -+-+-+--+
> | id   | name | tenant_id
>   | vip_address | provisioning_status | provider |
> +--+--+-
> -+-+-+--+
> | d087d3b4-afbe-4af6-8b31-5e86fc97da1b | lb1  |
> e59bb8f3bf9342aba02f9ba5804ed2fb | 10.0.1.9| ACTIVE  |
> octavia  |
> +--+--+-
> -+-+-+--+
>
> 

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

Trying to trace this, tempest calls the POST /servers//action API 
endpoint for the nova compute api.

https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82

Nova then takes the requests and tries to do this floating ip association using 
the neutron server api.

http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz

2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
[req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647 
2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate 
floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance 
d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out: ConnectTimeout: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out

Checking that timestamp in the neutron-server logs:
http://paste.openstack.org/show/626240/

We can see that during this timestamp right before at 23:12:30.377 and then 
after 23:12:35.611 everything seems to be doing fine.
So there is some connectivity issues to the neutron API from where the Nova API 
is running causing a timeout.

Now some more questions would be:

* Why is the return code 400? Are we being fooled or is it actually a 
connection timeout.
* Is the Neutron API stuck causing the failed connection? All talk are done 
over loopback so chance of a problem there is very low.
* Any firewall catching this? Not likely since the agent processes requests 
right before and after.

I can't find anything interesting in the overall other system logs that could 
explain that.
Back to the logs!

Best regards
Tobias


On 11/14/2017 08:35 AM, Tobias Urdin wrote:

Hello,

Same here, I will continue looking at this aswell.

Would be great if we could get some input from a neutron dev with good insight 
into the project.


Can we backtrace the timed out message from where it's thrown/returned.

Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out', u'code': 400}

I'm interested why we would get 400 code back, the floating ip operations 
should be async right so this would be the response from the API layer with 
information from the database that should

return more information.


Best regards

On 11/14/2017 07:41 AM, Arnaud MORIN wrote:
Hey,
Do you know if the bug appears on a specific Ubuntu / openstack version?
As far as I remember it was not related to the puppet branch?  I mean the bug 
is existing on master but also on newton puppet branches, right?

We are using Ubuntu in my company so we would love to see that continue ;)
I'll try to take a look again.

Cheers.

Le 14 nov. 2017 00:00, "Mohammed Naser" 
> a écrit :
Hi,

Hope that everyone had safe travels and enjoyed their time at Sydney
(and those who weren't there enjoyed a bit of quiet time!).  I'm just
sending this email if anyone had a chance to look more into this (or
perhaps we can get some help if there are any Canonical folks on the
list?)

I would be really sad if we had to drop Ubuntu/Debian support because
we cannot test it.  I think there are a lot of users out there using
it!  I'd be more than happy to provide any assistance/information in
troubleshooting this.

Thank you,
Mohammed

On Thu, Nov 2, 2017 at 1:10 PM, Mohammed Naser 
> wrote:
> On Thu, Nov 2, 2017 at 1:02 PM, Tobias Urdin 
> > wrote:
>> I've been staring at this for almost an hour now going through all the logs
>> and I can't really pin a point from
>>
>> where that error message is generated. Cannot find any references for the
>> timed out message that the API returns or the unable to associate part.
>>
>>
>> What I'm currently staring at is why would the instance fixed ip 172.24.5.17
>> be references as a network:router_gateway port in the OVS agent logs.
>>
>> 2017-10-29 23:19:27.591 11856 INFO
>> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>> [req-7274c6f7-18ef-420d-ad5a-9d0fe4eb35c6 - - - - -] Port
>> 053a625c-4227-41fb-9a26-45eda7bd2055 updated. Details: {'profile': {},
>> 'network_qos_policy_id': None, 'qos_policy_id': None,
>> 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':
>> 'f9647756-41ad-4ec5-af49-daefe410815e', 'segmentation_id': None,
>> 'fixed_ips': [{'subnet_id': 'a31c7115-1f3e-4220-8bdb-981b6df2e18c',
>> 'ip_address': '172.24.5.17'}], 'device_owner': u'network:router_gateway',
>> 'physical_network': u'external', 'mac_address': 'fa:16:3e:3b:ec:c3',
>> 'device': u'053a625c-4227-41fb-9a26-45eda7bd2055', 'port_security_enabled':
>> False, 'port_id': '053a625c-4227-41fb-9a26-45eda7bd2055', 'network_type':
>> u'flat',