Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Matthew Van Dijk]

Not much to add here - I commented in 
review.openstack.org/378063
that they could set the default value of the flag to revert behaviour.
- Matt

On Oct 11, 2016, at 10:21 PM, Armando M. 
mailto:arma...@gmail.com>> wrote:



On 11 October 2016 at 18:20, Sean M. Collins 
mailto:s...@coreitpro.com>> wrote:
Armando M. wrote:
> At this point I feel that changing the pool range is even less justified.
> If I had seen bug [4], I would have been against its fix, because you're
> absolutely right as the change being not backward compatible.

https://review.openstack.org/#/c/356026 was written by someone on the Trove 
team to
help them with their CI jobs IIRC.

CC'ing Matthew since he has more context. I went into the Trove channel
and asked them about reverting 356026. It doesn't seem like an option at
this point.

http://eavesdrop.openstack.org/irclogs/%23openstack-trove/%23openstack-trove.2016-09-30.log.html#t2016-09-30T17:53:08

A revert with no follow up is clearly not a viable option most of the times, 
and we clearly dug ourselves too deep now with [1]. Rather than making the use 
of subnet pools conditional as done in [1], IMO we should have made [2] 
conditional to preserve the existing provisioning behavior and let Trove 
override.

[1] Ic89ceca76afda67da5545111972c3348011f294f
[2] https://review.openstack.org/#/c/356026/




--
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Armando M.]

On 11 October 2016 at 18:20, Sean M. Collins  wrote:

> Armando M. wrote:
> > At this point I feel that changing the pool range is even less justified.
> > If I had seen bug [4], I would have been against its fix, because you're
> > absolutely right as the change being not backward compatible.
>
> https://review.openstack.org/#/c/356026 was written by someone on the
> Trove team to
> help them with their CI jobs IIRC.
>
> CC'ing Matthew since he has more context. I went into the Trove channel
> and asked them about reverting 356026. It doesn't seem like an option at
> this point.
>
> http://eavesdrop.openstack.org/irclogs/%23openstack-trove/%
> 23openstack-trove.2016-09-30.log.html#t2016-09-30T17:53:08


A revert with no follow up is clearly not a viable option most of the
times, and we clearly dug ourselves too deep now with [1]. Rather than
making the use of subnet pools conditional as done in [1], IMO we should
have made [2] conditional to preserve the existing provisioning behavior
and let Trove override.

[1] Ic89ceca76afda67da5545111972c3348011f294f
[2] https://review.openstack.org/#/c/356026/


>
>
> --
> Sean M. Collins
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Sean M. Collins]

Armando M. wrote:
> At this point I feel that changing the pool range is even less justified.
> If I had seen bug [4], I would have been against its fix, because you're
> absolutely right as the change being not backward compatible.

https://review.openstack.org/#/c/356026 was written by someone on the Trove 
team to
help them with their CI jobs IIRC.

CC'ing Matthew since he has more context. I went into the Trove channel
and asked them about reverting 356026. It doesn't seem like an option at
this point.

http://eavesdrop.openstack.org/irclogs/%23openstack-trove/%23openstack-trove.2016-09-30.log.html#t2016-09-30T17:53:08


-- 
Sean M. Collins

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Clark Boylan]

On Tue, Oct 11, 2016, at 05:58 PM, Carl Baldwin wrote:
> On Tue, Oct 11, 2016 at 6:26 PM, Clark Boylan 
> wrote:
> 
> > On Tue, Oct 11, 2016, at 05:14 PM, Carl Baldwin wrote:
> > > I found this [1] added here [2]. Since the provider is using 10.0.0.0/8
> > > on
> > > the public interface, and the subnet pool is also 10.0.0.0/8 the call to
> > > ip
> > > route replace eliminates the route through the public interface.
> >
> > It is also an issue if you have multiple subnets within 10/8 and rely on
> > your default route to route to them (though I don't think we currently
> > have any of these in the gate currently since rackspace sets more
> > specific routes for the subnets used for private networking there). This
> > would be the case if using a neutron setup with tenant networking and
> > floating IPs like we had with the old hpcloud.
> >
> 
> Any time we fiddle with the host's routing table with address ranges that
> we make up, we run the risk of clashing with something "real". What do we
> do? I guess we can try our best to pick ranges that shouldn't clash and
> try
> to minimize how much of the space we use up. Then, for those that get a
> clash, we provide the option to override. That's kind of what we've been
> doing, right? We could try other tricks to pick more obscure IP ranges
> that
> shouldn't be used anywhere (like 100.64.0.0/10 or the ranges reserved for
> documentation, which we do for ipv6) but with so many people competing
> for
> such a small space, I don't know if we can ever pick anything that won't
> ever clash. It just doesn't feel like there is a good answer.

Right my whole argument behind reusing FIXED_RANGE is its the var that
we have set for every devstack test job to avoid such conflicts. Its
value is currently conflict free as far as I know and has been set
intentionally to avoid the various conflicts we have run into over time.
This doesn't mean we will never run into new conflicts but it gives us a
single place to very clearly signal "this is useable". Maybe we need to
add a meta var that FIXED_RANGE and SUBNET POOL ranges are derived from.
Essentially set a value that is a large subnet that isn't conflicting
with anything and have the other ranges split it evenly. Then maintain
backward compat by using any hard set sub ranges that are supplied.

> 
> Could we instead modify devstack to use a different network namespace (as
> in ip nets) to represent the "external" network? I think the real problem
> is that we're trying to use the host's real network as the "external
> network" for our testing. We are combining any combination of real world
> subnets with made up hard-coded default address ranges. If we separated
> the
> two, then there would be no chance for conflict. I'm just thinking out
> loud
> here. There could be all kinds of difficulty with this idea.

Yup that is indeed the problem. Another related issue that makes this
extra turtly is we are running neutron within clouds running neutron. As
a result chance for conflicts is probably higher than if you were just
running on a random host on a random subnet (since we are probably more
likely to leak defaults between neutrons). It would definitely be neat
to separate these things more though. We have been slowly separating the
layer 2 devices (was sort of forced on us initially by OVH's use of /32s
on eth0 and nova nets inability to properly handle that). Probably worth
further exploring that for layer 3 as well though as you say this could
be difficult.

Clark


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Carl Baldwin]

On Tue, Oct 11, 2016 at 6:26 PM, Clark Boylan  wrote:

> On Tue, Oct 11, 2016, at 05:14 PM, Carl Baldwin wrote:
> > I found this [1] added here [2]. Since the provider is using 10.0.0.0/8
> > on
> > the public interface, and the subnet pool is also 10.0.0.0/8 the call to
> > ip
> > route replace eliminates the route through the public interface.
>
> It is also an issue if you have multiple subnets within 10/8 and rely on
> your default route to route to them (though I don't think we currently
> have any of these in the gate currently since rackspace sets more
> specific routes for the subnets used for private networking there). This
> would be the case if using a neutron setup with tenant networking and
> floating IPs like we had with the old hpcloud.
>

Any time we fiddle with the host's routing table with address ranges that
we make up, we run the risk of clashing with something "real". What do we
do? I guess we can try our best to pick ranges that shouldn't clash and try
to minimize how much of the space we use up. Then, for those that get a
clash, we provide the option to override. That's kind of what we've been
doing, right? We could try other tricks to pick more obscure IP ranges that
shouldn't be used anywhere (like 100.64.0.0/10 or the ranges reserved for
documentation, which we do for ipv6) but with so many people competing for
such a small space, I don't know if we can ever pick anything that won't
ever clash. It just doesn't feel like there is a good answer.

Could we instead modify devstack to use a different network namespace (as
in ip nets) to represent the "external" network? I think the real problem
is that we're trying to use the host's real network as the "external
network" for our testing. We are combining any combination of real world
subnets with made up hard-coded default address ranges. If we separated the
two, then there would be no chance for conflict. I'm just thinking out loud
here. There could be all kinds of difficulty with this idea.

Carl
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Armando M.]

On 11 October 2016 at 17:05, Clark Boylan  wrote:

> On Tue, Oct 11, 2016, at 05:01 PM, Armando M. wrote:
> > On 11 October 2016 at 16:54, Clark Boylan  wrote:
> >
> > > On Tue, Oct 11, 2016, at 04:51 PM, Armando M. wrote:
> > > > On 11 October 2016 at 16:43, Clark Boylan 
> wrote:
> > > >
> > > > > On Tue, Oct 11, 2016, at 04:32 PM, Armando M. wrote:
> > > > > > On 11 October 2016 at 14:09, Clark Boylan 
> > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > Currently multinode testing + neutron is broken in clouds that
> use
> > > > > > > portions of 10.0.0.0/8 for their networking due to route
> conflicts
> > > > > with
> > > > > > > devstack + neutron deployments. https://bugs.launchpad.net/bug
> > > > > s/1629133
> > > > > > > is tracking the issue for us. I would like to see this get
> resolved
> > > > > > > properly before we do further work on multinode testing as it
> is
> > > > > > > difficult to review and determine what failures are legit vs
> which
> > > > > > > failures are related to this bug and whether or not a specific
> > > > > multinode
> > > > > > > test has decided to workaround the issue.
> > > > > > >
> > > > > > > The change to use subnet pools in devstack is a non backward
> > > compatible
> > > > > > > change for devstack currently and it doesn't appear to have
> been
> > > > > > > documented in devstack at all. Would be great if we can
> finally fix
> > > > > this
> > > > > > > and get testing back to working and however we fix it ensure
> that
> > > > > > > devstack has the appropriate documentation.
> > > > > > >
> > > > > >
> > > > > > What is holding [1] back? Merging that would resolve the issue,
> then
> > > we
> > > > > > can
> > > > > > drill down into why subnetpools interfere with the underlying
> > > networking
> > > > > > setup. I have asked Carl to look into broken build [2]
> > > > > >
> > > > > > [1] https://review.openstack.org/#/c/379543/
> > > > > > [2]
> > > > > > http://logs.openstack.org/78/381278/2/check/gate-tempest-dsv
> > > > > m-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz
> > > > >
> > > > > Yours is one of the two -1's on the change :) I think that devstack
> > > core
> > > > > is probably holding back due to the two -1s there. If we are ok
> with
> > > > > iterating on making it better rather than all in one shot maybe
> that
> > > > > change is good for now and we can update the reviews?
> > > > >
> > > >
> > > > Well, that means the ball is in the contributor's court, who is
> supposed
> > > > to
> > > > address reviewers' concerns :)
> > > >
> > > The comments on the change with -1's are opposed to doing what the
> > > change does. I don't know how I can possibly address them.
> > >
> >
> > Then say so on the review and I am happy to rephrase to make sure I get
> > my
> > message across correctly. If you let the review rot how can you expect it
> > to make progress? That's like Openstack 101
>
> It does say so clearly in the commit itself:
>
> "It turns out that many people have already chosen fixed ranges that
> work for their environments. They have done so to avoid conflicts with
> existing networking ranges and routing. The change to use 10.0.0.0/8 and
> apply routes for this network to neutron interfaces prevents anyone from
> using networks within that range for anything else.
>
> For example imagine you have a cloud provider that provides private IPv4
> addresses allocated out of ranges in 10.0.0.0/8 and they do all public
> networking over ipv6. This change to devstack prevents you from using
> those private networks properly after starting neutron with devstack.
> However, in situations like this FIXED_RANGE should already represent a
> safe range to use so just use it."
>
> The comments both refuse to allow FIXED_RANGE as the default and its the
> only reason I pushed that patch. I personally believe this is the right
> fix, if neutron/devstack disagree they should feel welcome to propose
> alterantives, but I am not going to not use FIXED_RANGE when it was the
> whole point of the change in the first place.
>
>
Okay at least I am glad we got this clarified :)

This discussion gave me the opportunity to drill down further,  and looking
at this more closely, the issue seems to stem from [1], which fiddles with
the routing table and it caused other troubles [2]. The original use of
subnetpools as introduced by [3] should not have side effects whatsoever.
At this point I feel that changing the pool range is even less justified.
If I had seen bug [4], I would have been against its fix, because you're
absolutely right as the change being not backward compatible.

[1] https://review.openstack.org/#/c/356026
[2] https://bugs.launchpad.net/devstack/+bug/1628267
[3] https://review.openstack.org/#/c/282559/
[4] https://bugs.launchpad.net/devstack/+bug/1613717


> Clark
>
> __
> OpenStack Development Mailing List (not for usage questi

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Clark Boylan]

On Tue, Oct 11, 2016, at 05:14 PM, Carl Baldwin wrote:
> I found this [1] added here [2]. Since the provider is using 10.0.0.0/8
> on
> the public interface, and the subnet pool is also 10.0.0.0/8 the call to
> ip
> route replace eliminates the route through the public interface.

It is also an issue if you have multiple subnets within 10/8 and rely on
your default route to route to them (though I don't think we currently
have any of these in the gate currently since rackspace sets more
specific routes for the subnets used for private networking there). This
would be the case if using a neutron setup with tenant networking and
floating IPs like we had with the old hpcloud.

> 
> Carl
> 
> [1] https://github.com/openstack-dev/devstack/blob/
> 6c55227595228bc37b91a1dbc665ec704cbf4c56/lib/neutron_
> plugins/services/l3#L376
> [2] https://review.openstack.org/#/c/356026/4/lib/neutron_
> plugins/services/l3
> 
> On Tue, Oct 11, 2016 at 5:32 PM, Armando M.  wrote:
> 
> >
> >
> > On 11 October 2016 at 14:09, Clark Boylan  wrote:
> >
> >> Hello everyone,
> >>
> >> Currently multinode testing + neutron is broken in clouds that use
> >> portions of 10.0.0.0/8 for their networking due to route conflicts with
> >> devstack + neutron deployments. https://bugs.launchpad.net/bugs/1629133
> >> is tracking the issue for us. I would like to see this get resolved
> >> properly before we do further work on multinode testing as it is
> >> difficult to review and determine what failures are legit vs which
> >> failures are related to this bug and whether or not a specific multinode
> >> test has decided to workaround the issue.
> >>
> >> The change to use subnet pools in devstack is a non backward compatible
> >> change for devstack currently and it doesn't appear to have been
> >> documented in devstack at all. Would be great if we can finally fix this
> >> and get testing back to working and however we fix it ensure that
> >> devstack has the appropriate documentation.
> >>
> >
> > What is holding [1] back? Merging that would resolve the issue, then we
> > can drill down into why subnetpools interfere with the underlying
> > networking setup. I have asked Carl to look into broken build [2]
> >
> > [1] https://review.openstack.org/#/c/379543/
> > [2] http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-
> > multinode-full-ubuntu-xenial/7f82862/console.html.gz
> >
> >
> >> Thank you,
> >> Clark


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Carl Baldwin]

I found this [1] added here [2]. Since the provider is using 10.0.0.0/8 on
the public interface, and the subnet pool is also 10.0.0.0/8 the call to ip
route replace eliminates the route through the public interface.

Carl

[1] https://github.com/openstack-dev/devstack/blob/
6c55227595228bc37b91a1dbc665ec704cbf4c56/lib/neutron_
plugins/services/l3#L376
[2] https://review.openstack.org/#/c/356026/4/lib/neutron_
plugins/services/l3

On Tue, Oct 11, 2016 at 5:32 PM, Armando M.  wrote:

>
>
> On 11 October 2016 at 14:09, Clark Boylan  wrote:
>
>> Hello everyone,
>>
>> Currently multinode testing + neutron is broken in clouds that use
>> portions of 10.0.0.0/8 for their networking due to route conflicts with
>> devstack + neutron deployments. https://bugs.launchpad.net/bugs/1629133
>> is tracking the issue for us. I would like to see this get resolved
>> properly before we do further work on multinode testing as it is
>> difficult to review and determine what failures are legit vs which
>> failures are related to this bug and whether or not a specific multinode
>> test has decided to workaround the issue.
>>
>> The change to use subnet pools in devstack is a non backward compatible
>> change for devstack currently and it doesn't appear to have been
>> documented in devstack at all. Would be great if we can finally fix this
>> and get testing back to working and however we fix it ensure that
>> devstack has the appropriate documentation.
>>
>
> What is holding [1] back? Merging that would resolve the issue, then we
> can drill down into why subnetpools interfere with the underlying
> networking setup. I have asked Carl to look into broken build [2]
>
> [1] https://review.openstack.org/#/c/379543/
> [2] http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-
> multinode-full-ubuntu-xenial/7f82862/console.html.gz
>
>
>> Thank you,
>> Clark
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Clark Boylan]

On Tue, Oct 11, 2016, at 05:01 PM, Armando M. wrote:
> On 11 October 2016 at 16:54, Clark Boylan  wrote:
> 
> > On Tue, Oct 11, 2016, at 04:51 PM, Armando M. wrote:
> > > On 11 October 2016 at 16:43, Clark Boylan  wrote:
> > >
> > > > On Tue, Oct 11, 2016, at 04:32 PM, Armando M. wrote:
> > > > > On 11 October 2016 at 14:09, Clark Boylan 
> > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Currently multinode testing + neutron is broken in clouds that use
> > > > > > portions of 10.0.0.0/8 for their networking due to route conflicts
> > > > with
> > > > > > devstack + neutron deployments. https://bugs.launchpad.net/bug
> > > > s/1629133
> > > > > > is tracking the issue for us. I would like to see this get resolved
> > > > > > properly before we do further work on multinode testing as it is
> > > > > > difficult to review and determine what failures are legit vs which
> > > > > > failures are related to this bug and whether or not a specific
> > > > multinode
> > > > > > test has decided to workaround the issue.
> > > > > >
> > > > > > The change to use subnet pools in devstack is a non backward
> > compatible
> > > > > > change for devstack currently and it doesn't appear to have been
> > > > > > documented in devstack at all. Would be great if we can finally fix
> > > > this
> > > > > > and get testing back to working and however we fix it ensure that
> > > > > > devstack has the appropriate documentation.
> > > > > >
> > > > >
> > > > > What is holding [1] back? Merging that would resolve the issue, then
> > we
> > > > > can
> > > > > drill down into why subnetpools interfere with the underlying
> > networking
> > > > > setup. I have asked Carl to look into broken build [2]
> > > > >
> > > > > [1] https://review.openstack.org/#/c/379543/
> > > > > [2]
> > > > > http://logs.openstack.org/78/381278/2/check/gate-tempest-dsv
> > > > m-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz
> > > >
> > > > Yours is one of the two -1's on the change :) I think that devstack
> > core
> > > > is probably holding back due to the two -1s there. If we are ok with
> > > > iterating on making it better rather than all in one shot maybe that
> > > > change is good for now and we can update the reviews?
> > > >
> > >
> > > Well, that means the ball is in the contributor's court, who is supposed
> > > to
> > > address reviewers' concerns :)
> > >
> > The comments on the change with -1's are opposed to doing what the
> > change does. I don't know how I can possibly address them.
> >
> 
> Then say so on the review and I am happy to rephrase to make sure I get
> my
> message across correctly. If you let the review rot how can you expect it
> to make progress? That's like Openstack 101

It does say so clearly in the commit itself:

"It turns out that many people have already chosen fixed ranges that
work for their environments. They have done so to avoid conflicts with
existing networking ranges and routing. The change to use 10.0.0.0/8 and
apply routes for this network to neutron interfaces prevents anyone from
using networks within that range for anything else.

For example imagine you have a cloud provider that provides private IPv4
addresses allocated out of ranges in 10.0.0.0/8 and they do all public
networking over ipv6. This change to devstack prevents you from using
those private networks properly after starting neutron with devstack.
However, in situations like this FIXED_RANGE should already represent a
safe range to use so just use it."

The comments both refuse to allow FIXED_RANGE as the default and its the
only reason I pushed that patch. I personally believe this is the right
fix, if neutron/devstack disagree they should feel welcome to propose
alterantives, but I am not going to not use FIXED_RANGE when it was the
whole point of the change in the first place.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Armando M.]

On 11 October 2016 at 16:54, Clark Boylan  wrote:

> On Tue, Oct 11, 2016, at 04:51 PM, Armando M. wrote:
> > On 11 October 2016 at 16:43, Clark Boylan  wrote:
> >
> > > On Tue, Oct 11, 2016, at 04:32 PM, Armando M. wrote:
> > > > On 11 October 2016 at 14:09, Clark Boylan 
> wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > Currently multinode testing + neutron is broken in clouds that use
> > > > > portions of 10.0.0.0/8 for their networking due to route conflicts
> > > with
> > > > > devstack + neutron deployments. https://bugs.launchpad.net/bug
> > > s/1629133
> > > > > is tracking the issue for us. I would like to see this get resolved
> > > > > properly before we do further work on multinode testing as it is
> > > > > difficult to review and determine what failures are legit vs which
> > > > > failures are related to this bug and whether or not a specific
> > > multinode
> > > > > test has decided to workaround the issue.
> > > > >
> > > > > The change to use subnet pools in devstack is a non backward
> compatible
> > > > > change for devstack currently and it doesn't appear to have been
> > > > > documented in devstack at all. Would be great if we can finally fix
> > > this
> > > > > and get testing back to working and however we fix it ensure that
> > > > > devstack has the appropriate documentation.
> > > > >
> > > >
> > > > What is holding [1] back? Merging that would resolve the issue, then
> we
> > > > can
> > > > drill down into why subnetpools interfere with the underlying
> networking
> > > > setup. I have asked Carl to look into broken build [2]
> > > >
> > > > [1] https://review.openstack.org/#/c/379543/
> > > > [2]
> > > > http://logs.openstack.org/78/381278/2/check/gate-tempest-dsv
> > > m-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz
> > >
> > > Yours is one of the two -1's on the change :) I think that devstack
> core
> > > is probably holding back due to the two -1s there. If we are ok with
> > > iterating on making it better rather than all in one shot maybe that
> > > change is good for now and we can update the reviews?
> > >
> >
> > Well, that means the ball is in the contributor's court, who is supposed
> > to
> > address reviewers' concerns :)
> >
> The comments on the change with -1's are opposed to doing what the
> change does. I don't know how I can possibly address them.
>

Then say so on the review and I am happy to rephrase to make sure I get my
message across correctly. If you let the review rot how can you expect it
to make progress? That's like Openstack 101


>
> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Clark Boylan]

On Tue, Oct 11, 2016, at 04:51 PM, Armando M. wrote:
> On 11 October 2016 at 16:43, Clark Boylan  wrote:
> 
> > On Tue, Oct 11, 2016, at 04:32 PM, Armando M. wrote:
> > > On 11 October 2016 at 14:09, Clark Boylan  wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > Currently multinode testing + neutron is broken in clouds that use
> > > > portions of 10.0.0.0/8 for their networking due to route conflicts
> > with
> > > > devstack + neutron deployments. https://bugs.launchpad.net/bug
> > s/1629133
> > > > is tracking the issue for us. I would like to see this get resolved
> > > > properly before we do further work on multinode testing as it is
> > > > difficult to review and determine what failures are legit vs which
> > > > failures are related to this bug and whether or not a specific
> > multinode
> > > > test has decided to workaround the issue.
> > > >
> > > > The change to use subnet pools in devstack is a non backward compatible
> > > > change for devstack currently and it doesn't appear to have been
> > > > documented in devstack at all. Would be great if we can finally fix
> > this
> > > > and get testing back to working and however we fix it ensure that
> > > > devstack has the appropriate documentation.
> > > >
> > >
> > > What is holding [1] back? Merging that would resolve the issue, then we
> > > can
> > > drill down into why subnetpools interfere with the underlying networking
> > > setup. I have asked Carl to look into broken build [2]
> > >
> > > [1] https://review.openstack.org/#/c/379543/
> > > [2]
> > > http://logs.openstack.org/78/381278/2/check/gate-tempest-dsv
> > m-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz
> >
> > Yours is one of the two -1's on the change :) I think that devstack core
> > is probably holding back due to the two -1s there. If we are ok with
> > iterating on making it better rather than all in one shot maybe that
> > change is good for now and we can update the reviews?
> >
> 
> Well, that means the ball is in the contributor's court, who is supposed
> to
> address reviewers' concerns :)
> 
The comments on the change with -1's are opposed to doing what the
change does. I don't know how I can possibly address them.

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Armando M.]

On 11 October 2016 at 16:43, Clark Boylan  wrote:

> On Tue, Oct 11, 2016, at 04:32 PM, Armando M. wrote:
> > On 11 October 2016 at 14:09, Clark Boylan  wrote:
> >
> > > Hello everyone,
> > >
> > > Currently multinode testing + neutron is broken in clouds that use
> > > portions of 10.0.0.0/8 for their networking due to route conflicts
> with
> > > devstack + neutron deployments. https://bugs.launchpad.net/bug
> s/1629133
> > > is tracking the issue for us. I would like to see this get resolved
> > > properly before we do further work on multinode testing as it is
> > > difficult to review and determine what failures are legit vs which
> > > failures are related to this bug and whether or not a specific
> multinode
> > > test has decided to workaround the issue.
> > >
> > > The change to use subnet pools in devstack is a non backward compatible
> > > change for devstack currently and it doesn't appear to have been
> > > documented in devstack at all. Would be great if we can finally fix
> this
> > > and get testing back to working and however we fix it ensure that
> > > devstack has the appropriate documentation.
> > >
> >
> > What is holding [1] back? Merging that would resolve the issue, then we
> > can
> > drill down into why subnetpools interfere with the underlying networking
> > setup. I have asked Carl to look into broken build [2]
> >
> > [1] https://review.openstack.org/#/c/379543/
> > [2]
> > http://logs.openstack.org/78/381278/2/check/gate-tempest-dsv
> m-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz
>
> Yours is one of the two -1's on the change :) I think that devstack core
> is probably holding back due to the two -1s there. If we are ok with
> iterating on making it better rather than all in one shot maybe that
> change is good for now and we can update the reviews?
>

Well, that means the ball is in the contributor's court, who is supposed to
address reviewers' concerns :)


> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Clark Boylan]

On Tue, Oct 11, 2016, at 04:32 PM, Armando M. wrote:
> On 11 October 2016 at 14:09, Clark Boylan  wrote:
> 
> > Hello everyone,
> >
> > Currently multinode testing + neutron is broken in clouds that use
> > portions of 10.0.0.0/8 for their networking due to route conflicts with
> > devstack + neutron deployments. https://bugs.launchpad.net/bugs/1629133
> > is tracking the issue for us. I would like to see this get resolved
> > properly before we do further work on multinode testing as it is
> > difficult to review and determine what failures are legit vs which
> > failures are related to this bug and whether or not a specific multinode
> > test has decided to workaround the issue.
> >
> > The change to use subnet pools in devstack is a non backward compatible
> > change for devstack currently and it doesn't appear to have been
> > documented in devstack at all. Would be great if we can finally fix this
> > and get testing back to working and however we fix it ensure that
> > devstack has the appropriate documentation.
> >
> 
> What is holding [1] back? Merging that would resolve the issue, then we
> can
> drill down into why subnetpools interfere with the underlying networking
> setup. I have asked Carl to look into broken build [2]
> 
> [1] https://review.openstack.org/#/c/379543/
> [2]
> http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz

Yours is one of the two -1's on the change :) I think that devstack core
is probably holding back due to the two -1s there. If we are ok with
iterating on making it better rather than all in one shot maybe that
change is good for now and we can update the reviews?

Clark

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Multinode testing with devstack and neutron broken

[Armando M.]

On 11 October 2016 at 14:09, Clark Boylan  wrote:

> Hello everyone,
>
> Currently multinode testing + neutron is broken in clouds that use
> portions of 10.0.0.0/8 for their networking due to route conflicts with
> devstack + neutron deployments. https://bugs.launchpad.net/bugs/1629133
> is tracking the issue for us. I would like to see this get resolved
> properly before we do further work on multinode testing as it is
> difficult to review and determine what failures are legit vs which
> failures are related to this bug and whether or not a specific multinode
> test has decided to workaround the issue.
>
> The change to use subnet pools in devstack is a non backward compatible
> change for devstack currently and it doesn't appear to have been
> documented in devstack at all. Would be great if we can finally fix this
> and get testing back to working and however we fix it ensure that
> devstack has the appropriate documentation.
>

What is holding [1] back? Merging that would resolve the issue, then we can
drill down into why subnetpools interfere with the underlying networking
setup. I have asked Carl to look into broken build [2]

[1] https://review.openstack.org/#/c/379543/
[2]
http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/7f82862/console.html.gz


> Thank you,
> Clark
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev