Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-09-22 Thread Anne Gentle
On Thu, Aug 28, 2014 at 12:22 PM, Richard Woo richardwoo2...@gmail.com
wrote:

 I have another question about incubator proposal, for CLI and GUI. Do we
 imply that the incubator feature will need to branch python-neutron client,
 Horizon, and or Nova ( if changes are needed)?




This is as good a place for the docs perspective as any on this thread.

I think it's great to increase your dev velocity, but in the proposal I
need to see a well-thought-out plan for when documentation should be on
docs.openstack.org for deployers and users and when documentation should be
in the API reference page at http://developer.openstack.org/api-ref.html. A
section addressing the timing and requirements would be sufficient.

The docs affected are:
CLI Reference http://docs.openstack.org/cli-reference/content/
Config Reference
http://docs.openstack.org/icehouse/config-reference/content/
User Guide http://docs.openstack.org/user-guide/content/
Admin User Guide http://docs.openstack.org/user-guide-admin/content/
Cloud Admin User Guide http://docs.openstack.org/admin-guide-cloud/content/
API Reference http://developer.openstack.org/api-ref.html

I won't argue that the Install Guide should be included as these items
aren't exactly happy path quite yet.

Also in the wiki page, I see a line saying  link to commits in private
trees is acceptable -- is it really? How would readers get to it?

Thanks,
Anne



 On Tue, Aug 26, 2014 at 7:09 PM, James E. Blair cor...@inaugust.com
 wrote:

 Hi,

 After reading https://wiki.openstack.org/wiki/Network/Incubator I have
 some thoughts about the proposed workflow.

 We have quite a bit of experience and some good tools around splitting
 code out of projects and into new projects.  But we don't generally do a
 lot of importing code into projects.  We've done this once, to my
 recollection, in a way that preserved history, and that was with the
 switch to keystone-lite.

 It wasn't easy; it's major git surgery and would require significant
 infra-team involvement any time we wanted to do it.

 However, reading the proposal, it occurred to me that it's pretty clear
 that we expect these tools to be able to operate outside of the Neutron
 project itself, to even be releasable on their own.  Why not just stick
 with that?  In other words, the goal of this process should be to create
 separate projects with their own development lifecycle that will
 continue indefinitely, rather than expecting the code itself to merge
 into the neutron repo.

 This has advantages in simplifying workflow and making it more
 consistent.  Plus it builds on known integration mechanisms like APIs
 and python project versions.

 But more importantly, it helps scale the neutron project itself.  I
 think that a focused neutron core upon which projects like these can
 build on in a reliable fashion would be ideal.

 -Jim

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-28 Thread Jay Pipes

On 08/27/2014 04:28 PM, Kevin Benton wrote:

What are you talking about? The only reply was from me clarifying that
one of the purposes of the incubator was for components of neutron that
are experimental but are intended to be merged.


Right. The special unicorns.

 In that case it might

not make sense to have a life cycle of their own in another repo
indefinitely.


The main reasons these experimental components don't make sense to 
live in their own repo indefinitely are:


a) Neutron's design doesn't make it easy or straightforward to 
build/layer other things on top of it, or:


b) The experimental piece of code intends to replace whole-hog a large 
chunk of Neutron's existing codebase, or:


c) The experimental piece of code relies so heavily on inconsistent, 
unversioned internal interface and plugin calls that it cannot be 
designed externally due to the fragility of those interfaces


Fixing a) is the solution to these problems. An incubator area where 
experimental components can live will just continue to mask the true 
problem domain, which is that Neutron's design is cumbersome to build on 
top of, and its cross-component interfaces need to be versioned, made 
consistent, and cleaned up to use versioned data structures instead of 
passing random nested dicts of randomly-prefixed string key/values.


Frankly, we're going through a similar problem in Nova right now. There 
is a group of folks who believe that separating the nova-scheduler code 
into the Gantt project will magically make placement decision code and 
solver components *easier* to work on (because the pace of coding can be 
increased if there wasn't that pesky nova-core review process). But this 
is not correct, IMO. Separating out the scheduler into its own project 
before internal interfaces and data structures are cleaned up and 
versioned will just lead to greater technical debt and an increase in 
frustration on the part of Nova developers and scheduler developers alike.


-jay


On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com
mailto:jaypi...@gmail.com wrote:

On 08/26/2014 07:09 PM, James E. Blair wrote:

Hi,

After reading
https://wiki.openstack.org/__wiki/Network/Incubator
https://wiki.openstack.org/wiki/Network/Incubator I have
some thoughts about the proposed workflow.

We have quite a bit of experience and some good tools around
splitting
code out of projects and into new projects.  But we don't
generally do a
lot of importing code into projects.  We've done this once, to my
recollection, in a way that preserved history, and that was with the
switch to keystone-lite.

It wasn't easy; it's major git surgery and would require significant
infra-team involvement any time we wanted to do it.

However, reading the proposal, it occurred to me that it's
pretty clear
that we expect these tools to be able to operate outside of the
Neutron
project itself, to even be releasable on their own.  Why not
just stick
with that?  In other words, the goal of this process should be
to create
separate projects with their own development lifecycle that will
continue indefinitely, rather than expecting the code itself to
merge
into the neutron repo.

This has advantages in simplifying workflow and making it more
consistent.  Plus it builds on known integration mechanisms like
APIs
and python project versions.

But more importantly, it helps scale the neutron project itself.  I
think that a focused neutron core upon which projects like these can
build on in a reliable fashion would be ideal.


Despite replies to you saying that certain branches of Neutron
development work are special unicorns, I wanted to say I *fully*
support your above statement.

Best,
-jay



_
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
Kevin Benton


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-28 Thread Kevin Benton
Right. The special unicorns.

Repeating this without defining it isn't helping anything.

b) The experimental piece of code intends to replace whole-hog a large
chunk of Neutron's existing codebase, or:

In the DVR example I gave this is is the only relevant reason. Regardless
of how well the internal Neutron APIs are defined, this same problem would
have arisen. DVR completely changed the reference L3 service plugin, which
lives in the main tree.

A well-defined, versioned internal API would not have helped any of the
issues I brought up. The 'experimental' part of the DVR work isn't
referring to its internal API modifications, its referring to how the L3
service plugin will integrate with the data plane.


On Thu, Aug 28, 2014 at 7:45 AM, Jay Pipes jaypi...@gmail.com wrote:

 On 08/27/2014 04:28 PM, Kevin Benton wrote:

 What are you talking about? The only reply was from me clarifying that
 one of the purposes of the incubator was for components of neutron that
 are experimental but are intended to be merged.


 Right. The special unicorns.


  In that case it might

 not make sense to have a life cycle of their own in another repo
 indefinitely.


 The main reasons these experimental components don't make sense to live
 in their own repo indefinitely are:

 a) Neutron's design doesn't make it easy or straightforward to build/layer
 other things on top of it, or:

 b) The experimental piece of code intends to replace whole-hog a large
 chunk of Neutron's existing codebase, or:

 c) The experimental piece of code relies so heavily on inconsistent,
 unversioned internal interface and plugin calls that it cannot be designed
 externally due to the fragility of those interfaces

 Fixing a) is the solution to these problems. An incubator area where
 experimental components can live will just continue to mask the true
 problem domain, which is that Neutron's design is cumbersome to build on
 top of, and its cross-component interfaces need to be versioned, made
 consistent, and cleaned up to use versioned data structures instead of
 passing random nested dicts of randomly-prefixed string key/values.

 Frankly, we're going through a similar problem in Nova right now. There is
 a group of folks who believe that separating the nova-scheduler code into
 the Gantt project will magically make placement decision code and solver
 components *easier* to work on (because the pace of coding can be increased
 if there wasn't that pesky nova-core review process). But this is not
 correct, IMO. Separating out the scheduler into its own project before
 internal interfaces and data structures are cleaned up and versioned will
 just lead to greater technical debt and an increase in frustration on the
 part of Nova developers and scheduler developers alike.

 -jay

  On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com
 mailto:jaypi...@gmail.com wrote:

 On 08/26/2014 07:09 PM, James E. Blair wrote:

 Hi,

 After reading
 https://wiki.openstack.org/__wiki/Network/Incubator

 https://wiki.openstack.org/wiki/Network/Incubator I have
 some thoughts about the proposed workflow.

 We have quite a bit of experience and some good tools around
 splitting
 code out of projects and into new projects.  But we don't
 generally do a
 lot of importing code into projects.  We've done this once, to my
 recollection, in a way that preserved history, and that was with
 the
 switch to keystone-lite.

 It wasn't easy; it's major git surgery and would require
 significant
 infra-team involvement any time we wanted to do it.

 However, reading the proposal, it occurred to me that it's
 pretty clear
 that we expect these tools to be able to operate outside of the
 Neutron
 project itself, to even be releasable on their own.  Why not
 just stick
 with that?  In other words, the goal of this process should be
 to create
 separate projects with their own development lifecycle that will
 continue indefinitely, rather than expecting the code itself to
 merge
 into the neutron repo.

 This has advantages in simplifying workflow and making it more
 consistent.  Plus it builds on known integration mechanisms like
 APIs
 and python project versions.

 But more importantly, it helps scale the neutron project itself.
 I
 think that a focused neutron core upon which projects like these
 can
 build on in a reliable fashion would be ideal.


 Despite replies to you saying that certain branches of Neutron
 development work are special unicorns, I wanted to say I *fully*
 support your above statement.

 Best,
 -jay



 _
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.__org
 

Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-28 Thread Mark McClain

On Aug 28, 2014, at 10:45 AM, Jay Pipes jaypi...@gmail.com wrote:

 On 08/27/2014 04:28 PM, Kevin Benton wrote:
 What are you talking about? The only reply was from me clarifying that
 one of the purposes of the incubator was for components of neutron that
 are experimental but are intended to be merged.
 
 Right. The special unicorns.
 
  In that case it might
 not make sense to have a life cycle of their own in another repo
 indefinitely.
 
 The main reasons these experimental components don't make sense to live in 
 their own repo indefinitely are:
 
 a) Neutron's design doesn't make it easy or straightforward to build/layer 
 other things on top of it, or:
 

Correct and this something I want the team to address in Kilo.  Many of the L7 
services would be easier to build if we invest some time early in the cycle to 
establishing a well defined interface for a few items.  I’m sure the LBaaS team 
has good feedback to share with everyone.

 b) The experimental piece of code intends to replace whole-hog a large chunk 
 of Neutron's existing codebase, or:
 
 c) The experimental piece of code relies so heavily on inconsistent, 
 unversioned internal interface and plugin calls that it cannot be designed 
 externally due to the fragility of those interfaces

I’m glad Jim reminded us of the pain of merging histories and the availability 
of feature branches.  I think for items where we’re replacing large chunks of 
code feature branches make lots of sense.

 
 Fixing a) is the solution to these problems. An incubator area where 
 experimental components can live will just continue to mask the true 
 problem domain, which is that Neutron's design is cumbersome to build on top 
 of, and its cross-component interfaces need to be versioned, made consistent, 
 and cleaned up to use versioned data structures instead of passing random 
 nested dicts of randomly-prefixed string key/values.
 
 Frankly, we're going through a similar problem in Nova right now. There is a 
 group of folks who believe that separating the nova-scheduler code into the 
 Gantt project will magically make placement decision code and solver 
 components *easier* to work on (because the pace of coding can be increased 
 if there wasn't that pesky nova-core review process). But this is not 
 correct, IMO. Separating out the scheduler into its own project before 
 internal interfaces and data structures are cleaned up and versioned will 
 just lead to greater technical debt and an increase in frustration on the 
 part of Nova developers and scheduler developers alike.

Right hopefully the incubator will allow us to develop components that should 
be independent from the start without incurring too much debt.

 
 -jay
 
 On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com
 mailto:jaypi...@gmail.com wrote:
 
On 08/26/2014 07:09 PM, James E. Blair wrote:
 
Hi,
 
After reading
https://wiki.openstack.org/__wiki/Network/Incubator
https://wiki.openstack.org/wiki/Network/Incubator I have
some thoughts about the proposed workflow.
 
We have quite a bit of experience and some good tools around
splitting
code out of projects and into new projects.  But we don't
generally do a
lot of importing code into projects.  We've done this once, to my
recollection, in a way that preserved history, and that was with the
switch to keystone-lite.
 
It wasn't easy; it's major git surgery and would require significant
infra-team involvement any time we wanted to do it.
 
However, reading the proposal, it occurred to me that it's
pretty clear
that we expect these tools to be able to operate outside of the
Neutron
project itself, to even be releasable on their own.  Why not
just stick
with that?  In other words, the goal of this process should be
to create
separate projects with their own development lifecycle that will
continue indefinitely, rather than expecting the code itself to
merge
into the neutron repo.
 
This has advantages in simplifying workflow and making it more
consistent.  Plus it builds on known integration mechanisms like
APIs
and python project versions.
 
But more importantly, it helps scale the neutron project itself.  I
think that a focused neutron core upon which projects like these can
build on in a reliable fashion would be ideal.
 
 
Despite replies to you saying that certain branches of Neutron
development work are special unicorns, I wanted to say I *fully*
support your above statement.
 
Best,
-jay
 
 
 
_
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
mailto:OpenStack-dev@lists.openstack.org
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 
 

Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-28 Thread Richard Woo
I have another question about incubator proposal, for CLI and GUI. Do we
imply that the incubator feature will need to branch python-neutron client,
Horizon, and or Nova ( if changes are needed)?




On Tue, Aug 26, 2014 at 7:09 PM, James E. Blair cor...@inaugust.com wrote:

 Hi,

 After reading https://wiki.openstack.org/wiki/Network/Incubator I have
 some thoughts about the proposed workflow.

 We have quite a bit of experience and some good tools around splitting
 code out of projects and into new projects.  But we don't generally do a
 lot of importing code into projects.  We've done this once, to my
 recollection, in a way that preserved history, and that was with the
 switch to keystone-lite.

 It wasn't easy; it's major git surgery and would require significant
 infra-team involvement any time we wanted to do it.

 However, reading the proposal, it occurred to me that it's pretty clear
 that we expect these tools to be able to operate outside of the Neutron
 project itself, to even be releasable on their own.  Why not just stick
 with that?  In other words, the goal of this process should be to create
 separate projects with their own development lifecycle that will
 continue indefinitely, rather than expecting the code itself to merge
 into the neutron repo.

 This has advantages in simplifying workflow and making it more
 consistent.  Plus it builds on known integration mechanisms like APIs
 and python project versions.

 But more importantly, it helps scale the neutron project itself.  I
 think that a focused neutron core upon which projects like these can
 build on in a reliable fashion would be ideal.

 -Jim

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-28 Thread Jeremy Stanley
On 2014-08-28 08:31:26 -0700 (-0700), Kevin Benton wrote:
[...]
 DVR completely changed the reference L3 service plugin, which
 lives in the main tree. 
 
 A well-defined, versioned internal API would not have helped any
 of the issues I brought up.
[...]

Except, perhaps, insofar as that (in some ideal world) it might
allow the reference L3 service plugin to be extracted from the main
tree and developed within a separate source code repository with its
own life cycle.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-28 Thread Kevin Benton
Yes, in theory all of the plugins should be removable from the core neutron
repo. So then it would only need to be responsible for the APIs, db models,
etc. However, IIRC there are no plans to move any reference plugins from
the tree.


On Thu, Aug 28, 2014 at 11:20 AM, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2014-08-28 08:31:26 -0700 (-0700), Kevin Benton wrote:
 [...]
  DVR completely changed the reference L3 service plugin, which
  lives in the main tree.
 
  A well-defined, versioned internal API would not have helped any
  of the issues I brought up.
 [...]

 Except, perhaps, insofar as that (in some ideal world) it might
 allow the reference L3 service plugin to be extracted from the main
 tree and developed within a separate source code repository with its
 own life cycle.
 --
 Jeremy Stanley

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kevin Benton
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread loy wolfe
On Wed, Aug 27, 2014 at 2:44 PM, Kevin Benton blak...@gmail.com wrote:

 Incubator doesn't mean being kicked out of tree, it just mean that the
 API and resource model needs to be baked for fast iteration, and can't be
 put in tree temporarily.

 That was exactly my point about developing a major feature like DVR. Even
 with a limited API change (the new distributed flag), it has an impact on
 the the router/agent DB resource model and currently isn't compatible
 with VLAN based ML2 deployments. It's not exactly a hidden optimization
 like an improvement to some RPC handling code.


Flag is only for admin use, tenant can't see it, and the default policy for
router is setup by config file.

It COULD be compatible for DVR work on vlan, but there were some bugs
several months before, that DVR mac are not written successfully on the
egress packet, I'm not sure if it is fixed in the merged code.


 A huge piece of the DVR development had to happen in Neutron forks and 40+
 revision chains of Gerrit patches. It was very difficult to follow without
 being heavily involved with the L3 team. This would have been a
 great candidate to develop in the incubator if it existed at the time. It
 would have been easier to try as it was developed and to explore the entire
 codebase. Also, more people could have been contributing bug fixes and
 improvements since an entire section of code wouldn't be 'owned' by one
 person like it is with the author of a Gerrit review.

 For DVR, as it has no influence on tenant facing API resource model, it
 works as the built-in backend, and this feature has accepted wide common
 interests,

 As was pointed out before, common interest has nothing to do with
 incubation. Incubation is to rapidly iterate on a new feature for Neutron.
 It shouldn't be restricted to API changes, it should be used for any major
 new features that are possible to develop outside of the Neutron core. If
 we are going to have this new incubator tool, we should use it to the
 fullest extent possible.



 On Tue, Aug 26, 2014 at 6:19 PM, loy wolfe loywo...@gmail.com wrote:

 Incubator doesn't mean being kicked out of tree, it just mean that the
 API and resource model needs to be baked for fast iteration, and can't be
 put in tree temporarily. As kyle has said, incubator is not talking about
 moving 3rd drivers out of tree, which is in another thread.

 For DVR, as it has no influence on tenant facing API resource model, it
 works as the built-in backend, and this feature has accepted wide common
 interests, it's just the internal performance optimization tightly coupled
 with existing code, so it should be developed in tree.


 On Wed, Aug 27, 2014 at 8:08 AM, Kevin Benton blak...@gmail.com wrote:

 From what I understand, the intended projects for the incubator can't
 operate without neutron because they are just extensions/plugins/drivers.

 For example, if the DVR modifications to the reference reference L3
 plugin weren't already being developed in the tree, DVR could have been
 developed in the incubator and then merged into Neutron once the bugs were
 ironed out so a huge string of Gerrit patches didn't need to be tracked. If
 that had happened, would it make sense to keep the L3 plugin as a
 completely separate project or merge it? I understand this is the approach
 the load balancer folks took by making Octavia a separate project, but I
 think it can still operate on its own, where the reference L3 plugin (and
 many of the other incubator projects) are just classes that expect to be
 able to make core Neutron calls.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Kevin Benton

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Kevin Benton
Flag is only for admin use, tenant can't see it, and the default policy
for router is setup by config file.

It's still a public API that will have to follow a deprecation cycle. If a
new API was going to be introduced for admins to control the distributed
nature of routers, it would have been nice to introduce a little more
control than distributed=True/False (e.g. how many SNAT IPs are consumed,
etc). At this point we are stuck because any new API entries would require
a blueprint and the feature proposal freeze deadline is long gone.

It COULD be compatible for DVR work on vlan, but there were some bugs
several months before, that DVR mac are not written successfully on the
egress packet, I'm not sure if it is fixed in the merged code.

The current DVR wiki[1] still shows that the tenant_network_type has to be
VXLAN. I didn't know about this issue until just recently and I'm not sure
if there are plans yet to fix it for Juno. If it were in an incubation tree
somewhere and not on Gerrit for the majority of the cycle, the bug could
have been worked on in parallel by other volunteers interested in VLAN
support. Now that DVR is solidifying, VLAN support may even require a
blueprint unless it's blessed by the right cores, which would mean people
with VLAN deployments would not be able to use DVR until Kilo is released
next next May.

The whole point is that there is nowhere to work on big features like this
in a fast, iterative, and open manner. The incubator could be a perfect
place for this type of work.


1. https://wiki.openstack.org/wiki/Neutron/DVR/HowTo


On Wed, Aug 27, 2014 at 1:22 AM, loy wolfe loywo...@gmail.com wrote:




 On Wed, Aug 27, 2014 at 2:44 PM, Kevin Benton blak...@gmail.com wrote:

 Incubator doesn't mean being kicked out of tree, it just mean that the
 API and resource model needs to be baked for fast iteration, and can't be
 put in tree temporarily.

 That was exactly my point about developing a major feature like DVR. Even
 with a limited API change (the new distributed flag), it has an impact on
 the the router/agent DB resource model and currently isn't compatible
 with VLAN based ML2 deployments. It's not exactly a hidden optimization
 like an improvement to some RPC handling code.


 Flag is only for admin use, tenant can't see it, and the default policy
 for router is setup by config file.

 It COULD be compatible for DVR work on vlan, but there were some bugs
 several months before, that DVR mac are not written successfully on the
 egress packet, I'm not sure if it is fixed in the merged code.


 A huge piece of the DVR development had to happen in Neutron forks and
 40+ revision chains of Gerrit patches. It was very difficult to follow
 without being heavily involved with the L3 team. This would have been a
 great candidate to develop in the incubator if it existed at the time. It
 would have been easier to try as it was developed and to explore the entire
 codebase. Also, more people could have been contributing bug fixes and
 improvements since an entire section of code wouldn't be 'owned' by one
 person like it is with the author of a Gerrit review.

 For DVR, as it has no influence on tenant facing API resource model, it
 works as the built-in backend, and this feature has accepted wide common
 interests,

 As was pointed out before, common interest has nothing to do with
 incubation. Incubation is to rapidly iterate on a new feature for Neutron.
 It shouldn't be restricted to API changes, it should be used for any major
 new features that are possible to develop outside of the Neutron core. If
 we are going to have this new incubator tool, we should use it to the
 fullest extent possible.



 On Tue, Aug 26, 2014 at 6:19 PM, loy wolfe loywo...@gmail.com wrote:

 Incubator doesn't mean being kicked out of tree, it just mean that the
 API and resource model needs to be baked for fast iteration, and can't be
 put in tree temporarily. As kyle has said, incubator is not talking about
 moving 3rd drivers out of tree, which is in another thread.

 For DVR, as it has no influence on tenant facing API resource model, it
 works as the built-in backend, and this feature has accepted wide common
 interests, it's just the internal performance optimization tightly coupled
 with existing code, so it should be developed in tree.


 On Wed, Aug 27, 2014 at 8:08 AM, Kevin Benton blak...@gmail.com wrote:

 From what I understand, the intended projects for the incubator can't
 operate without neutron because they are just extensions/plugins/drivers.

 For example, if the DVR modifications to the reference reference L3
 plugin weren't already being developed in the tree, DVR could have been
 developed in the incubator and then merged into Neutron once the bugs were
 ironed out so a huge string of Gerrit patches didn't need to be tracked. If
 that had happened, would it make sense to keep the L3 plugin as a
 completely separate project or merge it? I understand this is the approach
 the 

Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Jay Pipes

On 08/26/2014 07:09 PM, James E. Blair wrote:

Hi,

After reading https://wiki.openstack.org/wiki/Network/Incubator I have
some thoughts about the proposed workflow.

We have quite a bit of experience and some good tools around splitting
code out of projects and into new projects.  But we don't generally do a
lot of importing code into projects.  We've done this once, to my
recollection, in a way that preserved history, and that was with the
switch to keystone-lite.

It wasn't easy; it's major git surgery and would require significant
infra-team involvement any time we wanted to do it.

However, reading the proposal, it occurred to me that it's pretty clear
that we expect these tools to be able to operate outside of the Neutron
project itself, to even be releasable on their own.  Why not just stick
with that?  In other words, the goal of this process should be to create
separate projects with their own development lifecycle that will
continue indefinitely, rather than expecting the code itself to merge
into the neutron repo.

This has advantages in simplifying workflow and making it more
consistent.  Plus it builds on known integration mechanisms like APIs
and python project versions.

But more importantly, it helps scale the neutron project itself.  I
think that a focused neutron core upon which projects like these can
build on in a reliable fashion would be ideal.


Despite replies to you saying that certain branches of Neutron 
development work are special unicorns, I wanted to say I *fully* support 
your above statement.


Best,
-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Kevin Benton
What are you talking about? The only reply was from me clarifying that one
of the purposes of the incubator was for components of neutron that are
experimental but are intended to be merged. In that case it might not make
sense to have a life cycle of their own in another repo indefinitely.


On Wed, Aug 27, 2014 at 11:52 AM, Jay Pipes jaypi...@gmail.com wrote:

 On 08/26/2014 07:09 PM, James E. Blair wrote:

 Hi,

 After reading https://wiki.openstack.org/wiki/Network/Incubator I have
 some thoughts about the proposed workflow.

 We have quite a bit of experience and some good tools around splitting
 code out of projects and into new projects.  But we don't generally do a
 lot of importing code into projects.  We've done this once, to my
 recollection, in a way that preserved history, and that was with the
 switch to keystone-lite.

 It wasn't easy; it's major git surgery and would require significant
 infra-team involvement any time we wanted to do it.

 However, reading the proposal, it occurred to me that it's pretty clear
 that we expect these tools to be able to operate outside of the Neutron
 project itself, to even be releasable on their own.  Why not just stick
 with that?  In other words, the goal of this process should be to create
 separate projects with their own development lifecycle that will
 continue indefinitely, rather than expecting the code itself to merge
 into the neutron repo.

 This has advantages in simplifying workflow and making it more
 consistent.  Plus it builds on known integration mechanisms like APIs
 and python project versions.

 But more importantly, it helps scale the neutron project itself.  I
 think that a focused neutron core upon which projects like these can
 build on in a reliable fashion would be ideal.


 Despite replies to you saying that certain branches of Neutron development
 work are special unicorns, I wanted to say I *fully* support your above
 statement.

 Best,
 -jay



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kevin Benton
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Kevin Benton
I agree that components isolated to one service or something along those
lines where there is a clear plugin point in Neutron, it might make sense
to separate them permanently. However, at that point why even bother with
the Neutron incubator when a new project can be started?

The feature branch idea sounds interesting for the one-time big
experimental changes. Are there any other projects that do this right now?


On Wed, Aug 27, 2014 at 12:30 PM, James E. Blair cor...@inaugust.com
wrote:

 Kevin Benton blak...@gmail.com writes:

  From what I understand, the intended projects for the incubator can't
  operate without neutron because they are just extensions/plugins/drivers.

 I could have phrased that better.  What I meant was that they could
 operate without being actually in the Neutron repo, not that they could
 not operate without Neutron itself.

 The proposal for the incubator is that extensions be developed outside
 of the Neutron repo.  My proposed refinement is that they stay outside
 of the Neutron repo.  They live their entire lives as extension modules
 in separate projects.

  For example, if the DVR modifications to the reference reference L3
 plugin
  weren't already being developed in the tree, DVR could have been
 developed
  in the incubator and then merged into Neutron once the bugs were ironed
 out
  so a huge string of Gerrit patches didn't need to be tracked. If that had
  happened, would it make sense to keep the L3 plugin as a completely
  separate project or merge it? I understand this is the approach the load
  balancer folks took by making Octavia a separate project, but I think it
  can still operate on its own, where the reference L3 plugin (and many of
  the other incubator projects) are just classes that expect to be able to
  make core Neutron calls.

 The list of Juno/Kilo candidates doesn't seem to have projects that are
 quite so low-level.

 If a feature is going to become part of the neutron core, then it should
 be developed in the neutron repository.  If we need a place to land code
 that isn't master, it's actually far easier to just use a feature branch
 on the neutron repo.  Commits can land there as needed, master can be
 periodically merged into it, and when the feature is ready, the feature
 branch can be merged into master.

 I think between those two options: incubate/spin-out components that are
 high-level enough not to have deep integration in the neutron core, and
 using feature branches for large experimental changes to the core
 itself, we can handle the problems the incubator repo is intended to
 address.

 -Jim

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Kevin Benton
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Ivar Lazzaro
Quoting Stefano from a different thread [0]:

The rationale for the separate repository is that Neutron's code needs a
 lot of love for the *existing* codebase, before new features can be
 added (and before core team can accept more responsibilities for it).
 But progress is unstoppable: new features are being proposed every cycle
 and reviewers bandwidth is not infinite.


The long term purpose of the incubator should be to gather all the big new
features that may require fast iteration before becoming stable and being
eventually merged into the main tree. This is especially important for
core new features, that may have a higher impact on Neutron's stability.
A great example was made by Kevin with DVR.

[0]
http://lists.openstack.org/pipermail/openstack-dev/2014-August/044224.html


On Wed, Aug 27, 2014 at 1:32 PM, Kevin Benton blak...@gmail.com wrote:

 I agree that components isolated to one service or something along those
 lines where there is a clear plugin point in Neutron, it might make sense
 to separate them permanently. However, at that point why even bother with
 the Neutron incubator when a new project can be started?

 The feature branch idea sounds interesting for the one-time big
 experimental changes. Are there any other projects that do this right now?


 On Wed, Aug 27, 2014 at 12:30 PM, James E. Blair cor...@inaugust.com
 wrote:

 Kevin Benton blak...@gmail.com writes:

  From what I understand, the intended projects for the incubator can't
  operate without neutron because they are just
 extensions/plugins/drivers.

 I could have phrased that better.  What I meant was that they could
 operate without being actually in the Neutron repo, not that they could
 not operate without Neutron itself.

 The proposal for the incubator is that extensions be developed outside
 of the Neutron repo.  My proposed refinement is that they stay outside
 of the Neutron repo.  They live their entire lives as extension modules
 in separate projects.

  For example, if the DVR modifications to the reference reference L3
 plugin
  weren't already being developed in the tree, DVR could have been
 developed
  in the incubator and then merged into Neutron once the bugs were ironed
 out
  so a huge string of Gerrit patches didn't need to be tracked. If that
 had
  happened, would it make sense to keep the L3 plugin as a completely
  separate project or merge it? I understand this is the approach the load
  balancer folks took by making Octavia a separate project, but I think it
  can still operate on its own, where the reference L3 plugin (and many of
  the other incubator projects) are just classes that expect to be able to
  make core Neutron calls.

 The list of Juno/Kilo candidates doesn't seem to have projects that are
 quite so low-level.

 If a feature is going to become part of the neutron core, then it should
 be developed in the neutron repository.  If we need a place to land code
 that isn't master, it's actually far easier to just use a feature branch
 on the neutron repo.  Commits can land there as needed, master can be
 periodically merged into it, and when the feature is ready, the feature
 branch can be merged into master.

 I think between those two options: incubate/spin-out components that are
 high-level enough not to have deep integration in the neutron core, and
 using feature branches for large experimental changes to the core
 itself, we can handle the problems the incubator repo is intended to
 address.

 -Jim

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Kevin Benton

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Jeremy Stanley
On 2014-08-27 16:05:36 -0700 (-0700), Joe Gordon wrote:
 We have feature branches? How do we use them? I am not even sure
 if feature branches were evaluated as an option when the incubator
 was proposed.

Keystone's new API used a feature branch, the EC work in Swift did,
so did the Gearman integration for Zuul... however it's also an
option not to be taken lightly and comes with its own set of unique
challenges.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Mandeep Dhami
Also note that one of the features of the incubator is that it can be
packaged/released independently (like python-neutronclient) and a feature
branch is not designed for that workflow.

Regards,
Mandeep



On Wed, Aug 27, 2014 at 4:55 PM, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2014-08-27 16:05:36 -0700 (-0700), Joe Gordon wrote:
  We have feature branches? How do we use them? I am not even sure
  if feature branches were evaluated as an option when the incubator
  was proposed.

 Keystone's new API used a feature branch, the EC work in Swift did,
 so did the Gearman integration for Zuul... however it's also an
 option not to be taken lightly and comes with its own set of unique
 challenges.
 --
 Jeremy Stanley

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Jeremy Stanley
On 2014-08-27 16:53:39 -0700 (-0700), Clark Boylan wrote:
[...]
 I thought there was a wiki article on how they work but I can't
 find it. Maybe someone else can link it here.
[...]

https://wiki.openstack.org/wiki/GerritJenkinsGit#Merge_Commits

-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Jeremy Stanley
On 2014-08-27 16:59:54 -0700 (-0700), Mandeep Dhami wrote:
 Also note that one of the features of the incubator is that it can be
 packaged/released independently (like python-neutronclient) and a feature
 branch is not designed for that workflow.

Good point, and that definitely is a reason to just keep those
pieces in their own separate Git repositories outside of the core
Neutron repository in perpetuity (even after they graduate from
incubation). One package per repository. That should be chiseled in
stone somewhere.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Dean Troyer
On Wed, Aug 27, 2014 at 6:59 PM, Mandeep Dhami dh...@noironetworks.com
wrote:


 Also note that one of the features of the incubator is that it can be
 packaged/released independently (like python-neutronclient) and a feature
 branch is not designed for that workflow.


Which is a good reason to not use the word 'incubator'.  It also has the
connotations from Oslo of being 'copy-pasta' code.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-27 Thread Ivar Lazzaro
On Wed, Aug 27, 2014 at 5:06 PM, Jeremy Stanley fu...@yuggoth.org wrote:

 Good point, and that definitely is a reason to just keep those
 pieces in their own separate Git repositories outside of the core
 Neutron repository in perpetuity (even after they graduate from
 incubation). One package per repository. That should be chiseled in
 stone somewhere.


Just asking, wasn't there a proposal about doing something similar for all
the neutron services? (l3/fw/lb/vnp ...)

Ivar.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-26 Thread Kevin Benton
From what I understand, the intended projects for the incubator can't
operate without neutron because they are just extensions/plugins/drivers.

For example, if the DVR modifications to the reference reference L3 plugin
weren't already being developed in the tree, DVR could have been developed
in the incubator and then merged into Neutron once the bugs were ironed out
so a huge string of Gerrit patches didn't need to be tracked. If that had
happened, would it make sense to keep the L3 plugin as a completely
separate project or merge it? I understand this is the approach the load
balancer folks took by making Octavia a separate project, but I think it
can still operate on its own, where the reference L3 plugin (and many of
the other incubator projects) are just classes that expect to be able to
make core Neutron calls.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [infra] [neutron] [tc] Neutron Incubator workflow

2014-08-26 Thread loy wolfe
Incubator doesn't mean being kicked out of tree, it just mean that the API
and resource model needs to be baked for fast iteration, and can't be put
in tree temporarily. As kyle has said, incubator is not talking about
moving 3rd drivers out of tree, which is in another thread.

For DVR, as it has no influence on tenant facing API resource model, it
works as the built-in backend, and this feature has accepted wide common
interests, it's just the internal performance optimization tightly coupled
with existing code, so it should be developed in tree.


On Wed, Aug 27, 2014 at 8:08 AM, Kevin Benton blak...@gmail.com wrote:

 From what I understand, the intended projects for the incubator can't
 operate without neutron because they are just extensions/plugins/drivers.

 For example, if the DVR modifications to the reference reference L3 plugin
 weren't already being developed in the tree, DVR could have been developed
 in the incubator and then merged into Neutron once the bugs were ironed out
 so a huge string of Gerrit patches didn't need to be tracked. If that had
 happened, would it make sense to keep the L3 plugin as a completely
 separate project or merge it? I understand this is the approach the load
 balancer folks took by making Octavia a separate project, but I think it
 can still operate on its own, where the reference L3 plugin (and many of
 the other incubator projects) are just classes that expect to be able to
 make core Neutron calls.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev