On May 22, 2014, at 1:59 PM, Mandeep Dhami <[email protected]> wrote:
Maru's concerns are that:
1. It is large
2. It is complex
As per the discussion in the irc meeting today, I hope it is clear now that
eventual size and complexity are not real issue. Rather, I am concerned at how
we get there.
I keep talking about 'iterating in the open', and want to make it clear what I
mean by this. It involves proposing a reviewable patch to openstack gerrit,
working with reviewers to get the patch merged, and then incorporating their
feedback into the overall design to drive the implementation of future patches.
'Iterating in the open' does not imply working outside of gerrit to create a
monolithic codebase that needs to be manually decomposed into reviewable chunks
at the end. I understand that this may be an effective way to create a POC,
but it is not an effective way to produce code that can be merged into Neutron.
Core reviewers have a mandate to ensure the quality of every patch, and their
feedback is likely to have an impact on subsequent implementation.
And Armando's related concerns are:
3. Could dev/review cycles be better spent on refactoring
4. If refactored neutron was available, would a simpler option become more
viable
Let me address them in that order.
1. Re: It is large
Group policy has an ambitious goal - provide devop teams with policy based controls that
are usable at scale and with automation (say a higher governance layer like Congress).
The fact that meeting a large challenge requires more code is natural. We understand that
challenge, and that is why we did a prototype (as PoC that was demonstrated on the
summit). And based on that learning we are incrementally creating patches for building
the group based policy. Just because a task is large, we as neutron can not shy away from
building it. That will only drive people who need it out side neutron (as we are seeing
with the frustration that the LBaaS team had because they have a requirement that is
"large" as well).
Again, the amount of code is not the problem. How code is introduced into the
tree, and how the design is socialized (both with developers and users), _is_
of critical importance. Neutron is not alone in requiring an 'iterate in the
open' approach - it is a characteristic common to many open source projects.
2. Re: It is complex
Complexity depends on the context. Our goal was to make the end-user's life
simpler (and more automated). To achieve some of that simplicity, we required a
little more complexity in the implementation. We decide to make that arbitrage
- a little higher complexity in implementation to allow for simpler usage. But
we were careful and did not want to impose that complexity on every use case -
hence a lot of that is optional (and exercised only if the use case needs it).
Unfortunately the model, has to model all of it so as it not add complexity
later in upgrade and backward compatibility issues. We choose to do
architecture upfront, and then implement it incrementally.
Doing upfront architecture is fine, so long as the architecture also evolves in
response to feedback from the review process in gerrit. Similarly, incremental
implementation is not enough - it needs to happen in gerrit. And to be clear,
the tool is not the critical factor. When I say gerrit, I mean that each patch
needs to receive core reviewer attention and that subsequent patches
incorporate their feedback.
The team came up with the model currently in model based on that review and
evaluation all the proposals in the document that you refer. It is easy to make
general comments, but unless you participate in the process and sign up to
writing the code, those comments are not going to help with solving the
original problem. And this _is_ open-source. If you disagree, please write code
and the community can decide for itself as to what model is actually simple to
use for them. Curtailing efforts from other developers just because their
engineering trade-offs are different from what you believe your use-case needs
is not why we like open source. We enjoy the mode where different developers
try different things, we experiment, and the software evolves to what the user
demands. Or maybe, multiple models live in harmony. Let the users decide that.
You are correct in saying that it is not my job to decide what you or other
developers do. It is, however, my role as a Neutron core reviewer to ensure
that we make good use of the resources available to us to meet the project's
commitments. If I believe that the approach chosen to implement a given
Neutron feature has the potential to starve other priorities of resources, then
I have a responsibility to voice that concern and push back. You're free to
implement whatever you want outside of the tree, but participating in the
Neutron community means accepting the norms that we have adopted. If you don't
like them, you are free to work to effect change, but it won't happen by simply
wishing it so.
And as per the summit session, there appears to be significant user demand for
a subset of the POC that represents the 'link-based' approach. If we are to
talk about taking user interests into account, I think the starting point would
be implementing what they are asking for first and and evolving more complex
solutions from there.
m.
3. Re: Could dev/review cycles be better spent on refactoring
I think that most people agree that policy control is an important feature that
fundamentally improves neutron (by solving the automation and scale issues). In a large
project, multiple sub-projects can, and for a healthy project should, work in parallel. I
understand that the neutron core team is stretched. But we still need to be able to
balance the needs of today (paying off the technical debt/existing-issues by doing
refactoring) with needs of tomorrow (new features like GP and LBaaS). GP effort was
started in Havana, and now we are trying to get this in Juno. I think that is reasonable
and a long enough cycle for a "high priority" project to be able to get some
core attention. Again I refer to LBaaS experience, as they struggled with very similar
issues.
4. Re: If refactored neutron was available, would a simpler option become more
viable
We would love to be able to answer that question. We have been trying to
understand the refactoring work to understand this (see another ML thread) and
we are open to understanding your position on that. We will call the ad-hoc
meeting that you suggested and we would like to understand the refactoring work
that might be reused for simpler policy implementation. At the same time, we
would like to build on what is available today, and when the required
refactored neutron becomes available (say Juno or K-release), we are more than
happy to adapt to it at that time. Serializing all development around an effort
that is still in inception phase is not a good solution. We are looking forward
to participating in the core refactoring work, and based on the final spec that
come up with, we would love to be able to eventually make the policy
implementation simpler.
Regards,
Mandeep
On Thu, May 22, 2014 at 11:44 AM, Armando M. <[email protected]> wrote:
I would second Maru's concerns, and I would also like to add the following:
We need to acknowledge the fact that there are certain architectural
aspects of Neutron as a project that need to be addressed; at the
summit we talked about the core refactoring, a task oriented API, etc.
To me these items have been neglected far too much over the past and
would need a higher priority and a lot more attention during the Juno
cycle. Being stretched as we are I wonder if dev/review cycles
wouldn't be better spent devoting more time to these efforts rather
than GP.
That said, I appreciate that GP is important and needs to move
forward, but at the same time I am thinking that there must be a
better way for addressing it and yet relieve some of the pressure that
GP complexity imposes to the Neutron team. One aspect it was discussed
at the summit was that the type of approach shown in [2] and [3]
below, was chosen because of lack of proper integration hooks...so I
am advocating: let's talk about those first before ruling them out in
favor of a monolithic approach that seems to violate some engineering
principles, like modularity and loose decoupling of system components.
I think we didn't have enough time during the summit to iron out some
of the concerns voiced here, and it seems like the IRC meeting for
Group Policy would not be the right venue to try and establish a
common ground among the people driving this effort and the rest of the
core team.
Shall we try and have an ad-hoc meeting and an ad-hoc agenda to find a
consensus?
Many thanks,
Armando
On 22 May 2014 11:38, Maru Newby <[email protected]> wrote:
On May 22, 2014, at 11:03 AM, Maru Newby <[email protected]> wrote:
At the summit session last week for group-based policy, there were many
concerns voiced about the approach being undertaken. I think those concerns
deserve a wider audience, and I'm going to highlight some of them here.
The primary concern seemed to be related to the complexity of the approach
implemented for the POC. A number of session participants voiced concern that
the simpler approach documented in the original proposal [1] (described in the
section titled 'Policies applied between groups') had not been implemented in
addition to or instead of what appeared in the POC (described in the section
titled 'Policies applied as a group API'). The simpler approach was considered
by those participants as having the advantage of clarity and immediate
usefulness, whereas the complex approach was deemed hard to understand and
without immediate utility.
A secondary but no less important concern is related to the impact on Neutron
of the approach implemented in the POC. The POC was developed monolithically,
without oversight through gerrit, and the resulting patches were excessive in
size (~4700 [2] and ~1500 [3] lines). Such large patches are effectively
impossible to review. Even broken down into reviewable chunks, though, it does
not seem realistic to target juno-1 for merging this kind of complexity. The
impact on stability could be considerable, and it is questionable whether the
necessary review effort should be devoted to fast-tracking group-based policy
at all, let alone an approach that is considered by many to be unnecessarily
complicated.
The blueprint for group policy [4] is currently listed as a 'High' priority.
With the above concerns in mind, does it make sense to continue prioritizing an
effort that at present would seem to require considerably more resources than
the benefit it appears to promise?
Maru
1: https://etherpad.openstack.org/p/group-based-policy
Apologies, this link is to the summit session etherpad. The link to the
original proposal is:
https://docs.google.com/document/d/1ZbOFxAoibZbJmDWx1oOrOsDcov6Cuom5aaBIrupCD9E/edit
2: https://review.openstack.org/93853
3: https://review.openstack.org/93935
4: https://blueprints.launchpad.net/neutron/+spec/group-based-policy-abstraction
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
