subject:"Re\: \[openstack\-dev\] \[tc\]\[all\] Big tent\? \(Related to Plugins for all\)"

On Wed, 2016-07-20 at 20:01 +, Fox, Kevin M wrote:
> I would argue Linus, yes. As he's constantly stepping up when a
> subsistem tries and breaks something for a user or creates a user
> facing mess and says, no, subsystem, no. breaking userspace is
> unacceptable, or, we're not adding support for an api we have to
> support forever thats very poorly designed.

Those are all vetos.  He doesn't compel one subsystem to accept the
patches of another, for instance.

>  Yes, he defers a lot to subsystem maintainers, as they have
> generally gotten the message of paying close attention to that kind
> of thing over time, and he hasn't had to speak up so much anymore.
> The rest really is best left up to the subsystems. But someone has to
> keep an eye on the big picture. The users of the whole thing. Users
> care about the linux kernel as a whole, and less so about any given
> subsystem.

He says "don't build this" (veto) he doesn't say "do build that"
(compulsion).  The problem I've heard a lot of people describing on
this thread is the latter: difficulty of getting one group to pay
attention to the needs of another.  Your "overarching Architectural
group with some power to define what a user is" is something like this.

The only power in Linux is the power to say "no".  The only way an
individual or a group builds acceptance for their own patches is on
their own.  Architectural decisions in this model are driven locally
not globally.

James


> Thanks,
> Kevin
> 
> From: James Bottomley [james.bottom...@hansenpartnership.com]
> Sent: Wednesday, July 20, 2016 12:42 PM
> To: OpenStack Development Mailing List (not for usage questions);
> Clint Byrum
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins
> for all)
> 
> On Wed, 2016-07-20 at 18:18 +, Fox, Kevin M wrote:
> > I wish it was so simple. Its not.
> > 
> > There is a good coding practice:
> > "The code is done, not when there is nothing more to add, but
> > nothing
> > more to remove"
> > 
> > Some of the more mature projects are in this mode of thinking now.
> > (which is mostly good, really). They don't want to add features
> > unless they see it as a benefit to their users. This is the
> > problem,
> > there is a disconnect between the view of Project X's users, and
> > greater view of OpenStack Users.
> > 
> > Even accepting the smallest of patches to work towards the goal is
> > unacceptable to projects if they believe it is not a helpful
> > feature
> > to their perceived user base, or helpful enough to them to justify
> > adding more code to their project. Or the feeling that "just push
> > it
> > to a new project or a different one is better". This fractured view
> > of OpenStack users at the project level is preventing progress on
> > OpenStack as a whole.
> > 
> > Only an overarching Architectural group with some power to define
> > what a user is, or the TC can address those sorts of issues.
> 
> I'll concede this requirement if you can point out to me who this
> group
> is for the Linux Kernel.  If you're tempted to say "Linus", it's most
> certainly not: while he does care about some architectural decisions,
> he emphatically avoids most of them, which leaves the subsystem
> maintainers (some equivalence to openstack projects/PTLs) doing this
> on
> a case by case basis.
> 
> James
> 
> > Thanks,
> > Kevin
> > 
> > From: James Bottomley [james.bottom...@hansenpartnership.com]
> > Sent: Wednesday, July 20, 2016 9:57 AM
> > To: OpenStack Development Mailing List (not for usage questions);
> > Clint Byrum
> > Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to
> > Plugins
> > for all)
> > 
> > On Wed, 2016-07-20 at 16:08 +, Fox, Kevin M wrote:
> > > +1 to the finding of a middle ground.
> > 
> > Thanks ... I have actually been an enterprise architect (I just
> > keep
> > very quiet about it when talking Open Source).
> > 
> > > The problem I've seen with your suggested OpenSource solution is
> > > the
> > > current social monetary system of OpenStack makes it extremely
> > > difficult.
> > > 
> > > Each project currently prints its own currency. Reviews. It takes
> > > quite a few Reviews (time/effort) on a project to gain enough
> > > currency that you get payed attention to. And, one project
> > > doesn't
> > > honor another projects currency.
> > 
> > OK, I accept your analogy, even though

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

And maybe this raises an interesting defininition mismatch in the conversation.

There is archetectural stuff like, do we support 7 different web frameworks, or 
do we standardize on flask... python vs go.

Theres also the architectural stuff at the, what interactive surface do you 
expose to users/operators. How consistant is it. Does it have all the features, 
no matter where they are implemented to do work.

They are somewhat related at the op level when debugging is involved.

But Im much more concerned with the latter archetecture getting attention then 
the former.

Thanks,
Kevin


From: James Bottomley
Sent: Wednesday, July 20, 2016 12:42:27 PM
To: OpenStack Development Mailing List (not for usage questions); Clint Byrum
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 18:18 +, Fox, Kevin M wrote:
> I wish it was so simple. Its not.
>
> There is a good coding practice:
> "The code is done, not when there is nothing more to add, but nothing
> more to remove"
>
> Some of the more mature projects are in this mode of thinking now.
> (which is mostly good, really). They don't want to add features
> unless they see it as a benefit to their users. This is the problem,
> there is a disconnect between the view of Project X's users, and
> greater view of OpenStack Users.
>
> Even accepting the smallest of patches to work towards the goal is
> unacceptable to projects if they believe it is not a helpful feature
> to their perceived user base, or helpful enough to them to justify
> adding more code to their project. Or the feeling that "just push it
> to a new project or a different one is better". This fractured view
> of OpenStack users at the project level is preventing progress on
> OpenStack as a whole.
>
> Only an overarching Architectural group with some power to define
> what a user is, or the TC can address those sorts of issues.

I'll concede this requirement if you can point out to me who this group
is for the Linux Kernel.  If you're tempted to say "Linus", it's most
certainly not: while he does care about some architectural decisions,
he emphatically avoids most of them, which leaves the subsystem
maintainers (some equivalence to openstack projects/PTLs) doing this on
a case by case basis.

James

> Thanks,
> Kevin
> 
> From: James Bottomley [james.bottom...@hansenpartnership.com]
> Sent: Wednesday, July 20, 2016 9:57 AM
> To: OpenStack Development Mailing List (not for usage questions);
> Clint Byrum
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins
> for all)
>
> On Wed, 2016-07-20 at 16:08 +, Fox, Kevin M wrote:
> > +1 to the finding of a middle ground.
>
> Thanks ... I have actually been an enterprise architect (I just keep
> very quiet about it when talking Open Source).
>
> > The problem I've seen with your suggested OpenSource solution is
> > the
> > current social monetary system of OpenStack makes it extremely
> > difficult.
> >
> > Each project currently prints its own currency. Reviews. It takes
> > quite a few Reviews (time/effort) on a project to gain enough
> > currency that you get payed attention to. And, one project doesn't
> > honor another projects currency.
>
> OK, I accept your analogy, even though I would view currency as the
> will to create and push patches.
>
> The problem you describe: getting the recipients to listen and accept
> your patches, is also a common one.  The first essential is simple
> minimal patches because they're hard to reject.
>
> Once you've overcome the reject barrier, there's the indifference one
> (no-one says no, but no-one says yes).
>
> Overcoming indifference involves partly knowing who to bother and
> when
> (In openstack, it's quite easy since you know who the core reviewers
> are) and also building a consensus for the patch; usually this
> involves
> finding other people who need the feature and getting them to pipe up
> (if you can't find other projects, then you can get potential users
> to
> do this) even better if they help you write the patches.  Ideally,
> you
> build your consensus before you actually push the patch set.
>  Sometimes
> building consensus involves looking beyond your particular need to a
> bigger one that would satisfy you but also pulls other people in.
>
> > When these sorts of major cross project things need to be done
> > though, you need to have folks with capital in many different
> > projects and its very difficult to amass that much.
> >
> > There is no OpenStack level currency (other then being elected as a
> > TC member) that gets one project to s

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

I would argue Linus, yes. As he's constantly stepping up when a subsistem tries 
and breaks something for a user or creates a user facing mess and says, no, 
subsystem, no. breaking userspace is unacceptable, or, we're not adding support 
for an api we have to support forever thats very poorly designed. Yes, he 
defers a lot to subsystem maintainers, as they have generally gotten the 
message of paying close attention to that kind of thing over time, and he 
hasn't had to speak up so much anymore. The rest really is best left up to the 
subsystems. But someone has to keep an eye on the big picture. The users of the 
whole thing. Users care about the linux kernel as a whole, and less so about 
any given subsystem.

Thanks,
Kevin

From: James Bottomley [james.bottom...@hansenpartnership.com]
Sent: Wednesday, July 20, 2016 12:42 PM
To: OpenStack Development Mailing List (not for usage questions); Clint Byrum
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 18:18 +, Fox, Kevin M wrote:
> I wish it was so simple. Its not.
>
> There is a good coding practice:
> "The code is done, not when there is nothing more to add, but nothing
> more to remove"
>
> Some of the more mature projects are in this mode of thinking now.
> (which is mostly good, really). They don't want to add features
> unless they see it as a benefit to their users. This is the problem,
> there is a disconnect between the view of Project X's users, and
> greater view of OpenStack Users.
>
> Even accepting the smallest of patches to work towards the goal is
> unacceptable to projects if they believe it is not a helpful feature
> to their perceived user base, or helpful enough to them to justify
> adding more code to their project. Or the feeling that "just push it
> to a new project or a different one is better". This fractured view
> of OpenStack users at the project level is preventing progress on
> OpenStack as a whole.
>
> Only an overarching Architectural group with some power to define
> what a user is, or the TC can address those sorts of issues.

I'll concede this requirement if you can point out to me who this group
is for the Linux Kernel.  If you're tempted to say "Linus", it's most
certainly not: while he does care about some architectural decisions,
he emphatically avoids most of them, which leaves the subsystem
maintainers (some equivalence to openstack projects/PTLs) doing this on
a case by case basis.

James

> Thanks,
> Kevin
> 
> From: James Bottomley [james.bottom...@hansenpartnership.com]
> Sent: Wednesday, July 20, 2016 9:57 AM
> To: OpenStack Development Mailing List (not for usage questions);
> Clint Byrum
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins
> for all)
>
> On Wed, 2016-07-20 at 16:08 +, Fox, Kevin M wrote:
> > +1 to the finding of a middle ground.
>
> Thanks ... I have actually been an enterprise architect (I just keep
> very quiet about it when talking Open Source).
>
> > The problem I've seen with your suggested OpenSource solution is
> > the
> > current social monetary system of OpenStack makes it extremely
> > difficult.
> >
> > Each project currently prints its own currency. Reviews. It takes
> > quite a few Reviews (time/effort) on a project to gain enough
> > currency that you get payed attention to. And, one project doesn't
> > honor another projects currency.
>
> OK, I accept your analogy, even though I would view currency as the
> will to create and push patches.
>
> The problem you describe: getting the recipients to listen and accept
> your patches, is also a common one.  The first essential is simple
> minimal patches because they're hard to reject.
>
> Once you've overcome the reject barrier, there's the indifference one
> (no-one says no, but no-one says yes).
>
> Overcoming indifference involves partly knowing who to bother and
> when
> (In openstack, it's quite easy since you know who the core reviewers
> are) and also building a consensus for the patch; usually this
> involves
> finding other people who need the feature and getting them to pipe up
> (if you can't find other projects, then you can get potential users
> to
> do this) even better if they help you write the patches.  Ideally,
> you
> build your consensus before you actually push the patch set.
>  Sometimes
> building consensus involves looking beyond your particular need to a
> bigger one that would satisfy you but also pulls other people in.
>
> > When these sorts of major cross project things need to be done
> > though, you need to have folks with capital in many different
> > p

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 18:18 +, Fox, Kevin M wrote:
> I wish it was so simple. Its not.
> 
> There is a good coding practice:
> "The code is done, not when there is nothing more to add, but nothing
> more to remove"
> 
> Some of the more mature projects are in this mode of thinking now.
> (which is mostly good, really). They don't want to add features
> unless they see it as a benefit to their users. This is the problem,
> there is a disconnect between the view of Project X's users, and
> greater view of OpenStack Users.
> 
> Even accepting the smallest of patches to work towards the goal is
> unacceptable to projects if they believe it is not a helpful feature
> to their perceived user base, or helpful enough to them to justify
> adding more code to their project. Or the feeling that "just push it
> to a new project or a different one is better". This fractured view
> of OpenStack users at the project level is preventing progress on
> OpenStack as a whole.
> 
> Only an overarching Architectural group with some power to define
> what a user is, or the TC can address those sorts of issues.

I'll concede this requirement if you can point out to me who this group
is for the Linux Kernel.  If you're tempted to say "Linus", it's most
certainly not: while he does care about some architectural decisions,
he emphatically avoids most of them, which leaves the subsystem
maintainers (some equivalence to openstack projects/PTLs) doing this on
a case by case basis.

James

> Thanks,
> Kevin
> 
> From: James Bottomley [james.bottom...@hansenpartnership.com]
> Sent: Wednesday, July 20, 2016 9:57 AM
> To: OpenStack Development Mailing List (not for usage questions);
> Clint Byrum
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins
> for all)
> 
> On Wed, 2016-07-20 at 16:08 +, Fox, Kevin M wrote:
> > +1 to the finding of a middle ground.
> 
> Thanks ... I have actually been an enterprise architect (I just keep
> very quiet about it when talking Open Source).
> 
> > The problem I've seen with your suggested OpenSource solution is
> > the
> > current social monetary system of OpenStack makes it extremely
> > difficult.
> > 
> > Each project currently prints its own currency. Reviews. It takes
> > quite a few Reviews (time/effort) on a project to gain enough
> > currency that you get payed attention to. And, one project doesn't
> > honor another projects currency.
> 
> OK, I accept your analogy, even though I would view currency as the
> will to create and push patches.
> 
> The problem you describe: getting the recipients to listen and accept
> your patches, is also a common one.  The first essential is simple
> minimal patches because they're hard to reject.
> 
> Once you've overcome the reject barrier, there's the indifference one
> (no-one says no, but no-one says yes).
> 
> Overcoming indifference involves partly knowing who to bother and
> when
> (In openstack, it's quite easy since you know who the core reviewers
> are) and also building a consensus for the patch; usually this
> involves
> finding other people who need the feature and getting them to pipe up
> (if you can't find other projects, then you can get potential users
> to
> do this) even better if they help you write the patches.  Ideally,
> you
> build your consensus before you actually push the patch set. 
>  Sometimes
> building consensus involves looking beyond your particular need to a
> bigger one that would satisfy you but also pulls other people in.
> 
> > When these sorts of major cross project things need to be done
> > though, you need to have folks with capital in many different
> > projects and its very difficult to amass that much.
> > 
> > There is no OpenStack level currency (other then being elected as a
> > TC member) that gets one project to stop and take the time to
> > carefully consider what someone is saying when bringing up cross
> > project issues.
> > 
> > Without some shared currency, then the problem becomes, the person
> > invested in getting a solution, can propose and prototype and
> > implement the feature all they want (often several times), but it
> > never actually is accepted into the projects because a project:
> >  a) doesn't take the time to really understand the problem, feeling
> > its trivial and just not accepting any reviews
> >  b) doesn't take the time to really understand the problem, so
> > minimize it in their mind to a 'you should just use existing thing
> > X...' or a heavy handily propose alternate solutions that really
> > aren't solutions.
> >

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 21:24 +0300, Duncan Thomas wrote:
> On 20 July 2016 at 19:57, James Bottomley <
> james.bottom...@hansenpartnership.com> wrote:
> 
> > 
> > OK, I accept your analogy, even though I would view currency as the
> > will to create and push patches.
> > 
> > The problem you describe: getting the recipients to listen and
> > accept
> > your patches, is also a common one.  The first essential is simple
> > minimal patches because they're hard to reject.
> > 
> > Once you've overcome the reject barrier, there's the indifference
> > one
> > (no-one says no, but no-one says yes).
> > 
> > [snip]
> 
> The trouble with drive-by architecture patches (or large feature 
> patches of any kind)

OK, there's an assumption here: that the patch is large.

>  is that it is often better *not* to merge them if you don't think
> the contributor is  going to stick around for a while. This changes
> are usually intrusive, and have repercussions that take time to
> discover. It's often difficult to keep a change clean when the
> original author isn't around to review the follow-on work.

I understand (and do agree with) the Maintenance problem.  However, if
you're trying to change architecture, and you do it by small patches,
it looks easily reversible and is a lot easier to gain acceptance for. 
 The key is to think small not big (the latter being the temptation for
architecture) even if the end result will eventually be big.

Let me give you an example from ancient history.  The biggest
architectural change I pushed into Linux was switching from bus
specific to generic device APIs.  I did it at the time because I had a
SCSI device in the PA-RISC architecture that could sit on a variety of
internal busses and I didn't want a huge driver with five sets of
partially duplicated code for five different busses.

Here's the first patch (both dated 21 Dec 2002)

generic device DMA API

27 files changed, 750 insertions(+), 155 deletions(-)

And the second

allow pci primary busses to have parents in the device model

2 files changed, 14 insertions(+), 5 deletions(-)

The first is large because I'm actually introducing yet another
parallel DMA API (50% of it is documentation).  It looks OK because all
our other DMA APIs were bus specific.

The second is the key change because it allows cascades of bus
hierarchies of different types.

These two patches are easy to gain acceptance for because they're just
introducing new APIs and not altering existing stuff.  They're small
and easily revertible if I happened to disappear (or they proved not to
work out in practice).  They were, however, enough to allow me to
convert the PA-RISC architecture to the new model and write the driver
I wanted for the 53c700 chip.

The end point we're at today, where practically every device API in
Linux uses generic devices, is no longer revertible.  If it were done
as one patch, it would touch about 80% of all the files in Linux and be
larger than the kernel itself.  However, I didn't do that: I pushed the
minimum viable patch set that would support what I wanted to do (plus
the use cases for a couple of other interested parties I picked up
along the way).

James

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-20 Thread Joshua Harlow


Duncan Thomas wrote:



On 20 July 2016 at 19:57, James Bottomley
> wrote:


OK, I accept your analogy, even though I would view currency as the
will to create and push patches.

The problem you describe: getting the recipients to listen and accept
your patches, is also a common one.  The first essential is simple
minimal patches because they're hard to reject.

Once you've overcome the reject barrier, there's the indifference one
(no-one says no, but no-one says yes).

[snip]

The trouble with drive-by architecture patches (or large feature patches
of any kind) is that it is often better *not* to merge them if you don't
think the contributor is  going to stick around for a while. This
changes are usually intrusive, and have repercussions that take time to
discover. It's often difficult to keep a change clean when the original
author isn't around to review the follow-on work.


Agreed, and knowing where yahoo and HP(e) are at right now (with regards 
to openstack and ...) these kind of things are a little more prevalent 
(with regards to quota, tasks...) now-a-days (for better or worse); not 
how I want it to be but it's reality.


Which I guess is why I'd be nice to have cross-project architecture 
'standardization' (? for lack of better word) with a more long term 
strategic vision (vs localized tactical visions). Such things are 
obviously not hard to get going (and are equally hard to sustain).




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-20 Thread Duncan Thomas

On 20 July 2016 at 19:57, James Bottomley <
james.bottom...@hansenpartnership.com> wrote:

>
> OK, I accept your analogy, even though I would view currency as the
> will to create and push patches.
>
> The problem you describe: getting the recipients to listen and accept
> your patches, is also a common one.  The first essential is simple
> minimal patches because they're hard to reject.
>
> Once you've overcome the reject barrier, there's the indifference one
> (no-one says no, but no-one says yes).
>
> [snip]

The trouble with drive-by architecture patches (or large feature patches of
any kind) is that it is often better *not* to merge them if you don't think
the contributor is  going to stick around for a while. This changes are
usually intrusive, and have repercussions that take time to discover. It's
often difficult to keep a change clean when the original author isn't
around to review the follow-on work.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

I wish it was so simple. Its not.

There is a good coding practice:
"The code is done, not when there is nothing more to add, but nothing more to 
remove"

Some of the more mature projects are in this mode of thinking now. (which is 
mostly good, really). They don't want to add features unless they see it as a 
benefit to their users. This is the problem, there is a disconnect between the 
view of Project X's users, and greater view of OpenStack Users.

Even accepting the smallest of patches to work towards the goal is unacceptable 
to projects if they believe it is not a helpful feature to their perceived user 
base, or helpful enough to them to justify adding more code to their project. 
Or the feeling that "just push it to a new project or a different one is 
better". This fractured view of OpenStack users at the project level is 
preventing progress on OpenStack as a whole.

Only an overarching Architectural group with some power to define what a user 
is, or the TC can address those sorts of issues.

Thanks,
Kevin

From: James Bottomley [james.bottom...@hansenpartnership.com]
Sent: Wednesday, July 20, 2016 9:57 AM
To: OpenStack Development Mailing List (not for usage questions); Clint Byrum
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 16:08 +, Fox, Kevin M wrote:
> +1 to the finding of a middle ground.

Thanks ... I have actually been an enterprise architect (I just keep
very quiet about it when talking Open Source).

> The problem I've seen with your suggested OpenSource solution is the
> current social monetary system of OpenStack makes it extremely
> difficult.
>
> Each project currently prints its own currency. Reviews. It takes
> quite a few Reviews (time/effort) on a project to gain enough
> currency that you get payed attention to. And, one project doesn't
> honor another projects currency.

OK, I accept your analogy, even though I would view currency as the
will to create and push patches.

The problem you describe: getting the recipients to listen and accept
your patches, is also a common one.  The first essential is simple
minimal patches because they're hard to reject.

Once you've overcome the reject barrier, there's the indifference one
(no-one says no, but no-one says yes).

Overcoming indifference involves partly knowing who to bother and when
(In openstack, it's quite easy since you know who the core reviewers
are) and also building a consensus for the patch; usually this involves
finding other people who need the feature and getting them to pipe up
(if you can't find other projects, then you can get potential users to
do this) even better if they help you write the patches.  Ideally, you
build your consensus before you actually push the patch set.  Sometimes
building consensus involves looking beyond your particular need to a
bigger one that would satisfy you but also pulls other people in.

> When these sorts of major cross project things need to be done
> though, you need to have folks with capital in many different
> projects and its very difficult to amass that much.
>
> There is no OpenStack level currency (other then being elected as a
> TC member) that gets one project to stop and take the time to
> carefully consider what someone is saying when bringing up cross
> project issues.
>
> Without some shared currency, then the problem becomes, the person
> invested in getting a solution, can propose and prototype and
> implement the feature all they want (often several times), but it
> never actually is accepted into the projects because a project:
>  a) doesn't take the time to really understand the problem, feeling
> its trivial and just not accepting any reviews
>  b) doesn't take the time to really understand the problem, so
> minimize it in their mind to a 'you should just use existing thing
> X...' or a heavy handily propose alternate solutions that really
> aren't solutions.
>  c) they decide its better handled by another project and stall/block
> reviews, trying to push the implementation to go elsewhere. When
> multiple projects decide this, the ball just keeps getting bounced
> around without any solution for years.
>
> The status quo is not working here. The current governance structure
> doesn't support progress.

What you'll find you've described above is a process that doesn't
support drive by coders at all and, by extension therefore, doesn't
welcome newcomers, which is a big problem, but one I thought OpenStack
was tackling?

The bounce problem is annoying but not insuperable.  It mostly occurs
where there's overlap.  Often the best method for coping is to field
the bounce: produce the patch for the other project.  If it's smaller
and neater, perhaps the bounce was correct.  If it's bigger and uglier,
get the other project to say so and you

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-20 Thread Clint Byrum

Excerpts from James Bottomley's message of 2016-07-20 08:31:34 -0700:
> So this is where the Open Source method takes over.  Change is produced
> by those people who most care about it because they're invested.  To
> take your Cinder example, you're unlikely to find them within Cinder
> because any project has inertia that resists change.  It takes the
> energy of the outside group X to force the change to Y, but this means
> that X often gets to propose, develop and even code Y.  Essentially
> they become drive by coders for Cinder.  This is where Open Source
> differs from Enterprise because you have the code and the access, you
> can do this.  However, you have to remember the inertia problem and
> build what you're trying to do as incrementally as possible: the larger
> the change, the bigger the resistance to it.  It's also a good test of
> the value of the change: if group X can't really be bothered to code it
> (and Cinder doesn't want it) then perhaps there's not really enough
> value in it anyway and it shouldn't happen.
> 

Thanks so much for stating this James. I couldn't agree more. A group
that can actually _accomplish_ change, and not just suggest it, is
exactly what we're working to start with the architecture WG. There are
plenty of people with the will to change, and I feel strongly that if
those people are given a structure and support, then they'll find the
time and space to complete these objectives.

I just want to make one nuance point about Cinder changes: the recent
DLM work, done outside any architecture working group, did actually come
from both inside and outside Cinder. The Cinder team realized something
was happening that would perhaps affect everyone, and raised it to the
cross-project level, which helped external individuals get involved. So
these initiatives can come from either direction. The key is that enough
momentum builds up to counter the natural inertia that you mentioned in
your message.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 16:08 +, Fox, Kevin M wrote:
> +1 to the finding of a middle ground.

Thanks ... I have actually been an enterprise architect (I just keep
very quiet about it when talking Open Source).

> The problem I've seen with your suggested OpenSource solution is the
> current social monetary system of OpenStack makes it extremely
> difficult.
> 
> Each project currently prints its own currency. Reviews. It takes
> quite a few Reviews (time/effort) on a project to gain enough
> currency that you get payed attention to. And, one project doesn't
> honor another projects currency.

OK, I accept your analogy, even though I would view currency as the
will to create and push patches.

The problem you describe: getting the recipients to listen and accept
your patches, is also a common one.  The first essential is simple
minimal patches because they're hard to reject.

Once you've overcome the reject barrier, there's the indifference one
(no-one says no, but no-one says yes).

Overcoming indifference involves partly knowing who to bother and when
(In openstack, it's quite easy since you know who the core reviewers
are) and also building a consensus for the patch; usually this involves
finding other people who need the feature and getting them to pipe up
(if you can't find other projects, then you can get potential users to
do this) even better if they help you write the patches.  Ideally, you
build your consensus before you actually push the patch set.  Sometimes
building consensus involves looking beyond your particular need to a
bigger one that would satisfy you but also pulls other people in.

> When these sorts of major cross project things need to be done
> though, you need to have folks with capital in many different
> projects and its very difficult to amass that much.
> 
> There is no OpenStack level currency (other then being elected as a
> TC member) that gets one project to stop and take the time to
> carefully consider what someone is saying when bringing up cross
> project issues.
> 
> Without some shared currency, then the problem becomes, the person
> invested in getting a solution, can propose and prototype and
> implement the feature all they want (often several times), but it
> never actually is accepted into the projects because a project:
>  a) doesn't take the time to really understand the problem, feeling
> its trivial and just not accepting any reviews
>  b) doesn't take the time to really understand the problem, so
> minimize it in their mind to a 'you should just use existing thing
> X...' or a heavy handily propose alternate solutions that really
> aren't solutions.
>  c) they decide its better handled by another project and stall/block
> reviews, trying to push the implementation to go elsewhere. When
> multiple projects decide this, the ball just keeps getting bounced
> around without any solution for years.
> 
> The status quo is not working here. The current governance structure
> doesn't support progress.

What you'll find you've described above is a process that doesn't
support drive by coders at all and, by extension therefore, doesn't
welcome newcomers, which is a big problem, but one I thought OpenStack
was tackling?

The bounce problem is annoying but not insuperable.  It mostly occurs
where there's overlap.  Often the best method for coping is to field
the bounce: produce the patch for the other project.  If it's smaller
and neater, perhaps the bounce was correct.  If it's bigger and uglier,
get the other project to say so and you now have a solid reason to go
back and say "we tried what you suggested and here's why it didn't
work".  Morally, you're now on higher ground because incorrect
technical advice is a personal failure for whoever bounced you (make
sure to capitalise on it if they try another bounce).

James

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

+1 to the finding of a middle ground.

The problem I've seen with your suggested OpenSource solution is the current 
social monetary system of OpenStack makes it extremely difficult.

Each project currently prints its own currency. Reviews. It takes quite a few 
Reviews (time/effort) on a project to gain enough currency that you get payed 
attention to. And, one project doesn't honor another projects currency.

When these sorts of major cross project things need to be done though, you need 
to have folks with capital in many different projects and its very difficult to 
amass that much.

There is no OpenStack level currency (other then being elected as a TC member) 
that gets one project to stop and take the time to carefully consider what 
someone is saying when bringing up cross project issues.

Without some shared currency, then the problem becomes, the person invested in 
getting a solution, can propose and prototype and implement the feature all 
they want (often several times), but it never actually is accepted into the 
projects because a project:
 a) doesn't take the time to really understand the problem, feeling its trivial 
and just not accepting any reviews
 b) doesn't take the time to really understand the problem, so minimize it in 
their mind to a 'you should just use existing thing X...' or a heavy handily 
propose alternate solutions that really aren't solutions.
 c) they decide its better handled by another project and stall/block reviews, 
trying to push the implementation to go elsewhere. When multiple projects 
decide this, the ball just keeps getting bounced around without any solution 
for years.

The status quo is not working here. The current governance structure doesn't 
support progress.

Thanks,
Kevin

From: James Bottomley [james.bottom...@hansenpartnership.com]
Sent: Wednesday, July 20, 2016 8:31 AM
To: OpenStack Development Mailing List (not for usage questions); Clint Byrum
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 11:58 +0200, Julien Danjou wrote:
> On Tue, Jul 19 2016, Clint Byrum wrote:
>
> > Perhaps if we form and start working together as a group, we can
> > disect why nothing happened, build consensus on the most important
> > thing to do next, and actually fix some architectural problems.

Architecture is often blamed for lack of interlock but most people
forget that the need for interlock often isn't appreciated until after
the components are built.  This is why a lot of people embrace agile
methodology (an excuse for not seeing the problem a priori).

Conversely, architects who try to foresee all interlocks often end up
with a combinatorial explosion that makes it a huge burden simply to
begin the project (and then often get sidelined as 'powerpoint
engineers').

The trick is to do something in the middle: try to foresee and build
the most common interlocks, but sidestep the combinatorial explosion by
building something simple enough to adapt to any interlock requirement
that arises after completion.

> >  The social structure that teams have is a huge part of the
> > deadlock we find ourselves in with certain controversial changes.
> > The idea is to unroll the dependency loop and start _somewhere_
> > rather than where a lot of these efforts die: starting
> > _everywhere_.
>
> I agree with your analysis, but I fail to see how e.g. a group of
> people X stating that Y should work this way in Cinder is going to
> achieve any change if nobody from Cinder is in X from the beginning.
>
> This is basically what seems to happen in many [working] groups as
> far as I can see.

So this is where the Open Source method takes over.  Change is produced
by those people who most care about it because they're invested.  To
take your Cinder example, you're unlikely to find them within Cinder
because any project has inertia that resists change.  It takes the
energy of the outside group X to force the change to Y, but this means
that X often gets to propose, develop and even code Y.  Essentially
they become drive by coders for Cinder.  This is where Open Source
differs from Enterprise because you have the code and the access, you
can do this.  However, you have to remember the inertia problem and
build what you're trying to do as incrementally as possible: the larger
the change, the bigger the resistance to it.  It's also a good test of
the value of the change: if group X can't really be bothered to code it
(and Cinder doesn't want it) then perhaps there's not really enough
value in it anyway and it shouldn't happen.

This latter observation is also an improvement over the enterprise
methods because enterprise architects do often mandate interlocks that
later turn out to be unnecessary (or at least of a lot less value than
was initially thought).

I suppose the executive summary of the above is that I don't

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Wed, 2016-07-20 at 11:58 +0200, Julien Danjou wrote:
> On Tue, Jul 19 2016, Clint Byrum wrote:
> 
> > Perhaps if we form and start working together as a group, we can 
> > disect why nothing happened, build consensus on the most important 
> > thing to do next, and actually fix some architectural problems.

Architecture is often blamed for lack of interlock but most people
forget that the need for interlock often isn't appreciated until after
the components are built.  This is why a lot of people embrace agile
methodology (an excuse for not seeing the problem a priori).

Conversely, architects who try to foresee all interlocks often end up
with a combinatorial explosion that makes it a huge burden simply to
begin the project (and then often get sidelined as 'powerpoint
engineers').

The trick is to do something in the middle: try to foresee and build
the most common interlocks, but sidestep the combinatorial explosion by
building something simple enough to adapt to any interlock requirement
that arises after completion.

> >  The social structure that teams have is a huge part of the
> > deadlock we find ourselves in with certain controversial changes.
> > The idea is to unroll the dependency loop and start _somewhere_
> > rather than where a lot of these efforts die: starting
> > _everywhere_.
> 
> I agree with your analysis, but I fail to see how e.g. a group of 
> people X stating that Y should work this way in Cinder is going to 
> achieve any change if nobody from Cinder is in X from the beginning.
> 
> This is basically what seems to happen in many [working] groups as 
> far as I can see.

So this is where the Open Source method takes over.  Change is produced
by those people who most care about it because they're invested.  To
take your Cinder example, you're unlikely to find them within Cinder
because any project has inertia that resists change.  It takes the
energy of the outside group X to force the change to Y, but this means
that X often gets to propose, develop and even code Y.  Essentially
they become drive by coders for Cinder.  This is where Open Source
differs from Enterprise because you have the code and the access, you
can do this.  However, you have to remember the inertia problem and
build what you're trying to do as incrementally as possible: the larger
the change, the bigger the resistance to it.  It's also a good test of
the value of the change: if group X can't really be bothered to code it
(and Cinder doesn't want it) then perhaps there's not really enough
value in it anyway and it shouldn't happen.

This latter observation is also an improvement over the enterprise
methods because enterprise architects do often mandate interlocks that
later turn out to be unnecessary (or at least of a lot less value than
was initially thought).

I suppose the executive summary of the above is that I don't think
you're describing a bug, I think you're describing a feature.

James

signature.asc
Description: This is a digitally signed message part
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-20 Thread Julien Danjou

On Tue, Jul 19 2016, Clint Byrum wrote:

> Perhaps if we form and start working together as a group, we can disect
> why nothing happened, build consensus on the most important thing to do
> next, and actually fix some architectural problems. The social structure
> that teams have is a huge part of the deadlock we find ourselves in
> with certain controversial changes. The idea is to unroll the dependency
> loop and start _somewhere_ rather than where a lot of these efforts die:
> starting _everywhere_.

I agree with your analysis, but I fail to see how e.g. a group of people
X stating that Y should work this way in Cinder is going to achieve any
change if nobody from Cinder is in X from the beginning.

This is basically what seems to happen in many [working] groups as far
as I can see.

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-20 Thread Thierry Carrez

Clint Byrum wrote:

[...]
But what I thought what the TC's job was, was benevolent dictators, which each subproject (or subsystem in linux terms)
are required to give up final say to, so that sometimes the projects have to sacrifice a bit so that the whole can
flourish and those benevolent dictators are elected for a time, by the OpenStack community. (Actually, I think that
kind of makes it a Democratic Republic... but I digress) Maybe I misunderstood what the TC's about. But I think we
still do need some folks elected by the community to be involved in making sure OpenStack as a whole has a cohesive
technical architecture that actually addresses user problems and that have some power to to stop the "this feature
belongs in this project", "no, it belongs in that project", "no, lets spawn 3 new projects to cover
that case" kinds of things, make the difficult decision, and ask a project to help the community out by going
along with "a solution" and we all can move on. Critical features have been stu

in this space for years and OpenStacks competitors have had solutions for
years.

You're right, this is the TC's job. However, the TC does it more by
exception, rather than by default. So while Linus and the subsystem
leaders in the kernel look after changes in general, the TC is there to
catch things that bubble out of the processes in place. So, I think the
TC needs contributors to bring _specific_ things that need to be handled,
and they will. They're just not going to be able to stand at the gate
and review every spec... this process only scales to the velocity and
breadth that OpenStack has if we get contributors involved.

It's always a balancing act -- we want the TC to keep an eye on the big
picture, provide guidance, encourage convergence, and stop the bucket
when needed. At the same time, developers don't want a set of people
that do not have specific experience in a their project to constantly
interfere with project development with random mandates.

However, I'd agree that the pendulum currently is still in the "not
enough" territory rather than in the "too much" territory. We have a
number of initiatives brewing though (clarifying principles, defining
release goals, release stewards, architecture WG, PTG event...) which I
think will improve things in the right direction, let's see how those
pan out. Please be patient while we are rolling those out.

So in summary: yes there still are issues, but it is not a simple
problem, and we are working on it.

--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Clint Byrum

Excerpts from Fox, Kevin M's message of 2016-07-19 21:59:29 +:
> Yeah. I'm not saying any project has done it out of malice. Everyone's just 
> doing whats best for their project. But it does not seem like there is an 
> overarching entity watching over the whole or (pushing, encouraging, 
> enticing, whatever words are appropriate here) projects to work on the 
> commons anymore. It use to be that incubating projects were pushed to help 
> the other projects integrate with them by the incubating project being 
> strongly encouraged to write the integrations themselves as part of the 
> incubation process. Now it seems like each project just spawns and then hopes 
> someone else will do the legwork. Using the carrot of incubation to further 
> the commons is not an ideal solution, but it was at least something.
> 
> Linux has an overarching entity, Linus for that task. He's there to make sure 
> that someone is really paying attention to integration of the whole thing 
> into a cohesive, usable whole. Sometimes he pushes back and makes sure 
> commons work happens as part of features getting in to ensure commons work 
> still gets done. I'm not advocating a benevolent dictator for OpenStack 
> though.
> 
> But what I thought what the TC's job was, was benevolent dictators, which 
> each subproject (or subsystem in linux terms) are required to give up final 
> say to, so that sometimes the projects have to sacrifice a bit so that the 
> whole can flourish and those benevolent dictators are elected for a time, by 
> the OpenStack community. (Actually, I think  that kind of makes it a 
> Democratic Republic... but I digress) Maybe I misunderstood what the TC's 
> about. But I think we still do need some folks elected by the community to be 
> involved in making sure OpenStack as a whole has a cohesive technical 
> architecture that actually addresses user problems and that have some power 
> to to stop the "this feature belongs in this project", "no, it belongs in 
> that project", "no, lets spawn 3 new projects to cover that case" kinds of 
> things, make the difficult decision, and ask a project to help the community 
> out by going along with "a solution" and we all can move on. Critical 
> features have been stuck 
>  in this space for years and OpenStacks competitors have had solutions for 
> years. 

You're right, this is the TC's job. However, the TC does it more by
exception, rather than by default. So while Linus and the subsystem
leaders in the kernel look after changes in general, the TC is there to
catch things that bubble out of the processes in place. So, I think the
TC needs contributors to bring _specific_ things that need to be handled,
and they will. They're just not going to be able to stand at the gate
and review every spec... this process only scales to the velocity and
breadth that OpenStack has if we get contributors involved.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Fox, Kevin M

Yeah. I'm not saying any project has done it out of malice. Everyone's just 
doing whats best for their project. But it does not seem like there is an 
overarching entity watching over the whole or (pushing, encouraging, enticing, 
whatever words are appropriate here) projects to work on the commons anymore. 
It use to be that incubating projects were pushed to help the other projects 
integrate with them by the incubating project being strongly encouraged to 
write the integrations themselves as part of the incubation process. Now it 
seems like each project just spawns and then hopes someone else will do the 
legwork. Using the carrot of incubation to further the commons is not an ideal 
solution, but it was at least something.

Linux has an overarching entity, Linus for that task. He's there to make sure 
that someone is really paying attention to integration of the whole thing into 
a cohesive, usable whole. Sometimes he pushes back and makes sure commons work 
happens as part of features getting in to ensure commons work still gets done. 
I'm not advocating a benevolent dictator for OpenStack though.

But what I thought what the TC's job was, was benevolent dictators, which each 
subproject (or subsystem in linux terms) are required to give up final say to, 
so that sometimes the projects have to sacrifice a bit so that the whole can 
flourish and those benevolent dictators are elected for a time, by the 
OpenStack community. (Actually, I think  that kind of makes it a Democratic 
Republic... but I digress) Maybe I misunderstood what the TC's about. But I 
think we still do need some folks elected by the community to be involved in 
making sure OpenStack as a whole has a cohesive technical architecture that 
actually addresses user problems and that have some power to to stop the "this 
feature belongs in this project", "no, it belongs in that project", "no, lets 
spawn 3 new projects to cover that case" kinds of things, make the difficult 
decision, and ask a project to help the community out by going along with "a 
solution" and we all can move on. Critical features have been stuck in this 
 space for years and OpenStacks competitors have had solutions for years.

Interest in OpenStack as a whole has leveled off in about 2014, while interest 
in other clouds have continued to grow:
https://www.google.com/trends/explore#q=%2Fm%2F0cm87w_%2C%20%2Fm%2F05nrgx%2C%20%2Fm%2F04y7lrx=q=Etc%2FGMT%2B7

If the trend of OpenStack as a whole is not rising, we need to figure out why 
and get the whole moving again. I think thats mostly the projects keeping their 
heads down and working on what they work on, and not much focus on the commons, 
or helping OpenStack as a whole get better, other then what just happens as a 
side effect of each project gaining features.

We need to figure out how to fix this.

Thanks,
Kevin

From: Zane Bitter [zbit...@redhat.com]
Sent: Tuesday, July 19, 2016 1:08 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On 14/07/16 16:30, Fox, Kevin M wrote:
> I'm going to go ahead and ask the difficult question now as the answer is 
> relevant to the attached proposal...
>
> Should we reconsider whether the big tent is the right approach going forward?
>
> There have been some major downsides I think to the Big Tent approach, such 
> as:
>  * Projects not working together because of fear of adding extra dependencies 
> Ops don't already have
>  * Reimplementing features, badly, over and over again in different projects 
> instead of standardizing something.
>  * More projects being created due to politics and not technical reasons.
>  * Less cross project communication
>  * Greater Operator pain at trying to assemble a more loose confederation of 
> projects into something useful.
>  * General pushing off more and more work to Operators/Users to deal with.
>  * Worse User experience as cross project issues aren't being addressed.
>  * Previously incubated projects such as Nova, Swift, etc getting a 
> disproportionate say in things as they have a greater current user base, and 
> its increasingly hard now for new projects to gain any traction.
>  * Much less community pressure on projects to work together to elevate 
> everyone. Architectural decisions are now made at individual project level 
> with little done at the OpenStack level.
>  * etc...

The thing is, all of these problems pre-date the big tent. Arguably they
were even worse before because so many projects were not only not
blessed by the TC as hard dependencies, but officially weren't even part
of OpenStack. Say what you want about the big tent, but at least we
haven't been treated to another Zaqar graduation review farce.

I think your complaint here is that the big tent hasn't done enough to
solve these pr

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Fox, Kevin M

Yeah. I'm not saying any project has done it out of malice. Everyone's just 
doing whats best for their project. But it does not seem like there is an 
overarching entity watching over the whole or (pushing, encouraging, enticing, 
whatever words are appropriate here) projects to work on the commons anymore. 
It use to be that incubating projects were pushed to help the other projects 
integrate with them by the incubating project being strongly encouraged to 
write the integrations themselves as part of the incubation process. Now it 
seems like each project just spawns and then hopes someone else will do the 
legwork. Using the carrot of incubation to further the commons is not an ideal 
solution, but it was at least something.

Linux has an overarching entity, Linus for that task. He's there to make sure 
that someone is really paying attention to integration of the whole thing into 
a cohesive, usable whole. Sometimes he pushes back and makes sure commons work 
happens as part of features getting in to ensure commons work still gets done. 
I'm not advocating a benevolent dictator for OpenStack though.

But what I thought what the TC's job was, was benevolent dictators, which each 
subproject (or subsystem in linux terms) are required to give up final say to, 
so that sometimes the projects have to sacrifice a bit so that the whole can 
flourish and those benevolent dictators are elected for a time, by the 
OpenStack community. (Actually, I think  that kind of makes it a Democratic 
Republic... but I digress) Maybe I misunderstood what the TC's about. But I 
think we still do need some folks elected by the community to be involved in 
making sure OpenStack as a whole has a cohesive technical architecture that 
actually addresses user problems and that have some power to to stop the "this 
feature belongs in this project", "no, it belongs in that project", "no, lets 
spawn 3 new projects to cover that case" kinds of things, make the difficult 
decision, and ask a project to help the community out by going along with "a 
solution" and we all can move on. Critical features have been stuck in this 
 space for years and OpenStacks competitors have had solutions for years. 

Interest in OpenStack as a whole has leveled off in about:

From: Zane Bitter [zbit...@redhat.com]
Sent: Tuesday, July 19, 2016 1:08 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On 14/07/16 16:30, Fox, Kevin M wrote:
> I'm going to go ahead and ask the difficult question now as the answer is 
> relevant to the attached proposal...
>
> Should we reconsider whether the big tent is the right approach going forward?
>
> There have been some major downsides I think to the Big Tent approach, such 
> as:
>  * Projects not working together because of fear of adding extra dependencies 
> Ops don't already have
>  * Reimplementing features, badly, over and over again in different projects 
> instead of standardizing something.
>  * More projects being created due to politics and not technical reasons.
>  * Less cross project communication
>  * Greater Operator pain at trying to assemble a more loose confederation of 
> projects into something useful.
>  * General pushing off more and more work to Operators/Users to deal with.
>  * Worse User experience as cross project issues aren't being addressed.
>  * Previously incubated projects such as Nova, Swift, etc getting a 
> disproportionate say in things as they have a greater current user base, and 
> its increasingly hard now for new projects to gain any traction.
>  * Much less community pressure on projects to work together to elevate 
> everyone. Architectural decisions are now made at individual project level 
> with little done at the OpenStack level.
>  * etc...

The thing is, all of these problems pre-date the big tent. Arguably they
were even worse before because so many projects were not only not
blessed by the TC as hard dependencies, but officially weren't even part
of OpenStack. Say what you want about the big tent, but at least we
haven't been treated to another Zaqar graduation review farce.

I think your complaint here is that the big tent hasn't done enough to
solve these problems. That's a fair complaint.

I think what you're suggesting is missing is some sort of TC imprimatur
to say that it's acceptable to take a hard dependency on a particular
project, which is effectively what graduation from incubation meant
previously (e.g. distributors were effectively, though not actually,
obliged to package it at this point if they weren't already). Of course
the only thing that can really _force_ people to adopt a project is
DefCore, and that comes with a major chicken-and-egg dilemma. It's true
that we've hollowed out most of the levels between just being "o

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Zane Bitter


On 14/07/16 16:30, Fox, Kevin M wrote:

I'm going to go ahead and ask the difficult question now as the answer is 
relevant to the attached proposal...

Should we reconsider whether the big tent is the right approach going forward?

There have been some major downsides I think to the Big Tent approach, such as:
 * Projects not working together because of fear of adding extra dependencies 
Ops don't already have
 * Reimplementing features, badly, over and over again in different projects 
instead of standardizing something.
 * More projects being created due to politics and not technical reasons.
 * Less cross project communication
 * Greater Operator pain at trying to assemble a more loose confederation of 
projects into something useful.
 * General pushing off more and more work to Operators/Users to deal with.
 * Worse User experience as cross project issues aren't being addressed.
 * Previously incubated projects such as Nova, Swift, etc getting a 
disproportionate say in things as they have a greater current user base, and 
its increasingly hard now for new projects to gain any traction.
 * Much less community pressure on projects to work together to elevate 
everyone. Architectural decisions are now made at individual project level with 
little done at the OpenStack level.
 * etc...


The thing is, all of these problems pre-date the big tent. Arguably they 
were even worse before because so many projects were not only not 
blessed by the TC as hard dependencies, but officially weren't even part 
of OpenStack. Say what you want about the big tent, but at least we 
haven't been treated to another Zaqar graduation review farce.


I think your complaint here is that the big tent hasn't done enough to 
solve these problems. That's a fair complaint.


I think what you're suggesting is missing is some sort of TC imprimatur 
to say that it's acceptable to take a hard dependency on a particular 
project, which is effectively what graduation from incubation meant 
previously (e.g. distributors were effectively, though not actually, 
obliged to package it at this point if they weren't already). Of course 
the only thing that can really _force_ people to adopt a project is 
DefCore, and that comes with a major chicken-and-egg dilemma. It's true 
that we've hollowed out most of the levels between just being "one of 
us" and being DefCore-approved.


I worried about this myself at the time the big tent was proposed. But I 
don't think it's a silver bullet - developers of new projects have 
always been reluctant to take hard dependencies (source: first-hand 
experience starting in 2012). It's *hard* to get operators to adopt your 
project. To get operators to adopt your project plus some other project 
is... well it's definitely never easier.


That said, I don't think any of the projects you mentioned elsewhere 
(e.g. the 4 with the own implementations of guest agents) are being 
deliberately intransigent. They're each just trying to find a 
good-enough solution in isolation. If a standard way of doing guest 
agents had existed before they started then I'm confident they (we) all 
would have used it, and if one cropped up now I hope none of them would 
resist efforts to at least support it as an option (i.e. with only soft 
dependencies).


The thing that's not happening, and has never happened, is that nobody 
is getting those groups together and trying to come up with a common 
standard that everything can move toward. I think Clint's proposed 
architecture working group is a step in the right direction here. I 
don't know if such questions, which in a sense concern the user-visible 
architecture as much as the backend architecture, would be considered in 
scope (or high-priority) by that group. It's possible we might need a 
separate working group to address the needs of what I've been calling 
autonomous applications (i.e. cloud-resident applications that use the 
cloud APIs to control their own infrastructure). But I'm reluctant to 
formally propose such a thing until we have had a chance to let the 
architecture group pioneer the territory :)


cheers,
Zane.


The overall goal of the Big Tent was to make the community more inclusive. That 
I think has mostly happened. Which is good. But It also seems to have fractured 
the community more into insular islands and made the system harder for our 
operators/users. Which is bad.

Maybe the benefits outweigh the drawbacks. I'm not sure. But I think its 
probably time to consider if it has been a net positive and should be further 
refined to try and address some of these problems, or a net negative and 
different approaches be explored.

Thanks,
Kevin



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Clint Byrum

Excerpts from Julien Danjou's message of 2016-07-19 09:30:36 +0200:
> On Mon, Jul 18 2016, Joshua Harlow wrote:
> 
> > Thus why I think the starting of the architecture working group is a good
> > thing; because I have a believe that people are forgetting among all of this
> > that such a group holds a lot of the keys to the kingdom (whether u, the
> > reader, want to admit that or not is well the readers problem) in openstack
> > (sorry and no disrespect to independent folks & contributors), but most of 
> > us
> > work for large companies that have architects (and leads) and if those
> > architects (and leads) can get together cross-company to aggregate and 
> > (agree
> > on) and solve actual problems then that really is IMHO the only way to keep 
> > our
> > projects healthy (assuming we can even do that at this stage).
> 
> I think it is a bit naive to think any working group is going to fix
> architectural problems. You know first hand what happened¹ with the Nova
> service group and tooz for example.
> 

Perhaps if we form and start working together as a group, we can disect
why nothing happened, build consensus on the most important thing to do
next, and actually fix some architectural problems. The social structure
that teams have is a huge part of the deadlock we find ourselves in
with certain controversial changes. The idea is to unroll the dependency
loop and start _somewhere_ rather than where a lot of these efforts die:
starting _everywhere_.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Fox, Kevin M

+1.

I think there have been a few efforts around the issue of the small tent thing 
described below, such as the http://www.openstack.org/brand/interop/ effort.

But those profiles seemed centred around standardizing the minimal common 
denominator parts of what's widely deployed today rather then coming up with a 
new standard for the pieces that should be required for a next generation 
iteration of the system. Without the latter, we just keep the same old stuff in 
the small tent, and no new stuff gets added.

Thanks,
Kevin

From: Ed Leafe [e...@leafe.com]
Sent: Monday, July 18, 2016 9:59 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Jul 18, 2016, at 12:39 PM, Fox, Kevin M <kevin@pnnl.gov> wrote:

> I'm arguing the opposite. It should be a requirement to use the OpenStack 
> Secret Store.
>
> Instead, we're trying to make it optional and either reimplementing large 
> swaths of it in individual projects to make it optional, or skip security all 
> together.
>
> If it wasn't optional, then:
> * Keystone could rely on it being there for password hashing
> * Magnum can rely on it for security.
> * lbaas can rely on it
> * App developers can rely on it being there
> * ...
>
> Same with Zaqar. It was a requirement then,
> * Guest agents of heat, sahara, mangum, trove, etc could all rely on it being 
> there and wouldn't have to deal with workarounds
> * Horizon could rely on it being there for dynamic ui
> * App developers can rely on it being there.
>
> Yes, it seems like more work for an operator to have to install "one more 
> thing", but the savings it provides by not having to do that "one more thing" 
> in 7 different projects pays for it.

There is a lot to be said for code simplification. When Glance split out of 
Nova, the knowledge that Glance would be required to be there meant the the 
code in Nova for interacting with it didn’t have to contain tons of ‘if :  else ”. Knowing that that code was 
unnecessary made writing the interaction much simpler (and less tedious to 
read). So if we added, say, Barbican as a requirement, all of the other 
projects that need to handle secrets (a lot of them!) could simply program to 
the Barbican interface, and be done with it.

The discussion of whether project X should be required is a whole ‘nother 
topic. But I want to emphasize that often the utility of having a standard is 
grossly underestimated.

In this discussion, the issue that seems to be arising is that the Big Tent 
doesn’t provide for a path for a project to be reconsidered as an OpenStack 
standard, in the way that Nova and Glance and Swift are. The Small Tent 
“incubation” process designed to identify which projects were mature enough to 
be considered “one of us”, when “us” was the group of core projects. This was 
unnecessarily harsh, and pushed away many worthy efforts. In an attempt to 
improve this, the definition of “one of us” was broadened so that lots of 
interesting projects could be part of the community.

What I think is causing some confusion (and friction) is that OpenStack needs 
two “us”s: what projects are part of our ecosystem and build software in the 
same open way (the Big Tent concept), and which are a fundamental component of 
the overall OpenStack software, which all other projects can assume are 
available (the Small Tent concept).

To that end, it seems necessary to define a migration path for a project to 
become required. Of course, most will never need to be required, but I think a 
case could be made that several projects (Barbican and Zaqar are the obvious 
ones here) have matured to the point where it makes sense to standardize on 
them. They are mature, and no other projects have come along to challenge their 
designs. At some point, it makes sense to stop pretending that there are 
options, and establish a standard instead.

-- Ed Leafe

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Joshua Harlow


Julien Danjou wrote:

On Mon, Jul 18 2016, Joshua Harlow wrote:


Thus why I think the starting of the architecture working group is a good
thing; because I have a believe that people are forgetting among all of this
that such a group holds a lot of the keys to the kingdom (whether u, the
reader, want to admit that or not is well the readers problem) in openstack
(sorry and no disrespect to independent folks&  contributors), but most of us
work for large companies that have architects (and leads) and if those
architects (and leads) can get together cross-company to aggregate and (agree
on) and solve actual problems then that really is IMHO the only way to keep our
projects healthy (assuming we can even do that at this stage).


I think it is a bit naive to think any working group is going to fix
architectural problems. You know first hand what happened¹ with the Nova
service group and tooz for example.

¹  Nothing.



Totally fair, but I still 'somewhat believe' that something good may 
come out of this. No I don't have a lot (I can probably count them on 
one hand) success stories with regards to this kind of cross-project 
work but it's not so easy to pin-point why things happened this way 
(without getting into speculation/theorizing territory).


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Doug Hellmann

Excerpts from Ed Leafe's message of 2016-07-18 21:59:44 -0700:
> On Jul 18, 2016, at 12:39 PM, Fox, Kevin M  wrote:
> 
> > I'm arguing the opposite. It should be a requirement to use the OpenStack 
> > Secret Store.
> > 
> > Instead, we're trying to make it optional and either reimplementing large 
> > swaths of it in individual projects to make it optional, or skip security 
> > all together.
> > 
> > If it wasn't optional, then:
> > * Keystone could rely on it being there for password hashing
> > * Magnum can rely on it for security.
> > * lbaas can rely on it
> > * App developers can rely on it being there
> > * ...
> > 
> > Same with Zaqar. It was a requirement then,
> > * Guest agents of heat, sahara, mangum, trove, etc could all rely on it 
> > being there and wouldn't have to deal with workarounds
> > * Horizon could rely on it being there for dynamic ui
> > * App developers can rely on it being there.
> > 
> > Yes, it seems like more work for an operator to have to install "one more 
> > thing", but the savings it provides by not having to do that "one more 
> > thing" in 7 different projects pays for it.
> 
> There is a lot to be said for code simplification. When Glance split out of 
> Nova, the knowledge that Glance would be required to be there meant the the 
> code in Nova for interacting with it didn’t have to contain tons of ‘if 
> :  else ”. Knowing that that 
> code was unnecessary made writing the interaction much simpler (and less 
> tedious to read). So if we added, say, Barbican as a requirement, all of the 
> other projects that need to handle secrets (a lot of them!) could simply 
> program to the Barbican interface, and be done with it.
> 
> The discussion of whether project X should be required is a whole ‘nother 
> topic. But I want to emphasize that often the utility of having a standard is 
> grossly underestimated.
> 
> In this discussion, the issue that seems to be arising is that the Big Tent 
> doesn’t provide for a path for a project to be reconsidered as an OpenStack 
> standard, in the way that Nova and Glance and Swift are. The Small Tent 
> “incubation” process designed to identify which projects were mature enough 
> to be considered “one of us”, when “us” was the group of core projects. This 
> was unnecessarily harsh, and pushed away many worthy efforts. In an attempt 
> to improve this, the definition of “one of us” was broadened so that lots of 
> interesting projects could be part of the community.

This came up during the discussion of co-gating that was part of
the big tent implementation, IIRC we settled on the policy that
teams are encouraged to rely on the projects produced by other
community teams *when they are comfortable adding co-gating to
ensure continued compatibility*, but there was no rule that such
relationships must be put in place because we don't want projects
to depend on new, potentially unstable, services before they are
ready.

So, the nova and glance teams are currently comfortable co-gating,
and nova can therefore depend on glance being present. Projects
handling secrets would need to negotiate similar functional test
co-gating relationships with the barbican team. At that point it
would be safe to have barbican as a required dependency of those
other projects.

Doug


> 
> What I think is causing some confusion (and friction) is that OpenStack needs 
> two “us”s: what projects are part of our ecosystem and build software in the 
> same open way (the Big Tent concept), and which are a fundamental component 
> of the overall OpenStack software, which all other projects can assume are 
> available (the Small Tent concept). 
> 
> To that end, it seems necessary to define a migration path for a project to 
> become required. Of course, most will never need to be required, but I think 
> a case could be made that several projects (Barbican and Zaqar are the 
> obvious ones here) have matured to the point where it makes sense to 
> standardize on them. They are mature, and no other projects have come along 
> to challenge their designs. At some point, it makes sense to stop pretending 
> that there are options, and establish a standard instead.
> 
> 
> -- Ed Leafe
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-19 Thread Julien Danjou

On Mon, Jul 18 2016, Joshua Harlow wrote:

> Thus why I think the starting of the architecture working group is a good
> thing; because I have a believe that people are forgetting among all of this
> that such a group holds a lot of the keys to the kingdom (whether u, the
> reader, want to admit that or not is well the readers problem) in openstack
> (sorry and no disrespect to independent folks & contributors), but most of us
> work for large companies that have architects (and leads) and if those
> architects (and leads) can get together cross-company to aggregate and (agree
> on) and solve actual problems then that really is IMHO the only way to keep 
> our
> projects healthy (assuming we can even do that at this stage).

I think it is a bit naive to think any working group is going to fix
architectural problems. You know first hand what happened¹ with the Nova
service group and tooz for example.

¹  Nothing.

-- 
Julien Danjou
// Free Software hacker
// https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-18 Thread Ed Leafe

On Jul 18, 2016, at 12:39 PM, Fox, Kevin M  wrote:

> I'm arguing the opposite. It should be a requirement to use the OpenStack 
> Secret Store.
> 
> Instead, we're trying to make it optional and either reimplementing large 
> swaths of it in individual projects to make it optional, or skip security all 
> together.
> 
> If it wasn't optional, then:
> * Keystone could rely on it being there for password hashing
> * Magnum can rely on it for security.
> * lbaas can rely on it
> * App developers can rely on it being there
> * ...
> 
> Same with Zaqar. It was a requirement then,
> * Guest agents of heat, sahara, mangum, trove, etc could all rely on it being 
> there and wouldn't have to deal with workarounds
> * Horizon could rely on it being there for dynamic ui
> * App developers can rely on it being there.
> 
> Yes, it seems like more work for an operator to have to install "one more 
> thing", but the savings it provides by not having to do that "one more thing" 
> in 7 different projects pays for it.

There is a lot to be said for code simplification. When Glance split out of 
Nova, the knowledge that Glance would be required to be there meant the the 
code in Nova for interacting with it didn’t have to contain tons of ‘if :  else ”. Knowing that that code was 
unnecessary made writing the interaction much simpler (and less tedious to 
read). So if we added, say, Barbican as a requirement, all of the other 
projects that need to handle secrets (a lot of them!) could simply program to 
the Barbican interface, and be done with it.

The discussion of whether project X should be required is a whole ‘nother 
topic. But I want to emphasize that often the utility of having a standard is 
grossly underestimated.

In this discussion, the issue that seems to be arising is that the Big Tent 
doesn’t provide for a path for a project to be reconsidered as an OpenStack 
standard, in the way that Nova and Glance and Swift are. The Small Tent 
“incubation” process designed to identify which projects were mature enough to 
be considered “one of us”, when “us” was the group of core projects. This was 
unnecessarily harsh, and pushed away many worthy efforts. In an attempt to 
improve this, the definition of “one of us” was broadened so that lots of 
interesting projects could be part of the community.

What I think is causing some confusion (and friction) is that OpenStack needs 
two “us”s: what projects are part of our ecosystem and build software in the 
same open way (the Big Tent concept), and which are a fundamental component of 
the overall OpenStack software, which all other projects can assume are 
available (the Small Tent concept). 

To that end, it seems necessary to define a migration path for a project to 
become required. Of course, most will never need to be required, but I think a 
case could be made that several projects (Barbican and Zaqar are the obvious 
ones here) have matured to the point where it makes sense to standardize on 
them. They are mature, and no other projects have come along to challenge their 
designs. At some point, it makes sense to stop pretending that there are 
options, and establish a standard instead.

-- Ed Leafe

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-18 Thread Joshua Harlow

I applaud this (since I know this kind of question and work is really 
really hard). So thanks for starting and keeping 'hard' questions going 
because without someone pushing the limit here (and/or raising the 
questions) I too fear what u fear and that we may be obsoleting 
ourselves by continuing down the path we are on (death by a thousand 
cuts anyone?)


In general I'd also like to know how cross-project things will be any 
different with the TC work, tags, release goals or other? I have a glass 
is half full feeling that those kind of things are 'to soft' or 'to 
late' but maybe I should just think more positive, but then I'm sort of 
in the camp that much *much* more aggressive action must be taken here.


Thus why I think the starting of the architecture working group is a 
good thing; because I have a believe that people are forgetting among 
all of this that such a group holds a lot of the keys to the kingdom 
(whether u, the reader, want to admit that or not is well the readers 
problem) in openstack (sorry and no disrespect to independent folks & 
contributors), but most of us work for large companies that have 
architects (and leads) and if those architects (and leads) can get 
together cross-company to aggregate and (agree on) and solve actual 
problems then that really is IMHO the only way to keep our projects 
healthy (assuming we can even do that at this stage).


Sadly I'm not sure if it's already to late for an architecture working 
group of which one of its goals (IMHO) is to do said analysis & 
aggregation of 'worker bees' to actually make fixes and solutions 
appear; but I'll remain hopefully on that (for now).


Fox, Kevin M wrote:

I feel like I'm just the messenger here. Please don't shoot me. I'm generally 
trying to provide good, though hard feedback to better a perceived problem. 
Many others would just stay silent or leave rather then have the difficult 
discussion.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-18 Thread Thierry Carrez


Fox, Kevin M wrote:

[...]
I feel like I'm just the messenger here. Please don't shoot me. I'm generally 
trying to provide good, though hard feedback to better a perceived problem. 
Many others would just stay silent or leave rather then have the difficult 
discussion.


And I think we all appreciate that :)


[...]
The openness of the big tent has allowed more choice in the ecosystem, which 
seems good, but I think it has also stalled development on the sorts of cross 
project issues that were important to our users and are now looking elsewhere 
for solutions.


As others have mentioned, we are always adapting and continuously 
improving. Now that we completed the community-centric redefinition of 
OpenStack (the "big tent"), we are I think in a better position to push 
the sort of cross-project initiatives and normalization that you are 
looking for. For example, we talked recently about having the Technical 
Committee set "release goals" to drive simple coherence and visible 
improvements across all the components of the stack.


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

The current incarnation of the Instance User spec: 
https://review.openstack.org/#/c/93/

Thanks,
Kevin

From: Michael Still [mi...@stillhq.com]
Sent: Monday, July 18, 2016 10:13 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)


On 16 Jul 2016 1:27 PM, "Thomas Herve" 
<the...@redhat.com<mailto:the...@redhat.com>> wrote:
>
> On Fri, Jul 15, 2016 at 8:36 PM, Fox, Kevin M 
> <kevin@pnnl.gov<mailto:kevin@pnnl.gov>> wrote:
> > Some specific things:
> >
> > Magnum trying to not use Barbican as it adds an addition dependency. See 
> > the discussion on the devel mailing list for details.
> >
> > Horizon discussions at the summit around wanting to use Zaqar for dynamic 
> > ui updates instead of polling, but couldn't depend on a non widely deployed 
> > subsystem.
> >
> > Each Advanced OpenStack Service implementing a guest controller 
> > communication channel that are incompatible with each other and work around 
> > communications issues differently. This makes a lot more pain for Ops to 
> > debug or architect a viable solution. For example:
> >  * Sahara uses ssh from the controllers to the vms. This does not play well 
> > with tenant networks. They have tried to work around this several ways:
> > * require every vm to have a floating ip. (Unnecessary attack surface)
> > * require the controller to be on the one and only network node (Uses 
> > ip netns exec to tunnel. Doesn't work for large clouds)
> > * Double tunnel ssh via the controller vm's. so some vms have fips, 
> > some don't. Better then all, but still not good.
> >   * Trove uses Rabbit for the guest agent to talk back to the controllers. 
> > This has traffic going the right direction to work well with tenant 
> > networks.
> > * But Rabbit is not multitenant so a security risk if any user can get 
> > into any one of the database vm's.
> > Really, I believe the right solution is to use a multitenant aware message 
> > queue so that the guest agent can pull in the right direction for tenant 
> > networks, and not have the security risk. We have such a system already, 
> > Zaqar, but its not widely deployed and projects don't want to depend on 
> > other projects that aren't widely deployed.
>
> While I agree using Barbican/Zaqar for those would be fantastic, this
> is easily solved: just optionally depend on it. Heat provides features
> that use Zaqar, which are not just present when Zaqar is not there.
> For example, if people in the Trove project were convinced that Zaqar
> was the best solution to their problem, I think they would find a way
> to provide an implementation using it. I don't think they are, though.
> Whatever the reasons may be.
>
> > The lack of Instance Users has caused lots of projects to try and work 
> > around the lack thereof. I know for sure, Mangum, Heat, and Trove work 
> > around the lack. I'm positive others have too. As an operator, it makes me 
> > have to very carefully consider all the tradeoffs each project made, and 
> > decide if I can accept the same risk they assumed. Since each is different, 
> > thats much harder.
>
> Instance users would be nice. Nobody just provided a good solution. I
> know you tried, but I don't think you succeeded. We need a good
> implementation, easy to understand, and maybe this will move forward.

I think I need a more complete definition of instance users to know what you're 
talking about here. Is this the instance locking stuff Bruno proposed a while 
ago?

> > I'm sure there are more examples. but I hope you get I'm not just trying to 
> > troll.
> >
> > The TC did apply inconsistant rules on letting projects in. That was for 
> > sure a negative before the big tent. But it also provided a way to apply 
> > pressure to projects to fix some of the issues that multiple projects face, 
> > and that plague user/operators and raise the whole community up, and that 
> > has fallen to the wayside since. Which is a big negative now. Maybe that 
> > could be bolted on top of the Big Tent I don't know.
>
> So I think that's the root of the issue here. You assume we can
> "pressure" people do something you think is right. But that's not how
> opensource works. You either implement the solution to your problem,
> or convince someone else, but you don't force anybody. I think that's
> what the TC recognized and changed, to the correct path. Could project
> be better integrated, so that we have a more coherent "product"? Sure.
> But you don't achieve that with more gover

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

Kfox> Responding to this inline. Sorry, its outlook. Not my choice. :/

-Original Message-
From: Thomas Herve [mailto:the...@redhat.com] 
Sent: Saturday, July 16, 2016 1:22 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

On Fri, Jul 15, 2016 at 8:36 PM, Fox, Kevin M <kevin@pnnl.gov> wrote:
> Some specific things:
>
> Magnum trying to not use Barbican as it adds an addition dependency. See the 
> discussion on the devel mailing list for details.
>
> Horizon discussions at the summit around wanting to use Zaqar for dynamic ui 
> updates instead of polling, but couldn't depend on a non widely deployed 
> subsystem.
>
> Each Advanced OpenStack Service implementing a guest controller communication 
> channel that are incompatible with each other and work around communications 
> issues differently. This makes a lot more pain for Ops to debug or architect 
> a viable solution. For example:
>  * Sahara uses ssh from the controllers to the vms. This does not play well 
> with tenant networks. They have tried to work around this several ways:
> * require every vm to have a floating ip. (Unnecessary attack surface)
> * require the controller to be on the one and only network node (Uses ip 
> netns exec to tunnel. Doesn't work for large clouds)
> * Double tunnel ssh via the controller vm's. so some vms have fips, some 
> don't. Better then all, but still not good.
>   * Trove uses Rabbit for the guest agent to talk back to the controllers. 
> This has traffic going the right direction to work well with tenant networks.
> * But Rabbit is not multitenant so a security risk if any user can get 
> into any one of the database vm's.
> Really, I believe the right solution is to use a multitenant aware message 
> queue so that the guest agent can pull in the right direction for tenant 
> networks, and not have the security risk. We have such a system already, 
> Zaqar, but its not widely deployed and projects don't want to depend on other 
> projects that aren't widely deployed.

While I agree using Barbican/Zaqar for those would be fantastic, this is easily 
solved: just optionally depend on it. Heat provides features that use Zaqar, 
which are not just present when Zaqar is not there.
For example, if people in the Trove project were convinced that Zaqar was the 
best solution to their problem, I think they would find a way to provide an 
implementation using it. I don't think they are, though.
Whatever the reasons may be.

kfox> The problem with making it optional, is it's a chicken and the egg 
problem. Everyone makes it optional, so they all have to implement fallback 
code. Operators see its optional, so they don't deploy the optional stuff as 
that's "more work". Users can't depend on it, since its not there on most 
systems. They don't tend to ask for it to be installed. Then the new project 
never gains any traction. It's a vicious cycle.

> The lack of Instance Users has caused lots of projects to try and work around 
> the lack thereof. I know for sure, Mangum, Heat, and Trove work around the 
> lack. I'm positive others have too. As an operator, it makes me have to very 
> carefully consider all the tradeoffs each project made, and decide if I can 
> accept the same risk they assumed. Since each is different, thats much harder.

Instance users would be nice. Nobody just provided a good solution. I know you 
tried, but I don't think you succeeded. We need a good implementation, easy to 
understand, and maybe this will move forward.

kfox> The problem is, it's a hard thing to solve, and I believe the current 
organization of OpenStack provides significant barrier to solving it.  I've 
tried for years to solve it, and have basically given up. This is part of the 
big tent issue. I'm moving more and more stuff that requires Instance Users to 
non OpenStack systems, which will be a majority of my workload at some point if 
things continue. This is the kind of thing OpenStack is really suffering from 
right now, and will only get worse as it continues to be swept under the rug.

> I'm sure there are more examples. but I hope you get I'm not just trying to 
> troll.
>
> The TC did apply inconsistant rules on letting projects in. That was for sure 
> a negative before the big tent. But it also provided a way to apply pressure 
> to projects to fix some of the issues that multiple projects face, and that 
> plague user/operators and raise the whole community up, and that has fallen 
> to the wayside since. Which is a big negative now. Maybe that could be bolted 
> on top of the Big Tent I don't know.

So I think that's the root of the issue here. You assume we can "pressure" 
people do something you think is right. But that's not how

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

2016-07-18 Thread Thomas Herve

On Mon, Jul 18, 2016 at 7:13 PM, Michael Still  wrote:
> On 16 Jul 2016 1:27 PM, "Thomas Herve"  wrote:
>>
>> On Fri, Jul 15, 2016 at 8:36 PM, Fox, Kevin M  wrote:
>> > The lack of Instance Users has caused lots of projects to try and work
>> > around the lack thereof. I know for sure, Mangum, Heat, and Trove work
>> > around the lack. I'm positive others have too. As an operator, it makes me
>> > have to very carefully consider all the tradeoffs each project made, and
>> > decide if I can accept the same risk they assumed. Since each is different,
>> > thats much harder.
>>
>> Instance users would be nice. Nobody just provided a good solution. I
>> know you tried, but I don't think you succeeded. We need a good
>> implementation, easy to understand, and maybe this will move forward.
>
> I think I need a more complete definition of instance users to know what
> you're talking about here. Is this the instance locking stuff Bruno proposed
> a while ago?

I believe this is what's Kevin talks about:
https://review.openstack.org/#/c/93/

-- 
Thomas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

I'm arguing the opposite. It should be a requirement to use the OpenStack 
Secret Store.

Instead, we're trying to make it optional and either reimplementing large 
swaths of it in individual projects to make it optional, or skip security all 
together.

If it wasn't optional, then:
 * Keystone could rely on it being there for password hashing
 * Magnum can rely on it for security.
 * lbaas can rely on it
 * App developers can rely on it being there
 * ...

Same with Zaqar. It was a requirement then,
 * Guest agents of heat, sahara, mangum, trove, etc could all rely on it being 
there and wouldn't have to deal with workarounds
 * Horizon could rely on it being there for dynamic ui
 * App developers can rely on it being there.

Yes, it seems like more work for an operator to have to install "one more 
thing", but the savings it provides by not having to do that "one more thing" 
in 7 different projects pays for it.

Rather then focus on sharing code by adding required project dependencies, 
we're just implementing things in each project to workaround the lack thereof 
in a general project and then everyone suffers for it.

Thanks,
Kevin


From: Hongbin Lu [hongbin...@huawei.com]
Sent: Friday, July 15, 2016 10:41 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

No, Magnum still uses Barbican as an optional dependency, and I believe nobody 
has proposed to remove Barbican entirely. I have no position about big tent but 
using Magnum as an example of "projects are not working together" is 
inappropriate.

Best regards,
Hongbin

> -Original Message-
> From: Fox, Kevin M [mailto:kevin@pnnl.gov]
> Sent: July-15-16 2:37 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins
> for all)
>
> Some specific things:
>
> Magnum trying to not use Barbican as it adds an addition dependency.
> See the discussion on the devel mailing list for details.
>
> Horizon discussions at the summit around wanting to use Zaqar for
> dynamic ui updates instead of polling, but couldn't depend on a non
> widely deployed subsystem.
>
> Each Advanced OpenStack Service implementing a guest controller
> communication channel that are incompatible with each other and work
> around communications issues differently. This makes a lot more pain
> for Ops to debug or architect a viable solution. For example:
>  * Sahara uses ssh from the controllers to the vms. This does not play
> well with tenant networks. They have tried to work around this several
> ways:
> * require every vm to have a floating ip. (Unnecessary attack
> surface)
> * require the controller to be on the one and only network node
> (Uses ip netns exec to tunnel. Doesn't work for large clouds)
> * Double tunnel ssh via the controller vm's. so some vms have fips,
> some don't. Better then all, but still not good.
>   * Trove uses Rabbit for the guest agent to talk back to the
> controllers. This has traffic going the right direction to work well
> with tenant networks.
> * But Rabbit is not multitenant so a security risk if any user can
> get into any one of the database vm's.
> Really, I believe the right solution is to use a multitenant aware
> message queue so that the guest agent can pull in the right direction
> for tenant networks, and not have the security risk. We have such a
> system already, Zaqar, but its not widely deployed and projects don't
> want to depend on other projects that aren't widely deployed.
>
> The lack of Instance Users has caused lots of projects to try and work
> around the lack thereof. I know for sure, Mangum, Heat, and Trove work
> around the lack. I'm positive others have too. As an operator, it makes
> me have to very carefully consider all the tradeoffs each project made,
> and decide if I can accept the same risk they assumed. Since each is
> different, thats much harder.
>
> I'm sure there are more examples. but I hope you get I'm not just
> trying to troll.
>
> The TC did apply inconsistant rules on letting projects in. That was
> for sure a negative before the big tent. But it also provided a way to
> apply pressure to projects to fix some of the issues that multiple
> projects face, and that plague user/operators and raise the whole
> community up, and that has fallen to the wayside since. Which is a big
> negative now. Maybe that could be bolted on top of the Big Tent I don't
> know.
>
> I could write a very long description about the state of being an
> OpenStack App developer too that touches on all the problems with
> getting a consistent target and all the cross project c

Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)