Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-26 Thread Joe Gordon
On Wed, Jan 21, 2015 at 5:03 AM, Sean Dague s...@dague.net wrote:

 On 01/20/2015 08:15 PM, Robert Collins wrote:
  On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
  ...
  This ml thread came up in the TC meeting today and I am responding here
  to catch the thread up with the meeting. The soft update option is the
  suggested fix for non openstack projects that want to have most of their
  requirements managed by global requirements.
 
  For the project structure reform opening things up we should consider
  loosening the criteria to get on the list and make it primarily based on
  technical criteria such as py3k support, license compatibility, upstream
  support/activity, and so on (basically the current criteria with less of
  a focus on where the project comes from if it is otherwise healthy).
  Then individual projects would choose the subset they need to depend on.
  This model should be viable with different domains as well if we go that
  route.
 
  The following is not from the TC meeting but addressing other portions
  of this conversation:
 
  At least one concern with this option is that as the number of total
  requirements goes up is the difficulty in debugging installation
  conflicts becomes more difficult too. I have suggested that we could
  write tools to help with this. Install bisection based on pip logs for
  example, but these tools are still theoretical so I may be
  overestimating their usefulness.
 
  To address the community scaling aspect I think you push a lot of work
  back on deployers/users if we don't curate requirements for anything
  that ends up tagged as production ready (or whatever the equivalent
  tag becomes). Essentially we are saying this doesn't scale for us so
  now you deal with the fallout. Have fun, which isn't very friendly to
  people consuming the software. We already have an absurd number of
  requirements and management of them has appeared to scale. I don't
  foresee my workload going up if we open up the list as suggested.
 
  Perhaps I missed something, but the initial request wasn't about
  random packages, it was about other stackforge clients - these are
  things in the ecosystem! I'm glad we have technical solutions, but it
  just seems odd to me that adding them would ever have been
  controversial.

 Well, I think Clark and I have different opinions of how much of a pain
 unwinding the requirements are, and how long these tend to leave the
 gate broken. I am happy to also put it in a somebody elses problem
 field for resolving the issues. :)

 Honestly, I think we're actually at a different point, where we need to
 stop assuming that the sane way to deal with python is to install it
 into system libraries, and just put every service in a venv and get rid
 of global requirements entirely. Global requirements was a scaling fix
 for getting to 10 coexisting projects. I don't think it actually works
 well with 50 ecosystem projects. Which is why I proposed the domains
 solution instead.


++ using per service virtual environments would help us avoid a whole class
of nasty issues. On the flip side doing this makes things harder for
distros to find a set of non-conflicting dependencies etc.


  On the pip solver side, joe gordon was working on a thing to install a
  fixed set of packages by bypassing the pip resolver... not sure how
  thats progressing.

 I think if we are talking seriously about bypassing the pip resolver, we
 should step back and think about that fact. Because now we're producting
 a custom installation process that will produce an answer for us, which
 is completely different than any answer that anyone else is getting for
 how to get a coherent system.


Fully agreed, I am looking into avoiding pips dependency solver for stable
branches only right now. But using per service venvs would be even better.



 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-26 Thread Robert Collins
TripleO has done per service venvs for a couple years now, and it
doesn't solve the fragility issue that our unbounded deps cause. It
avoids most but not all conflicting deps within OpenStack, and none of
the 'upstream broke us' cases.

-Rob

On 27 January 2015 at 09:01, Joe Gordon joe.gord...@gmail.com wrote:


 On Wed, Jan 21, 2015 at 5:03 AM, Sean Dague s...@dague.net wrote:

 On 01/20/2015 08:15 PM, Robert Collins wrote:
  On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
  ...
  This ml thread came up in the TC meeting today and I am responding here
  to catch the thread up with the meeting. The soft update option is the
  suggested fix for non openstack projects that want to have most of
  their
  requirements managed by global requirements.
 
  For the project structure reform opening things up we should consider
  loosening the criteria to get on the list and make it primarily based
  on
  technical criteria such as py3k support, license compatibility,
  upstream
  support/activity, and so on (basically the current criteria with less
  of
  a focus on where the project comes from if it is otherwise healthy).
  Then individual projects would choose the subset they need to depend
  on.
  This model should be viable with different domains as well if we go
  that
  route.
 
  The following is not from the TC meeting but addressing other portions
  of this conversation:
 
  At least one concern with this option is that as the number of total
  requirements goes up is the difficulty in debugging installation
  conflicts becomes more difficult too. I have suggested that we could
  write tools to help with this. Install bisection based on pip logs for
  example, but these tools are still theoretical so I may be
  overestimating their usefulness.
 
  To address the community scaling aspect I think you push a lot of work
  back on deployers/users if we don't curate requirements for anything
  that ends up tagged as production ready (or whatever the equivalent
  tag becomes). Essentially we are saying this doesn't scale for us so
  now you deal with the fallout. Have fun, which isn't very friendly to
  people consuming the software. We already have an absurd number of
  requirements and management of them has appeared to scale. I don't
  foresee my workload going up if we open up the list as suggested.
 
  Perhaps I missed something, but the initial request wasn't about
  random packages, it was about other stackforge clients - these are
  things in the ecosystem! I'm glad we have technical solutions, but it
  just seems odd to me that adding them would ever have been
  controversial.

 Well, I think Clark and I have different opinions of how much of a pain
 unwinding the requirements are, and how long these tend to leave the
 gate broken. I am happy to also put it in a somebody elses problem
 field for resolving the issues. :)

 Honestly, I think we're actually at a different point, where we need to
 stop assuming that the sane way to deal with python is to install it
 into system libraries, and just put every service in a venv and get rid
 of global requirements entirely. Global requirements was a scaling fix
 for getting to 10 coexisting projects. I don't think it actually works
 well with 50 ecosystem projects. Which is why I proposed the domains
 solution instead.


 ++ using per service virtual environments would help us avoid a whole class
 of nasty issues. On the flip side doing this makes things harder for distros
 to find a set of non-conflicting dependencies etc.


  On the pip solver side, joe gordon was working on a thing to install a
  fixed set of packages by bypassing the pip resolver... not sure how
  thats progressing.

 I think if we are talking seriously about bypassing the pip resolver, we
 should step back and think about that fact. Because now we're producting
 a custom installation process that will produce an answer for us, which
 is completely different than any answer that anyone else is getting for
 how to get a coherent system.


 Fully agreed, I am looking into avoiding pips dependency solver for stable
 branches only right now. But using per service venvs would be even better.



 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack 

Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-26 Thread Ben Nemec

On 01/26/2015 02:29 PM, Robert Collins wrote:
 TripleO has done per service venvs for a couple years now, and it
 doesn't solve the fragility issue that our unbounded deps cause. It
 avoids most but not all conflicting deps within OpenStack, and none of
 the 'upstream broke us' cases.

Note that a lot of us (our CI included) are no longer running with
separate venvs due to the time it takes to build them.

See common-venv from
http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_gate_test.sh#n21

Also, I don't see how this could work with distro packages (I'm not
saying there isn't a way, but I don't know what it would be), so I think
that's something I would definitely want an answer on before we decide
this is the way forward.

-Ben

 
 -Rob
 
 On 27 January 2015 at 09:01, Joe Gordon joe.gord...@gmail.com wrote:


 On Wed, Jan 21, 2015 at 5:03 AM, Sean Dague s...@dague.net wrote:

 On 01/20/2015 08:15 PM, Robert Collins wrote:
 On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
 ...
 This ml thread came up in the TC meeting today and I am responding here
 to catch the thread up with the meeting. The soft update option is the
 suggested fix for non openstack projects that want to have most of
 their
 requirements managed by global requirements.

 For the project structure reform opening things up we should consider
 loosening the criteria to get on the list and make it primarily based
 on
 technical criteria such as py3k support, license compatibility,
 upstream
 support/activity, and so on (basically the current criteria with less
 of
 a focus on where the project comes from if it is otherwise healthy).
 Then individual projects would choose the subset they need to depend
 on.
 This model should be viable with different domains as well if we go
 that
 route.

 The following is not from the TC meeting but addressing other portions
 of this conversation:

 At least one concern with this option is that as the number of total
 requirements goes up is the difficulty in debugging installation
 conflicts becomes more difficult too. I have suggested that we could
 write tools to help with this. Install bisection based on pip logs for
 example, but these tools are still theoretical so I may be
 overestimating their usefulness.

 To address the community scaling aspect I think you push a lot of work
 back on deployers/users if we don't curate requirements for anything
 that ends up tagged as production ready (or whatever the equivalent
 tag becomes). Essentially we are saying this doesn't scale for us so
 now you deal with the fallout. Have fun, which isn't very friendly to
 people consuming the software. We already have an absurd number of
 requirements and management of them has appeared to scale. I don't
 foresee my workload going up if we open up the list as suggested.

 Perhaps I missed something, but the initial request wasn't about
 random packages, it was about other stackforge clients - these are
 things in the ecosystem! I'm glad we have technical solutions, but it
 just seems odd to me that adding them would ever have been
 controversial.

 Well, I think Clark and I have different opinions of how much of a pain
 unwinding the requirements are, and how long these tend to leave the
 gate broken. I am happy to also put it in a somebody elses problem
 field for resolving the issues. :)

 Honestly, I think we're actually at a different point, where we need to
 stop assuming that the sane way to deal with python is to install it
 into system libraries, and just put every service in a venv and get rid
 of global requirements entirely. Global requirements was a scaling fix
 for getting to 10 coexisting projects. I don't think it actually works
 well with 50 ecosystem projects. Which is why I proposed the domains
 solution instead.


 ++ using per service virtual environments would help us avoid a whole class
 of nasty issues. On the flip side doing this makes things harder for distros
 to find a set of non-conflicting dependencies etc.


 On the pip solver side, joe gordon was working on a thing to install a
 fixed set of packages by bypassing the pip resolver... not sure how
 thats progressing.

 I think if we are talking seriously about bypassing the pip resolver, we
 should step back and think about that fact. Because now we're producting
 a custom installation process that will produce an answer for us, which
 is completely different than any answer that anyone else is getting for
 how to get a coherent system.


 Fully agreed, I am looking into avoiding pips dependency solver for stable
 branches only right now. But using per service venvs would be even better.



 -Sean

 --
 Sean Dague
 http://dague.net

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 

Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-26 Thread Robert Collins
On 27 January 2015 at 09:56, Clint Byrum cl...@fewbar.com wrote:

 moved post to bottom for us backwards folk who see the quotes in
 original order

/pedant

 TripleO has done per service venvs for a couple years now, and it
 doesn't solve the fragility issue that our unbounded deps cause. It
 avoids most but not all conflicting deps within OpenStack, and none of
 the 'upstream broke us' cases.


 Note that we are not testing per-service venvs anymore because of the
 extreme high cost of building the controller images with so many separate
 venvs. We just put the openstack namespaced pieces in one big openstack
 venv now.

Yes, but the experience we have about the limitations of per-service
venvs is still relevant, no? Are you really saying 'do per-service
venvs because they work well', or are you agreeing with me that they
don't solve the problems plaguing the gate?

-Rob


-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-26 Thread Chris Dent

On Tue, 27 Jan 2015, Robert Collins wrote:


Yes, but the experience we have about the limitations of per-service
venvs is still relevant, no? Are you really saying 'do per-service
venvs because they work well', or are you agreeing with me that they
don't solve the problems plaguing the gate?


To take this topic even further from its original subject I'd just
like to remind at least myself that the reason we have problems in the
gate is because there are real problems that actually exist or will
exist soon and that catching them before they leak out is is presumably
exactly why we have a gate.

Given a lot more resources we should always pip -U and never put upper
version caps in requirements.txt. Then when stuff breaks do the good
work to untangle the mess and fix the problem. Not just for us but for
the commonweal.

Breaking stuff tends to be how stuff gets fixed.

Yes, I'm talking idealistic impractical hooey here, but I think it is
important to remember these things. So whereas earlier I said I was
keen on virtualenvs, now that I actually think about it some I think
we're better off creating a chaotic crucible of everything in one fiery
cauldron of truth and using breakage in the gate as a way of marshalling
resources.

Do I expect to see that happen? Sadly, not really.

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-26 Thread Clint Byrum
Excerpts from Robert Collins's message of 2015-01-26 12:29:37 -0800:
 On 27 January 2015 at 09:01, Joe Gordon joe.gord...@gmail.com wrote:
 
 
  On Wed, Jan 21, 2015 at 5:03 AM, Sean Dague s...@dague.net wrote:
 
  On 01/20/2015 08:15 PM, Robert Collins wrote:
   On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
   ...
   This ml thread came up in the TC meeting today and I am responding here
   to catch the thread up with the meeting. The soft update option is the
   suggested fix for non openstack projects that want to have most of
   their
   requirements managed by global requirements.
  
   For the project structure reform opening things up we should consider
   loosening the criteria to get on the list and make it primarily based
   on
   technical criteria such as py3k support, license compatibility,
   upstream
   support/activity, and so on (basically the current criteria with less
   of
   a focus on where the project comes from if it is otherwise healthy).
   Then individual projects would choose the subset they need to depend
   on.
   This model should be viable with different domains as well if we go
   that
   route.
  
   The following is not from the TC meeting but addressing other portions
   of this conversation:
  
   At least one concern with this option is that as the number of total
   requirements goes up is the difficulty in debugging installation
   conflicts becomes more difficult too. I have suggested that we could
   write tools to help with this. Install bisection based on pip logs for
   example, but these tools are still theoretical so I may be
   overestimating their usefulness.
  
   To address the community scaling aspect I think you push a lot of work
   back on deployers/users if we don't curate requirements for anything
   that ends up tagged as production ready (or whatever the equivalent
   tag becomes). Essentially we are saying this doesn't scale for us so
   now you deal with the fallout. Have fun, which isn't very friendly to
   people consuming the software. We already have an absurd number of
   requirements and management of them has appeared to scale. I don't
   foresee my workload going up if we open up the list as suggested.
  
   Perhaps I missed something, but the initial request wasn't about
   random packages, it was about other stackforge clients - these are
   things in the ecosystem! I'm glad we have technical solutions, but it
   just seems odd to me that adding them would ever have been
   controversial.
 
  Well, I think Clark and I have different opinions of how much of a pain
  unwinding the requirements are, and how long these tend to leave the
  gate broken. I am happy to also put it in a somebody elses problem
  field for resolving the issues. :)
 
  Honestly, I think we're actually at a different point, where we need to
  stop assuming that the sane way to deal with python is to install it
  into system libraries, and just put every service in a venv and get rid
  of global requirements entirely. Global requirements was a scaling fix
  for getting to 10 coexisting projects. I don't think it actually works
  well with 50 ecosystem projects. Which is why I proposed the domains
  solution instead.
 
 
  ++ using per service virtual environments would help us avoid a whole class
  of nasty issues. On the flip side doing this makes things harder for distros
  to find a set of non-conflicting dependencies etc.
 
 
   On the pip solver side, joe gordon was working on a thing to install a
   fixed set of packages by bypassing the pip resolver... not sure how
   thats progressing.
 
  I think if we are talking seriously about bypassing the pip resolver, we
  should step back and think about that fact. Because now we're producting
  a custom installation process that will produce an answer for us, which
  is completely different than any answer that anyone else is getting for
  how to get a coherent system.
 
 
  Fully agreed, I am looking into avoiding pips dependency solver for stable
  branches only right now. But using per service venvs would be even better.
 

moved post to bottom for us backwards folk who see the quotes in
original order
 TripleO has done per service venvs for a couple years now, and it
 doesn't solve the fragility issue that our unbounded deps cause. It
 avoids most but not all conflicting deps within OpenStack, and none of
 the 'upstream broke us' cases.
 

Note that we are not testing per-service venvs anymore because of the
extreme high cost of building the controller images with so many separate
venvs. We just put the openstack namespaced pieces in one big openstack
venv now.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-21 Thread Robert Collins
On 22 January 2015 at 02:48, Chris Dent chd...@redhat.com wrote:
 On Wed, 21 Jan 2015, Sean Dague wrote:

 On the pip solver side, joe gordon was working on a thing to install a
 fixed set of packages by bypassing the pip resolver... not sure how
 thats progressing.


 I think if we are talking seriously about bypassing the pip resolver, we
 should step back and think about that fact. Because now we're producting
 a custom installation process that will produce an answer for us, which
 is completely different than any answer that anyone else is getting for
 how to get a coherent system.

Actually buildout is precisely that today - you specify exact versions
and get them. pip's inability to duplicate that behaviour is one of
the reasons buildout is still a thing; bypassing the resolver to get
the same behaviour *for the same reasons!* isn't at all odd.
Undesirable perhaps - and we should work on pip upstream too - but not
odd.

 Agreed. Skipping the pip resolver will move OpenStack and friends further
 down into a weird world of its own rather than Python norms.

See above - I disagree. Python norms are -very- broad, and buildout
has a healthy ecosystem within Python web shops, for exactly the
issues we're trying to solve here.

 If possible we should try to resolve the complexity, not bandaid the
 problems caused by the complexity.

 If we can't do that then per service virtualenv (or even containers)
 provides a far more contained[1] bandage.

Per service virtualenvs are good IMO, but orthogonal to the unbounded
thing we have today, which we have largely *because* of pip not having
a resolver. (Thats driven by random things getting upgraded because of
transitive dependencies - caused by the resolver).

 Related: I personally find it completely bewildering that things are
 managed and run in such an unbounded fashion. My intuition is that
 this is the result of many/most projects being on the same release
 cycle and the belief that my master must integrate with your master. We
 should release (far) more often and my master should integrate with your
 lastest release.

Yes, I would like to see that, as a complement to the trunk-trunk
tests. We need to know that something we're about to release won't
break folk as well as knowing that the thing we're about to release
works with the releases of other things that are already out there.

-Rob


-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-21 Thread Boris Pavlovic
Clarkb,

As Robert said, you are missing the point.

I didn't say that Rally wants this lib so it should be in global
requirements.

I asked only about python clients of stackforge projects that are regarding
all rules:
(Like py3k support, license, are in projects.txt and so on).  From my point
of view, if clients are regarding all reules there is no compatibility
issues = it is easy to add them to g-r. Am I wrong?


All,

Sorry I wasn't able to be on meeting yesterday.


First of all, we don't have any issues with having Murano/Mistral
benchmarks and testing them in gates and supporting them in our tree. (We
already have rally-dsvm-murano and rally-dsvm-mistral dsvm jobs that works
well)

The real issue is very bad user  dev experience caused by soft decencies.

Let's take a look at actions of typical user (running Murano benchamrk):
1) Try to run tests for Murano
2) Input validation error: MuranoClient is missing please install it
3) WTF???

Let's take a look at actions of end developer (writing Murano benchmark):
1) Try to make new Murano benchmark
2) Need to import some exceptions from Murano client
3) All unit tests are failing..
4) WTF???

Adding extra steps/conditions that are required for using product are
adding exponential complexity to it. So having 5 such steps/conditions will
make product inconsumable.

So the only thing that I would like is to remove one extra step by adding
good python clients to g-r.


Best regards,
Boris Pavlovic




On Wed, Jan 21, 2015 at 4:15 AM, Robert Collins robe...@robertcollins.net
wrote:

 On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
 ...
  This ml thread came up in the TC meeting today and I am responding here
  to catch the thread up with the meeting. The soft update option is the
  suggested fix for non openstack projects that want to have most of their
  requirements managed by global requirements.
 
  For the project structure reform opening things up we should consider
  loosening the criteria to get on the list and make it primarily based on
  technical criteria such as py3k support, license compatibility, upstream
  support/activity, and so on (basically the current criteria with less of
  a focus on where the project comes from if it is otherwise healthy).
  Then individual projects would choose the subset they need to depend on.
  This model should be viable with different domains as well if we go that
  route.
 
  The following is not from the TC meeting but addressing other portions
  of this conversation:
 
  At least one concern with this option is that as the number of total
  requirements goes up is the difficulty in debugging installation
  conflicts becomes more difficult too. I have suggested that we could
  write tools to help with this. Install bisection based on pip logs for
  example, but these tools are still theoretical so I may be
  overestimating their usefulness.
 
  To address the community scaling aspect I think you push a lot of work
  back on deployers/users if we don't curate requirements for anything
  that ends up tagged as production ready (or whatever the equivalent
  tag becomes). Essentially we are saying this doesn't scale for us so
  now you deal with the fallout. Have fun, which isn't very friendly to
  people consuming the software. We already have an absurd number of
  requirements and management of them has appeared to scale. I don't
  foresee my workload going up if we open up the list as suggested.

 Perhaps I missed something, but the initial request wasn't about
 random packages, it was about other stackforge clients - these are
 things in the ecosystem! I'm glad we have technical solutions, but it
 just seems odd to me that adding them would ever have been
 controversial.

 On the pip solver side, joe gordon was working on a thing to install a
 fixed set of packages by bypassing the pip resolver... not sure how
 thats progressing.

 -Rob

 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-21 Thread Sean Dague
On 01/20/2015 08:15 PM, Robert Collins wrote:
 On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
 ...
 This ml thread came up in the TC meeting today and I am responding here
 to catch the thread up with the meeting. The soft update option is the
 suggested fix for non openstack projects that want to have most of their
 requirements managed by global requirements.

 For the project structure reform opening things up we should consider
 loosening the criteria to get on the list and make it primarily based on
 technical criteria such as py3k support, license compatibility, upstream
 support/activity, and so on (basically the current criteria with less of
 a focus on where the project comes from if it is otherwise healthy).
 Then individual projects would choose the subset they need to depend on.
 This model should be viable with different domains as well if we go that
 route.

 The following is not from the TC meeting but addressing other portions
 of this conversation:

 At least one concern with this option is that as the number of total
 requirements goes up is the difficulty in debugging installation
 conflicts becomes more difficult too. I have suggested that we could
 write tools to help with this. Install bisection based on pip logs for
 example, but these tools are still theoretical so I may be
 overestimating their usefulness.

 To address the community scaling aspect I think you push a lot of work
 back on deployers/users if we don't curate requirements for anything
 that ends up tagged as production ready (or whatever the equivalent
 tag becomes). Essentially we are saying this doesn't scale for us so
 now you deal with the fallout. Have fun, which isn't very friendly to
 people consuming the software. We already have an absurd number of
 requirements and management of them has appeared to scale. I don't
 foresee my workload going up if we open up the list as suggested.
 
 Perhaps I missed something, but the initial request wasn't about
 random packages, it was about other stackforge clients - these are
 things in the ecosystem! I'm glad we have technical solutions, but it
 just seems odd to me that adding them would ever have been
 controversial.

Well, I think Clark and I have different opinions of how much of a pain
unwinding the requirements are, and how long these tend to leave the
gate broken. I am happy to also put it in a somebody elses problem
field for resolving the issues. :)

Honestly, I think we're actually at a different point, where we need to
stop assuming that the sane way to deal with python is to install it
into system libraries, and just put every service in a venv and get rid
of global requirements entirely. Global requirements was a scaling fix
for getting to 10 coexisting projects. I don't think it actually works
well with 50 ecosystem projects. Which is why I proposed the domains
solution instead.

 On the pip solver side, joe gordon was working on a thing to install a
 fixed set of packages by bypassing the pip resolver... not sure how
 thats progressing.

I think if we are talking seriously about bypassing the pip resolver, we
should step back and think about that fact. Because now we're producting
a custom installation process that will produce an answer for us, which
is completely different than any answer that anyone else is getting for
how to get a coherent system.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-21 Thread Chris Dent

On Wed, 21 Jan 2015, Sean Dague wrote:

On the pip solver side, joe gordon was working on a thing to install a
fixed set of packages by bypassing the pip resolver... not sure how
thats progressing.


I think if we are talking seriously about bypassing the pip resolver, we
should step back and think about that fact. Because now we're producting
a custom installation process that will produce an answer for us, which
is completely different than any answer that anyone else is getting for
how to get a coherent system.


Agreed. Skipping the pip resolver will move OpenStack and friends further
down into a weird world of its own rather than Python norms.

If possible we should try to resolve the complexity, not bandaid the
problems caused by the complexity.

If we can't do that then per service virtualenv (or even containers)
provides a far more contained[1] bandage.

Related: I personally find it completely bewildering that things are
managed and run in such an unbounded fashion. My intuition is that
this is the result of many/most projects being on the same release
cycle and the belief that my master must integrate with your master. We
should release (far) more often and my master should integrate with your
lastest release.

[1] Oh, my sides.

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-20 Thread Clark Boylan
On Wed, Jan 14, 2015, at 04:50 AM, Sean Dague wrote:
 On 01/14/2015 03:55 AM, Flavio Percoco wrote:
  On 13/01/15 14:31 -0500, Sean Dague wrote:
  On 01/13/2015 02:10 PM, Jay Pipes wrote:
  On 01/13/2015 12:39 PM, Zane Bitter wrote:
  On 13/01/15 10:01, Jeremy Stanley wrote:
  On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
  Why doesn't rally just remove itself from projects.txt, then there
  would be no restrictions on what it adds.
 
  I second this recommendation. If Rally wants to depend on things
  which are not part of OpenStack and not required to run or test
  OpenStack in an official capacity (and since it's not an official
  OpenStack project it's entirely within its rights to want to do
  that), then forcing it to comply with the list of requirements for
  official OpenStack projects is an unnecessarily restrictive choice.
 
  FWIW I don't really agree with this advice. The purpose of
  openstack/requirements is to ensure that it's possible to create a
  distribution of OpenStack without conflicting version requirements, not
  to force every project to use every dependency listed. As such, some
  co-ordination around client libraries for related projects seems
  like it
  ought to be an uncontroversial addition. (Obviously it's easy to
  imagine
  potential additions that would legitimately be highly controversial,
  but
  IMHO this isn't one of them.) Saying that we require people to use our
  tools to get into the club but that our tools are not going to
  accommodate them in any way until they are in sounds a bit too close to
  Go away to my ears.
 
  +1
 
  I think as we grow global-requirements probably needs to be broken up
  into functional lists. What's allowed in base-iass, what's allowed in
  application services, what's allowed in docs, etc, etc.
 
  The complexity of keeping the g-r list actually working goes up sort of
  n^2 as the size of it, because pip doesn't have a real solver. This is a
  barely functional set of plate spinning so just add everything misses
  the point that the breaks get more and more difficult to unwind.
 
  If someone is up for stepping up and contributing to breaking up into
  domains, please do. But I think while this remains a global list we
  really do need to be a bit careful here.
  
  Do we really need a per-domain g-r?
  
  With the upcoming changes in the openstack governance model, I believe
  this won't scale even if we break it into per-domain basis. Some
  questions that need answeres are:
  
  - Who's going to review the per domain g-r ?
  
  If it's a centralized team, then it won't definitely scale. If it's
  the domain team then, what's the real point of having these
  requirements?
  
  - How are we going to support the creation and maintnance of these g-r
   in the gate? Are they actually worth it?
  
  
  While I understand why we have g-r, I really don't think it'll scale
  for a broader community as we're envisioning it. With that in mind,
  I'd probably recommend to have 1 requirements group for the base-caas
  integrated gate and let other projects and groups have their own
  requirements. That is, non base-caas projects should get the versions
  of their common requirements from g-r so that they won't conflict if
  they're installed alongside with any of the base-caas projects. But
  other than those base requirements, I'd let non base-caas projects
  have what they need (which is pretty much what heat's team has done
  with the contrib packages, AFAICT).
  
  I know this opens the doors for version conflicts between the
  requirements of non base-caas projects but I think this is a more
  welcoming (and non-blocking) way to do things until we've a better
  one.
  
  Hope the above makes some sense I blame the lack of coffee if it
  doesn't,
  Flavio
 
 We have the base infrastructure of that in devstack today with a SOFT
 requirements update -
 https://github.com/openstack-dev/devstack/blob/master/functions-common#L1696-L1704
 
 If you are not in projects.txt, and you set your REQUIREMENTS_MODE=soft
 in your devstack run, we will leave extra dependencies untouched.
 Probably a few more bits required to make it really easy to expose, but
 that direction is also feasable.
 
 The reason I brought up domains though is that some groups really want
 the facility to have common requirements lists across a domain. Like the
 OpenStack Docs team. They've currently inserted a ton of stuff in g-r
 that really shouldn't be there because they have a lot of git trees, and
 the synchronization is a nice feature.
 
 However, if there were domain specific lists, that would be fine. A
 collection of projects that want to coordinate could all agree on a
 domain specific set of lists, and off we go. Maybe that list wouldn't
 even need to be contained in the main repo, it could be in a sub repo so
 have different approvers.
 
This ml thread came up in the TC meeting today and I am responding here
to catch the thread up with the meeting. 

Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-20 Thread Robert Collins
On 21 January 2015 at 10:21, Clark Boylan cboy...@sapwetik.org wrote:
...
 This ml thread came up in the TC meeting today and I am responding here
 to catch the thread up with the meeting. The soft update option is the
 suggested fix for non openstack projects that want to have most of their
 requirements managed by global requirements.

 For the project structure reform opening things up we should consider
 loosening the criteria to get on the list and make it primarily based on
 technical criteria such as py3k support, license compatibility, upstream
 support/activity, and so on (basically the current criteria with less of
 a focus on where the project comes from if it is otherwise healthy).
 Then individual projects would choose the subset they need to depend on.
 This model should be viable with different domains as well if we go that
 route.

 The following is not from the TC meeting but addressing other portions
 of this conversation:

 At least one concern with this option is that as the number of total
 requirements goes up is the difficulty in debugging installation
 conflicts becomes more difficult too. I have suggested that we could
 write tools to help with this. Install bisection based on pip logs for
 example, but these tools are still theoretical so I may be
 overestimating their usefulness.

 To address the community scaling aspect I think you push a lot of work
 back on deployers/users if we don't curate requirements for anything
 that ends up tagged as production ready (or whatever the equivalent
 tag becomes). Essentially we are saying this doesn't scale for us so
 now you deal with the fallout. Have fun, which isn't very friendly to
 people consuming the software. We already have an absurd number of
 requirements and management of them has appeared to scale. I don't
 foresee my workload going up if we open up the list as suggested.

Perhaps I missed something, but the initial request wasn't about
random packages, it was about other stackforge clients - these are
things in the ecosystem! I'm glad we have technical solutions, but it
just seems odd to me that adding them would ever have been
controversial.

On the pip solver side, joe gordon was working on a thing to install a
fixed set of packages by bypassing the pip resolver... not sure how
thats progressing.

-Rob

-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-14 Thread Flavio Percoco

On 13/01/15 14:31 -0500, Sean Dague wrote:

On 01/13/2015 02:10 PM, Jay Pipes wrote:

On 01/13/2015 12:39 PM, Zane Bitter wrote:

On 13/01/15 10:01, Jeremy Stanley wrote:

On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:

Why doesn't rally just remove itself from projects.txt, then there
would be no restrictions on what it adds.


I second this recommendation. If Rally wants to depend on things
which are not part of OpenStack and not required to run or test
OpenStack in an official capacity (and since it's not an official
OpenStack project it's entirely within its rights to want to do
that), then forcing it to comply with the list of requirements for
official OpenStack projects is an unnecessarily restrictive choice.


FWIW I don't really agree with this advice. The purpose of
openstack/requirements is to ensure that it's possible to create a
distribution of OpenStack without conflicting version requirements, not
to force every project to use every dependency listed. As such, some
co-ordination around client libraries for related projects seems like it
ought to be an uncontroversial addition. (Obviously it's easy to imagine
potential additions that would legitimately be highly controversial, but
IMHO this isn't one of them.) Saying that we require people to use our
tools to get into the club but that our tools are not going to
accommodate them in any way until they are in sounds a bit too close to
Go away to my ears.


+1


I think as we grow global-requirements probably needs to be broken up
into functional lists. What's allowed in base-iass, what's allowed in
application services, what's allowed in docs, etc, etc.

The complexity of keeping the g-r list actually working goes up sort of
n^2 as the size of it, because pip doesn't have a real solver. This is a
barely functional set of plate spinning so just add everything misses
the point that the breaks get more and more difficult to unwind.

If someone is up for stepping up and contributing to breaking up into
domains, please do. But I think while this remains a global list we
really do need to be a bit careful here.


Do we really need a per-domain g-r?

With the upcoming changes in the openstack governance model, I believe
this won't scale even if we break it into per-domain basis. Some
questions that need answeres are:

- Who's going to review the per domain g-r ?

If it's a centralized team, then it won't definitely scale. If it's
the domain team then, what's the real point of having these
requirements?

- How are we going to support the creation and maintnance of these g-r
 in the gate? Are they actually worth it?


While I understand why we have g-r, I really don't think it'll scale
for a broader community as we're envisioning it. With that in mind,
I'd probably recommend to have 1 requirements group for the base-caas
integrated gate and let other projects and groups have their own
requirements. That is, non base-caas projects should get the versions
of their common requirements from g-r so that they won't conflict if
they're installed alongside with any of the base-caas projects. But
other than those base requirements, I'd let non base-caas projects
have what they need (which is pretty much what heat's team has done
with the contrib packages, AFAICT).

I know this opens the doors for version conflicts between the
requirements of non base-caas projects but I think this is a more
welcoming (and non-blocking) way to do things until we've a better
one.

Hope the above makes some sense I blame the lack of coffee if it
doesn't,
Flavio

That said, I'd like to suggest another possible workaround: in Heat we
keep resource plugins for non-official projects in the /contrib tree,
and each plugin has it's own setup.cfg and requirements.txt. For example:

http://git.openstack.org/cgit/openstack/heat/tree/contrib/heat_barbican

So the user has the option to install each plugin, and it comes with its
own requirements, independent of the main Heat installation. Perhaps
Rally could consider something similar.


Seems like a very reasonable solution to me.

Thanks for bringing it up,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
@flaper87
Flavio Percoco


pgpSTIBS8VTg9.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-14 Thread Sean Dague
On 01/14/2015 03:55 AM, Flavio Percoco wrote:
 On 13/01/15 14:31 -0500, Sean Dague wrote:
 On 01/13/2015 02:10 PM, Jay Pipes wrote:
 On 01/13/2015 12:39 PM, Zane Bitter wrote:
 On 13/01/15 10:01, Jeremy Stanley wrote:
 On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
 Why doesn't rally just remove itself from projects.txt, then there
 would be no restrictions on what it adds.

 I second this recommendation. If Rally wants to depend on things
 which are not part of OpenStack and not required to run or test
 OpenStack in an official capacity (and since it's not an official
 OpenStack project it's entirely within its rights to want to do
 that), then forcing it to comply with the list of requirements for
 official OpenStack projects is an unnecessarily restrictive choice.

 FWIW I don't really agree with this advice. The purpose of
 openstack/requirements is to ensure that it's possible to create a
 distribution of OpenStack without conflicting version requirements, not
 to force every project to use every dependency listed. As such, some
 co-ordination around client libraries for related projects seems
 like it
 ought to be an uncontroversial addition. (Obviously it's easy to
 imagine
 potential additions that would legitimately be highly controversial,
 but
 IMHO this isn't one of them.) Saying that we require people to use our
 tools to get into the club but that our tools are not going to
 accommodate them in any way until they are in sounds a bit too close to
 Go away to my ears.

 +1

 I think as we grow global-requirements probably needs to be broken up
 into functional lists. What's allowed in base-iass, what's allowed in
 application services, what's allowed in docs, etc, etc.

 The complexity of keeping the g-r list actually working goes up sort of
 n^2 as the size of it, because pip doesn't have a real solver. This is a
 barely functional set of plate spinning so just add everything misses
 the point that the breaks get more and more difficult to unwind.

 If someone is up for stepping up and contributing to breaking up into
 domains, please do. But I think while this remains a global list we
 really do need to be a bit careful here.
 
 Do we really need a per-domain g-r?
 
 With the upcoming changes in the openstack governance model, I believe
 this won't scale even if we break it into per-domain basis. Some
 questions that need answeres are:
 
 - Who's going to review the per domain g-r ?
 
 If it's a centralized team, then it won't definitely scale. If it's
 the domain team then, what's the real point of having these
 requirements?
 
 - How are we going to support the creation and maintnance of these g-r
  in the gate? Are they actually worth it?
 
 
 While I understand why we have g-r, I really don't think it'll scale
 for a broader community as we're envisioning it. With that in mind,
 I'd probably recommend to have 1 requirements group for the base-caas
 integrated gate and let other projects and groups have their own
 requirements. That is, non base-caas projects should get the versions
 of their common requirements from g-r so that they won't conflict if
 they're installed alongside with any of the base-caas projects. But
 other than those base requirements, I'd let non base-caas projects
 have what they need (which is pretty much what heat's team has done
 with the contrib packages, AFAICT).
 
 I know this opens the doors for version conflicts between the
 requirements of non base-caas projects but I think this is a more
 welcoming (and non-blocking) way to do things until we've a better
 one.
 
 Hope the above makes some sense I blame the lack of coffee if it
 doesn't,
 Flavio

We have the base infrastructure of that in devstack today with a SOFT
requirements update -
https://github.com/openstack-dev/devstack/blob/master/functions-common#L1696-L1704

If you are not in projects.txt, and you set your REQUIREMENTS_MODE=soft
in your devstack run, we will leave extra dependencies untouched.
Probably a few more bits required to make it really easy to expose, but
that direction is also feasable.

The reason I brought up domains though is that some groups really want
the facility to have common requirements lists across a domain. Like the
OpenStack Docs team. They've currently inserted a ton of stuff in g-r
that really shouldn't be there because they have a lot of git trees, and
the synchronization is a nice feature.

However, if there were domain specific lists, that would be fine. A
collection of projects that want to coordinate could all agree on a
domain specific set of lists, and off we go. Maybe that list wouldn't
even need to be contained in the main repo, it could be in a sub repo so
have different approvers.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-14 Thread Davanum Srinivas
Flavio,

fyi, nova-docker is aspiring to be part of nova tree itself, but we
had a couple of python libraries we needed for docker that are not in
global, so we used the soft requirements to do that.

https://github.com/stackforge/nova-docker/blob/master/contrib/devstack/gate_hook.sh#L16

thanks,
dims

On Wed, Jan 14, 2015 at 7:50 AM, Sean Dague s...@dague.net wrote:
 On 01/14/2015 03:55 AM, Flavio Percoco wrote:
 On 13/01/15 14:31 -0500, Sean Dague wrote:
 On 01/13/2015 02:10 PM, Jay Pipes wrote:
 On 01/13/2015 12:39 PM, Zane Bitter wrote:
 On 13/01/15 10:01, Jeremy Stanley wrote:
 On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
 Why doesn't rally just remove itself from projects.txt, then there
 would be no restrictions on what it adds.

 I second this recommendation. If Rally wants to depend on things
 which are not part of OpenStack and not required to run or test
 OpenStack in an official capacity (and since it's not an official
 OpenStack project it's entirely within its rights to want to do
 that), then forcing it to comply with the list of requirements for
 official OpenStack projects is an unnecessarily restrictive choice.

 FWIW I don't really agree with this advice. The purpose of
 openstack/requirements is to ensure that it's possible to create a
 distribution of OpenStack without conflicting version requirements, not
 to force every project to use every dependency listed. As such, some
 co-ordination around client libraries for related projects seems
 like it
 ought to be an uncontroversial addition. (Obviously it's easy to
 imagine
 potential additions that would legitimately be highly controversial,
 but
 IMHO this isn't one of them.) Saying that we require people to use our
 tools to get into the club but that our tools are not going to
 accommodate them in any way until they are in sounds a bit too close to
 Go away to my ears.

 +1

 I think as we grow global-requirements probably needs to be broken up
 into functional lists. What's allowed in base-iass, what's allowed in
 application services, what's allowed in docs, etc, etc.

 The complexity of keeping the g-r list actually working goes up sort of
 n^2 as the size of it, because pip doesn't have a real solver. This is a
 barely functional set of plate spinning so just add everything misses
 the point that the breaks get more and more difficult to unwind.

 If someone is up for stepping up and contributing to breaking up into
 domains, please do. But I think while this remains a global list we
 really do need to be a bit careful here.

 Do we really need a per-domain g-r?

 With the upcoming changes in the openstack governance model, I believe
 this won't scale even if we break it into per-domain basis. Some
 questions that need answeres are:

 - Who's going to review the per domain g-r ?

 If it's a centralized team, then it won't definitely scale. If it's
 the domain team then, what's the real point of having these
 requirements?

 - How are we going to support the creation and maintnance of these g-r
  in the gate? Are they actually worth it?


 While I understand why we have g-r, I really don't think it'll scale
 for a broader community as we're envisioning it. With that in mind,
 I'd probably recommend to have 1 requirements group for the base-caas
 integrated gate and let other projects and groups have their own
 requirements. That is, non base-caas projects should get the versions
 of their common requirements from g-r so that they won't conflict if
 they're installed alongside with any of the base-caas projects. But
 other than those base requirements, I'd let non base-caas projects
 have what they need (which is pretty much what heat's team has done
 with the contrib packages, AFAICT).

 I know this opens the doors for version conflicts between the
 requirements of non base-caas projects but I think this is a more
 welcoming (and non-blocking) way to do things until we've a better
 one.

 Hope the above makes some sense I blame the lack of coffee if it
 doesn't,
 Flavio

 We have the base infrastructure of that in devstack today with a SOFT
 requirements update -
 https://github.com/openstack-dev/devstack/blob/master/functions-common#L1696-L1704

 If you are not in projects.txt, and you set your REQUIREMENTS_MODE=soft
 in your devstack run, we will leave extra dependencies untouched.
 Probably a few more bits required to make it really easy to expose, but
 that direction is also feasable.

 The reason I brought up domains though is that some groups really want
 the facility to have common requirements lists across a domain. Like the
 OpenStack Docs team. They've currently inserted a ton of stuff in g-r
 that really shouldn't be there because they have a lot of git trees, and
 the synchronization is a nice feature.

 However, if there were domain specific lists, that would be fine. A
 collection of projects that want to coordinate could all agree on a
 domain specific set of lists, and off we go. Maybe that list 

Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Zane Bitter

On 13/01/15 10:01, Jeremy Stanley wrote:

On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:

Why doesn't rally just remove itself from projects.txt, then there
would be no restrictions on what it adds.


I second this recommendation. If Rally wants to depend on things
which are not part of OpenStack and not required to run or test
OpenStack in an official capacity (and since it's not an official
OpenStack project it's entirely within its rights to want to do
that), then forcing it to comply with the list of requirements for
official OpenStack projects is an unnecessarily restrictive choice.


FWIW I don't really agree with this advice. The purpose of 
openstack/requirements is to ensure that it's possible to create a 
distribution of OpenStack without conflicting version requirements, not 
to force every project to use every dependency listed. As such, some 
co-ordination around client libraries for related projects seems like it 
ought to be an uncontroversial addition. (Obviously it's easy to imagine 
potential additions that would legitimately be highly controversial, but 
IMHO this isn't one of them.) Saying that we require people to use our 
tools to get into the club but that our tools are not going to 
accommodate them in any way until they are in sounds a bit too close to 
Go away to my ears.



That said, I'd like to suggest another possible workaround: in Heat we 
keep resource plugins for non-official projects in the /contrib tree, 
and each plugin has it's own setup.cfg and requirements.txt. For example:


http://git.openstack.org/cgit/openstack/heat/tree/contrib/heat_barbican

So the user has the option to install each plugin, and it comes with its 
own requirements, independent of the main Heat installation. Perhaps 
Rally could consider something similar.


cheers,
Zane.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Sean Dague
On 01/13/2015 02:10 PM, Jay Pipes wrote:
 On 01/13/2015 12:39 PM, Zane Bitter wrote:
 On 13/01/15 10:01, Jeremy Stanley wrote:
 On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
 Why doesn't rally just remove itself from projects.txt, then there
 would be no restrictions on what it adds.

 I second this recommendation. If Rally wants to depend on things
 which are not part of OpenStack and not required to run or test
 OpenStack in an official capacity (and since it's not an official
 OpenStack project it's entirely within its rights to want to do
 that), then forcing it to comply with the list of requirements for
 official OpenStack projects is an unnecessarily restrictive choice.

 FWIW I don't really agree with this advice. The purpose of
 openstack/requirements is to ensure that it's possible to create a
 distribution of OpenStack without conflicting version requirements, not
 to force every project to use every dependency listed. As such, some
 co-ordination around client libraries for related projects seems like it
 ought to be an uncontroversial addition. (Obviously it's easy to imagine
 potential additions that would legitimately be highly controversial, but
 IMHO this isn't one of them.) Saying that we require people to use our
 tools to get into the club but that our tools are not going to
 accommodate them in any way until they are in sounds a bit too close to
 Go away to my ears.
 
 +1

I think as we grow global-requirements probably needs to be broken up
into functional lists. What's allowed in base-iass, what's allowed in
application services, what's allowed in docs, etc, etc.

The complexity of keeping the g-r list actually working goes up sort of
n^2 as the size of it, because pip doesn't have a real solver. This is a
barely functional set of plate spinning so just add everything misses
the point that the breaks get more and more difficult to unwind.

If someone is up for stepping up and contributing to breaking up into
domains, please do. But I think while this remains a global list we
really do need to be a bit careful here.

 That said, I'd like to suggest another possible workaround: in Heat we
 keep resource plugins for non-official projects in the /contrib tree,
 and each plugin has it's own setup.cfg and requirements.txt. For example:

 http://git.openstack.org/cgit/openstack/heat/tree/contrib/heat_barbican

 So the user has the option to install each plugin, and it comes with its
 own requirements, independent of the main Heat installation. Perhaps
 Rally could consider something similar.
 
 Seems like a very reasonable solution to me.
 
 Thanks for bringing it up,
 -jay
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Jay Pipes

On 01/13/2015 12:39 PM, Zane Bitter wrote:

On 13/01/15 10:01, Jeremy Stanley wrote:

On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:

Why doesn't rally just remove itself from projects.txt, then there
would be no restrictions on what it adds.


I second this recommendation. If Rally wants to depend on things
which are not part of OpenStack and not required to run or test
OpenStack in an official capacity (and since it's not an official
OpenStack project it's entirely within its rights to want to do
that), then forcing it to comply with the list of requirements for
official OpenStack projects is an unnecessarily restrictive choice.


FWIW I don't really agree with this advice. The purpose of
openstack/requirements is to ensure that it's possible to create a
distribution of OpenStack without conflicting version requirements, not
to force every project to use every dependency listed. As such, some
co-ordination around client libraries for related projects seems like it
ought to be an uncontroversial addition. (Obviously it's easy to imagine
potential additions that would legitimately be highly controversial, but
IMHO this isn't one of them.) Saying that we require people to use our
tools to get into the club but that our tools are not going to
accommodate them in any way until they are in sounds a bit too close to
Go away to my ears.


+1


That said, I'd like to suggest another possible workaround: in Heat we
keep resource plugins for non-official projects in the /contrib tree,
and each plugin has it's own setup.cfg and requirements.txt. For example:

http://git.openstack.org/cgit/openstack/heat/tree/contrib/heat_barbican

So the user has the option to install each plugin, and it comes with its
own requirements, independent of the main Heat installation. Perhaps
Rally could consider something similar.


Seems like a very reasonable solution to me.

Thanks for bringing it up,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Serg Melikyan
I support this, it will help to not break compatibility with
global-requirements for sole purpose to include python client for other
stackforge project.

On Tue, Jan 13, 2015 at 12:14 AM, Boris Pavlovic bo...@pavlovic.me wrote:

 Hello TC,

 I would like to propose to allow adding all python-clients from stackforge
 (that are regarding global-requirements) to global requirements.

 It doesn't cost anything and simplifies life for everybody on stackforge.

 P.S. We already have billions libs in global requirements that aren't even
 on stackforge.
 Having few more or less doesn't make any sense...


 Best regards,
 Boris Pavlovic

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
http://mirantis.com | smelik...@mirantis.com

+7 (495) 640-4904, 0261
+7 (903) 156-0836
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Flavio Percoco

On 13/01/15 01:14 +0400, Boris Pavlovic wrote:

Hello TC, 

I would like to propose to allow adding all python-clients from stackforge
(that are regarding global-requirements) to global requirements. 


I believe this is not a thing for the TC to decided but the dev
community, especially the infra team, requirement's cores, etc.

What python-clients are you referring to? Are you referring to libs
that live in stackforge or dependencies for things that live in
stackforge?

Do you have an (or many) example?

Flavio

--
@flaper87
Flavio Percoco


pgps2sjt5rSu5.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Boris Pavlovic
Flavio,

Current 2 python clients are Mistral and Murano that I would like to use in
Rally.

I can't add them to requirements = so I need to work everywhere in
lazy-mode + print exceptions to end users
e.g. you don't have this client, so install it - which is really not user
friendly.

Best regards,
Boris Pavlovic

On Tue, Jan 13, 2015 at 1:46 PM, Flavio Percoco fla...@redhat.com wrote:

 On 13/01/15 01:14 +0400, Boris Pavlovic wrote:

 Hello TC,

 I would like to propose to allow adding all python-clients from stackforge
 (that are regarding global-requirements) to global requirements.


 I believe this is not a thing for the TC to decided but the dev
 community, especially the infra team, requirement's cores, etc.

 What python-clients are you referring to? Are you referring to libs
 that live in stackforge or dependencies for things that live in
 stackforge?

 Do you have an (or many) example?

 Flavio

 --
 @flaper87
 Flavio Percoco

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Sean Dague
On 01/13/2015 06:56 AM, Boris Pavlovic wrote:
 Flavio, 
 
 Current 2 python clients are Mistral and Murano that I would like to use
 in Rally. 
 
 I can't add them to requirements = so I need to work everywhere in
 lazy-mode + print exceptions to end users 
 e.g. you don't have this client, so install it - which is really not
 user friendly.

Why doesn't rally just remove itself from projects.txt, then there would
be no restrictions on what it adds.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Jeremy Stanley
On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
 Why doesn't rally just remove itself from projects.txt, then there
 would be no restrictions on what it adds.

I second this recommendation. If Rally wants to depend on things
which are not part of OpenStack and not required to run or test
OpenStack in an official capacity (and since it's not an official
OpenStack project it's entirely within its rights to want to do
that), then forcing it to comply with the list of requirements for
official OpenStack projects is an unnecessarily restrictive choice.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Boris Pavlovic
Jeremy, Sean,


First of all because I would like to make a Rally official part of
OpenStack and if I remove
Rally from project.txt this won't happen.

On other side I would like to make OpenStack better. And OpenStack includes
as well
projects on StackForge. That for some reason were rejected. In my opinion
all projects
in equal portion deserver good testing framework.
And I as PTL of Rally would like to adopt it for everybody.


P.S. It's sad that in one thread we are talking about making Big Tenant and
how it is cool
to remove programs. In another we are blocking adding python clients from
projects
that are not Core and suggesting another making incompatible with
OpenStack.


Best regards,
Boris Pavlovic


On Tue, Jan 13, 2015 at 6:01 PM, Jeremy Stanley fu...@yuggoth.org wrote:

 On 2015-01-13 07:46:38 -0500 (-0500), Sean Dague wrote:
  Why doesn't rally just remove itself from projects.txt, then there
  would be no restrictions on what it adds.

 I second this recommendation. If Rally wants to depend on things
 which are not part of OpenStack and not required to run or test
 OpenStack in an official capacity (and since it's not an official
 OpenStack project it's entirely within its rights to want to do
 that), then forcing it to comply with the list of requirements for
 official OpenStack projects is an unnecessarily restrictive choice.
 --
 Jeremy Stanley

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][python-clients] More freedom for all python clients

2015-01-13 Thread Jeremy Stanley
On 2015-01-13 19:53:28 +0400 (+0400), Boris Pavlovic wrote:
[...]
 P.S. It's sad that in one thread we are talking about making Big
 Tenant and how it is cool  to remove programs. In another we are
 blocking adding python clients from projects  that are not Core
 and suggesting another making incompatible with OpenStack. 

The big tent proposal is not just a free-for-all, and will entail
most of the same barriers to entry for officialness as you currently
see for programs (aside from addresses similar needs as a current
official team). Quality control and selectiveness of design will
still be extremely important, and I don't see our
openstack/requirements tooling, workflow and choices changing
significantly in the face of that.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev