Re: [openstack-dev] [Infra][Solum][Mistral] New class of requirements for Stackforge projects

2014-06-25 Thread Adrian Otto 
Ok,

I submitted and abandoned a couple of reviews[1][2] for a solution aimed to 
meet my goals without adding a new per-project requirements file. The flaw with 
this approach is that pip may install other requirements when installing the 
one(s) loaded from the fallback mirror, and those may conflict with the ones 
loaded from the primary mirror.

After discussing this further in #openstack-infra this evening, we should give 
serious consideration to adding python-mistralclient to global requirements. I 
have posted a review[3] for that to get input from the requirements review team.

Thanks,

Adrian

[1] https://review.openstack.org/102716
[2] https://review.openstack.org/102719
[3] https://review.openstack.org/102738

On Jun 25, 2014, at 9:51 PM, Matthew Oliver 
mailto:m...@oliver.net.au>> wrote:


On Jun 26, 2014 12:12 PM, "Angus Salkeld" 
mailto:angus.salk...@rackspace.com>> wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 25/06/14 15:13, Clark Boylan wrote:
> > On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto 
> > mailto:adrian.o...@rackspace.com>> wrote:
> >> Hello,
> >>
> >> Solum has run into a constraint with the current scheme for requirements 
> >> management within the OpenStack CI system. We have a proposal for dealing 
> >> with this constraint that involves making a contribution to 
> >> openstack-infra. This message explains the constraint, and our proposal 
> >> for addressing it.
> >>
> >> == Background ==
> >>
> >> OpenStack uses a list of global requirements in the requirements repo[1], 
> >> and each project has it’s own requirements.txt and test-requirements.txt 
> >> files. The requirements are satisfied by gate jobs using pip configured to 
> >> use the pypi.openstack.org mirror, which is 
> >> periodically updated with new content from 
> >> pypi.python.org. One motivation for doing this is 
> >> that pypi.python.org may not be as fast or as 
> >> reliable as a local mirror. The gate/check jobs for the projects use the 
> >> OpenStack internal pypi mirror to ensure stability.
> >>
> >> The OpenStack CI system will sync up the requirements across all the 
> >> official projects and will create reviews in the participating projects 
> >> for any mis-matches. Solum is one of these projects, and enjoys this 
> >> feature.
> >>
> >> Another motivation is so that users of OpenStack will have one single set 
> >> of python package requirements/dependencies to install and run the 
> >> individual OpenStack components.
> >>
> >> == Problem ==
> >>
> >> Stackforge projects listed in openstack/requirements/projects.txt that 
> >> decide to depend on each other (for example, Solum wanting to list 
> >> mistralclient as a requirement) are unable to, because they are not yet 
> >> integrated, and are not listed in 
> >> openstack/requirements/global-requirements.txt yet. This means that in 
> >> order to depend on each other, a project must withdraw from projects.txt 
> >> and begin using pip with pypi.poython.org to 
> >> satisfy all of their requirements.I strongly dislike this option.
> >>
> >> Mistral is still evolving rapidly, and we don’t think it makes sense for 
> >> them to pursue integration wight now. The upstream distributions who 
> >> include packages to support OpenStack will also prefer not to deal with a 
> >> requirement that will be cutting a new version every week or two in order 
> >> to satisfy evolving needs as Solum and other consumers of Mistral help 
> >> refine how it works.
> >>
> >> == Proposal ==
> >>
> >> We want the best of both worlds. We want the freedom to innovate and use 
> >> new software for a limited selection of stackforge projects, and still use 
> >> the OpenStack pypi server to satisfy my regular requirements. We want the 
> >> speed and reliability of using our local mirror, and users of Solum to use 
> >> a matching set of requirements for all the things that we use, and 
> >> integrated projects use. We want to continue getting the reviews that 
> >> bring us up to date with new requirements versions.
> >>
> >> We propose that we submit an enhancement to the gate/check job setup that 
> >> will:
> >>
> >> 1) Begin (as it does today) by satisfying global-requirements.txt and my 
> >> local project’s requirements.txt and test-requirements.txt using the local 
> >> OpenStack pypi mirror.
> >> 2) After all requirements are satisfied, check the name of my project. If 
> >> it begins with ‘stackforge/‘ then look for a stackforge-requirements.txt 
> >> file. If one exists, reconfigure pip to switch to use 
> >> pypi.python.org, and satisfy the requirements 
> >> listed in the file. We will list mistralclient there, and get the latest 
> >> tagged/released version of that.
> >>
> > I am reasonably sure that if you remove yourself from the
> > openstack/requirements project list

Re: [openstack-dev] [Infra][Solum][Mistral] New class of requirements for Stackforge projects

2014-06-25 Thread Matthew Oliver 
On Jun 26, 2014 12:12 PM, "Angus Salkeld" 
wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 25/06/14 15:13, Clark Boylan wrote:
> > On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto 
wrote:
> >> Hello,
> >>
> >> Solum has run into a constraint with the current scheme for
requirements management within the OpenStack CI system. We have a proposal
for dealing with this constraint that involves making a contribution to
openstack-infra. This message explains the constraint, and our proposal for
addressing it.
> >>
> >> == Background ==
> >>
> >> OpenStack uses a list of global requirements in the requirements
repo[1], and each project has it’s own requirements.txt and
test-requirements.txt files. The requirements are satisfied by gate jobs
using pip configured to use the pypi.openstack.org mirror, which is
periodically updated with new content from pypi.python.org. One motivation
for doing this is that pypi.python.org may not be as fast or as reliable as
a local mirror. The gate/check jobs for the projects use the OpenStack
internal pypi mirror to ensure stability.
> >>
> >> The OpenStack CI system will sync up the requirements across all the
official projects and will create reviews in the participating projects for
any mis-matches. Solum is one of these projects, and enjoys this feature.
> >>
> >> Another motivation is so that users of OpenStack will have one single
set of python package requirements/dependencies to install and run the
individual OpenStack components.
> >>
> >> == Problem ==
> >>
> >> Stackforge projects listed in openstack/requirements/projects.txt that
decide to depend on each other (for example, Solum wanting to list
mistralclient as a requirement) are unable to, because they are not yet
integrated, and are not listed in
openstack/requirements/global-requirements.txt yet. This means that in
order to depend on each other, a project must withdraw from projects.txt
and begin using pip with pypi.poython.org to satisfy all of their
requirements.I strongly dislike this option.
> >>
> >> Mistral is still evolving rapidly, and we don’t think it makes sense
for them to pursue integration wight now. The upstream distributions who
include packages to support OpenStack will also prefer not to deal with a
requirement that will be cutting a new version every week or two in order
to satisfy evolving needs as Solum and other consumers of Mistral help
refine how it works.
> >>
> >> == Proposal ==
> >>
> >> We want the best of both worlds. We want the freedom to innovate and
use new software for a limited selection of stackforge projects, and still
use the OpenStack pypi server to satisfy my regular requirements. We want
the speed and reliability of using our local mirror, and users of Solum to
use a matching set of requirements for all the things that we use, and
integrated projects use. We want to continue getting the reviews that bring
us up to date with new requirements versions.
> >>
> >> We propose that we submit an enhancement to the gate/check job setup
that will:
> >>
> >> 1) Begin (as it does today) by satisfying global-requirements.txt and
my local project’s requirements.txt and test-requirements.txt using the
local OpenStack pypi mirror.
> >> 2) After all requirements are satisfied, check the name of my project.
If it begins with ‘stackforge/‘ then look for a stackforge-requirements.txt
file. If one exists, reconfigure pip to switch to use pypi.python.org, and
satisfy the requirements listed in the file. We will list mistralclient
there, and get the latest tagged/released version of that.
> >>
> > I am reasonably sure that if you remove yourself from the
> > openstack/requirements project list this is basically how it will
> > work. Pip is configured to use the OpenStack mirror and fall back on
> > pypi.python.org for packages not available on the OpenStack mirror
> > [2]. So I don't think there is any work to do here with additional
> > requirements files. It should just work. Adding a new requirements
> > file will just make things more confusing for packagers and consumers
> > of your software.
>
> Adrian I know this is not the optimal solution, but I think this is
> the most pragmatic solution (esp. given we need to progress and not be
held
> up by this), most stackforge projects are in the same boat as us.
> As far as pypi breakages (most are easily fixable by restricting the
> package versions if we get an issue with a new release
> of *random-api-breaking-package*).
>

I've looked through the infra choose mirror code, and Clark is correct. If
the project isn't in the projects.txt file they will only access to
pypi.openstack.org however if removed then it will first check
pypi.openstack.org and then fall back to to pypi.python.org. I think the
only real solution is what Angus mentioned, remove yourself from
projects.txt at least until all your dependencies can be provided by
pypi.openstack.org or another solution is put into place. In the mean time
you can at least progress and

Re: [openstack-dev] [Infra][Solum][Mistral] New class of requirements for Stackforge projects

2014-06-25 Thread Angus Salkeld 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 25/06/14 15:13, Clark Boylan wrote:
> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto  
> wrote:
>> Hello,
>>
>> Solum has run into a constraint with the current scheme for requirements 
>> management within the OpenStack CI system. We have a proposal for dealing 
>> with this constraint that involves making a contribution to openstack-infra. 
>> This message explains the constraint, and our proposal for addressing it.
>>
>> == Background ==
>>
>> OpenStack uses a list of global requirements in the requirements repo[1], 
>> and each project has it’s own requirements.txt and test-requirements.txt 
>> files. The requirements are satisfied by gate jobs using pip configured to 
>> use the pypi.openstack.org mirror, which is periodically updated with new 
>> content from pypi.python.org. One motivation for doing this is that 
>> pypi.python.org may not be as fast or as reliable as a local mirror. The 
>> gate/check jobs for the projects use the OpenStack internal pypi mirror to 
>> ensure stability.
>>
>> The OpenStack CI system will sync up the requirements across all the 
>> official projects and will create reviews in the participating projects for 
>> any mis-matches. Solum is one of these projects, and enjoys this feature.
>>
>> Another motivation is so that users of OpenStack will have one single set of 
>> python package requirements/dependencies to install and run the individual 
>> OpenStack components.
>>
>> == Problem ==
>>
>> Stackforge projects listed in openstack/requirements/projects.txt that 
>> decide to depend on each other (for example, Solum wanting to list 
>> mistralclient as a requirement) are unable to, because they are not yet 
>> integrated, and are not listed in 
>> openstack/requirements/global-requirements.txt yet. This means that in order 
>> to depend on each other, a project must withdraw from projects.txt and begin 
>> using pip with pypi.poython.org to satisfy all of their requirements.I 
>> strongly dislike this option.
>>
>> Mistral is still evolving rapidly, and we don’t think it makes sense for 
>> them to pursue integration wight now. The upstream distributions who include 
>> packages to support OpenStack will also prefer not to deal with a 
>> requirement that will be cutting a new version every week or two in order to 
>> satisfy evolving needs as Solum and other consumers of Mistral help refine 
>> how it works.
>>
>> == Proposal ==
>>
>> We want the best of both worlds. We want the freedom to innovate and use new 
>> software for a limited selection of stackforge projects, and still use the 
>> OpenStack pypi server to satisfy my regular requirements. We want the speed 
>> and reliability of using our local mirror, and users of Solum to use a 
>> matching set of requirements for all the things that we use, and integrated 
>> projects use. We want to continue getting the reviews that bring us up to 
>> date with new requirements versions.
>>
>> We propose that we submit an enhancement to the gate/check job setup that 
>> will:
>>
>> 1) Begin (as it does today) by satisfying global-requirements.txt and my 
>> local project’s requirements.txt and test-requirements.txt using the local 
>> OpenStack pypi mirror.
>> 2) After all requirements are satisfied, check the name of my project. If it 
>> begins with ‘stackforge/‘ then look for a stackforge-requirements.txt file. 
>> If one exists, reconfigure pip to switch to use pypi.python.org, and satisfy 
>> the requirements listed in the file. We will list mistralclient there, and 
>> get the latest tagged/released version of that.
>>
> I am reasonably sure that if you remove yourself from the
> openstack/requirements project list this is basically how it will
> work. Pip is configured to use the OpenStack mirror and fall back on
> pypi.python.org for packages not available on the OpenStack mirror
> [2]. So I don't think there is any work to do here with additional
> requirements files. It should just work. Adding a new requirements
> file will just make things more confusing for packagers and consumers
> of your software.

Adrian I know this is not the optimal solution, but I think this is
the most pragmatic solution (esp. given we need to progress and not be held
up by this), most stackforge projects are in the same boat as us.
As far as pypi breakages (most are easily fixable by restricting the
package versions if we get an issue with a new release
of *random-api-breaking-package*).


>>
>> == Call To Action ==
>>
>> What do you think of this approach to satisfy a balance of interests? 
>> Everything remains the same for OpenStack projects, and Stackforge projects 
>> get a new feature that allows them to require software that has not yet been 
>> integrated. Are there even better options that we should consider?
>>
>> Thanks,
>>
>> Adrian Otto
>>
>>
>> References:
>> [1] https://review.openstack.org/openstack/requirements
> 
> For what it is worth the Infra team has also been l

Re: [openstack-dev] [Infra][Solum][Mistral] New class of requirements for Stackforge projects

2014-06-24 Thread Clark Boylan 
On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto  wrote:
> Hello,
>
> Solum has run into a constraint with the current scheme for requirements 
> management within the OpenStack CI system. We have a proposal for dealing 
> with this constraint that involves making a contribution to openstack-infra. 
> This message explains the constraint, and our proposal for addressing it.
>
> == Background ==
>
> OpenStack uses a list of global requirements in the requirements repo[1], and 
> each project has it’s own requirements.txt and test-requirements.txt files. 
> The requirements are satisfied by gate jobs using pip configured to use the 
> pypi.openstack.org mirror, which is periodically updated with new content 
> from pypi.python.org. One motivation for doing this is that pypi.python.org 
> may not be as fast or as reliable as a local mirror. The gate/check jobs for 
> the projects use the OpenStack internal pypi mirror to ensure stability.
>
> The OpenStack CI system will sync up the requirements across all the official 
> projects and will create reviews in the participating projects for any 
> mis-matches. Solum is one of these projects, and enjoys this feature.
>
> Another motivation is so that users of OpenStack will have one single set of 
> python package requirements/dependencies to install and run the individual 
> OpenStack components.
>
> == Problem ==
>
> Stackforge projects listed in openstack/requirements/projects.txt that decide 
> to depend on each other (for example, Solum wanting to list mistralclient as 
> a requirement) are unable to, because they are not yet integrated, and are 
> not listed in openstack/requirements/global-requirements.txt yet. This means 
> that in order to depend on each other, a project must withdraw from 
> projects.txt and begin using pip with pypi.poython.org to satisfy all of 
> their requirements.I strongly dislike this option.
>
> Mistral is still evolving rapidly, and we don’t think it makes sense for them 
> to pursue integration wight now. The upstream distributions who include 
> packages to support OpenStack will also prefer not to deal with a requirement 
> that will be cutting a new version every week or two in order to satisfy 
> evolving needs as Solum and other consumers of Mistral help refine how it 
> works.
>
> == Proposal ==
>
> We want the best of both worlds. We want the freedom to innovate and use new 
> software for a limited selection of stackforge projects, and still use the 
> OpenStack pypi server to satisfy my regular requirements. We want the speed 
> and reliability of using our local mirror, and users of Solum to use a 
> matching set of requirements for all the things that we use, and integrated 
> projects use. We want to continue getting the reviews that bring us up to 
> date with new requirements versions.
>
> We propose that we submit an enhancement to the gate/check job setup that 
> will:
>
> 1) Begin (as it does today) by satisfying global-requirements.txt and my 
> local project’s requirements.txt and test-requirements.txt using the local 
> OpenStack pypi mirror.
> 2) After all requirements are satisfied, check the name of my project. If it 
> begins with ‘stackforge/‘ then look for a stackforge-requirements.txt file. 
> If one exists, reconfigure pip to switch to use pypi.python.org, and satisfy 
> the requirements listed in the file. We will list mistralclient there, and 
> get the latest tagged/released version of that.
>
I am reasonably sure that if you remove yourself from the
openstack/requirements project list this is basically how it will
work. Pip is configured to use the OpenStack mirror and fall back on
pypi.python.org for packages not available on the OpenStack mirror
[2]. So I don't think there is any work to do here with additional
requirements files. It should just work. Adding a new requirements
file will just make things more confusing for packagers and consumers
of your software.
>
> == Call To Action ==
>
> What do you think of this approach to satisfy a balance of interests? 
> Everything remains the same for OpenStack projects, and Stackforge projects 
> get a new feature that allows them to require software that has not yet been 
> integrated. Are there even better options that we should consider?
>
> Thanks,
>
> Adrian Otto
>
>
> References:
> [1] https://review.openstack.org/openstack/requirements

For what it is worth the Infra team has also been looking at
potentially using something like bandersnatch to mirror all of pypi
which is now a possibility because OpenStack doesn't depend on
packages that are hosted external to pypi. We would then do
requirements enforcement via checks rather than explicit use of a
restricted mirror. There are some things to sort out like platform
dependent wheels (I am not sure that any OpenStack project directly
consumes these but I have found them to be quite handy) and the
potential need for more enforcement to keep this working, but I think
this is a possibility.

Clark

[2]

[openstack-dev] [Infra][Solum][Mistral] New class of requirements for Stackforge projects

2014-06-24 Thread Adrian Otto 
Hello,

Solum has run into a constraint with the current scheme for requirements 
management within the OpenStack CI system. We have a proposal for dealing with 
this constraint that involves making a contribution to openstack-infra. This 
message explains the constraint, and our proposal for addressing it.

== Background ==

OpenStack uses a list of global requirements in the requirements repo[1], and 
each project has it’s own requirements.txt and test-requirements.txt files. The 
requirements are satisfied by gate jobs using pip configured to use the 
pypi.openstack.org mirror, which is periodically updated with new content from 
pypi.python.org. One motivation for doing this is that pypi.python.org may not 
be as fast or as reliable as a local mirror. The gate/check jobs for the 
projects use the OpenStack internal pypi mirror to ensure stability.

The OpenStack CI system will sync up the requirements across all the official 
projects and will create reviews in the participating projects for any 
mis-matches. Solum is one of these projects, and enjoys this feature.

Another motivation is so that users of OpenStack will have one single set of 
python package requirements/dependencies to install and run the individual 
OpenStack components.

== Problem ==

Stackforge projects listed in openstack/requirements/projects.txt that decide 
to depend on each other (for example, Solum wanting to list mistralclient as a 
requirement) are unable to, because they are not yet integrated, and are not 
listed in openstack/requirements/global-requirements.txt yet. This means that 
in order to depend on each other, a project must withdraw from projects.txt and 
begin using pip with pypi.poython.org to satisfy all of their requirements.I 
strongly dislike this option.

Mistral is still evolving rapidly, and we don’t think it makes sense for them 
to pursue integration wight now. The upstream distributions who include 
packages to support OpenStack will also prefer not to deal with a requirement 
that will be cutting a new version every week or two in order to satisfy 
evolving needs as Solum and other consumers of Mistral help refine how it works.

== Proposal ==

We want the best of both worlds. We want the freedom to innovate and use new 
software for a limited selection of stackforge projects, and still use the 
OpenStack pypi server to satisfy my regular requirements. We want the speed and 
reliability of using our local mirror, and users of Solum to use a matching set 
of requirements for all the things that we use, and integrated projects use. We 
want to continue getting the reviews that bring us up to date with new 
requirements versions.

We propose that we submit an enhancement to the gate/check job setup that will:

1) Begin (as it does today) by satisfying global-requirements.txt and my local 
project’s requirements.txt and test-requirements.txt using the local OpenStack 
pypi mirror.
2) After all requirements are satisfied, check the name of my project. If it 
begins with ‘stackforge/‘ then look for a stackforge-requirements.txt file. If 
one exists, reconfigure pip to switch to use pypi.python.org, and satisfy the 
requirements listed in the file. We will list mistralclient there, and get the 
latest tagged/released version of that.

== Call To Action ==

What do you think of this approach to satisfy a balance of interests? 
Everything remains the same for OpenStack projects, and Stackforge projects get 
a new feature that allows them to require software that has not yet been 
integrated. Are there even better options that we should consider?

Thanks,

Adrian Otto


References:
[1] https://review.openstack.org/openstack/requirements
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev