Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-28 Thread Steven Dake (stdake) 
Tony

A+ on the explanation.

Thanks, dude, you rock!  That solves the exact problem.  I think what we need 
is an override for upper-constraints.txt in Kolla.

Regards
-steve

-Original Message-
From: Tony Breeds <t...@bakeyournoodle.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Monday, November 28, 2016 at 2:06 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [kolla][release][requirements] providing 
constraints for transitive dependencies

On Mon, Nov 28, 2016 at 08:15:17AM +, Steven Dake (stdake) wrote:
> Tony,
> 
> Are you indicating that all transitive dependencies (e.g. nova depends on 
x
> depends on y, y = version of dep we want to specify) are in
> global-requirements.txt?

So in order to reduce confusion I'll try to explain this in a little more
detail but the TL;DR is:

No only nova's direct requirements are in global-requirements.txt.  *all*
requirements are in upper-constraints.txt.

Now the longer version:

 * nova requires oslo.messaging, which is listed in 
openstack/nova:requirements.txt.
 * openstack/nova:requirements.txt is regularly updated by the proposal-bot 
with
   the relevant parts of global-requirements.txt
 * the nova project runs the 'check-requirements' jobs ensuring that
   everything in:
openstack/nova:requirements.txt
openstack/nova:test-requirements.txt
and relevant parts of openstack/nova:setup.cfg
   are listed in global-requirements.
 * As oslo.messaging is an OpenStack project it's bound by the same rules 
above
 * However any non-OpenStack libraries used by either nova or oslo.messaging
   are not in global-requirements.txt.  There is a nightly job that installs
   all of global-requirements.txt (and dependencies) and tracks the exact
   version of each library in a file ... upper-constraints.txt

So anything you need to install to use any part of OpenStack[1] will be 
listed
in upper-constraints.txt.

So if you really need to install version $x of library $foo you can edit
upper-constraints.txt to state that.

There are a couple of gotchas.  I recommend you use edit-constraints from
openstack-requirements (on pypi) to actually edit the constraints files.  
Also if
you're doing anything "too funky" you're probably best actually editing a
requirements and then using generate-constraints to actually generate the
constraints file.  That's time/network expensive but will result in a valid 
set
where editing the constraints file at will could result in an
incompatible and thus uninstallable system.

I really hope that helps and hasn't just confused the situation.

Yours Tony.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-28 Thread Tony Breeds 
On Mon, Nov 28, 2016 at 08:15:17AM +, Steven Dake (stdake) wrote:
> Tony,
> 
> Are you indicating that all transitive dependencies (e.g. nova depends on x
> depends on y, y = version of dep we want to specify) are in
> global-requirements.txt?

So in order to reduce confusion I'll try to explain this in a little more
detail but the TL;DR is:

No only nova's direct requirements are in global-requirements.txt.  *all*
requirements are in upper-constraints.txt.

Now the longer version:

 * nova requires oslo.messaging, which is listed in 
openstack/nova:requirements.txt.
 * openstack/nova:requirements.txt is regularly updated by the proposal-bot with
   the relevant parts of global-requirements.txt
 * the nova project runs the 'check-requirements' jobs ensuring that
   everything in:
openstack/nova:requirements.txt
openstack/nova:test-requirements.txt
and relevant parts of openstack/nova:setup.cfg
   are listed in global-requirements.
 * As oslo.messaging is an OpenStack project it's bound by the same rules above
 * However any non-OpenStack libraries used by either nova or oslo.messaging
   are not in global-requirements.txt.  There is a nightly job that installs
   all of global-requirements.txt (and dependencies) and tracks the exact
   version of each library in a file ... upper-constraints.txt

So anything you need to install to use any part of OpenStack[1] will be listed
in upper-constraints.txt.

So if you really need to install version $x of library $foo you can edit
upper-constraints.txt to state that.

There are a couple of gotchas.  I recommend you use edit-constraints from
openstack-requirements (on pypi) to actually edit the constraints files.  Also 
if
you're doing anything "too funky" you're probably best actually editing a
requirements and then using generate-constraints to actually generate the
constraints file.  That's time/network expensive but will result in a valid set
where editing the constraints file at will could result in an
incompatible and thus uninstallable system.

I really hope that helps and hasn't just confused the situation.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-28 Thread Steven Dake (stdake) 
Clark,

Cool didn’t know the transitive deps were specified in upper-constraints.txt.  
Learning new things 24/7 it seems ☺  I’m pretty sure we can work with that and 
just override the upper constraints file we use.

Regards
-steve


-Original Message-
From: Clark Boylan <cboy...@sapwetik.org>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Sunday, November 27, 2016 at 8:02 PM
To: "openstack-dev@lists.openstack.org" <openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [kolla][release][requirements] providing 
constraints for transitive dependencies

On Sun, Nov 27, 2016, at 06:53 PM, Jeffrey Zhang wrote:
> FWIW, python dependencies are fixed.
> Kolla is using the following command in Dockerfile for source code
> installation.
> 
> pip install -u /requirements/upper-constrains.txt /nova

I think you need a to use the -c option to set the constraints file.

> 
> then all python dependencies installed will use a fixed version from
> upper-constrains.txt.

Yup, constraints apply to transitive deps as well. Keep in mind you
don't have to use the published upper-constraints either, but they are
an easy way to use a known tested set that works with upstream gating.

> 
> [0]
> 
https://specs.openstack.org/openstack/openstack-specs/specs/requirements-management.html
> [1]
> 
https://github.com/openstack/requirements/blob/master/upper-constraints.txt
> 
> On Mon, Nov 28, 2016 at 10:41 AM, Steven Dake (stdake) <std...@cisco.com>
> wrote:
> 
> > Hey folks,
> >
> >
> >
> > I get a lot of requests for variance reduction of transitive 
dependencies
> > in Kolla’s containers.  As an example, we build from source nova.  Nova
> > itself we can specify a version to install in the containers during 
build
> > time.  Nova’s python dependencies, not so much.  Is there a best 
practice
> > for doing such in the python ecosystem?
> >
> >
> >
> > Binary distributions don’t typically suffer from this problem.  They
> > deliver one version of dependencies, and that is what you get.  That is
> > what a slew of folks are after with from source container builds.  Any
> > advice from the requirements team or release team welcome as the folks 
on
> > those teams have the most experience with this sort of thing.
> >
> >
> >
> > If this has been asked by someone else in a different context and
> > answered, a pointer to that discussion would work too J

> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-28 Thread Steven Dake (stdake) 
Tony,

Are you indicating that all transitive dependencies (e.g. nova depends on x 
depends on y, y = version of dep we want to specify) are in 
global-requirements.txt?

Regards
-steve


-Original Message-
From: Tony Breeds <t...@bakeyournoodle.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Date: Sunday, November 27, 2016 at 9:42 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Subject: Re: [openstack-dev] [kolla][release][requirements] providing 
constraints for transitive dependencies

On Mon, Nov 28, 2016 at 02:41:08AM +, Steven Dake (stdake) wrote:
> Hey folks,
> 
> I get a lot of requests for variance reduction of transitive dependencies 
in
> Kolla’s containers.  As an example, we build from source nova.  Nova 
itself
> we can specify a version to install in the containers during build time.
> Nova’s python dependencies, not so much.

I'm not certain I follow.  You're using upper-constraints for install time
consistent version selection, and that applies to *all* dependencies no 
matter
how many levels down they are.  You *could* extract and manipulate the 
upper-constraints
file for each container, and assuming 1 container 1 service that will 
probably
give you want you want.  The only trick is for data that is passed over the 
RPC
between services.  For example having different versions of oslo.context in 
the
nova conductor and nova-compute containers would be a bad thing.

Is that kinda what you're asking?

If not a more concrete example would be very helpful.

> Is there a best practice for doing such in the python ecosystem?

Nope.  in the python eco-system you more or less get what you ask for and 
it;s
up to you to find a spot on the prescriptive <-> flexible line.

Yours Tony.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-27 Thread Tony Breeds 
On Mon, Nov 28, 2016 at 02:41:08AM +, Steven Dake (stdake) wrote:
> Hey folks,
> 
> I get a lot of requests for variance reduction of transitive dependencies in
> Kolla’s containers.  As an example, we build from source nova.  Nova itself
> we can specify a version to install in the containers during build time.
> Nova’s python dependencies, not so much.

I'm not certain I follow.  You're using upper-constraints for install time
consistent version selection, and that applies to *all* dependencies no matter
how many levels down they are.  You *could* extract and manipulate the 
upper-constraints
file for each container, and assuming 1 container 1 service that will probably
give you want you want.  The only trick is for data that is passed over the RPC
between services.  For example having different versions of oslo.context in the
nova conductor and nova-compute containers would be a bad thing.

Is that kinda what you're asking?

If not a more concrete example would be very helpful.

> Is there a best practice for doing such in the python ecosystem?

Nope.  in the python eco-system you more or less get what you ask for and it;s
up to you to find a spot on the prescriptive <-> flexible line.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-27 Thread Jeffrey Zhang 
On Mon, Nov 28, 2016 at 11:02 AM, Clark Boylan  wrote:

> > pip install -u /requirements/upper-constrains.txt /nova
>
> I think you need a to use the -c option to set the constraints file.
>

Sorry, i made a mistake here. Kolla is using `-c` option, right now[0].

/var/lib/kolla/venv/bin/pip --no-cache-dir install --upgrade -c
requirements/upper-constraints.txt /keystone

[0]
https://github.com/openstack/kolla/blob/master/docker/keystone/keystone-base/Dockerfile.j2#L65



-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-27 Thread Clark Boylan 
On Sun, Nov 27, 2016, at 06:53 PM, Jeffrey Zhang wrote:
> FWIW, python dependencies are fixed.
> Kolla is using the following command in Dockerfile for source code
> installation.
> 
> pip install -u /requirements/upper-constrains.txt /nova

I think you need a to use the -c option to set the constraints file.

> 
> then all python dependencies installed will use a fixed version from
> upper-constrains.txt.

Yup, constraints apply to transitive deps as well. Keep in mind you
don't have to use the published upper-constraints either, but they are
an easy way to use a known tested set that works with upstream gating.

> 
> [0]
> https://specs.openstack.org/openstack/openstack-specs/specs/requirements-management.html
> [1]
> https://github.com/openstack/requirements/blob/master/upper-constraints.txt
> 
> On Mon, Nov 28, 2016 at 10:41 AM, Steven Dake (stdake) 
> wrote:
> 
> > Hey folks,
> >
> >
> >
> > I get a lot of requests for variance reduction of transitive dependencies
> > in Kolla’s containers.  As an example, we build from source nova.  Nova
> > itself we can specify a version to install in the containers during build
> > time.  Nova’s python dependencies, not so much.  Is there a best practice
> > for doing such in the python ecosystem?
> >
> >
> >
> > Binary distributions don’t typically suffer from this problem.  They
> > deliver one version of dependencies, and that is what you get.  That is
> > what a slew of folks are after with from source container builds.  Any
> > advice from the requirements team or release team welcome as the folks on
> > those teams have the most experience with this sort of thing.
> >
> >
> >
> > If this has been asked by someone else in a different context and
> > answered, a pointer to that discussion would work too J

> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][release][requirements] providing constraints for transitive dependencies

2016-11-27 Thread Jeffrey Zhang 
FWIW, python dependencies are fixed.
Kolla is using the following command in Dockerfile for source code
installation.

pip install -u /requirements/upper-constrains.txt /nova

then all python dependencies installed will use a fixed version from
upper-constrains.txt.

[0]
https://specs.openstack.org/openstack/openstack-specs/specs/requirements-management.html
[1]
https://github.com/openstack/requirements/blob/master/upper-constraints.txt

On Mon, Nov 28, 2016 at 10:41 AM, Steven Dake (stdake) 
wrote:

> Hey folks,
>
>
>
> I get a lot of requests for variance reduction of transitive dependencies
> in Kolla’s containers.  As an example, we build from source nova.  Nova
> itself we can specify a version to install in the containers during build
> time.  Nova’s python dependencies, not so much.  Is there a best practice
> for doing such in the python ecosystem?
>
>
>
> Binary distributions don’t typically suffer from this problem.  They
> deliver one version of dependencies, and that is what you get.  That is
> what a slew of folks are after with from source container builds.  Any
> advice from the requirements team or release team welcome as the folks on
> those teams have the most experience with this sort of thing.
>
>
>
> If this has been asked by someone else in a different context and
> answered, a pointer to that discussion would work too J
>
>
>
> Thanks!
>
> -steve
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev