Re: [Distutils] How to specify dependencies in Python

[Paul Moore]

On 17 January 2017 at 17:40, Jim Fulton  wrote:
>> I am speaking of both. And: I think requirements.txt is optional.
>
> Then I disagree with your statement. :)
>
> I should stop, but I'll take one more stab at this.
>
> It matters whether you're talking about components(/libraries) or whole
> systems (/applications).

FWIW, that was my point as well. Thomas, if you're not understanding
that instructions need to be different depending on whether you're
talking about components/libraries or systems/applications, then we're
not going to reach an agreement.

IMO, the "Quickstart" you quoted that triggered this is pretty much OK
as long as it's talking about developing applications. It's not clear
without context whether that's the case, but unless *you* understand
the distinction Jim and I are trying to make, then I doubt we'd agree
with any suggested rewording of that section that you might propose.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Jim Fulton]

On Tue, Jan 17, 2017 at 11:34 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

>
>
> Am 16.01.2017 um 18:06 schrieb Jim Fulton:
>
>>
>>
>> On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler <
>> guettl...@thomas-guettler.de >
>> wrote:
>>
>>
>>
>> Am 13.01.2017 um 16:25 schrieb Jim Fulton:
>>
>>
>>
>> On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler <
>> guettl...@thomas-guettler.de
>>  > guettl...@thomas-guettler.de
>> >> wrote:
>>
>> What is an application for you?
>>
>>
>> Another way to think about this, FWIW, is to distinguish between
>> the "whole system" (for which "Application" is
>> often a
>> useful shorthand), as opposed to components (aka libraries).
>>
>> It's important for components to be flexible, so they're
>> typically very flexible about versions of their
>> dependencies.
>>
>> For whole systems, OTOH, it's important that once a configuration
>> is tested, that the same configuration is used in
>> production, so systems typically pin all of their dependencies,
>> ideally extending to their environments (which is a
>> reason why container technology is attractive).
>>
>>
>> Yes, install_requires in setup.py should define flexible
>> dependencies, but requirements.txt should define fixed
>> dependencies via fixed version.
>>
>> Do you agree with my sentence from above?
>>
>>
>> Are you speaking of a component/library or whole system?
>>
>
> I am speaking of both. And: I think requirements.txt is optional.


Then I disagree with your statement. :)

I should stop, but I'll take one more stab at this.

It matters whether you're talking about components(/libraries) or whole
systems (/applications).

For components:

Consumers of a component need to be able to to determine the component's
dependencies.  The component uses install_required (and extras_require) for
this.  The version specifications in these dependencies should be as
flexible as possible, to allow reuse in as many whole systems as possible.

Developers of a component will use tools like pip and buildout to automate
their development activities.  For pip, that will usually entail one or
more requirements.txt files. For buildout, that will  entail one or more
(for different development activities) buildout configs and a single
versions.cfg.

For whole systems:

Many whole systems only assemble components.  For these systems, there is
no setup.py file (no python project).

If a whole system includes a Python project (that isn't distributed
separately), it's a matter of taste how much information is included in
setup.py. Personally, I would treat the Python project like a component
project and include its direct dependencies and minimal version constraints.

Typically a whole system managed with pip will use a requirements.txt file
(or possibly multiple) and a system  developed with buildout will have a
buildout config and a versions config.

A whole system could fix all of its dependent versions in install_requires
in a setup script, but that would be cumbersome. By using requirements.txt
or a buildout versions config, a developer can avail themselves of
automation to help maintain the files.

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Thomas Güttler]

Am 16.01.2017 um 18:06 schrieb Jim Fulton:



On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler > wrote:



Am 13.01.2017 um 16:25 schrieb Jim Fulton:



On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler 
 
>> wrote:

What is an application for you?


Another way to think about this, FWIW, is to distinguish between the "whole 
system" (for which "Application" is
often a
useful shorthand), as opposed to components (aka libraries).

It's important for components to be flexible, so they're typically very 
flexible about versions of their
dependencies.

For whole systems, OTOH, it's important that once a configuration is 
tested, that the same configuration is used in
production, so systems typically pin all of their dependencies, ideally 
extending to their environments (which is a
reason why container technology is attractive).


Yes, install_requires in setup.py should define flexible dependencies, but 
requirements.txt should define fixed
dependencies via fixed version.

Do you agree with my sentence from above?


Are you speaking of a component/library or whole system?


I am speaking of both. And: I think requirements.txt is optional.


--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Glyph Lefkowitz]

> On Jan 16, 2017, at 7:59 AM, Thomas Güttler  
> wrote:
> 
> I think requirements.txt should be the result of some kind of 
> Continous-Integration run.
> If all tests are successful, then requirements.txt should be created with 
> "pip freeze".
> 
> This means, that the Continous-Integration run does not use requirements.txt 
> to
> build its environment.
> 
> Next question: Where is the best place to store requirements.txt?
> 
> I think it should not be in the repo of a library. It should be somehow 
> outside.

I think I understand what you're trying to say here, but I think you have it 
backwards.

The best example I have made of the "right" way to do this is in this project: 
https://github.com/rackerlabs/mimic/ 

The project has both a setup.py and (several versions of) requirements.txt.

setup.py gives the abstract requirements for the project.  This is what should 
work, but has not necessarily been exactly tested to.

requirements.txt is the concrete requirements.  This is generated (more or 
less) by freezing an environment where one does `pip install .` to trigger the 
setup.py.

However, all continuous integration tests are run against the versions listed 
in requirements.txt (the details are here: 
https://github.com/rackerlabs/mimic/blob/5fae30d9e9a45c15f3f0a51fa436a7f25502b742/.travis/install.sh#L101-L105
 

 and here 
https://github.com/rackerlabs/mimic/blob/5fae30d9e9a45c15f3f0a51fa436a7f25502b742/.travis/run.sh#L18
 
)
 to ensure that new contributors always have a stable set of versions to test 
against, and their PR won't end up randomly having to fix some upgrade issue.

Finally, we use https://requires.io  to submit PRs every 
time one of the dependencies upgrades.  On a regular basis (every 6 months or 
so), these do cause errors; when they do, development unrelated to the version 
upgrade can continue on unimpeded, and the version-compatibility fix lives 
neatly in its own PR that solves just that problem.

It's a bit subtle to understand the distinction between setup.py and 
requirements.txt (https://caremad.io/posts/2013/07/setup-vs-requirement/ 
 is a good attempt, but 
I think stumbles over explaining some nuances that are obvious to dstufft and 
not to anyone else), but if you get them lined up correctly then a bunch of 
edge-cases work very nicely, and reasoning about deployment is much easier.

-glyph___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Jim Fulton]

On Mon, Jan 16, 2017 at 11:03 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

>
>
> Am 13.01.2017 um 16:25 schrieb Jim Fulton:
>
>>
>>
>> On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler <
>> guettl...@thomas-guettler.de >
>> wrote:
>>
>> What is an application for you?
>>
>>
>> Another way to think about this, FWIW, is to distinguish between the
>> "whole system" (for which "Application" is often a
>> useful shorthand), as opposed to components (aka libraries).
>>
>> It's important for components to be flexible, so they're typically very
>> flexible about versions of their dependencies.
>>
>> For whole systems, OTOH, it's important that once a configuration is
>> tested, that the same configuration is used in
>> production, so systems typically pin all of their dependencies, ideally
>> extending to their environments (which is a
>> reason why container technology is attractive).
>>
>
> Yes, install_requires in setup.py should define flexible dependencies, but
> requirements.txt should define fixed dependencies via fixed version.
>
> Do you agree with my sentence from above?


Are you speaking of a component/library or whole system?

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Paul Moore]

On 16 January 2017 at 15:59, Thomas Güttler
 wrote:
> Thank you for your explanation. For me this means, I should tell the
> maintainer
> that a library should specify it's dependencies via install_requires.

Well yes, but that doesn't mean the quoted comment from Quickstart is
wrong. It's not clear without context whether the comment is intended
to refer to applications using this library, or other libraries using
this library. It's even possible that the project author doesn't fully
understand the subtleties here.

> I think requirements.txt should be the result of some kind of
> Continous-Integration run.
> If all tests are successful, then requirements.txt should be created with
> "pip freeze".
>
> This means, that the Continous-Integration run does not use requirements.txt
> to
> build its environment.
>
> Next question: Where is the best place to store requirements.txt?
>
> I think it should not be in the repo of a library. It should be somehow
> outside.
>
> Up to now I a missing the right english term for it.

I'm not at all sure how true that is, but as I don't write this sort
of application, and certainly don't publish such things, I can't
really comment.

>> If you can point to the actual library you're referring to, it would
>> be easier to be specific :-)
>
>
> I did not tell the name here, since I don't want to offend the library
> maintainer.
> I think he does not like it, when people talk behind his back about his
> code.
>
> I will do so after we found a solid solution here.

OK, but I don't think we can find anything more concrete without more
information, and at some point you're going to end up having to give
enough information that we might as well just look at the project.
After all, it's (presumably) open source, so we can always look at the
code and submit PRs/issues anyway. It seems unlikely that the author
is going to be offended - if there's a risk, why not simply invite
him/her to participate in this thread?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Thomas Güttler]

Am 13.01.2017 um 16:25 schrieb Jim Fulton:



On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler > wrote:

What is an application for you?


Another way to think about this, FWIW, is to distinguish between the "whole system" (for 
which "Application" is often a
useful shorthand), as opposed to components (aka libraries).

It's important for components to be flexible, so they're typically very 
flexible about versions of their dependencies.

For whole systems, OTOH, it's important that once a configuration is tested, 
that the same configuration is used in
production, so systems typically pin all of their dependencies, ideally 
extending to their environments (which is a
reason why container technology is attractive).


Yes, install_requires in setup.py should define flexible dependencies, but requirements.txt should define fixed 
dependencies via fixed version.


Do you agree with my sentence from above?



--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Thomas Güttler]

Am 13.01.2017 um 14:07 schrieb Paul Moore:

On 13 January 2017 at 12:23, Thomas Güttler
 wrote:

Am 12.01.2017 um 13:43 schrieb Nick Coghlan:


On 12 January 2017 at 22:04, Thomas Güttler
 wrote:


I came across a python library which has docs, which start like this:

{{{

 Quickstart

 Include foolib in your requirements.txt file.

}}}

AFAIK dependencies should be specified via `install_requires` in
`setup.py`.



Applications and services don't necessarily have a setup.py file -
setup.py is more for pip-installable libraries and frameworks (see
https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on
that topic).



What is an application?

The django project uses this definition:
https://docs.djangoproject.com/en/1.10/ref/applications/

I guess you mean something else, if you talk about "application".

What is an application for you?


IMO, an "application" in this context is a standalone program
(implemented in Python). This is as opposed to a "library" which is
written to be imported in other Python code. Obviously, there's some
overlap - some "applications" provide a "programming API" that can be
imported, and some libraries provide command line entry points.


Hi Paul,

yes, I think the same way. There is some overlap.


A library (something that you import) should declare in its metadata
that you need to have its dependencies installed before it will work.
That lets you (the user of the library) be ignorant of the
dependencies of the library (which are implementation details as far
as you are concerned) - pip just sorts it out for you from the
metadata. To get the dependency metadata, the *library* should include
its dependencies in install_requires in its setup.py.


Thank you for your explanation. For me this means, I should tell the maintainer
that a library should specify it's dependencies via install_requires.



For applications on the other hand, you need to install exactly the
environment you tested, so you *don't* use dependency metadata.
Rather, you create a requirements.txt file that contains the exact
versions of everything (direct dependencies) your application needs,
then deploy your application script to a Python environment where
you've run "pip install -r requirements.txt" to set it up correctly.



I think requirements.txt should be the result of some kind of 
Continous-Integration run.
If all tests are successful, then requirements.txt should be created with "pip 
freeze".

This means, that the Continous-Integration run does not use requirements.txt to
build its environment.

Next question: Where is the best place to store requirements.txt?

I think it should not be in the repo of a library. It should be somehow outside.

Up to now I a missing the right english term for it.



As I say, there are lots of grey areas here. But from your
description, the library you found sounds like it should have
dependency metadata, and not need you to use a requirements file. On
the other hand, the "Quickstart" may be trying to tell people how to
use foolib in their *application*, in which case it's correct. (It's
oversimplified, as if you're writing a library that depends on foolib
you'd use install_requires, but simply changing the text to just say
"put foolib in your install_requires" is no better, as then you'd have
ignored the application use case...)

If you can point to the actual library you're referring to, it would
be easier to be specific :-)


I did not tell the name here, since I don't want to offend the library 
maintainer.
I think he does not like it, when people talk behind his back about his code.

I will do so after we found a solid solution here.

Regards,
  Thomas Güttler




--
Thomas Guettler http://www.thomas-guettler.de/
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Jim Fulton]

On Fri, Jan 13, 2017 at 7:23 AM, Thomas Güttler <
guettl...@thomas-guettler.de> wrote:

> What is an application for you?


Another way to think about this, FWIW, is to distinguish between the "whole
system" (for which "Application" is often a useful shorthand), as opposed
to components (aka libraries).

It's important for components to be flexible, so they're typically very
flexible about versions of their dependencies.

For whole systems, OTOH, it's important that once a configuration is
tested, that the same configuration is used in production, so systems
typically pin all of their dependencies, ideally extending to their
environments (which is a reason why container technology is attractive).

Jim

-- 
Jim Fulton
http://jimfulton.info
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Nathaniel Smith]

It's probably a good idea here to explicitly distinguish between the kind
of application that's a web app you deploy to sort of managed server
environment, and the kind of application that's a command line or GUI tool
that people download and run. The same word means radically different
things to different people :-). I believe Paul's advice below is
specifically addressing the former case, not the latter.

On Jan 13, 2017 5:07 AM, "Paul Moore"  wrote:

> On 13 January 2017 at 12:23, Thomas Güttler
>  wrote:
> > Am 12.01.2017 um 13:43 schrieb Nick Coghlan:
> >>
> >> On 12 January 2017 at 22:04, Thomas Güttler
> >>  wrote:
> >>>
> >>> I came across a python library which has docs, which start like this:
> >>>
> >>> {{{
> >>>
> >>>  Quickstart
> >>>
> >>>  Include foolib in your requirements.txt file.
> >>>
> >>> }}}
> >>>
> >>> AFAIK dependencies should be specified via `install_requires` in
> >>> `setup.py`.
> >>
> >>
> >> Applications and services don't necessarily have a setup.py file -
> >> setup.py is more for pip-installable libraries and frameworks (see
> >> https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on
> >> that topic).
> >
> >
> > What is an application?
> >
> > The django project uses this definition:
> > https://docs.djangoproject.com/en/1.10/ref/applications/
> >
> > I guess you mean something else, if you talk about "application".
> >
> > What is an application for you?
>
> IMO, an "application" in this context is a standalone program
> (implemented in Python). This is as opposed to a "library" which is
> written to be imported in other Python code. Obviously, there's some
> overlap - some "applications" provide a "programming API" that can be
> imported, and some libraries provide command line entry points.
>
> A library (something that you import) should declare in its metadata
> that you need to have its dependencies installed before it will work.
> That lets you (the user of the library) be ignorant of the
> dependencies of the library (which are implementation details as far
> as you are concerned) - pip just sorts it out for you from the
> metadata. To get the dependency metadata, the *library* should include
> its dependencies in install_requires in its setup.py.
>
> For applications on the other hand, you need to install exactly the
> environment you tested, so you *don't* use dependency metadata.
> Rather, you create a requirements.txt file that contains the exact
> versions of everything (direct dependencies) your application needs,
> then deploy your application script to a Python environment where
> you've run "pip install -r requirements.txt" to set it up correctly.
>
> As I say, there are lots of grey areas here. But from your
> description, the library you found sounds like it should have
> dependency metadata, and not need you to use a requirements file. On
> the other hand, the "Quickstart" may be trying to tell people how to
> use foolib in their *application*, in which case it's correct. (It's
> oversimplified, as if you're writing a library that depends on foolib
> you'd use install_requires, but simply changing the text to just say
> "put foolib in your install_requires" is no better, as then you'd have
> ignored the application use case...)
>
> If you can point to the actual library you're referring to, it would
> be easier to be specific :-)
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Paul Moore]

On 13 January 2017 at 12:23, Thomas Güttler
 wrote:
> Am 12.01.2017 um 13:43 schrieb Nick Coghlan:
>>
>> On 12 January 2017 at 22:04, Thomas Güttler
>>  wrote:
>>>
>>> I came across a python library which has docs, which start like this:
>>>
>>> {{{
>>>
>>>  Quickstart
>>>
>>>  Include foolib in your requirements.txt file.
>>>
>>> }}}
>>>
>>> AFAIK dependencies should be specified via `install_requires` in
>>> `setup.py`.
>>
>>
>> Applications and services don't necessarily have a setup.py file -
>> setup.py is more for pip-installable libraries and frameworks (see
>> https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on
>> that topic).
>
>
> What is an application?
>
> The django project uses this definition:
> https://docs.djangoproject.com/en/1.10/ref/applications/
>
> I guess you mean something else, if you talk about "application".
>
> What is an application for you?

IMO, an "application" in this context is a standalone program
(implemented in Python). This is as opposed to a "library" which is
written to be imported in other Python code. Obviously, there's some
overlap - some "applications" provide a "programming API" that can be
imported, and some libraries provide command line entry points.

A library (something that you import) should declare in its metadata
that you need to have its dependencies installed before it will work.
That lets you (the user of the library) be ignorant of the
dependencies of the library (which are implementation details as far
as you are concerned) - pip just sorts it out for you from the
metadata. To get the dependency metadata, the *library* should include
its dependencies in install_requires in its setup.py.

For applications on the other hand, you need to install exactly the
environment you tested, so you *don't* use dependency metadata.
Rather, you create a requirements.txt file that contains the exact
versions of everything (direct dependencies) your application needs,
then deploy your application script to a Python environment where
you've run "pip install -r requirements.txt" to set it up correctly.

As I say, there are lots of grey areas here. But from your
description, the library you found sounds like it should have
dependency metadata, and not need you to use a requirements file. On
the other hand, the "Quickstart" may be trying to tell people how to
use foolib in their *application*, in which case it's correct. (It's
oversimplified, as if you're writing a library that depends on foolib
you'd use install_requires, but simply changing the text to just say
"put foolib in your install_requires" is no better, as then you'd have
ignored the application use case...)

If you can point to the actual library you're referring to, it would
be easier to be specific :-)

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to specify dependencies in Python

[Nick Coghlan]

On 12 January 2017 at 22:04, Thomas Güttler
 wrote:
> I came across a python library which has docs, which start like this:
>
> {{{
>
>  Quickstart
>
>  Include foolib in your requirements.txt file.
>
> }}}
>
> AFAIK dependencies should be specified via `install_requires` in `setup.py`.

Applications and services don't necessarily have a setup.py file -
setup.py is more for pip-installable libraries and frameworks (see
https://caremad.io/posts/2013/07/setup-vs-requirement/ for more on
that topic).

Since the section is titled "Quickstart", it seems reasonable to gloss
over the fact that there are options other than requirements.txt,
especially as folks that have already gone to the effort of learning
how to make their software pip-installable are also likely to have
worked out how that impacts where you specify your dependencies.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig