Re: [Distutils] pipenv use cases

2018-01-17 Thread Nick Coghlan 
On 17 January 2018 at 12:51, Nick Coghlan  wrote:
> I think tox provides a good precedent for what's needed when it comes
> to effectively targeting multiple environments: you want a venv per
> target environment per project, not just a venv per project.

After double-checking the status quo, I'll note that simply doing
`pipenv install --dev` should already work pretty well for testing use
cases (even if you use environment markers):
https://docs.pipenv.org/advanced/#testing-projects

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pipenv use cases

2018-01-16 Thread Nick Coghlan 
On 17 January 2018 at 05:21, Brett Cannon  wrote:
> Well, technically we need consensus so we agree that what goes up on
> packaging.python.org makes sense; I'm not worried about the whole world. :)
> I mean I'm not even after specifically recommended testing practices and
> such since that seems outside the realm of packaging.python.org. I'm more
> interested in e.g. how should people handle venvs when they support more
> than one version of Python? Documenting not pinning your dependency
> versions. How do you specify what is required for testing and such so that
> you can generically know how to get those dependencies installed in your
> venv if you don't want to have your eventual wheel to include those
> test-only dependencies? Basically I'm after a tutorial on how to handle
> packaging while developing a library. I view "use tox, use pytest" as
> something under TIP's purview to create a tutorial/guide for.

Right, one of the things we worked out when Jon wrote the pipenv-based
application dependency management guide was that "Install this library
to my user directory so that I, the human Nick Coghlan, can use it in
my personal scripts" is a different task from declaring application
dependencies, and the subsequent feedback we've received (including
this thread) shows that library/framework dependency management is
slightly different again (and dependency management for portable
applications that work across multiple Python versions will look more
like the latter than the former).

I think tox provides a good precedent for what's needed when it comes
to effectively targeting multiple environments: you want a venv per
target environment per project, not just a venv per project.

However, while that's a configuration I'd personally like to see us
better *enable* in pipenv, I wouldn't want it to come at the expense
of the core notion of a "preferred deployment environment" as
represented by Pipfile.lock, and I'd also prefer that it didn't come
at the expense of making the lock file format itself more complex.

One way we could do that is to add a "--lockfile" option to both
"pipenv lock" and the "pipenv sync" subcommand proposed in
https://github.com/pypa/pipenv/issues/1255, and a "--venv" option to
the latter, such that the workflow for libraries/frameworks/portable
applications would be:

* use Pipfile/Pipfile.lock as normal for your "recommended development
environment" (e.g. the most recent supported Python version for access
to the most up to date developer tools, or perhaps the oldest
supported Python version to minimise the risk of breaking
compatibility). The "interactive" pipenv subcommands (install,
uninstall, update, run, shell) would all continue to work specifically
on the default lock file and venv.
* use "pipenv lock --python TARGET --lockfile PATH_TO_TARGET_LOCKFILE"
to generate target specific lockfiles for other environments
* use "pipenv sync --lockfile PATH_TO_TARGET_LOCKFILE --venv
PATH_TO_TARGET_VENV" to configure virtual environments based on those
alternate lock files

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pipenv use cases

2018-01-16 Thread Brett Cannon 
On Tue, 16 Jan 2018 at 10:43 Paul Moore  wrote:

> On 16 January 2018 at 17:36, Brett Cannon  wrote:
> > Is there a library developer workflow that's being promoted then
> somewhere
> > that I'm just not finding? Or does that need to be written for
> > packaging.python.org (which I might be willing to write; see below for
> > motivation)?
>
> Possibly not as such, but there's a fairly broad consensus over such
> things as:
>
> * Use tox to test your code against the versions you want to use
> * Declare dependencies with install_requires and leave the version
> requirements loose
>
> Maybe some other things I'm not thinking of right now. But basically
> most projects I see use a broadly similar structure, and they are all
> libraries. There's nowhere near as many good examples of Python
> applications that I know of, and little obvious consensus (for
> example, there's nothing I've ever seen that suggests a "standard"
> method of deploying applications - the nearest thing I know of is
> "write the app as a library and use a console entry point", which is
> pretty much the opposite of a separate app development standard :-)
>
>
> > At least from a VS Code perspective it would be great to have a target of
> > supporting the workflows as documented at packaging.python.org so people
> > know how they should generally structure things as well as making sure VS
> > Code always supports the modern workflows. Also being able to say "your
> > workflow is not normal" is also always helpful when dealing with feature
> > requests. :)
>
> That's my feeling too - if we have reasonable community consensus on
> "normal workflows", it's a lot easier to define boundaries on what
> applications like pipenv or VS Code will support. Consensus is hard to
> come by, though...
>

Well, technically *we* need consensus so we agree that what goes up on
packaging.python.org makes sense; I'm not worried about the whole world. :)
I mean I'm not even after specifically recommended testing practices and
such since that seems outside the realm of packaging.python.org. I'm more
interested in e.g. how should people handle venvs when they support more
than one version of Python? Documenting not pinning your dependency
versions. How do you specify what is required for testing and such so that
you can generically know how to get those dependencies installed in your
venv if you don't want to have your eventual wheel to include those
test-only dependencies? Basically I'm after a tutorial on how to handle
packaging while developing a library. I view "use tox, use pytest" as
something under TIP's purview to create a tutorial/guide for.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pipenv use cases

2018-01-16 Thread Paul Moore 
On 16 January 2018 at 17:36, Brett Cannon  wrote:
> Is there a library developer workflow that's being promoted then somewhere
> that I'm just not finding? Or does that need to be written for
> packaging.python.org (which I might be willing to write; see below for
> motivation)?

Possibly not as such, but there's a fairly broad consensus over such things as:

* Use tox to test your code against the versions you want to use
* Declare dependencies with install_requires and leave the version
requirements loose

Maybe some other things I'm not thinking of right now. But basically
most projects I see use a broadly similar structure, and they are all
libraries. There's nowhere near as many good examples of Python
applications that I know of, and little obvious consensus (for
example, there's nothing I've ever seen that suggests a "standard"
method of deploying applications - the nearest thing I know of is
"write the app as a library and use a console entry point", which is
pretty much the opposite of a separate app development standard :-)


> At least from a VS Code perspective it would be great to have a target of
> supporting the workflows as documented at packaging.python.org so people
> know how they should generally structure things as well as making sure VS
> Code always supports the modern workflows. Also being able to say "your
> workflow is not normal" is also always helpful when dealing with feature
> requests. :)

That's my feeling too - if we have reasonable community consensus on
"normal workflows", it's a lot easier to define boundaries on what
applications like pipenv or VS Code will support. Consensus is hard to
come by, though...

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pipenv use cases

2018-01-16 Thread Brett Cannon 
On Tue, 16 Jan 2018 at 02:45 Nick Coghlan  wrote:

> On 16 January 2018 at 20:22, Paul Moore  wrote:
> > On 16 January 2018 at 10:03, Nick Coghlan  wrote:
> >> On 16 January 2018 at 19:47, Paul Moore  wrote:
> >>> I think that if the pipenv docs had some better guidance on what use
> >>> cases it was intended to cover (and what it wasn't, in relation to the
> >>> broader range of pip+virtualenv use cases) that would help people
> >>> better understand its place in the ecosystem.
> >>
> >> That's fair, and making the PyPUG tutorial specifically about
> >> *application* dependency management was an initial step in that
> >> direction - for the library development use case, you're generally
> >> going to step out of pipenv's world as soon as you try to run your
> >> tests across multiple versions.
> >>
> >> The basic usage constraint is that pipenv expects you to control your
> >> target Python version, and it expects you to have exactly one - to go
> >> beyond that (as is needed for multi-version library support), you'll
> >> still need to drop down to the lower level tooling, either skipping
> >> the use of pipenv entirely, or else by using `pipfile lock
> >> --requirements` to shift levels.
> >
> > New subject as I don't want to hijack the original thread to rehash my
> > old issue, but I do want to make a couple of points on this (and I
> > agree, it's hard to find a good forum to discuss things like this as
> > it stands).
> >
> > 1. "pipenv expects you to control your target Python version" - I'm
> > not 100% sure what that means, but if it's saying "pipenv is only
> > really for code that will only be used with a single Python version"
> > then that's basically excluding a huge chunk of Python development.
> > Specifically, library development of all forms.
>
> Yes, that's deliberate. We want to target app developers as our
> initial audience, since library and framework developers have
> different needs (and for folks just starting out with library
> development, pipenv + the latest version of Python is actually fine -
> matrix testing only comes into play once folks actually want to
> support older versions, perhaps because the version they started out
> with is no longer the latest one).
>

Is there a library developer workflow that's being promoted then somewhere
that I'm just not finding? Or does that need to be written for
packaging.python.org (which I *might* be willing to write; see below for
motivation)?

At least from a VS Code perspective it would be great to have a target of
supporting the workflows as documented at packaging.python.org so people
know how they should generally structure things as well as making sure VS
Code always supports the modern workflows. Also being able to say "your
workflow is not normal" is also always helpful when dealing with feature
requests. :)

-Brett


>
> > Admittedly my
> > experience of what's "typical" is biased strongly by the communities I
> > work with, but conversely the "writing standalone apps in Python"
> > community doesn't really seem to have a web presence that we can
> > promote pipenv through, so it's becoming visible via the "general
> > Python development" community, which quite biased to libraries, and so
> > is not the right target audience, from what you say.
>
> As I noted in my original comment, pipenv only applies to library and
> framework development insofar as you can treat your test suite as an
> "app" to be deployed to developer's machine (the deployment mechanism
> is "git clone", but it's still a deployment).
>
> Where that currently breaks down is as soon as you're testing across
> multiple versions, where the combination of wheel files and
> environment markers means the current lock file format runs into
> trouble (but so do requirements files).
>
> > 2. My specific issues weren't outside that constraint - I *was*
> > writing code that would only ever need to run under Python 3.6. My
> > problem was the need for "local" build dependencies, which seems
> > entirely within that use case. In fairness, the pipenv devs weren't
> > totally against the functionality I needed, it's just not something
> > they had considered important. Maybe the problem here is that there
> > isn't a good set of "developing apps in Python" best practices (as
> > opposed to the library development situation - use install_requires,
> > test with tox, ...), so I didn't know the "recommended" solution to my
> > problem, that pipenv would have been expecting me to use. Maybe it's a
> > chicken-and-egg situation - the "best practice" is to use pipenv, but
> > until pipenv gets to encounter all the various use cases, that "best
> > practice" doesn't properly cover every situation...
>
> Local build dependencies are within scope, but pipenv doesn't
> magically fix the development resource constraint problem :)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com

Re: [Distutils] pipenv use cases

2018-01-16 Thread Paul Moore 
On 16 January 2018 at 10:44, Nick Coghlan  wrote:

> Yes, that's deliberate. We want to target app developers as our
> initial audience, since library and framework developers have
> different needs (and for folks just starting out with library
> development, pipenv + the latest version of Python is actually fine -
> matrix testing only comes into play once folks actually want to
> support older versions, perhaps because the version they started out
> with is no longer the latest one).

Having that up front in the pipenv docs/webpage would probably help
communicate the intention better. At the moment, it's pretty hard to
find. And the general "pipenv is cool!" enthusiasm (which I agree with
- it is :-)) tends to encourage people to try it out for whatever
their use case is (hence Chris' original post). Also, there's a quite
common recommendation around to "build your app as a library and have
its main program as a console script entry point" - tools like pip,
tox, pytest, pew, pipenv itself, etc take that approach, for example.
So separating "app developers" and "library developers" still leaves a
fairly large grey area.

> Local build dependencies are within scope, but pipenv doesn't
> magically fix the development resource constraint problem :)

Understood - as I said, the pipenv devs were very helpful, it just
took a bit of time to establish what I was trying to do, and that it
*was* something that they saw the need to support. Actually
implementing that support can take as long as it needs - I appreciate
the "so much to do, so little time" problem. Better publicised "Python
application development workflow" best practices might have helped
save a little of our time in establishing I had a valid use case and
they intended to support it but didn't yet. That's all I'm really
saying.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pipenv use cases

2018-01-16 Thread Nick Coghlan 
On 16 January 2018 at 20:22, Paul Moore  wrote:
> On 16 January 2018 at 10:03, Nick Coghlan  wrote:
>> On 16 January 2018 at 19:47, Paul Moore  wrote:
>>> I think that if the pipenv docs had some better guidance on what use
>>> cases it was intended to cover (and what it wasn't, in relation to the
>>> broader range of pip+virtualenv use cases) that would help people
>>> better understand its place in the ecosystem.
>>
>> That's fair, and making the PyPUG tutorial specifically about
>> *application* dependency management was an initial step in that
>> direction - for the library development use case, you're generally
>> going to step out of pipenv's world as soon as you try to run your
>> tests across multiple versions.
>>
>> The basic usage constraint is that pipenv expects you to control your
>> target Python version, and it expects you to have exactly one - to go
>> beyond that (as is needed for multi-version library support), you'll
>> still need to drop down to the lower level tooling, either skipping
>> the use of pipenv entirely, or else by using `pipfile lock
>> --requirements` to shift levels.
>
> New subject as I don't want to hijack the original thread to rehash my
> old issue, but I do want to make a couple of points on this (and I
> agree, it's hard to find a good forum to discuss things like this as
> it stands).
>
> 1. "pipenv expects you to control your target Python version" - I'm
> not 100% sure what that means, but if it's saying "pipenv is only
> really for code that will only be used with a single Python version"
> then that's basically excluding a huge chunk of Python development.
> Specifically, library development of all forms.

Yes, that's deliberate. We want to target app developers as our
initial audience, since library and framework developers have
different needs (and for folks just starting out with library
development, pipenv + the latest version of Python is actually fine -
matrix testing only comes into play once folks actually want to
support older versions, perhaps because the version they started out
with is no longer the latest one).

> Admittedly my
> experience of what's "typical" is biased strongly by the communities I
> work with, but conversely the "writing standalone apps in Python"
> community doesn't really seem to have a web presence that we can
> promote pipenv through, so it's becoming visible via the "general
> Python development" community, which quite biased to libraries, and so
> is not the right target audience, from what you say.

As I noted in my original comment, pipenv only applies to library and
framework development insofar as you can treat your test suite as an
"app" to be deployed to developer's machine (the deployment mechanism
is "git clone", but it's still a deployment).

Where that currently breaks down is as soon as you're testing across
multiple versions, where the combination of wheel files and
environment markers means the current lock file format runs into
trouble (but so do requirements files).

> 2. My specific issues weren't outside that constraint - I *was*
> writing code that would only ever need to run under Python 3.6. My
> problem was the need for "local" build dependencies, which seems
> entirely within that use case. In fairness, the pipenv devs weren't
> totally against the functionality I needed, it's just not something
> they had considered important. Maybe the problem here is that there
> isn't a good set of "developing apps in Python" best practices (as
> opposed to the library development situation - use install_requires,
> test with tox, ...), so I didn't know the "recommended" solution to my
> problem, that pipenv would have been expecting me to use. Maybe it's a
> chicken-and-egg situation - the "best practice" is to use pipenv, but
> until pipenv gets to encounter all the various use cases, that "best
> practice" doesn't properly cover every situation...

Local build dependencies are within scope, but pipenv doesn't
magically fix the development resource constraint problem :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] pipenv use cases

2018-01-16 Thread Paul Moore 
On 16 January 2018 at 10:03, Nick Coghlan  wrote:
> On 16 January 2018 at 19:47, Paul Moore  wrote:
>> I think that if the pipenv docs had some better guidance on what use
>> cases it was intended to cover (and what it wasn't, in relation to the
>> broader range of pip+virtualenv use cases) that would help people
>> better understand its place in the ecosystem.
>
> That's fair, and making the PyPUG tutorial specifically about
> *application* dependency management was an initial step in that
> direction - for the library development use case, you're generally
> going to step out of pipenv's world as soon as you try to run your
> tests across multiple versions.
>
> The basic usage constraint is that pipenv expects you to control your
> target Python version, and it expects you to have exactly one - to go
> beyond that (as is needed for multi-version library support), you'll
> still need to drop down to the lower level tooling, either skipping
> the use of pipenv entirely, or else by using `pipfile lock
> --requirements` to shift levels.

New subject as I don't want to hijack the original thread to rehash my
old issue, but I do want to make a couple of points on this (and I
agree, it's hard to find a good forum to discuss things like this as
it stands).

1. "pipenv expects you to control your target Python version" - I'm
not 100% sure what that means, but if it's saying "pipenv is only
really for code that will only be used with a single Python version"
then that's basically excluding a huge chunk of Python development.
Specifically, library development of all forms. Admittedly my
experience of what's "typical" is biased strongly by the communities I
work with, but conversely the "writing standalone apps in Python"
community doesn't really seem to have a web presence that we can
promote pipenv through, so it's becoming visible via the "general
Python development" community, which quite biased to libraries, and so
is not the right target audience, from what you say.

2. My specific issues weren't outside that constraint - I *was*
writing code that would only ever need to run under Python 3.6. My
problem was the need for "local" build dependencies, which seems
entirely within that use case. In fairness, the pipenv devs weren't
totally against the functionality I needed, it's just not something
they had considered important. Maybe the problem here is that there
isn't a good set of "developing apps in Python" best practices (as
opposed to the library development situation - use install_requires,
test with tox, ...), so I didn't know the "recommended" solution to my
problem, that pipenv would have been expecting me to use. Maybe it's a
chicken-and-egg situation - the "best practice" is to use pipenv, but
until pipenv gets to encounter all the various use cases, that "best
practice" doesn't properly cover every situation...

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig