Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

Good write-ups. Though it sounds like idna should just be moved into the
stdlib. (Agreed that for things like this codecs are not a great idea.) We
have other things that must be updated whenever the Unicode standard
changes and it seems that doing this in feature releases is typically fine,
and treating it as a bugfix (so doing it in bugfix releases is acceptable)
sounds fine to me too. Unless it were so security-critical that we'd want
to enable users to do such updates to less-supported Pythons or on a
tighter schedule (which is one of the reasons requests can't go into the
stdlib -- they have their own cert store).

On Wed, Nov 1, 2017 at 7:36 PM, Nathaniel Smith  wrote:

> On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum  wrote:
> > Can you write 1-2 paragraphs with the argument for each?
> >
> > On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith  wrote:
> >> - lxml
>
> My impression (probably others are more knowledgeable) is that lxml
> has more or less replaced the stdlib 'xml' package as the de facto
> standard -- sort of similar to the urllib2/requests situation. AFAIK
> lxml has never been proposed for stdlib inclusion and I believe the
> fact that it's all in Cython would be a barrier even if the
> maintainers were amenable. But it might be helpful to our users to put
> a box at the top of the 'xml' docs suggesting people check out 'lxml',
> similar to the one on the urllib2 docs.
>
> >> - numpy
>
> Numpy's arrays are a foundational data structure and de facto
> standard, and would probably fit naturally in the stdlib semantically,
> but for a number of logistical/implementational reasons it doesn't
> make sense to merge. Probably doesn't make much difference whether
> python-dev "blesses" it or not in practice, since there aren't any
> real competitors inside or outside the stdlib; it'd more just be an
> acknowledgement of the status quo.
>
> >> - cryptography
>
> Conceptually, core cryptographic operations are the kind of
> functionality that you might expect to see in the stdlib, but the
> unique sensitivity of crypto code makes this a bad idea. Historically
> there have been a variety of basic crypto packages for Python, but at
> this point IIUC the other ones are all considered
> obsolete-and-potentially-dangerous and the consensus is everyone
> should move to 'cryptography', so documenting that in this PEP might
> help send people in the right direction.
>
> >> - idna
>
> This is a bit of a funny one. IDNA functionality is pretty fundamental
> -- you need it to do unicode<->bytes conversions on hostnames, so
> basically anyone doing networking needs it. Python ships with some
> built-in IDNA functionality (as the "idna" codec), but it's using an
> obsolete standard (IDNA2003, versus the current IDNA2008, see
> bpo-17305), and IIRC Christian thinks the whole codec-based design is
> the wrong approach... basically what we have in the stdlib has been
> broken for most of a decade and there doesn't seem to be anyone
> interested in fixing it. So... in the long run the stdlib support
> should either be fixed or deprecated. I'm not sure which is better.
> (The argument for deprecating it would be that IIUC you need to update
> the tables whenever a new unicode standard comes out, and since it's a
> networking thing you want to keep in sync with the rest of the world,
> which is easier with a standalone library. I don't know how much this
> matters in practice.) But right now, this library is just better than
> the stdlib functionality, and it wouldn't hurt to document that.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nathaniel Smith]

On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum  wrote:
> Can you write 1-2 paragraphs with the argument for each?
>
> On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith  wrote:
>> - lxml

My impression (probably others are more knowledgeable) is that lxml
has more or less replaced the stdlib 'xml' package as the de facto
standard -- sort of similar to the urllib2/requests situation. AFAIK
lxml has never been proposed for stdlib inclusion and I believe the
fact that it's all in Cython would be a barrier even if the
maintainers were amenable. But it might be helpful to our users to put
a box at the top of the 'xml' docs suggesting people check out 'lxml',
similar to the one on the urllib2 docs.

>> - numpy

Numpy's arrays are a foundational data structure and de facto
standard, and would probably fit naturally in the stdlib semantically,
but for a number of logistical/implementational reasons it doesn't
make sense to merge. Probably doesn't make much difference whether
python-dev "blesses" it or not in practice, since there aren't any
real competitors inside or outside the stdlib; it'd more just be an
acknowledgement of the status quo.

>> - cryptography

Conceptually, core cryptographic operations are the kind of
functionality that you might expect to see in the stdlib, but the
unique sensitivity of crypto code makes this a bad idea. Historically
there have been a variety of basic crypto packages for Python, but at
this point IIUC the other ones are all considered
obsolete-and-potentially-dangerous and the consensus is everyone
should move to 'cryptography', so documenting that in this PEP might
help send people in the right direction.

>> - idna

This is a bit of a funny one. IDNA functionality is pretty fundamental
-- you need it to do unicode<->bytes conversions on hostnames, so
basically anyone doing networking needs it. Python ships with some
built-in IDNA functionality (as the "idna" codec), but it's using an
obsolete standard (IDNA2003, versus the current IDNA2008, see
bpo-17305), and IIRC Christian thinks the whole codec-based design is
the wrong approach... basically what we have in the stdlib has been
broken for most of a decade and there doesn't seem to be anyone
interested in fixing it. So... in the long run the stdlib support
should either be fixed or deprecated. I'm not sure which is better.
(The argument for deprecating it would be that IIUC you need to update
the tables whenever a new unicode standard comes out, and since it's a
networking thing you want to keep in sync with the rest of the world,
which is easier with a standalone library. I don't know how much this
matters in practice.) But right now, this library is just better than
the stdlib functionality, and it wouldn't hurt to document that.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nathaniel Smith]

On Oct 31, 2017 4:42 AM, "Nick Coghlan"  wrote:

On 31 October 2017 at 02:29, Guido van Rossum  wrote:

> What's your proposed process to arrive at the list of recommended packages?
>

I'm thinking it makes the most sense to treat inclusion in the recommended
packages list as a possible outcome of proposals for standard library
inclusion, rather than being something we'd provide a way to propose
specifically.

We'd only use it in cases where a proposal would otherwise meet the
criteria for stdlib inclusion, but the logistics of actually doing so don't
work for some reason.

Running the initial 5 proposals through that filter:

* six: a cross-version compatibility layer clearly needs to be outside the
standard library
* setuptools: we want to update this in line with the PyPA interop specs,
not the Python language version
* cffi: updates may be needed for PyPA interop specs, Python implementation
updates or C language definition updates
* requests: updates are more likely to be driven by changes in network
protocols and client platform APIs than Python language changes
* regex: we don't want two regex engines in the stdlib, transparently
replacing _sre would be difficult, and _sre is still good enough for most
purposes


Some other packages that might meet these criteria, or at least be useful
for honing them:

- lxml
- numpy
- cryptography
- idna

-n
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 1 November 2017 at 05:56, Guido van Rossum  wrote:

> On Tue, Oct 31, 2017 at 12:24 PM, MRAB  wrote:
>
>> At least Python 3.6 is only 1 year/release behind, which is fine!
>>
>
> OK, so presumably that argument doesn't preclude inclusion in the 3.7 (or
> later) stdlib. I'm beginning to warm up to the idea again... Maybe we
> should just bite the bullet. Nick, what do you think? Is it worth a small
> PEP?
>

I'm personally still in favour of swapping out the current _sre based
implementation for a _regex based implementation (such that 3.7+ still only
contained a single regex engine, and the stdlib re module and a PyPI regex
backport module could still happily coexist), and a PEP + draft patch would
be the way to do that.

The framing of a PEP for that approach would be "Replace the regex engine
backing the re module" rather than "Add regex to the standard library". The
existing engine could then potentially be spun out as a new "sre" project
on PyPI, such that folks that *were* depending on _sre internals had access
to an upgrade path that didn't necessarily require them porting their code
to a new regex engine (the PEP author wouldn't have to commit to doing that
work - we'd just ask the PyPI admins to reserve the name in case there
proved to be demand for such a project).

However, I'm also not one of the folks that would be on the hook for
handling any compatibility regressions that were subsequently reported
against the 3.7 re module, so I'd also take my +1 with a rather large grain
of salt - it's easy to be positive about a plan when the potential
downsides don't affect me personally :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

On Tue, Oct 31, 2017 at 12:24 PM, MRAB  wrote:

> On 2017-10-31 18:44, Guido van Rossum wrote:
>
>> On Tue, Oct 31, 2017 at 11:41 AM, MRAB > > wrote:
>>
>> regex gets updated when the Unicode Consortium releases an update.
>>
>>
>> Is it a feature that that is more frequently than Python releases? There
>> are other things in Python that must be updated whenever the UC releases an
>> update, and they get treated as features (or perhaps as bugfixes, I'm not
>> sure) but this means they generally don't get backported.
>>
>> Here's a list of the updates to Unicode:
>
> https://www.unicode.org/versions/enumeratedversions.html
>
> Roughly yearly.
>
> Those still on Python 2.7, for example, are now 8 years behind re Unicode.
>

Frankly I consider that a feature of the 2.7 end-of-life planning. :-)


> At least Python 3.6 is only 1 year/release behind, which is fine!
>

OK, so presumably that argument doesn't preclude inclusion in the 3.7 (or
later) stdlib. I'm beginning to warm up to the idea again... Maybe we
should just bite the bullet. Nick, what do you think? Is it worth a small
PEP?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[MRAB]

On 2017-10-31 18:44, Guido van Rossum wrote:
On Tue, Oct 31, 2017 at 11:41 AM, MRAB > wrote:


regex gets updated when the Unicode Consortium releases an update.


Is it a feature that that is more frequently than Python releases? 
There are other things in Python that must be updated whenever the UC 
releases an update, and they get treated as features (or perhaps as 
bugfixes, I'm not sure) but this means they generally don't get 
backported.



Here's a list of the updates to Unicode:

https://www.unicode.org/versions/enumeratedversions.html

Roughly yearly.

Those still on Python 2.7, for example, are now 8 years behind re Unicode.

At least Python 3.6 is only 1 year/release behind, which is fine!
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

On Tue, Oct 31, 2017 at 11:41 AM, MRAB  wrote:

> regex gets updated when the Unicode Consortium releases an update.
>

Is it a feature that that is more frequently than Python releases? There
are other things in Python that must be updated whenever the UC releases an
update, and they get treated as features (or perhaps as bugfixes, I'm not
sure) but this means they generally don't get backported.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[MRAB]

On 2017-10-31 11:42, Nick Coghlan wrote:
On 31 October 2017 at 02:29, Guido van Rossum > wrote:


What's your proposed process to arrive at the list of recommended
packages?


I'm thinking it makes the most sense to treat inclusion in the 
recommended packages list as a possible outcome of proposals for 
standard library inclusion, rather than being something we'd provide a 
way to propose specifically.


We'd only use it in cases where a proposal would otherwise meet the 
criteria for stdlib inclusion, but the logistics of actually doing so 
don't work for some reason.


Running the initial 5 proposals through that filter:

* six: a cross-version compatibility layer clearly needs to be outside 
the standard library
* setuptools: we want to update this in line with the PyPA interop 
specs, not the Python language version
* cffi: updates may be needed for PyPA interop specs, Python 
implementation updates or C language definition updates
* requests: updates are more likely to be driven by changes in network 
protocols and client platform APIs than Python language changes
* regex: we don't want two regex engines in the stdlib, transparently 
replacing _sre would be difficult, and _sre is still good enough for 
most purposes



regex gets updated when the Unicode Consortium releases an update.

[snip]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Wes Turner]

On Tuesday, October 31, 2017, Guido van Rossum  wrote:

> On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan  > wrote:
>
>> On 31 October 2017 at 02:29, Guido van Rossum > > wrote:
>>
>>> What's your proposed process to arrive at the list of recommended
>>> packages?
>>>
>>
>> I'm thinking it makes the most sense to treat inclusion in the
>> recommended packages list as a possible outcome of proposals for standard
>> library inclusion, rather than being something we'd provide a way to
>> propose specifically.
>>
>
> I don't think that gets you off the hook for a process proposal. We need
> some criteria to explain why a module should be on the recommended list --
> not just a ruling as to why it shouldn't be in the stdlib.
>
>
>> We'd only use it in cases where a proposal would otherwise meet the
>> criteria for stdlib inclusion, but the logistics of actually doing so don't
>> work for some reason.
>>
>
> But that would exclude most of the modules you mention below, since one of
> the criteria is that their development speed be matched with Python's
> release cycle. I think there must be some form of "popularity" combined
> with "best of breed". In particular I'd like to have a rule that explains
> why flask and Django would never make the list. (I don't know what that
> rule is, or I would tell you -- my gut tells me it's something to do with
> having their own community *and* competing for the same spot.)
>
> Running the initial 5 proposals through that filter:
>>
>> * six: a cross-version compatibility layer clearly needs to be outside
>> the standard library
>>
>
> Hm... Does six still change regularly? If not I think it *would* be a
> candidate for actual stdlib inclusion. Just like we added u"..." literals
> to Python 3.4.
>
>
>> * setuptools: we want to update this in line with the PyPA interop specs,
>> not the Python language version
>>
>
> But does that exclude stdlib inclusion? Why would those specs change, and
> why couldn't they wait for a new Python release?
>
>
>> * cffi: updates may be needed for PyPA interop specs, Python
>> implementation updates or C language definition updates
>>
>
> Hm, again, I don't recall that this was debated -- I think it's a failure
> that it's not in the stdlib.
>
>
>> * requests: updates are more likely to be driven by changes in network
>> protocols and client platform APIs than Python language changes
>>
>
> Here I agree. There's no alternative (except aiohttp, but that's
> asyncio-based) and it can't be in the stdlib because it's actively being
> developed.
>

What about certifi (SSL bundles (from requests (?)) on PyPi)
https://pypi.org/project/certifi/ ?


>
>
>> * regex: we don't want two regex engines in the stdlib, transparently
>> replacing _sre would be difficult, and _sre is still good enough for most
>> purposes
>>
>
> I think this needn't be recommended at all. For 99.9% of regular
> expression uses, re is just fine. Let's just work on a strategy for
> introducing regex into the stdlib.
>
>
>> Of the 5, I'd suggest that regex is the only one that could potentially
>> still make its way into the standard library some day - it would just
>> require someone with both the time and inclination to create a CPython
>> variant that used _regex instead of _sre as the default regex engine, and
>> then gathered evidence to show that it was "compatible enough" with _sre to
>> serve as the default engine for CPython.
>>
>> For the first four, there are compelling arguments that their drivers for
>> new feature additions are such that their release cycles shouldn't ever be
>> tied to the rate at which we update the Python language definition.
>>
>
> As you can tell from my arguing, the reasons need to be written up in more
> detail.
>
>
>> And is it really just going to be a list of names, or is there going to
>>> be some documentation (about the vetting, not about the contents of the
>>> packages) for each name?
>>>
>>
>> I'm thinking a new subsection in https://docs.python.org/devgui
>> de/stdlibchanges.html for "Recommended Third Party Packages" would make
>> sense, covering what I wrote above.
>>
>
> That's too well hidden for my taste.
>
>
>> It also occurred to me that since the recommendations are independent of
>> the Python version, they don't really belong in the version specific
>> documentation.
>>
>
> But that doesn't mean they can't (also) be listed there. (And each
> probably has its version dependencies.)
>
>
>> While the Developer's Guide isn't really the right place for the list
>> either (except as an easier way to answer "Why isn't  in the standard
>> library?" questions), it could be a good interim option until I get around
>> to actually writing a first draft of https://github.com/python/redi
>> stributor-guide/ (which I was talking to Barry about at the dev sprint,
>> but didn't end up 

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

OK, go ahead and write the PEP! I'm actually happy with the responses you
gave, so your last email will make a good start for some of the contents of
the PEP.

On Tue, Oct 31, 2017 at 9:19 AM, Nick Coghlan  wrote:

> On 1 November 2017 at 00:53, Guido van Rossum  wrote:
>
>> On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan  wrote:
>>
>>> On 31 October 2017 at 02:29, Guido van Rossum  wrote:
>>>
 What's your proposed process to arrive at the list of recommended
 packages?

>>>
>>> I'm thinking it makes the most sense to treat inclusion in the
>>> recommended packages list as a possible outcome of proposals for standard
>>> library inclusion, rather than being something we'd provide a way to
>>> propose specifically.
>>>
>>
>> I don't think that gets you off the hook for a process proposal. We need
>> some criteria to explain why a module should be on the recommended list --
>> not just a ruling as to why it shouldn't be in the stdlib.
>>
>
> The developer guide already has couple of sections on this aspect:
>
> * https://devguide.python.org/stdlibchanges/#acceptable-types-of-modules
> * https://devguide.python.org/stdlibchanges/#requirements
>
> I don't think either of those sections is actually quite right (since
> we've approved new modules that wouldn't meet them), but they're not
> terrible as a starting point in general, and they're accurate for the
> recommended packages use case.
>
>
>> We'd only use it in cases where a proposal would otherwise meet the
>>> criteria for stdlib inclusion, but the logistics of actually doing so don't
>>> work for some reason.
>>>
>>
>> But that would exclude most of the modules you mention below, since one
>> of the criteria is that their development speed be matched with Python's
>> release cycle. I think there must be some form of "popularity" combined
>> with "best of breed". In particular I'd like to have a rule that explains
>> why flask and Django would never make the list. (I don't know what that
>> rule is, or I would tell you -- my gut tells me it's something to do with
>> having their own community *and* competing for the same spot.)
>>
>
> The developer guide words this as "The module needs to be considered
> best-of-breed.". In some categories (like WSGI frameworks), there are
> inherent trade-offs that mean there will *never* be a single best-of-breed
> solution, since projects like Django, Flask, and Pyramid occupy
> deliberately different points in the space of available design decisions.
>
> Running the initial 5 proposals through that filter:
>>>
>>> * six: a cross-version compatibility layer clearly needs to be outside
>>> the standard library
>>>
>>
>> Hm... Does six still change regularly? If not I think it *would* be a
>> candidate for actual stdlib inclusion. Just like we added u"..." literals
>> to Python 3.4.
>>
>
> It still changes as folks porting new projects discover additional
> discrepancies between the 2.x and 3.x standard library layouts and
> behaviour (e.g. we found recently that the 3.x subprocess module's
> emulation of the old commands module APIs actually bit shifts the status
> codes relative to the 2.7 versions). The rate of change has definitely
> slowed down a lot since the early days, but it isn't zero.
>
> In addition, the only folks that need it are those that already care about
> older versions of Python - if you can start with whatever the latest
> version of Python is, and don't have any reason to support users still
> running older version, you can happily pretend those older versions don't
> exist, and hence don't need a compatibility library like six. As a separate
> library, it can just gracefully fade away as folks stop depending on it as
> they drop Python 2.7 support. By contrast, if we were to bring it into the
> 3.x standard library, then we'd eventually have to figure out when we could
> deprecate and remove it again.
>
>
>> * setuptools: we want to update this in line with the PyPA interop specs,
>>> not the Python language version
>>>
>>
>> But does that exclude stdlib inclusion? Why would those specs change, and
>> why couldn't they wait for a new Python release?
>>
>
> The specs mainly change when we want to offer publishers new capabilities
> while still maintaining compatibility with older installation clients (and
> vice-versa: we want folks still running Python 2.7 to be able to publish
> wheel files and use recently added metadata fields like
> Description-Content-Type).
>
> The reason we can't wait for new Python releases is because when we add
> such things, we need them to work on *all* supported Python releases
> (including 2.7 and security-release-only 3.x versions).
>
> There are also other drivers for setuptools updates, which include:
>
> - operating system build toolchain changes (e.g. finding new versions of
> Visual Studio or XCode)
> - changes to PyPI's operations (e.g. the legacy upload API getting turned

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 1 November 2017 at 00:53, Guido van Rossum  wrote:

> On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan  wrote:
>
>> On 31 October 2017 at 02:29, Guido van Rossum  wrote:
>>
>>> What's your proposed process to arrive at the list of recommended
>>> packages?
>>>
>>
>> I'm thinking it makes the most sense to treat inclusion in the
>> recommended packages list as a possible outcome of proposals for standard
>> library inclusion, rather than being something we'd provide a way to
>> propose specifically.
>>
>
> I don't think that gets you off the hook for a process proposal. We need
> some criteria to explain why a module should be on the recommended list --
> not just a ruling as to why it shouldn't be in the stdlib.
>

The developer guide already has couple of sections on this aspect:

* https://devguide.python.org/stdlibchanges/#acceptable-types-of-modules
* https://devguide.python.org/stdlibchanges/#requirements

I don't think either of those sections is actually quite right (since we've
approved new modules that wouldn't meet them), but they're not terrible as
a starting point in general, and they're accurate for the recommended
packages use case.


> We'd only use it in cases where a proposal would otherwise meet the
>> criteria for stdlib inclusion, but the logistics of actually doing so don't
>> work for some reason.
>>
>
> But that would exclude most of the modules you mention below, since one of
> the criteria is that their development speed be matched with Python's
> release cycle. I think there must be some form of "popularity" combined
> with "best of breed". In particular I'd like to have a rule that explains
> why flask and Django would never make the list. (I don't know what that
> rule is, or I would tell you -- my gut tells me it's something to do with
> having their own community *and* competing for the same spot.)
>

The developer guide words this as "The module needs to be considered
best-of-breed.". In some categories (like WSGI frameworks), there are
inherent trade-offs that mean there will *never* be a single best-of-breed
solution, since projects like Django, Flask, and Pyramid occupy
deliberately different points in the space of available design decisions.

Running the initial 5 proposals through that filter:
>>
>> * six: a cross-version compatibility layer clearly needs to be outside
>> the standard library
>>
>
> Hm... Does six still change regularly? If not I think it *would* be a
> candidate for actual stdlib inclusion. Just like we added u"..." literals
> to Python 3.4.
>

It still changes as folks porting new projects discover additional
discrepancies between the 2.x and 3.x standard library layouts and
behaviour (e.g. we found recently that the 3.x subprocess module's
emulation of the old commands module APIs actually bit shifts the status
codes relative to the 2.7 versions). The rate of change has definitely
slowed down a lot since the early days, but it isn't zero.

In addition, the only folks that need it are those that already care about
older versions of Python - if you can start with whatever the latest
version of Python is, and don't have any reason to support users still
running older version, you can happily pretend those older versions don't
exist, and hence don't need a compatibility library like six. As a separate
library, it can just gracefully fade away as folks stop depending on it as
they drop Python 2.7 support. By contrast, if we were to bring it into the
3.x standard library, then we'd eventually have to figure out when we could
deprecate and remove it again.


> * setuptools: we want to update this in line with the PyPA interop specs,
>> not the Python language version
>>
>
> But does that exclude stdlib inclusion? Why would those specs change, and
> why couldn't they wait for a new Python release?
>

The specs mainly change when we want to offer publishers new capabilities
while still maintaining compatibility with older installation clients (and
vice-versa: we want folks still running Python 2.7 to be able to publish
wheel files and use recently added metadata fields like
Description-Content-Type).

The reason we can't wait for new Python releases is because when we add
such things, we need them to work on *all* supported Python releases
(including 2.7 and security-release-only 3.x versions).

There are also other drivers for setuptools updates, which include:

- operating system build toolchain changes (e.g. finding new versions of
Visual Studio or XCode)
- changes to PyPI's operations (e.g. the legacy upload API getting turned
off due to persistent service stability problems, switching to HTTPS only
access)

With setuptools as a separate project, a whole lot of package publication
problems can be solved via "pip install --upgrade setuptools wheel" in a
virtual environment, which is a luxury we don't have with plain distutils.


>  * cffi: updates may be needed for PyPA interop specs, Python
> 

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

On Tue, Oct 31, 2017 at 4:42 AM, Nick Coghlan  wrote:

> On 31 October 2017 at 02:29, Guido van Rossum  wrote:
>
>> What's your proposed process to arrive at the list of recommended
>> packages?
>>
>
> I'm thinking it makes the most sense to treat inclusion in the recommended
> packages list as a possible outcome of proposals for standard library
> inclusion, rather than being something we'd provide a way to propose
> specifically.
>

I don't think that gets you off the hook for a process proposal. We need
some criteria to explain why a module should be on the recommended list --
not just a ruling as to why it shouldn't be in the stdlib.


> We'd only use it in cases where a proposal would otherwise meet the
> criteria for stdlib inclusion, but the logistics of actually doing so don't
> work for some reason.
>

But that would exclude most of the modules you mention below, since one of
the criteria is that their development speed be matched with Python's
release cycle. I think there must be some form of "popularity" combined
with "best of breed". In particular I'd like to have a rule that explains
why flask and Django would never make the list. (I don't know what that
rule is, or I would tell you -- my gut tells me it's something to do with
having their own community *and* competing for the same spot.)

Running the initial 5 proposals through that filter:
>
> * six: a cross-version compatibility layer clearly needs to be outside the
> standard library
>

Hm... Does six still change regularly? If not I think it *would* be a
candidate for actual stdlib inclusion. Just like we added u"..." literals
to Python 3.4.


> * setuptools: we want to update this in line with the PyPA interop specs,
> not the Python language version
>

But does that exclude stdlib inclusion? Why would those specs change, and
why couldn't they wait for a new Python release?


> * cffi: updates may be needed for PyPA interop specs, Python
> implementation updates or C language definition updates
>

Hm, again, I don't recall that this was debated -- I think it's a failure
that it's not in the stdlib.


> * requests: updates are more likely to be driven by changes in network
> protocols and client platform APIs than Python language changes
>

Here I agree. There's no alternative (except aiohttp, but that's
asyncio-based) and it can't be in the stdlib because it's actively being
developed.


> * regex: we don't want two regex engines in the stdlib, transparently
> replacing _sre would be difficult, and _sre is still good enough for most
> purposes
>

I think this needn't be recommended at all. For 99.9% of regular expression
uses, re is just fine. Let's just work on a strategy for introducing regex
into the stdlib.


> Of the 5, I'd suggest that regex is the only one that could potentially
> still make its way into the standard library some day - it would just
> require someone with both the time and inclination to create a CPython
> variant that used _regex instead of _sre as the default regex engine, and
> then gathered evidence to show that it was "compatible enough" with _sre to
> serve as the default engine for CPython.
>
> For the first four, there are compelling arguments that their drivers for
> new feature additions are such that their release cycles shouldn't ever be
> tied to the rate at which we update the Python language definition.
>

As you can tell from my arguing, the reasons need to be written up in more
detail.


> And is it really just going to be a list of names, or is there going to be
>> some documentation (about the vetting, not about the contents of the
>> packages) for each name?
>>
>
> I'm thinking a new subsection in https://docs.python.org/
> devguide/stdlibchanges.html for "Recommended Third Party Packages" would
> make sense, covering what I wrote above.
>

That's too well hidden for my taste.


> It also occurred to me that since the recommendations are independent of
> the Python version, they don't really belong in the version specific
> documentation.
>

But that doesn't mean they can't (also) be listed there. (And each probably
has its version dependencies.)


> While the Developer's Guide isn't really the right place for the list
> either (except as an easier way to answer "Why isn't  in the standard
> library?" questions), it could be a good interim option until I get around
> to actually writing a first draft of https://github.com/python/
> redistributor-guide/ (which I was talking to Barry about at the dev
> sprint, but didn't end up actually creating any content for since I went
> down a signal handling rabbit hole instead).
>

Hm, let's not put more arbitrary check boxes in the way of progress. Maybe
it can be an informational PEP that's occasionally updated?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of 

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 31 October 2017 at 02:29, Guido van Rossum  wrote:

> What's your proposed process to arrive at the list of recommended packages?
>

I'm thinking it makes the most sense to treat inclusion in the recommended
packages list as a possible outcome of proposals for standard library
inclusion, rather than being something we'd provide a way to propose
specifically.

We'd only use it in cases where a proposal would otherwise meet the
criteria for stdlib inclusion, but the logistics of actually doing so don't
work for some reason.

Running the initial 5 proposals through that filter:

* six: a cross-version compatibility layer clearly needs to be outside the
standard library
* setuptools: we want to update this in line with the PyPA interop specs,
not the Python language version
* cffi: updates may be needed for PyPA interop specs, Python implementation
updates or C language definition updates
* requests: updates are more likely to be driven by changes in network
protocols and client platform APIs than Python language changes
* regex: we don't want two regex engines in the stdlib, transparently
replacing _sre would be difficult, and _sre is still good enough for most
purposes

Of the 5, I'd suggest that regex is the only one that could potentially
still make its way into the standard library some day - it would just
require someone with both the time and inclination to create a CPython
variant that used _regex instead of _sre as the default regex engine, and
then gathered evidence to show that it was "compatible enough" with _sre to
serve as the default engine for CPython.

For the first four, there are compelling arguments that their drivers for
new feature additions are such that their release cycles shouldn't ever be
tied to the rate at which we update the Python language definition.


> And is it really just going to be a list of names, or is there going to be
> some documentation (about the vetting, not about the contents of the
> packages) for each name?
>

I'm thinking a new subsection in
https://docs.python.org/devguide/stdlibchanges.html for "Recommended Third
Party Packages" would make sense, covering what I wrote above.

It also occurred to me that since the recommendations are independent of
the Python version, they don't really belong in the version specific
documentation.

While the Developer's Guide isn't really the right place for the list
either (except as an easier way to answer "Why isn't  in the standard
library?" questions), it could be a good interim option until I get around
to actually writing a first draft of
https://github.com/python/redistributor-guide/ (which I was talking to
Barry about at the dev sprint, but didn't end up actually creating any
content for since I went down a signal handling rabbit hole instead).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

So what's your list? What would you put in requirements.txt? I'm fine with
doing it in the form of requirements.txt -- I am worried that when push
comes to shove, we won't be able to agree on what list of package names
should go into that file.

On Mon, Oct 30, 2017 at 12:08 PM, Juancarlo Añez  wrote:

>
>
> On Mon, Oct 30, 2017 at 12:29 PM, Guido van Rossum 
> wrote:
>
>> What's your proposed process to arrive at the list of recommended
>> packages? And is it really just going to be a list of names, or is there
>> going to be some documentation (about the vetting, not about the contents
>> of the packages) for each name?
>>
>
> As I see it, the bootstrap would be a requirements.txt (requirements.pip)
> with the packages that more experienced developers prefer to use over
> stdlib, pinned to package versions known to work well with the CPython
> release.
>
> I think this is a great idea, specially if it's an easy opt-in.
>
> For example, sometimes I go into a programming niche for a few years, and
> I'd very much appreciate a curated list of packages when I move to another
> niche. A pay-forward kind of thing.
>
> The downside to the idea is that the list of packages will need a curator
> (which could be python.org).
>
>
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Juancarlo Añez]

On Mon, Oct 30, 2017 at 12:29 PM, Guido van Rossum  wrote:

> What's your proposed process to arrive at the list of recommended
> packages? And is it really just going to be a list of names, or is there
> going to be some documentation (about the vetting, not about the contents
> of the packages) for each name?
>

As I see it, the bootstrap would be a requirements.txt (requirements.pip)
with the packages that more experienced developers prefer to use over
stdlib, pinned to package versions known to work well with the CPython
release.

I think this is a great idea, specially if it's an easy opt-in.

For example, sometimes I go into a programming niche for a few years, and
I'd very much appreciate a curated list of packages when I move to another
niche. A pay-forward kind of thing.

The downside to the idea is that the list of packages will need a curator
(which could be python.org).
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

What's your proposed process to arrive at the list of recommended packages?
And is it really just going to be a list of names, or is there going to be
some documentation (about the vetting, not about the contents of the
packages) for each name?

On Mon, Oct 30, 2017 at 9:08 AM, Nick Coghlan  wrote:

> On 31 October 2017 at 00:28, Guido van Rossum  wrote:
>
>> I just feel that when you're talking about an org like PayPal they can
>> take care of themselves and don't need our help. They will likely have
>> packages they want installed everywhere that would never make in on your
>> list. So it feels this whole discussion is a distraction and a waste of
>> time (yours, too).
>>
>
> Just because companies are big doesn't mean they necessarily have anyone
> internally that's already up to speed on the specifics of recommended
> practices in a sprawling open source community like Python's. The genesis
> of this idea is that I think we can offer a more consistent initial
> experience for those folks than "Here's PyPI and Google, y'all have fun
> now" (and in so doing, help folks writing books and online tutorials to
> feel more comfortable with the idea of assuming that libraries like
> requests will be available in even the most restrictive institutional
> environments that still allow the use of Python).
>
> One specific situation this idea is designed to help with is the one where:
>
> - there's a centrally managed Standard Operating Environment that dictates
> what gets installed
> - they've approved the python.org installers
> - they *haven't* approved anything else yet
>
> Now, a lot of large orgs simply won't get into that situation in the first
> place, since their own supplier management rules will push them towards a
> commercial redistributor, in which case they'll get their chosen
> redistributor's preferred package set, which will then typically cover at
> least a few hundred of the most popular PyPI packages.
>
> But some of them will start from the narrower "standard library only"
> baseline, and I spent enough time back at Boeing arguing for libraries to
> be added to our approved component list to appreciate the benefits of
> transitive declarations of trust ("we trust supplier X, they unambigously
> state their trust in supplier Y, so that's an additional point in favour of
> our also trusting supplier Y") when it comes time to make your case to your
> supplier management organisation. Such declarations still aren'y always
> sufficient, but they definitely don't hurt, and they sometimes help.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 31 October 2017 at 00:28, Guido van Rossum  wrote:

> I just feel that when you're talking about an org like PayPal they can
> take care of themselves and don't need our help. They will likely have
> packages they want installed everywhere that would never make in on your
> list. So it feels this whole discussion is a distraction and a waste of
> time (yours, too).
>

Just because companies are big doesn't mean they necessarily have anyone
internally that's already up to speed on the specifics of recommended
practices in a sprawling open source community like Python's. The genesis
of this idea is that I think we can offer a more consistent initial
experience for those folks than "Here's PyPI and Google, y'all have fun
now" (and in so doing, help folks writing books and online tutorials to
feel more comfortable with the idea of assuming that libraries like
requests will be available in even the most restrictive institutional
environments that still allow the use of Python).

One specific situation this idea is designed to help with is the one where:

- there's a centrally managed Standard Operating Environment that dictates
what gets installed
- they've approved the python.org installers
- they *haven't* approved anything else yet

Now, a lot of large orgs simply won't get into that situation in the first
place, since their own supplier management rules will push them towards a
commercial redistributor, in which case they'll get their chosen
redistributor's preferred package set, which will then typically cover at
least a few hundred of the most popular PyPI packages.

But some of them will start from the narrower "standard library only"
baseline, and I spent enough time back at Boeing arguing for libraries to
be added to our approved component list to appreciate the benefits of
transitive declarations of trust ("we trust supplier X, they unambigously
state their trust in supplier Y, so that's an additional point in favour of
our also trusting supplier Y") when it comes time to make your case to your
supplier management organisation. Such declarations still aren'y always
sufficient, but they definitely don't hurt, and they sometimes help.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 30 October 2017 at 04:56, Guido van Rossum  wrote:

> The two use cases you describe (scripters and teachers) leave me luke-warm
> -- scripters live in the wild west and can just pip install whatever
> (that's what it means to be scripting)
>

For personal scripting, we can install whatever, but "institutional
scripting" isn't the same thing - there we're scripting predefined
"Standard Operating Environments", like those Mahmoud Hashemi describes for
PayPal at the start of
https://www.paypal-engineering.com/2016/09/07/python-packaging-at-paypal/

"Just use PyInstaller" isn't an adequate answer in large-scale environments
because of the "zlib problem": you don't want to have to rebuild and
redeploy the world to handle a security update in a low level frequently
used component, you want to just update that component and have everything
else pick it up dynamically.

While "Just use conda" is excellent advice nowadays for any organisation
contemplating defining their own bespoke Python SOE (hence Mahmoud's post),
if that isn't being driven by the folks that already maintain the SOE (as
happened in PayPal's case) convincing an org to add a handful of python-dev
endorsed libraries to an established SOE is going to be easier than
rebasing their entire Python SOE on conda.


> and teachers tend to want a customized bundle anyway -- let the edu world
> get together and create their own recommended bundle.
>
> As long as it's not going to be bundled, i.e. there's just going to be
> some list of packages that we recommend to 3rd party repackagers, then I'm
> fine with it. But they must remain clearly marked as 3rd party packages in
> whatever docs we provide, and live in site-packages.
>

Yep, that was my intent, although it may not have been clear in my initial
proposal. I've filed two separate RFEs in relation to that:

* Documentation only: https://bugs.python.org/issue31898
* Regression testing resource: https://bugs.python.org/issue31899

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Paul Moore]

On 29 October 2017 at 20:44, Alex Walters  wrote:
> Writing scripts for non-developers, in an unmanaged environment (IT cant
> push a python install to the system) on windows means running pyinstaller
> et. al., on your script, if it has dependencies or not.  Its not worth it to
> walk someone through a python install to run a script, let alone installing
> optional dependencies.

Let's just say "not in the environments I work in", and leave it at that.

> Not to sound too crass, but there are only so many edge cases
> that a third party platform developer can be expected to care about (enough
> to bend over backwards to support).

I never suggested otherwise. I just wanted to point out that
"scripting in Python" covers a wider range of use cases than "can use
pip install". I'm not asking python-dev to support those use cases
(heck, I'm part of python-dev and *I* don't want to bend over
backwards to support them), but I do ask that people be careful not to
dismiss a group of users who are commonly under-represented in open
source mailing lists and communities.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Alex Walters]

Writing scripts for non-developers, in an unmanaged environment (IT cant
push a python install to the system) on windows means running pyinstaller
et. al., on your script, if it has dependencies or not.  Its not worth it to
walk someone through a python install to run a script, let alone installing
optional dependencies.  For linux, you are not limited to the standard
library, but what the distros have packaged - its easy enough to tell a
typical non-developer linux user "apt install python python-twisted, then
run the script".

For deployment to restricted environments, optional dependencies suggested
by python-dev is unlikely to be installed either.  Not that this matters, or
would matter for years, because IME those environments have ancient versions
of python that the developer is restricted to anyways.

And environments with limited internet access ... Bandersnatch on a portable
hard drive.  Not to sound too crass, but there are only so many edge cases
that a third party platform developer can be expected to care about (enough
to bend over backwards to support).

Putting a curated list of "These are good packages that solve a lot of
problems that the stdlib doesn't or makes complicated" on python.org is a
great idea.  Building tooling to make those part of a default installation
is not.

> -Original Message-
> From: Python-ideas [mailto:python-ideas-bounces+tritium-
> list=sdamon@python.org] On Behalf Of Paul Moore
> Sent: Sunday, October 29, 2017 4:26 PM
> To: Guido van Rossum <gu...@python.org>
> Cc: python-ideas@python.org
> Subject: Re: [Python-ideas] Defining an easily installable "Recommended
> baseline package set"
> 
> On 29 October 2017 at 18:56, Guido van Rossum <gu...@python.org> wrote:
> > The two use cases you describe (scripters and teachers) leave me luke-
> warm
> > -- scripters live in the wild west and can just pip install whatever
(that's
> > what it means to be scripting)
> 
> In my experience, "scripting" *does* include people for whom "being in
> a standard Python distribution" is a requirement.
> 
> Situations I have encountered:
> 
> 1. Servers where developers need to write administrative or monitoring
> scripts, but they don't control what's on that server. The Python
> installation that comes by default is all that's available.
> 2. Developers working in environments with limited internet access.
> For example, my employer has a Windows (NTLM) proxy that pip can't
> work with. Getting internet access for pip involves installing a
> separate application, and that's not always possible/desirable.
> 3. Developers writing scripts to be shared with non-developers. On
> Unix, this means "must work with the system Python", and on Windows
> "download and install Python from python.org" is typically all you can
> expect (although in that case "install Anaconda" is a possible
> alternative, although not one I've tried telling people to do myself).
> 
> Nick's proposal doesn't actually help for (1) or (2), as the problem
> there is that "pip install" won't work. And bundling a script with its
> (pure Python) dependencies, for example as a zipapp, is always a
> solution - although it's nowhere near as easy as simply copying a
> single-file script to the destination where it's to be run. So these
> situations don't actually matter in terms of the value of the
> proposals being discussed here. But I did want to dispute the idea
> that "scripters can just pip install whatever" is inherent to the idea
> of being a scripter - my experience is the opposite, that scripters
> are the people *least* able to simply pip install things.
> 
> Paul
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Antoine Pitrou]

On Sun, 29 Oct 2017 11:44:52 +
Paul Moore  wrote:

> On 29 October 2017 at 09:51, Antoine Pitrou  wrote:
> > On Sun, 29 Oct 2017 17:54:22 +1000
> > Nick Coghlan  wrote:  
> >> This means that
> >> if educators aren't teaching them, or redistributors aren't providing them,
> >> then they're actively doing their users a disservice  
> >
> > Which redistributors do not provide the requests library, for example?
> > regex is probably not as popular (mostly because re is good enough for
> > most purposes), but it still appears to be available from Ubuntu and
> > Anaconda.  
> 
> I know it's not what you meant, but "the python.org installers" is the
> obvious answer here.

Well, I'm not sure what Nick meant by the word exactly, but "to
redistribute" means "to distribute again", so the word looked aimed
at third-party vendors of Python rather than python.org :-)

Regards

Antoine.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

The two use cases you describe (scripters and teachers) leave me luke-warm
-- scripters live in the wild west and can just pip install whatever
(that's what it means to be scripting) and teachers tend to want a
customized bundle anyway -- let the edu world get together and create their
own recommended bundle.

As long as it's not going to be bundled, i.e. there's just going to be some
list of packages that we recommend to 3rd party repackagers, then I'm fine
with it. But they must remain clearly marked as 3rd party packages in
whatever docs we provide, and live in site-packages.

I would really like to see what you'd add to the list besides requests -- I
really don't see why the teaching use case would need the regex module
(unless it's a class in regular expressions).

--Guido

On Sun, Oct 29, 2017 at 12:54 AM, Nick Coghlan  wrote:

> On 29 October 2017 at 15:16, Guido van Rossum  wrote:
>
>> Why? What's wrong with pip install?
>>
>
> At a technical level, this would just be a really thin wrapper around 'pip
> install' (even thinner than ensurepip in general, since these libraries
> *wouldn't* be bundled for offline installation, only listed by name).
>
>
>> Why complicate things? Your motivation is really weak here. "beneficial"?
>> "difficult cases"?
>>
>
> The main recurring problems with "pip install" are a lack of
> discoverability and a potential lack of availability (depending on the
> environment).
>
> This then causes a couple of key undesirable outcomes:
>
> - folks using Python as a teaching language have to choose between
> teaching with just the standard library APIs, requiring that learners
> restrict themselves to a particular preconfigured learning environment, or
> make a detour into package management tools in order to ensure learners
> have access to the APIs they actually want to use (this isn't hypothetical
> - I was a technical reviewer for a book that justified teaching XML-RPC
> over HTTPS+JSON on the basis that xmlrpc was in the standard library, and
> requests wasn't)
> - folks using Python purely as a scripting language (i.e without app level
> dependency management) may end up having to restrict themselves to the
> standard library API, even when there's a well-established frequently
> preferred alternative for what they're doing (e.g. requests for API
> management, regex for enhanced regular expressions)
>
> The underlying problem is that our reasons for omitting these particular
> libraries from the standard library relate mainly to publisher side
> concerns like the logistics of ongoing bug fixing and support, *not* end
> user concerns like software reliability or API usability. This means that
> if educators aren't teaching them, or redistributors aren't providing them,
> then they're actively doing their users a disservice (as opposed to other
> cases like web frameworks and similar, where there are multiple competing
> options, you're only going to want one of them in any given application,
> and the relevant trade-offs between the available options depend greatly on
> exactly what you're doing)
>
> Now, the Python-for-data-science community have taken a particular
> direction around handling this, and there's an additional library set
> beyond the standard library that's pretty much taken for granted in a data
> science context. While conda has been the focal point for those efforts
> more recently, it started a long time ago with initiatives like Python(x,
> y) and the Enthought Python Distribution.
>
> Similarly, initiatives like Raspberry Pi are able to assume a particular
> learning environment (Raspbian in the Pi's case), rather than coping with
> arbitrary starting points.
>
> Curated lists like the "awesome-python" one that Stephan linked don't
> really help that much with the discoverability problem, since they become
> just another thing for people to learn: How do they find out such lists
> exist in the first place? Given such a list, how do they determine if the
> recommendations it offers are actually relevant to their needs? Since
> assessing a published package API against your needs as a user is a skill
> that has to be learned like any other, it can be a lot easier to get
> started in a more prescriptive environment that says "This is what you have
> to work with for now, we'll explain more about your options for branching
> out later".
>
> The proposal in this thread thus stems from asking the question "Who is
> going to be best positioned to offer authoritative advice on which third
> party modules may be preferable to their standard library counterparts for
> end users of Python?" and answering it with "The standard library module
> maintainers that are already responsible for deciding whether or not to
> place appropriate See Also links in the module documentation".
>
> All the proposal does is to suggest taking those existing recommendations
> from the documentation and converting them into a more readibly 

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Stefan Krah]

On Sun, Oct 29, 2017 at 10:10:21AM -0700, Brendan Barnwell wrote:
> Only now, since the libraries aren't in the stdlib, the Python devs
> won't really be able to do anything to fix that; all they could do
> is remove the offending package from the approved list.  In practice
> I think this is unlikely to happen, since the idea would be that
> Python devs would be judicious in awarding the seal of approval only
> to projects that are robust and not prone to breakage.  But it does
> change the nature of the approval from "we approve this *code* and
> we will fix it if it breaks" (which is how the existing stdlib
> works) to "we approve these *people* (the people working on requests
> or regex or whatever) and we will cease to do if they break their
> code".

Let's be realistic: If MRAB were to include regex in the stdlib, *he*
would be the one fixing things anyway.

And he'd get to fight feature requests and stylistic rewrites. :-)


Stefan Krah




___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Brendan Barnwell]

On 2017-10-29 00:54, Nick Coghlan wrote:

The proposal in this thread thus stems from asking the question "Who is
going to be best positioned to offer authoritative advice on which third
party modules may be preferable to their standard library counterparts
for end users of Python?" and answering it with "The standard library
module maintainers that are already responsible for deciding whether or
not to place appropriate See Also links in the module documentation".

All the proposal does is to suggest taking those existing
recommendations from the documentation and converting them into a more
readibly executable form.


	So it sounds like you're just saying that there should be one more 
"awesome-python" style list, except it should be official, with the seal 
of approval of Python itself.  The details of exactly how that would 
displayed to users are not so important as that it be easily accessible 
from python.org and clearly shown to be endorsed by the Python developers.


	I think that is a good idea.  The current situation is quite confusing 
in many cases.  You see a lot of questions on StackOverflow where 
someone is tearing their hair out trying to use urllib to do heavy 
lifting and the answer is "use requests instead".  Likewise people 
trying to implement matrix multiplication who should probably be using 
numpy,  As for regex, to be honest, I have yet to find any way in which 
it is not completely superior to the existing re library, and I find it 
somewhat frustrating that the "publisher-side" concerns you mention 
continue to hold it back from replacing re.


	The only problem I see with this sort of "Python seal of approval" 
library list is that it carries some of the same risks as incorporating 
such libraries into the stdlib.  Not all, but some.  For one thing, if 
people go to python.org and see a thing that says "You may also want to 
use these libraries that bear the Python Seal of Approval. . .", and 
then they download them, and something goes wrong (e.g., there is some 
kind of version conflict with another package), they will likely blame 
python.org (at least partially).  Only now, since the libraries aren't 
in the stdlib, the Python devs won't really be able to do anything to 
fix that; all they could do is remove the offending package from the 
approved list.  In practice I think this is unlikely to happen, since 
the idea would be that Python devs would be judicious in awarding the 
seal of approval only to projects that are robust and not prone to 
breakage.  But it does change the nature of the approval from "we 
approve this *code* and we will fix it if it breaks" (which is how the 
existing stdlib works) to "we approve these *people* (the people working 
on requests or regex or whatever) and we will cease to do if they break 
their code".


--
Brendan Barnwell
"Do not follow where the path may lead.  Go, instead, where there is no 
path, and leave a trail."

   --author unknown
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Koos Zevenhoven]

On Sun, Oct 29, 2017 at 1:44 PM, Paul Moore  wrote:

> On 29 October 2017 at 09:51, Antoine Pitrou  wrote:
> > On Sun, 29 Oct 2017 17:54:22 +1000
> > Nick Coghlan  wrote:
> >> All the proposal does is to suggest taking those existing
> recommendations
> >> from the documentation and converting them into a more readibly
> executable
> >> form.
> >
> > I'm curious what such a list looks like :-)
>
> I am also. I'd put requests on it immediately, but that's the only
> thing I consider obvious. regex is what triggered this, but I'm not
> sure it adds *that* much - it's a trade off between people who need
> the extra features and people confused over why we have two regex
> libraries available. After that, you're almost immediately into
> domain-specific answers, and it becomes tricky fast.
>

​If the list is otherwise not be long enough, maybe it will contain
something that is now in the stdlib?-)

-- Koos​


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Wes Turner]

On Sunday, October 29, 2017, Paul Moore  wrote:

> On 29 October 2017 at 10:40, Stephan Houben  > wrote:
> > Perhaps slightly off-topic, but I have sometimes wondered if
> > pip could not be made somewhat friendlier for the absolute newbie
> > and the classroom context.
> >
> > Some concrete proposals.
> >
> > 1. Add a function `pip` to the interactive interpreter
> >   (similar to how `help` is available).
> >
> >   def pip(args):
> >   import sys
> >   import subprocess
> >   subprocess.check_call([sys.executable, "-m", "pip"] +
> args.split())
> >
> >This allows people to install something using pip as long as they
> have a
> >Python prompt open, and avoids instructors to have to deal with
> > platform-specific
> >instructions for various shells. Also avoids confusion when multiple
> > Python interpreters
> >are available (it operates in the context of the current interpreter.)
>
> There are subtle issues around whether newly installed/upgraded
> packages are visible in a running Python interpreter.


If the package is already imported, reload() isn't sufficient; but
deepreload from IPython may be.

IPython also supports shell commands prefixed with '!':

! pip install -U requests regex IPython
! pip install -U --user psfblessedset1
%run pip install -U --user psfblessedset1
%run?

% autoreload?
# http://ipython.readthedocs.io/en/stable/config/extensions/autoreload.html

This:


> It's possible
> that this would result in *more* confusion than the current situation.


Why isn't it upgrading over top of the preinstalled set?

At least with bash, the shell command history is logged to .bash_history by
default. IMO, it's easier to assume that environment-modifying commands are
logged in the shell log; and that running a ``%logstart -o logged-cmds.py``
won't change the versions of packages it builds upon.

Is it bad form to put ``! pip install -U requests `` at the top of every
jupyter notebook?
``! pip install requests==version `` is definitely more reproducible.
(seeAlso: binder, jupyter/docker-stacks)

I'll just include these here unnecessarily from the other curated package
set discussions:

https://github.com/jupyter/docker-stacks

https://github.com/Kaggle/docker-python/blob/master/Dockerfile

https://binderhub.readthedocs.io/en/latest/

I can see the appeal of something like this, but it's not as simple as
> it looks. If you want to discuss this further, I'd definitely suggest
> making it a thread of its own.


> Personally, as a pip maintainer, I'm -0
> on this (possibly even -1).


Same.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Paul Moore]

On 29 October 2017 at 10:40, Stephan Houben  wrote:
> Perhaps slightly off-topic, but I have sometimes wondered if
> pip could not be made somewhat friendlier for the absolute newbie
> and the classroom context.
>
> Some concrete proposals.
>
> 1. Add a function `pip` to the interactive interpreter
>   (similar to how `help` is available).
>
>   def pip(args):
>   import sys
>   import subprocess
>   subprocess.check_call([sys.executable, "-m", "pip"] + args.split())
>
>This allows people to install something using pip as long as they have a
>Python prompt open, and avoids instructors to have to deal with
> platform-specific
>instructions for various shells. Also avoids confusion when multiple
> Python interpreters
>are available (it operates in the context of the current interpreter.)

There are subtle issues around whether newly installed/upgraded
packages are visible in a running Python interpreter. It's possible
that this would result in *more* confusion than the current situation.
I can see the appeal of something like this, but it's not as simple as
it looks. If you want to discuss this further, I'd definitely suggest
making it a thread of its own. Personally, as a pip maintainer, I'm -0
on this (possibly even -1).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Paul Moore]

On 29 October 2017 at 09:51, Antoine Pitrou  wrote:
> On Sun, 29 Oct 2017 17:54:22 +1000
> Nick Coghlan  wrote:
>> This means that
>> if educators aren't teaching them, or redistributors aren't providing them,
>> then they're actively doing their users a disservice
>
> Which redistributors do not provide the requests library, for example?
> regex is probably not as popular (mostly because re is good enough for
> most purposes), but it still appears to be available from Ubuntu and
> Anaconda.

I know it's not what you meant, but "the python.org installers" is the
obvious answer here. On Windows, if you say to someone "install
Python", they get the python.org distribution. Explicitly directing
them to Anaconda is an option, but that gives them a distinctly
different experience than "standard Python plus some best of breed
packages like requests" does.

>> All the proposal does is to suggest taking those existing recommendations
>> from the documentation and converting them into a more readibly executable
>> form.
>
> I'm curious what such a list looks like :-)

I am also. I'd put requests on it immediately, but that's the only
thing I consider obvious. regex is what triggered this, but I'm not
sure it adds *that* much - it's a trade off between people who need
the extra features and people confused over why we have two regex
libraries available. After that, you're almost immediately into
domain-specific answers, and it becomes tricky fast.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Paul Moore]

On 29 October 2017 at 07:54, Nick Coghlan  wrote:
> On 29 October 2017 at 15:16, Guido van Rossum  wrote:
>>
>> Why? What's wrong with pip install?
>
> At a technical level, this would just be a really thin wrapper around 'pip
> install' (even thinner than ensurepip in general, since these libraries
> *wouldn't* be bundled for offline installation, only listed by name).

I agree with Guido, this doesn't seem to add much as stated.

(From an earlier message of Nick's)
> At the same time, it would be beneficial to have a way to offer an even
> stronger recommendation to redistributors that we think full-featured
> general purpose Python scripting environments should offer the regex
> module as an opt-in alternative to the baseline re feature set, since that
> would also help with other currently difficult cases like the requests module.

This is key to me. The target area is *scripting*. Library and
application developers have their own ways of managing the dependency
handling problem, and they generally work fine. The people who don't
currently have a good solution are those who just use the system
install - i.e., people writing adhoc scripts, people writing code that
they want to share with other members of their organisation, via
"here's this file, just run it", and not as a full application. For
those people "not in a standard install" is a significantly higher
barrier to usage than elsewhere. "Here's a Python file, just install
Python and double click on the file" is a reasonable request. Running
pip may be beyond the capabilities of the recipient.

In my view, the key problems in this area are:

1. What users writing one-off scripts can expect to be available.
2. Corporate environments where "adding extra software" is a major hurdle.
3. Offline environments, where PyPI isn't available.

The solutions to these issues aren't so much around how we let people
know that they should do "pip install" (or "--install-recommended")
but rather around initial installs. To that end I'd suggest:

1. The python.org installers gain an "install recommended 3rd party
packages" option, that is true by default, which does "pip install
". This covers (1) and (2) above, as it makes
the recommended package set the norm, and ensures that they are
covered by an approval to "install Python".
2. For offline environments, we need to do a little more, but I'd
imagine having the installer look for wheels in the same directory as
the installer executable, and leave it to the user to put them there.
If wheels aren't available the installer should warn but continue.

For 3rd party distributions (Linux, Homebrew, Conda, ...) this gives a
clear message that Python users are entitled to expect these modules
to be available. Handling that expectation is down to those
distributions.

Things I haven't thought through:

1. System vs user site-packages. If we install (say) requests into the
system site-packages, the user could potentially need admin rights to
upgrade it. Or get into a situation where they have an older version
in site-packages and a newer one in user-site (which I don't believe
is a well-tested scenario),
2. Virtual environments. Should venv/virtualenv include the
recommended packages by default? Probably not, but that does make the
"in a virtual environment" experience different from the system Python
experience.
3. The suggestion above for offline environments isn't perfect, by any
means. Better solutions would be good here (but I don't think we want
to go down the bundling route like we did with pip...?).

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Stephan Houben]

Perhaps slightly off-topic, but I have sometimes wondered if
pip could not be made somewhat friendlier for the absolute newbie
and the classroom context.

Some concrete proposals.

1. Add a function `pip` to the interactive interpreter
  (similar to how `help` is available).

  def pip(args):
  import sys
  import subprocess
  subprocess.check_call([sys.executable, "-m", "pip"] + args.split())

   This allows people to install something using pip as long as they have a
   Python prompt open, and avoids instructors to have to deal with
platform-specific
   instructions for various shells. Also avoids confusion when multiple
Python interpreters
   are available (it operates in the context of the current interpreter.)

2. Add to Pypi package webpages a line like:

To install, execute this line in your Python interpreter:
 pip("install my-package --user")

  Make this copyable with a button (like Github allows you to copy the repo
name).

3. Add the notion of a "channel" to which one can "subscribe".

   This is actually nothing new, just a Pypi package with no code and just
a bunch of
   dependencies. But it allows me to say: Just enter:

  pip("install stephans-awesome-stuff --user")

  in your Python and you get a bunch of useful stuff.

Stephan

2017-10-29 8:54 GMT+01:00 Nick Coghlan :

> On 29 October 2017 at 15:16, Guido van Rossum  wrote:
>
>> Why? What's wrong with pip install?
>>
>
> At a technical level, this would just be a really thin wrapper around 'pip
> install' (even thinner than ensurepip in general, since these libraries
> *wouldn't* be bundled for offline installation, only listed by name).
>
>
>> Why complicate things? Your motivation is really weak here. "beneficial"?
>> "difficult cases"?
>>
>
> The main recurring problems with "pip install" are a lack of
> discoverability and a potential lack of availability (depending on the
> environment).
>
> This then causes a couple of key undesirable outcomes:
>
> - folks using Python as a teaching language have to choose between
> teaching with just the standard library APIs, requiring that learners
> restrict themselves to a particular preconfigured learning environment, or
> make a detour into package management tools in order to ensure learners
> have access to the APIs they actually want to use (this isn't hypothetical
> - I was a technical reviewer for a book that justified teaching XML-RPC
> over HTTPS+JSON on the basis that xmlrpc was in the standard library, and
> requests wasn't)
> - folks using Python purely as a scripting language (i.e without app level
> dependency management) may end up having to restrict themselves to the
> standard library API, even when there's a well-established frequently
> preferred alternative for what they're doing (e.g. requests for API
> management, regex for enhanced regular expressions)
>
> The underlying problem is that our reasons for omitting these particular
> libraries from the standard library relate mainly to publisher side
> concerns like the logistics of ongoing bug fixing and support, *not* end
> user concerns like software reliability or API usability. This means that
> if educators aren't teaching them, or redistributors aren't providing them,
> then they're actively doing their users a disservice (as opposed to other
> cases like web frameworks and similar, where there are multiple competing
> options, you're only going to want one of them in any given application,
> and the relevant trade-offs between the available options depend greatly on
> exactly what you're doing)
>
> Now, the Python-for-data-science community have taken a particular
> direction around handling this, and there's an additional library set
> beyond the standard library that's pretty much taken for granted in a data
> science context. While conda has been the focal point for those efforts
> more recently, it started a long time ago with initiatives like Python(x,
> y) and the Enthought Python Distribution.
>
> Similarly, initiatives like Raspberry Pi are able to assume a particular
> learning environment (Raspbian in the Pi's case), rather than coping with
> arbitrary starting points.
>
> Curated lists like the "awesome-python" one that Stephan linked don't
> really help that much with the discoverability problem, since they become
> just another thing for people to learn: How do they find out such lists
> exist in the first place? Given such a list, how do they determine if the
> recommendations it offers are actually relevant to their needs? Since
> assessing a published package API against your needs as a user is a skill
> that has to be learned like any other, it can be a lot easier to get
> started in a more prescriptive environment that says "This is what you have
> to work with for now, we'll explain more about your options for branching
> out later".
>
> The proposal in this thread thus stems from asking the question "Who is
> going to be best positioned 

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Antoine Pitrou]

On Sun, 29 Oct 2017 17:54:22 +1000
Nick Coghlan  wrote:
> 
> The underlying problem is that our reasons for omitting these particular
> libraries from the standard library relate mainly to publisher side
> concerns like the logistics of ongoing bug fixing and support, *not* end
> user concerns like software reliability or API usability.

They're both really.  One important consequence of a library being in
the stdlib is to tie it to the stdlib's release cycle, QA
infrastructure and compatibility requirements -- which more or less
solves many dependency and/or version pinning headaches.

> This means that
> if educators aren't teaching them, or redistributors aren't providing them,
> then they're actively doing their users a disservice

Which redistributors do not provide the requests library, for example?
regex is probably not as popular (mostly because re is good enough for
most purposes), but it still appears to be available from Ubuntu and
Anaconda.

> All the proposal does is to suggest taking those existing recommendations
> from the documentation and converting them into a more readibly executable
> form.

I'm curious what such a list looks like :-)

Regards

Antoine.


___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

On 29 October 2017 at 15:16, Guido van Rossum  wrote:

> Why? What's wrong with pip install?
>

At a technical level, this would just be a really thin wrapper around 'pip
install' (even thinner than ensurepip in general, since these libraries
*wouldn't* be bundled for offline installation, only listed by name).


> Why complicate things? Your motivation is really weak here. "beneficial"?
> "difficult cases"?
>

The main recurring problems with "pip install" are a lack of
discoverability and a potential lack of availability (depending on the
environment).

This then causes a couple of key undesirable outcomes:

- folks using Python as a teaching language have to choose between teaching
with just the standard library APIs, requiring that learners restrict
themselves to a particular preconfigured learning environment, or make a
detour into package management tools in order to ensure learners have
access to the APIs they actually want to use (this isn't hypothetical - I
was a technical reviewer for a book that justified teaching XML-RPC over
HTTPS+JSON on the basis that xmlrpc was in the standard library, and
requests wasn't)
- folks using Python purely as a scripting language (i.e without app level
dependency management) may end up having to restrict themselves to the
standard library API, even when there's a well-established frequently
preferred alternative for what they're doing (e.g. requests for API
management, regex for enhanced regular expressions)

The underlying problem is that our reasons for omitting these particular
libraries from the standard library relate mainly to publisher side
concerns like the logistics of ongoing bug fixing and support, *not* end
user concerns like software reliability or API usability. This means that
if educators aren't teaching them, or redistributors aren't providing them,
then they're actively doing their users a disservice (as opposed to other
cases like web frameworks and similar, where there are multiple competing
options, you're only going to want one of them in any given application,
and the relevant trade-offs between the available options depend greatly on
exactly what you're doing)

Now, the Python-for-data-science community have taken a particular
direction around handling this, and there's an additional library set
beyond the standard library that's pretty much taken for granted in a data
science context. While conda has been the focal point for those efforts
more recently, it started a long time ago with initiatives like Python(x,
y) and the Enthought Python Distribution.

Similarly, initiatives like Raspberry Pi are able to assume a particular
learning environment (Raspbian in the Pi's case), rather than coping with
arbitrary starting points.

Curated lists like the "awesome-python" one that Stephan linked don't
really help that much with the discoverability problem, since they become
just another thing for people to learn: How do they find out such lists
exist in the first place? Given such a list, how do they determine if the
recommendations it offers are actually relevant to their needs? Since
assessing a published package API against your needs as a user is a skill
that has to be learned like any other, it can be a lot easier to get
started in a more prescriptive environment that says "This is what you have
to work with for now, we'll explain more about your options for branching
out later".

The proposal in this thread thus stems from asking the question "Who is
going to be best positioned to offer authoritative advice on which third
party modules may be preferable to their standard library counterparts for
end users of Python?" and answering it with "The standard library module
maintainers that are already responsible for deciding whether or not to
place appropriate See Also links in the module documentation".

All the proposal does is to suggest taking those existing recommendations
from the documentation and converting them into a more readibly executable
form.

I'm not particularly wedded to any particular approach to making the
recommendations available in a more machine-friendly form, though - it's
just the "offer something more machine friendly than scraping the docs for
recommendation links" aspect that I'm interested in. For example, we could
skip touching ensurepip or venv at all, and instead limit this to a
documentation proposal to collect these recommendations from the
documentation, and publish them within the `venv` module docs as a
"recommended-libraries.txt" file (using pip's requirements.txt format).
That would be sufficient to allow straightforward 3rd party automation,
without necessarily committing to providing such automation ourselves.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: 

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Stephan Houben]

There are already curated lists of Python packages, such as:

https://github.com/vinta/awesome-python

Even better, if you don't agree, you can just clone it and create your own
;-)

Stephan

2017-10-29 6:46 GMT+01:00 Alex Walters <tritium-l...@sdamon.com>:

> At this point, I would punt to distutils-sig about curated packages, and
> pip tooling to support that, but they are bogged down as it stands with
> just getting warehouse up and running.  I don’t support putting specialized
> tooling in python itself to install curated packages, because that curation
> would be on the same release schedule as python itself.  Pip is an
> important special case, but it’s a special case to avoid further special
> cases.  If there was excess manpower on the packaging side, that’s where it
> should be developed.
>
>
>
> *From:* Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com@
> python.org] *On Behalf Of *Guido van Rossum
> *Sent:* Sunday, October 29, 2017 1:17 AM
> *To:* Nick Coghlan <ncogh...@gmail.com>
> *Cc:* python-ideas@python.org
> *Subject:* Re: [Python-ideas] Defining an easily installable "Recommended
> baseline package set"
>
>
>
> Why? What's wrong with pip install? Why complicate things? Your motivation
> is really weak here. "beneficial"? "difficult cases"?
>
>
>
> On Sat, Oct 28, 2017 at 8:57 PM, Nick Coghlan <ncogh...@gmail.com> wrote:
>
> Over on python-dev, the question of recommending MRAB's "regex" module
> over the standard library's "re" module for more advanced regular
> expressions recently came up again.
>
> Because of various logistical issues and backwards compatibility risks,
> it's highly unlikely that we'll ever be able to swap out the current _sre
> based re module implementation in the standard library for an
> implementation based on the regex module.
>
> At the same time, it would be beneficial to have a way to offer an even
> stronger recommendation to redistributors that we think full-featured
> general purpose Python scripting environments should offer the regex module
> as an opt-in alternative to the baseline re feature set, since that would
> also help with other currently difficult cases like the requests module.
>
> What I'm thinking is that we could make some relatively simple additions
> to the `ensurepip` and `venv` modules to help with this:
>
>
>
> 1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library
> module names containing dependency specifiers for recommended third party
> packages for particular tasks (e.g. "regex" as an enhanced alternative to
> "re", "requests" as an enhanced HTTPS-centric alternative to
> "urllib.request")
>
> 2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap
>
> 3. Add a corresponding `--install-recommended flag to the `python -m
> ensurepip` CLI
>
> 4. Add a corresponding `--install-recommended flag to the `python -m venv`
> CLI (when combined with `--without-pip`, this would run pip directly from
> the bundled wheel file to do the installations)
>
> We'd also need either a new informational PEP or else a section in the
> developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES`
> are up to the individual module maintainers (hence keying the mapping by
> standard library module name, rather than having a single flat list for the
> entire standard library).
>
> For redistributors with weak dependency support, these reference
> interpreter level recommendations could become redistributor level
> recommendations. Redistributors without weak dependency support could still
> make a distinction between "default" installations (which would include
> them) and "minimal" installations (which would exclude them).
>
> Folks writing scripts and example code for independent distribution (i.e.
> no explicitly declared dependencies) could then choose between relying on
> just the standard library (as now), or on the standard library plus
> independently versioned recommended packages.
>
>
>
> Cheers,
>
> Nick.
>
>
> --
>
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
>
> --
>
> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Alex Walters]

At this point, I would punt to distutils-sig about curated packages, and pip 
tooling to support that, but they are bogged down as it stands with just 
getting warehouse up and running.  I don’t support putting specialized tooling 
in python itself to install curated packages, because that curation would be on 
the same release schedule as python itself.  Pip is an important special case, 
but it’s a special case to avoid further special cases.  If there was excess 
manpower on the packaging side, that’s where it should be developed.

 

From: Python-ideas 
[mailto:python-ideas-bounces+tritium-list=sdamon@python.org] On Behalf Of 
Guido van Rossum
Sent: Sunday, October 29, 2017 1:17 AM
To: Nick Coghlan <ncogh...@gmail.com>
Cc: python-ideas@python.org
Subject: Re: [Python-ideas] Defining an easily installable "Recommended 
baseline package set"

 

Why? What's wrong with pip install? Why complicate things? Your motivation is 
really weak here. "beneficial"? "difficult cases"?

 

On Sat, Oct 28, 2017 at 8:57 PM, Nick Coghlan <ncogh...@gmail.com 
<mailto:ncogh...@gmail.com> > wrote:

Over on python-dev, the question of recommending MRAB's "regex" module over the 
standard library's "re" module for more advanced regular expressions recently 
came up again.

Because of various logistical issues and backwards compatibility risks, it's 
highly unlikely that we'll ever be able to swap out the current _sre based re 
module implementation in the standard library for an implementation based on 
the regex module.

At the same time, it would be beneficial to have a way to offer an even 
stronger recommendation to redistributors that we think full-featured general 
purpose Python scripting environments should offer the regex module as an 
opt-in alternative to the baseline re feature set, since that would also help 
with other currently difficult cases like the requests module.

What I'm thinking is that we could make some relatively simple additions to the 
`ensurepip` and `venv` modules to help with this:

 

1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library 
module names containing dependency specifiers for recommended third party 
packages for particular tasks (e.g. "regex" as an enhanced alternative to "re", 
"requests" as an enhanced HTTPS-centric alternative to "urllib.request")

2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap

3. Add a corresponding `--install-recommended flag to the `python -m ensurepip` 
CLI

4. Add a corresponding `--install-recommended flag to the `python -m venv` CLI 
(when combined with `--without-pip`, this would run pip directly from the 
bundled wheel file to do the installations)

We'd also need either a new informational PEP or else a section in the 
developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES` 
are up to the individual module maintainers (hence keying the mapping by 
standard library module name, rather than having a single flat list for the 
entire standard library).

For redistributors with weak dependency support, these reference interpreter 
level recommendations could become redistributor level recommendations. 
Redistributors without weak dependency support could still make a distinction 
between "default" installations (which would include them) and "minimal" 
installations (which would exclude them).

Folks writing scripts and example code for independent distribution (i.e. no 
explicitly declared dependencies) could then choose between relying on just the 
standard library (as now), or on the standard library plus independently 
versioned recommended packages.

 

Cheers,

Nick.



-- 

Nick Coghlan   |   ncogh...@gmail.com <mailto:ncogh...@gmail.com>|   
Brisbane, Australia


___
Python-ideas mailing list
Python-ideas@python.org <mailto:Python-ideas@python.org> 
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/




-- 

--Guido van Rossum (python.org/~guido <http://python.org/~guido> )

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Defining an easily installable "Recommended baseline package set"

[Guido van Rossum]

Why? What's wrong with pip install? Why complicate things? Your motivation
is really weak here. "beneficial"? "difficult cases"?

On Sat, Oct 28, 2017 at 8:57 PM, Nick Coghlan  wrote:

> Over on python-dev, the question of recommending MRAB's "regex" module
> over the standard library's "re" module for more advanced regular
> expressions recently came up again.
>
> Because of various logistical issues and backwards compatibility risks,
> it's highly unlikely that we'll ever be able to swap out the current _sre
> based re module implementation in the standard library for an
> implementation based on the regex module.
>
> At the same time, it would be beneficial to have a way to offer an even
> stronger recommendation to redistributors that we think full-featured
> general purpose Python scripting environments should offer the regex module
> as an opt-in alternative to the baseline re feature set, since that would
> also help with other currently difficult cases like the requests module.
>
> What I'm thinking is that we could make some relatively simple additions
> to the `ensurepip` and `venv` modules to help with this:
>
> 1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library
> module names containing dependency specifiers for recommended third party
> packages for particular tasks (e.g. "regex" as an enhanced alternative to
> "re", "requests" as an enhanced HTTPS-centric alternative to
> "urllib.request")
>
> 2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap
>
> 3. Add a corresponding `--install-recommended flag to the `python -m
> ensurepip` CLI
>
> 4. Add a corresponding `--install-recommended flag to the `python -m venv`
> CLI (when combined with `--without-pip`, this would run pip directly from
> the bundled wheel file to do the installations)
>
> We'd also need either a new informational PEP or else a section in the
> developer guide to say that the contents of `ensurepip.RECOMMENDED_PACKAGES`
> are up to the individual module maintainers (hence keying the mapping by
> standard library module name, rather than having a single flat list for the
> entire standard library).
>
> For redistributors with weak dependency support, these reference
> interpreter level recommendations could become redistributor level
> recommendations. Redistributors without weak dependency support could still
> make a distinction between "default" installations (which would include
> them) and "minimal" installations (which would exclude them).
>
> Folks writing scripts and example code for independent distribution (i.e.
> no explicitly declared dependencies) could then choose between relying on
> just the standard library (as now), or on the standard library plus
> independently versioned recommended packages.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Defining an easily installable "Recommended baseline package set"

[Nick Coghlan]

Over on python-dev, the question of recommending MRAB's "regex" module over
the standard library's "re" module for more advanced regular expressions
recently came up again.

Because of various logistical issues and backwards compatibility risks,
it's highly unlikely that we'll ever be able to swap out the current _sre
based re module implementation in the standard library for an
implementation based on the regex module.

At the same time, it would be beneficial to have a way to offer an even
stronger recommendation to redistributors that we think full-featured
general purpose Python scripting environments should offer the regex module
as an opt-in alternative to the baseline re feature set, since that would
also help with other currently difficult cases like the requests module.

What I'm thinking is that we could make some relatively simple additions to
the `ensurepip` and `venv` modules to help with this:

1. Add a ensurepip.RECOMMENDED_PACKAGES mapping keyed by standard library
module names containing dependency specifiers for recommended third party
packages for particular tasks (e.g. "regex" as an enhanced alternative to
"re", "requests" as an enhanced HTTPS-centric alternative to
"urllib.request")

2. Add a new `install_recommended` boolean flag to ensurepip.bootstrap

3. Add a corresponding `--install-recommended flag to the `python -m
ensurepip` CLI

4. Add a corresponding `--install-recommended flag to the `python -m venv`
CLI (when combined with `--without-pip`, this would run pip directly from
the bundled wheel file to do the installations)

We'd also need either a new informational PEP or else a section in the
developer guide to say that the contents of
`ensurepip.RECOMMENDED_PACKAGES` are up to the individual module
maintainers (hence keying the mapping by standard library module name,
rather than having a single flat list for the entire standard library).

For redistributors with weak dependency support, these reference
interpreter level recommendations could become redistributor level
recommendations. Redistributors without weak dependency support could still
make a distinction between "default" installations (which would include
them) and "minimal" installations (which would exclude them).

Folks writing scripts and example code for independent distribution (i.e.
no explicitly declared dependencies) could then choose between relying on
just the standard library (as now), or on the standard library plus
independently versioned recommended packages.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/