Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Nathaniel Smith]

On Sat, Dec 1, 2018, 06:56 Steve Holden  We* should probably do more collectively to point people at
> production-quality third-party modules, as I believe we currently do with
> pipenv which, while not a part of the standard library, is still
> recommended in the documentation as the preferred method of dependency
> management.
>

Small correction: the only "official" recommendation for pipenv is that
packaging.python.org (which is maintained by pypa, not python-dev) contains
several tutorials, and one of them discusses how to use pipenv. For a while
Kenneth used this as justification for telling everyone pipenv was "the
officially recommended install tool", and this created a lot of ill will,
so the pipenv team has been working on rolling that back.

A better precedent is requests. There was a long discussion a few years ago
about whether requests should move to the stdlib, and the outcome was that
it didn't, but the urllib docs got a note added recommending the use of
requests, which you can see here:

https://docs.python.org/3/library/urllib.request.html#module-urllib.request

Personally I would have phrased the note more strongly, but my perspective
is skewed by having tried to understand the internals. I'm glad urllib has
helped a lot of people solve their problems, but there's also a lot of ways
that it's flat out broken.

Anyway, I agree that there are probably other places where the docs could
use this technique.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steve Holden]

We* should probably do more collectively to point people at
production-quality third-party modules, as I believe we currently do with
pipenv which, while not a part of the standard library, is still
recommended in the documentation as the preferred method of dependency
management. We should also be even more strident when a library module is a
basic version, not to be used for production purposes.

This inevitably means, however, that there will be lag in the
documentation, which generally speaking lags current best practices.

Steve Holden

* I am not a significant contributor to the code base.



On Fri, Nov 30, 2018 at 9:02 PM Glenn Linderman 
wrote:

> On 11/29/2018 2:10 PM, Andrew Svetlov wrote:
>
> Neither http.client nor http.server doesn't support compression
> (gzip/compress/deflate) at all.
> I doubt if we want to add this feature: for client better to use requests
> or, well, aiohttp.
> The same for servers: almost any production ready web server from PyPI
> supports compression.
>
> What production ready web servers exist on PyPi? Are there any that don't
> bring lots of baggage, their own enhanced way of doing things? The nice
> thing about the http.server is that it does things in a standard-conforming
> way, the bad thing about it is that it doesn't implement all the standards,
> and isn't maintained very well.
>
> From just reading PyPi, it is hard to discover whether a particular
> package is production-ready or not.
>
> I had used CherryPy for a while, but at the time it didn't support Python
> 3, and to use the same scripts behind CherryPy or Apache CGI (my deployment
> target, because that was what web hosts provided) became difficult for
> complex scripts so I reverted to http.server with a few private
> extensions (private because no one merged the bugs I reported some 3
> versions of Python-development-process ago; back then I submitted patches,
> but I haven't had time to keep up with the churn of technologies Pythondev
> has used since Python 3 came out, which is when I started using Python, and
> I'm sure the submitted patches have bit-rotted by now).
>
> When I google "python web server" the first hit is the doc page for
> http.server, the second is a wiki page that mentions CherryPy and a bunch
> of others, but the descriptions, while terse, mostly point out some special
> capabilities of the server, making it seem like you not only get a web
> server, but a philosophy. I just want a web server. The last one, Waitress,
> is the only one that doesn't seem to have a philosophy in its description.
>
> So it would be nice if http.server and http.client could get some basic
> improvements to be complete, or if the docs could point to a replacement
> that is a complete server, but without a philosophy or framework
> (bloatware) to have to learn and/or work around.
>
> Glenn
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Fri, 30 Nov 2018 13:00:37 -0800
Glenn Linderman  wrote:
> 
> So it would be nice if http.server and http.client could get some basic 
> improvements to be complete, or if the docs could point to a replacement 
> that is a complete server, but without a philosophy or framework 
> (bloatware) to have to learn and/or work around.

Why do you think http.server is any different?  If you want to
implement your own Web service with http.server you must implement your
own handler class, which is not very different from writing a handler
for a third-party HTTP server.  Maybe you're so used to this way of
doing that you find it "natural", but in reality http.server is as
opinionated as any other HTTP server implementation.

By the way, there is a framework in the stdlib, it's called asyncio.
And I'm sure you'll find production-ready HTTP servers written for
it :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Glenn Linderman]

On 11/29/2018 2:10 PM, Andrew Svetlov wrote:
Neither http.client nor http.server doesn't support compression 
(gzip/compress/deflate) at all.
I doubt if we want to add this feature: for client better to use 
requests or, well, aiohttp.
The same for servers: almost any production ready web server from PyPI 
supports compression.


What production ready web servers exist on PyPi? Are there any that 
don't bring lots of baggage, their own enhanced way of doing things? The 
nice thing about the http.server is that it does things in a 
standard-conforming way, the bad thing about it is that it doesn't 
implement all the standards, and isn't maintained very well.


From just reading PyPi, it is hard to discover whether a particular 
package is production-ready or not.


I had used CherryPy for a while, but at the time it didn't support 
Python 3, and to use the same scripts behind CherryPy or Apache CGI (my 
deployment target, because that was what web hosts provided) became 
difficult for complex scripts so I reverted to http.server with a 
few private extensions (private because no one merged the bugs I 
reported some 3 versions of Python-development-process ago; back then I 
submitted patches, but I haven't had time to keep up with the churn of 
technologies Pythondev has used since Python 3 came out, which is when I 
started using Python, and I'm sure the submitted patches have bit-rotted 
by now).


When I google "python web server" the first hit is the doc page for 
http.server, the second is a wiki page that mentions CherryPy and a 
bunch of others, but the descriptions, while terse, mostly point out 
some special capabilities of the server, making it seem like you not 
only get a web server, but a philosophy. I just want a web server. The 
last one, Waitress, is the only one that doesn't seem to have a 
philosophy in its description.


So it would be nice if http.server and http.client could get some basic 
improvements to be complete, or if the docs could point to a replacement 
that is a complete server, but without a philosophy or framework 
(bloatware) to have to learn and/or work around.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Andrew Svetlov]

On Fri, Nov 30, 2018 at 2:12 AM Brett Cannon  wrote:

>
>
> On Thu, 29 Nov 2018 at 14:12, Andrew Svetlov 
> wrote:
>
>> Neither http.client nor http.server doesn't support compression
>> (gzip/compress/deflate) at all.
>> I doubt if we want to add this feature: for client better to use requests
>> or, well, aiohttp.
>> The same for servers: almost any production ready web server from PyPI
>> supports compression.
>>
>
> There was actually a PR to add compressions support to http.server but I
> closed it in the name of maintainability since http.server, as you said,
> isn't for production use so compression isn't critical.
>
> I remember this PR and agree with your decision.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Brett Cannon]

On Thu, 29 Nov 2018 at 14:12, Andrew Svetlov 
wrote:

> Neither http.client nor http.server doesn't support compression
> (gzip/compress/deflate) at all.
> I doubt if we want to add this feature: for client better to use requests
> or, well, aiohttp.
> The same for servers: almost any production ready web server from PyPI
> supports compression.
>

There was actually a PR to add compressions support to http.server but I
closed it in the name of maintainability since http.server, as you said,
isn't for production use so compression isn't critical.

-Brett


>
> I don't insist on adding brotli to standard library.
> There is officiall brotli library on PyPI from google, binary wheels are
> provided.
> Unfortunately installing from a tarball requires C++ compiler
> On other hand writing a binding in pure C looks very easy.
>
> On Thu, Nov 29, 2018 at 10:30 PM Gregory P. Smith  wrote:
>
>>
>> On Thu, Nov 29, 2018 at 2:58 AM Andrew Svetlov 
>> wrote:
>>
>>> 5 cents about lz4 alternatives: Broli (mentioned above) is widely
>>> supported by web.
>>>
>>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
>>> mentions it along with gzip and deflate methods.
>>> I don't recall lz4 or Zstd metioning in this context.
>>>
>>> Both Chrome/Chromium and Firefox accepts it by default (didn't check
>>> Microsoft products yet).
>>>
>>
>> Acceptance by multiple popular browsers is a good reason to *also*
>> propose brotli support in the stdlib. Though it'd probably make sense to
>> actually _support_ Accept-Encoding based on available compression modules
>> within the stdlib http.client (and potentially server) as a prerequisite
>> for that reasoning.
>> https://github.com/python/cpython/blob/master/Lib/http/client.py#L1168.
>>
>> -gps
>>
>>
>>> P.S.
>>> I worked with lz4 python binding a year ago.
>>> It sometimes crashed to core dump when used in multithreaded environment
>>> (we used to run compressor/decompresson with asyncio by
>>> loop.run_in_executor() call).
>>> I hope the bug is fixed now, have no update for the current state.
>>>
>>> On Thu, Nov 29, 2018 at 12:04 PM INADA Naoki 
>>> wrote:
>>>
 On Thu, Nov 29, 2018 at 6:27 AM Steven D'Aprano 
 wrote:
 >
 > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
 >
 > > PyPI makes getting more algorithms easy.
 >
 > Can we please stop over-generalising like this? PyPI makes getting
 > more algorithms easy for *SOME* people. (Sorry for shouting, but you
 > just pressed one of my buttons.)

 I don't think this is over-generalising.

 If "get it from PyPI" is not easy enough, why not adding hundreds of
 famous libraries?
 Because we can't maintain all of them well.

 When considering adding new format (not only compression, but also
 serialization like toml), I think it should be stable, widely used, and
 will
 be used widely for a long time.  If we want to use the format in Python
 core
 or Python stdlib, it's good reasoning too.  gzip and json are good
 example.

 When we say "we can use PyPI", it means "are there enough reasons
 make the package special enough to add to stdlib?"  We don't mean
 "everyone can use PyPI."

 Regards,
 --
 INADA Naoki  
 ___
 Python-Dev mailing list
 Python-Dev@python.org
 https://mail.python.org/mailman/listinfo/python-dev
 Unsubscribe:
 https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com

>>>
>>>
>>> --
>>> Thanks,
>>> Andrew Svetlov
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>>
>>
>
> --
> Thanks,
> Andrew Svetlov
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Andrew Svetlov]

On Thu, Nov 29, 2018 at 11:22 PM Steve Dower  wrote:

> FWIW, Brotli has been supported in Microsoft Edge since early last year:
>
>
> https://blogs.windows.com/msedgedev/2016/12/20/introducing-brotli-compression/
>
>
Thanks, good to know.

-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Andrew Svetlov]

Neither http.client nor http.server doesn't support compression
(gzip/compress/deflate) at all.
I doubt if we want to add this feature: for client better to use requests
or, well, aiohttp.
The same for servers: almost any production ready web server from PyPI
supports compression.

I don't insist on adding brotli to standard library.
There is officiall brotli library on PyPI from google, binary wheels are
provided.
Unfortunately installing from a tarball requires C++ compiler
On other hand writing a binding in pure C looks very easy.

On Thu, Nov 29, 2018 at 10:30 PM Gregory P. Smith  wrote:

>
> On Thu, Nov 29, 2018 at 2:58 AM Andrew Svetlov 
> wrote:
>
>> 5 cents about lz4 alternatives: Broli (mentioned above) is widely
>> supported by web.
>>
>> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
>> mentions it along with gzip and deflate methods.
>> I don't recall lz4 or Zstd metioning in this context.
>>
>> Both Chrome/Chromium and Firefox accepts it by default (didn't check
>> Microsoft products yet).
>>
>
> Acceptance by multiple popular browsers is a good reason to *also*
> propose brotli support in the stdlib. Though it'd probably make sense to
> actually _support_ Accept-Encoding based on available compression modules
> within the stdlib http.client (and potentially server) as a prerequisite
> for that reasoning.
> https://github.com/python/cpython/blob/master/Lib/http/client.py#L1168.
>
> -gps
>
>
>> P.S.
>> I worked with lz4 python binding a year ago.
>> It sometimes crashed to core dump when used in multithreaded environment
>> (we used to run compressor/decompresson with asyncio by
>> loop.run_in_executor() call).
>> I hope the bug is fixed now, have no update for the current state.
>>
>> On Thu, Nov 29, 2018 at 12:04 PM INADA Naoki 
>> wrote:
>>
>>> On Thu, Nov 29, 2018 at 6:27 AM Steven D'Aprano 
>>> wrote:
>>> >
>>> > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
>>> >
>>> > > PyPI makes getting more algorithms easy.
>>> >
>>> > Can we please stop over-generalising like this? PyPI makes getting
>>> > more algorithms easy for *SOME* people. (Sorry for shouting, but you
>>> > just pressed one of my buttons.)
>>>
>>> I don't think this is over-generalising.
>>>
>>> If "get it from PyPI" is not easy enough, why not adding hundreds of
>>> famous libraries?
>>> Because we can't maintain all of them well.
>>>
>>> When considering adding new format (not only compression, but also
>>> serialization like toml), I think it should be stable, widely used, and
>>> will
>>> be used widely for a long time.  If we want to use the format in Python
>>> core
>>> or Python stdlib, it's good reasoning too.  gzip and json are good
>>> example.
>>>
>>> When we say "we can use PyPI", it means "are there enough reasons
>>> make the package special enough to add to stdlib?"  We don't mean
>>> "everyone can use PyPI."
>>>
>>> Regards,
>>> --
>>> INADA Naoki  
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>>>
>>
>>
>> --
>> Thanks,
>> Andrew Svetlov
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>
>

-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steve Dower]

On 29Nov2018 1230, Gregory P. Smith wrote:


On Thu, Nov 29, 2018 at 2:58 AM Andrew Svetlov > wrote:


5 cents about lz4 alternatives: Broli (mentioned above) is widely
supported by web.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
mentions it along with gzip and deflate methods.
I don't recall lz4 or Zstd metioning in this context.

Both Chrome/Chromium and Firefox accepts it by default (didn't check
Microsoft products yet).


Acceptance by multiple popular browsers is a good reason to /also/ 
propose brotli support in the stdlib. Though it'd probably make sense to 
actually _support_ Accept-Encoding based on available compression 
modules within the stdlib http.client (and potentially server) as a 
prerequisite for that reasoning. 
https://github.com/python/cpython/blob/master/Lib/http/client.py#L1168.


FWIW, Brotli has been supported in Microsoft Edge since early last year:

https://blogs.windows.com/msedgedev/2016/12/20/introducing-brotli-compression/


-gps

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Gregory P. Smith]

On Thu, Nov 29, 2018 at 2:58 AM Andrew Svetlov 
wrote:

> 5 cents about lz4 alternatives: Broli (mentioned above) is widely
> supported by web.
>
> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
> mentions it along with gzip and deflate methods.
> I don't recall lz4 or Zstd metioning in this context.
>
> Both Chrome/Chromium and Firefox accepts it by default (didn't check
> Microsoft products yet).
>

Acceptance by multiple popular browsers is a good reason to *also* propose
brotli support in the stdlib. Though it'd probably make sense to actually
_support_ Accept-Encoding based on available compression modules within the
stdlib http.client (and potentially server) as a prerequisite for that
reasoning.
https://github.com/python/cpython/blob/master/Lib/http/client.py#L1168.

-gps


> P.S.
> I worked with lz4 python binding a year ago.
> It sometimes crashed to core dump when used in multithreaded environment
> (we used to run compressor/decompresson with asyncio by
> loop.run_in_executor() call).
> I hope the bug is fixed now, have no update for the current state.
>
> On Thu, Nov 29, 2018 at 12:04 PM INADA Naoki 
> wrote:
>
>> On Thu, Nov 29, 2018 at 6:27 AM Steven D'Aprano 
>> wrote:
>> >
>> > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
>> >
>> > > PyPI makes getting more algorithms easy.
>> >
>> > Can we please stop over-generalising like this? PyPI makes getting
>> > more algorithms easy for *SOME* people. (Sorry for shouting, but you
>> > just pressed one of my buttons.)
>>
>> I don't think this is over-generalising.
>>
>> If "get it from PyPI" is not easy enough, why not adding hundreds of
>> famous libraries?
>> Because we can't maintain all of them well.
>>
>> When considering adding new format (not only compression, but also
>> serialization like toml), I think it should be stable, widely used, and
>> will
>> be used widely for a long time.  If we want to use the format in Python
>> core
>> or Python stdlib, it's good reasoning too.  gzip and json are good
>> example.
>>
>> When we say "we can use PyPI", it means "are there enough reasons
>> make the package special enough to add to stdlib?"  We don't mean
>> "everyone can use PyPI."
>>
>> Regards,
>> --
>> INADA Naoki  
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>>
>
>
> --
> Thanks,
> Andrew Svetlov
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[David Mertz]

On Thu, Nov 29, 2018, 2:55 PM Paul Moore  ... and some users need a single, unambiguous choice for the
> "official, complete" distribution. Which need the current stdlib
> serves extremely well.
>

Except it doesn't. At least not for a large swatch of users.

10 years ago, what I wanted in Python was pretty much entirely in the
stdlib. The contents of stdlib haven't changed that much since then, but MY
needs have.

For what I do personally, a distribution without NumPy, Pandas, Numba,
scikit-learn, and matplotlib is unusably incomplete. On the other hand, I
rarely care about Django, Twisted, Whoosh, or Sphinx. But some users need
those things, and even lots of supporting packages in their ecosystems.

What makes a "complete distribution?" It really depends on context. The
stdlib is an extremely good compromise, but it absolutely is a compromise.
I feel like there is plenty of room for different purpose-driven supersets
of the stdlib to make different compromises. Steve Dower lists 10 or so
such distros; what they have in common is that SOMEONE, decided to curate a
collection... which does not need any approval from the PSF or the core
developers.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Paul Moore]

On Thu, 29 Nov 2018 at 17:52, Nathaniel Smith  wrote:
>
> On Thu, Nov 29, 2018, 08:34 Antoine Pitrou >
>>
>> Le 29/11/2018 à 17:25, Steve Dower a écrit :
>> >
>> > My experience is that the first group would benefit from a larger
>> > _standard distribution_, which is not necessarily the same thing as a
>> > larger stdlib.
>> >
>> > I'm firmly on the "smaller core, larger distribution" side of things,
>> > where we as the core team take responsibility for the bare minimum
>> > needed to be an effective language and split more functionality out to
>> > individual libraries.
>>
>> We may ask ourselves if there is really a large difference between a
>> "standard distribution" and a "standard library".  The primary
>> difference seems to be that the distribution is modular, while the
>> stdlib is not.
>
>
> Some differences that come to mind:

Ha. That arrived after I'd sent my other email. I'm not going to
discuss these points case by case, except to say that there are some
definite aspects of your interpretation of a "standard distribution"
that are in conflict with what would work best for my use case.[1]

Regardless of the details that we discuss here, it seems self-evident
to me that any proposal to move away from the current "core+stdlib"
distribution would be a significant change and would absolutely
require a PEP, which would have to cover all of these trade-offs in
detail. But until there is such a PEP, that clearly explains the
precise variant of "standard distribution" model that is being
proposed, I don't see much point in getting into details. There's
simply too much likelihood of people talking past each other because
of differing assumptions.

Paul

[1] Which, to reiterate, remains just one particular environment, plus
a lot of extrapolation that there are "many other enterprise
environments like mine"[2]
[2] Oh, I really hope not - pity the poor souls who would work there :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Paul Moore]

On Thu, 29 Nov 2018 at 19:08, Nathaniel Smith  wrote:
>
> On Thu, Nov 29, 2018, 10:32 Antoine Pitrou >
>> On Thu, 29 Nov 2018 09:49:32 -0800
>> Nathaniel Smith  wrote:
>> >
>> > There are a lot of challenges to switching to a "standard distribution"
>> > model. I'm not certain it's the best option. But what I like about it is
>> > that it could potentially reduce the conflict between what our different
>> > user groups need, instead of playing zero-sum tug-of-war every time this
>> > comes up.
>>
>> There is no conflict between what different _user_ groups need.
>> Including lz4 in the stdlib will not create a conflict for people who
>> prefer numpy or cryptography.
>
>
> Some users need as much functionality as possible in the standard download. 
> Some users need the best quality, most up to date software. The current 
> stdlib design makes it impossible to serve both sets of users well.

... and some users need a single, unambiguous choice for the
"official, complete" distribution. Which need the current stdlib
serves extremely well.

> The conflict is less extreme for software that's stable, tightly scoped, and 
> ubiquitous, like zlib or json; maybe lz4 is in the same place. But every time 
> we talk about adding a new package, it turns into a flashpoint for these 
> underlying tensions. I'm talking about the underlying issue, not about lz4 in 
> particular.

Agreed, there are functional areas that are less foundational, and
more open to debate. But historically, we've been very good at
achieving a balance in those areas. In exploring alternatives, let's
not lose sight of the fact that the stdlib has been a huge success, so
we know we *can* deliver an extremely successful distribution based on
that model, no matter how much it might trigger regular debates :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

Le 29/11/2018 à 20:05, Nathaniel Smith a écrit :
> On Thu, Nov 29, 2018, 10:32 Antoine Pitrou   wrote:
> 
> On Thu, 29 Nov 2018 09:49:32 -0800
> Nathaniel Smith mailto:n...@pobox.com>> wrote:
> >
> > There are a lot of challenges to switching to a "standard
> distribution"
> > model. I'm not certain it's the best option. But what I like about
> it is
> > that it could potentially reduce the conflict between what our
> different
> > user groups need, instead of playing zero-sum tug-of-war every
> time this
> > comes up.
> 
> There is no conflict between what different _user_ groups need.
> Including lz4 in the stdlib will not create a conflict for people who
> prefer numpy or cryptography.
> 
> Some users need as much functionality as possible in the standard
> download. Some users need the best quality, most up to date software.
> The current stdlib design makes it impossible to serve both sets of
> users well.

But it doesn't have to.  Obviously not every need can be solved by the
stdlib, unless every Python package that has at least one current user
is put in the stdlib.

So, yes, there's a discussion for each concretely proposed package about
whether it's sufficiently useful (and stable etc.) to be put in the
stdlib.  Every time it's a balancing act, and obviously it's an
imperfect decision.  That doesn't mean it cannot be done.  You'd
actually end up having the same discussions if designing a distribution,
only the overall bar would probably be lower.

By the way, I'll argue that putting modules in the stdlib does produce
better quality modules.  For example, putting multiprocessing in the
stdlib uncovered many defects that were previously largely unseen
(thanks in part to the extended CI that Python runs on).  Improving it
was a sometimes painful process, but today multiprocessing's quality is
far beyond what it was when originally included, and it can be
reasonably considered rock-solid within its intended domain.

One argument against putting lz4 in the stdlib is that compression
algorithms come by and, unless there's a large use base already, can
disappear quickly if they get outdated by some newer and better
algorithm.  I'm surprised I haven't seen that argument yet.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Nathaniel Smith]

On Thu, Nov 29, 2018, 10:32 Antoine Pitrou  On Thu, 29 Nov 2018 09:49:32 -0800
> Nathaniel Smith  wrote:
> >
> > There are a lot of challenges to switching to a "standard distribution"
> > model. I'm not certain it's the best option. But what I like about it is
> > that it could potentially reduce the conflict between what our different
> > user groups need, instead of playing zero-sum tug-of-war every time this
> > comes up.
>
> There is no conflict between what different _user_ groups need.
> Including lz4 in the stdlib will not create a conflict for people who
> prefer numpy or cryptography.


Some users need as much functionality as possible in the standard download.
Some users need the best quality, most up to date software. The current
stdlib design makes it impossible to serve both sets of users well.

The conflict is less extreme for software that's stable, tightly scoped,
and ubiquitous, like zlib or json; maybe lz4 is in the same place. But
every time we talk about adding a new package, it turns into a flashpoint
for these underlying tensions. I'm talking about the underlying issue, not
about lz4 in particular.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Paul Moore]

On Thu, 29 Nov 2018 at 16:28, Steve Dower  wrote:
> My experience is that the first group would benefit from a larger
> _standard distribution_, which is not necessarily the same thing as a
> larger stdlib.
>
> I'm firmly on the "smaller core, larger distribution" side of things,
> where we as the core team take responsibility for the bare minimum
> needed to be an effective language and split more functionality out to
> individual libraries. We then also prepare/recommend a standard
> distribution that bundles many of these libraries by default (Anaconda
> style), as well as a minimal one that is a better starting point for
> low-footprint systems (Miniconda style) or embedding into other apps.

For the environments I'm familiar with, a "large standard
distribution" would be just as acceptable as a "large standard
library". However, we have to be *very* careful that we understand
each other if we're going to make a fine distinction like this. So,
specifically, my requirements are:

1. The installers shipped from python.org as the "official" Windows
builds would need to be the "standard distribution", not a stripped
down core. People not familiar with the nuances (colleagues who want
to use a Python script I wrote, corporate auditors making decisions on
"what's allowed", etc) can't be expected to deal with "Python, but not
the one at python.org, this one instead" or even "Python, but make
sure you get this non-default download".[1]
2. The various other distributions (Anaconda, ActiveState, ...) would
need to buy into, and ship, the "standard distribution" (having to
deal with debates around "I have Python", "No you don't, that
distribution has different libraries" is problematic for grass-roots
adoption - well, for any sort of consistent experience). This is
probably both the most difficult requirement to achieve, and the one
I'd have the best chance of being somewhat flexible over. But only
"somewhat" flexible - we'd rapidly get bogged down in debating
questions on a module-by-module basis...
3. The distribution needs to be versioned as a whole. A key point of a
"standard set of modules" is not to have to deal with a combinatorial
explosion of version management issues. Even "Python x.y.z with
library a.b.c" offers some risks, unless the library is pretty stable
(slow pace of change is, of course, one of the problems with the
stdlib that proponents of a decoupled library hope to solve...)

At this point, I've talked myself into a position where I don't see
any practical difference between a stdlib and a standard distribution.
So what I think I need is for someone to describe a concrete proposal
for a "standard distribution", and explain clearly how it differs from
the stdlib, and where it *fails* to meet the above criteria. Then, and
only then, can I form a clear view on whether I would be OK with their
version of a "standard distribution".

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Thu, 29 Nov 2018 09:49:32 -0800
Nathaniel Smith  wrote:
> 
> There are a lot of challenges to switching to a "standard distribution"
> model. I'm not certain it's the best option. But what I like about it is
> that it could potentially reduce the conflict between what our different
> user groups need, instead of playing zero-sum tug-of-war every time this
> comes up.

There is no conflict between what different _user_ groups need.
Including lz4 in the stdlib will not create a conflict for people who
prefer numpy or cryptography.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Nathaniel Smith]

On Thu, Nov 29, 2018, 08:34 Antoine Pitrou 
> Le 29/11/2018 à 17:25, Steve Dower a écrit :
> >
> > My experience is that the first group would benefit from a larger
> > _standard distribution_, which is not necessarily the same thing as a
> > larger stdlib.
> >
> > I'm firmly on the "smaller core, larger distribution" side of things,
> > where we as the core team take responsibility for the bare minimum
> > needed to be an effective language and split more functionality out to
> > individual libraries.
>
> We may ask ourselves if there is really a large difference between a
> "standard distribution" and a "standard library".  The primary
> difference seems to be that the distribution is modular, while the
> stdlib is not.
>

Some differences that come to mind:

- a standard distribution provides a clear path for creating and managing
subsets, which is useful when disk/download weight is an issue. (This is
another situation that only affects some people and is easy for most of us
to forget about.) I guess this is your "modular" point?

- a standard distribution lets those who *do* have internet access install
upgrades incrementally as needed.

- This may be controversial, but my impression is that keeping package
development outside the stdlib almost always produces better packages in
the long run. You get faster feedback cycles, more responsive maintainers,
and it's just easier to find people to help maintain one focused package in
an area they care about than it is to get new core devs on board. Of course
there's probably some survivorship bias here too (urllib is worse than
requests, but is it worse than the average third-party package http
client?). But that's my impression.

Concretely: requests, pip, numpy, setuptools are all examples of packages
that should *obviously* be included in any self-respecting set of
batteries, but where we're not willing to put them in the stdlib. Obviously
we aren't doing a great job of supporting offline users, regardless of
whether we add lz4.

There are a lot of challenges to switching to a "standard distribution"
model. I'm not certain it's the best option. But what I like about it is
that it could potentially reduce the conflict between what our different
user groups need, instead of playing zero-sum tug-of-war every time this
comes up.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Christian Heimes]

On 29/11/2018 17.32, Antoine Pitrou wrote:
> We may ask ourselves if there is really a large difference between a
> "standard distribution" and a "standard library".  The primary
> difference seems to be that the distribution is modular, while the
> stdlib is not.

Yes, there is a huge difference between a larger distribution and a
stdlib. I'm going to ignore all legal issues and human drama of "choose
your favorite kid", but start with a simple example.

I'm sure we can all agree that the requests library should be part of an
extended Python distribution. After all it's the best and easiest HTTP
library for Python. I'm using it on a daily basis. However I would
require some changes to align it with other stdlib modules. Among others
requests and urllib3 would have to obey sys.flags.ignore_environment and
accept an SSLContext object. Requests depends on certifi to provide a CA
trust store. I would definitely veto against the inclusion of certifi,
because it's not just dangerous but also breaks my code.

If we would keep the standard distribution of Python as it is and just
have a Python SIG offer an additional extended distribution on
python.org, then I don't have to care about the quality and security of
additional code. The Python core team would neither own the code nor
takes responsibility of the code. Instead the extended distribution SIG
merely set the quality standards and leave the maintance burden to the
original owners. In case a library doesn't keep up or has severe flaws,
the SIG may even decide to remove a package from the extended distribution.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

Le 29/11/2018 à 17:25, Steve Dower a écrit :
> 
> My experience is that the first group would benefit from a larger 
> _standard distribution_, which is not necessarily the same thing as a 
> larger stdlib.
> 
> I'm firmly on the "smaller core, larger distribution" side of things, 
> where we as the core team take responsibility for the bare minimum 
> needed to be an effective language and split more functionality out to 
> individual libraries.

We may ask ourselves if there is really a large difference between a
"standard distribution" and a "standard library".  The primary
difference seems to be that the distribution is modular, while the
stdlib is not.

As for reviews and CI, though, we would probably want a standard
distribution to be high-quality, so to go through a review process and
be tested by the same buildbot fleet as the stdlib.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steve Dower]

On 29Nov2018 0254, Antoine Pitrou wrote:

I'd like to point the discussion is asymmetric here.

On the one hand, people who don't have access to PyPI would _really_
benefit from a larger stdlib with more batteries included.

On the other hand, people who have access to PyPI _don't_ benefit from
having a slim stdlib.  There's nothing virtuous or advantageous about
having _less_ batteries included.  Python doesn't become magically
faster or more powerful by including less in its standard
distribution: the best it does is make the distribution slightly
smaller.

So there's really one bunch of people arguing for practical benefits,
and another bunch of people arguing for mostly aesthetical or
philosophical reasons.


My experience is that the first group would benefit from a larger 
_standard distribution_, which is not necessarily the same thing as a 
larger stdlib.


I'm firmly on the "smaller core, larger distribution" side of things, 
where we as the core team take responsibility for the bare minimum 
needed to be an effective language and split more functionality out to 
individual libraries. We then also prepare/recommend a standard 
distribution that bundles many of these libraries by default (Anaconda 
style), as well as a minimal one that is a better starting point for 
low-footprint systems (Miniconda style) or embedding into other apps.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Paul Moore]

On Thu, 29 Nov 2018 at 15:52, Oleg Broytman  wrote:
>
> On Thu, Nov 29, 2018 at 09:36:51AM -0500, Benjamin Peterson 
>  wrote:
> > - stdlib modules become a permanent maintenance burden to CPython core 
> > developers.
>
>Add ditributions maintainers here.

Well, given that "you shouldn't use pip install on your
distribution-installed Python", I'm not sure that's as clear cut a
factor as it first seems. Yes, I know people should use virtual
environments, or use "--user" installs, but distributions still have a
maintenance burden providing distro-packaged modules, whether they are
in the stdlib or not. If unittest were removed from the stdlib, I'd
confidently expect that distributions would still provide a
python3-unittest package, for example...

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Oleg Broytman]

On Thu, Nov 29, 2018 at 09:36:51AM -0500, Benjamin Peterson 
 wrote:
> - stdlib modules become a permanent maintenance burden to CPython core 
> developers.

   Add ditributions maintainers here.

Oleg.
-- 
Oleg Broytmanhttps://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Paul Moore]

On Thu, 29 Nov 2018 at 14:56, Benjamin Peterson  wrote:
> While I'm sympathetic to users in such situations, I'm not sure how much we 
> can really help them. These are the sorts of users who are likely to still be 
> stuck using Python 2.6. Any stdlib improvements we discuss and implement 
> today are easily a decade away from benefiting users in restrictive 
> environments. On that kind of timescale, it's very hard to know what to do, 
> especially since, as Paul says, we don't hear much feedback from such users.

As a user in that situation, I can confirm that there *are* situations
where I am stuck using older versions of Python (2.6? Ha - luxury! I
have to use 2.4 on some of our systems!) But there are also situations
where I can do a one-off install of the latest version of Python
(typically by copying install files from one machine to another, until
I get it to a machine with no internet access) but installing
additional modules, while possible (by similar means) is too painful
to be worthwhile. Also, when sharing scripts with other users who are
able to handle "download and run the installer from python.org", but
for whom "pip install" (including the proxy configuration to let pip
see the internet, which isn't needed for the installer because our
browsers auto-configure) is impossibly hard. So "latest Python, no
PyPI access" is not an unrealistic model for me to work to.

I can't offer more than one person's feedback, but my experience is a
real-life situation. And it's one where I've been able to push Python
over other languages (such as Perl) *precisely* because the stdlib
provides a big chunk of built in functionality as part of the base
install. If we hadn't been able to use Python with its stdlib, I'd
have probably had to argue for Java (for the same "large stdlib"
reasons).

> The model these users are living in is increasingly at odds with how software 
> is written and used these days.

I'd dispute that. It's increasingly at odds with how *open source*
software, and modern web applications are written (and my experience
is just as limited in the opposite direction, so I'm sure this
statement is just as inaccurate as the one I'm disputing ;-)). But
from what I can see there's a huge base of people in "enterprise"
companies (the people who 20 years ago were still writing COBOL) who
are just starting to use "modern" tools like Python, usually in a
"grass roots" fashion, but whose corporate infrastructure hasn't got a
clue how to deal with the sort of "everything is downloaded from the
web as needed" environment that web-based development is embracing.

> Browsers and Rust are updated every month. The node.js world is a frenzy of 
> change. Are these users' environments running 5 year old browsers? I hope not 
> for security's sake.

For people in those environments, I hope not as well. For people in
locked down environments where upgrading internal applications is a
massive project, and upgrading browsers without suffcient testing
could potentially break old but key business applications, and for
whom "limit internet access" is a draconian but mostly effective
solution, slow change and limited connectivity is a practical
solution.

It frustrates me enormously, and I hate having to argue that it's the
right solution, but it's certainly *not* as utterly wrong-headed as
some people try to argue. Priorities differ.

> At some point, languorous corporate environments will have to catch up to how 
> modern software development is done to avoid seriously hampering their 
> employees' effectiveness (and security!).

And employees making a grass-roots effort to do so by gradually
introducing modern tools like Python are part of that process. Making
it harder to demonstrate benefits without needing infrastructure-level
changes is not helping them do so. It's not necessarily Python's role
to help that process, admittedly - but I personally have that goal,
and therefore I'll continue to argue that the benefits of having a
comprehensive stdlib are worth it.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steven D'Aprano]

On Thu, Nov 29, 2018 at 09:53:30AM -0500, Benjamin Peterson wrote:

[...]
> > This is not an argument either for or against adding LZ4, I have no 
> > opinion either way. But it is a reminder that "just get it from PyPI" 
> > represents an extremely privileged position that not all Python users 
> > are capable of taking, and we shouldn't be so blase about abandoning 
> > those who can't to future std lib improvements.
> 
> While I'm sympathetic to users in such situations, I'm not sure how 
> much we can really help them. These are the sorts of users who are 
> likely to still be stuck using Python 2.6.

Not necessarily. They may have approval for the latest approved vendor 
software, but not for third party packages.

Nick has been quiet lately, it would be good to hear from him, because I 
expect Red Hat's corporate userbase will be an informative example. I 
hear Red Hat has now moved to Python 3 for its system Python, so I 
expect that many of their corporate users will be, or will soon be, 
running Python 3.x too.

In any case, it's not the users stuck on 2.6 *now* that I'm so concerned 
about, since as you say there's nothing we can do for them. Its the 
users stuck on 3.9 ten years from now, who are missing out on 
functionality because we keep saying "its easy to get it from PyPI".

(To be clear, this isn't a plea to add everything to the stdlib.)


> Any stdlib improvements we 
> discuss and implement today are easily a decade away from benefiting 
> users in restrictive environments. On that kind of timescale, it's 
> very hard to know what to do, especially since, as Paul says, we don't 
> hear much feedback from such users.

Indeed, it isn't easy to know where to draw the line. That's what makes 
the PyPI argument all the more pernicious: it makes it *seem* easy, 
"just get it from PyPI", because it is easy for we privileged users who 
control the computers we use.


> The model these users are living in is increasingly at odds with how 
> software is written and used these days. Browsers and Rust are updated 
> every month. The node.js world is a frenzy of change. 

The node.js world is hardly the paragon of excellence in software 
development we ought to emulate.


> Are these users' 
> environments running 5 year old browsers? I hope not for security's 
> sake. At some point, languorous corporate environments will have to 
> catch up to how modern software development is done to avoid seriously 
> hampering their employees' effectiveness (and security!).

Arguably, modern software development will have to slow down and stop 
introducing a dozen zero-day exploits every week. (Only half tongue in 
cheek.)

Browsers are at the interface of the most hostile, exposed part of the 
computer ecosystem. Not all software operates in such an exposed 
position, and *stability* remains important for many uses and users.

I don't see that this is really relevant to Python though, not unless 
you're thinking about accelerating the pace of new releases.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Thu, 29 Nov 2018 10:05:47 -0500
"Benjamin Peterson"  wrote:
> On Thu, Nov 29, 2018, at 08:45, Antoine Pitrou wrote:
> > 
> > Le 29/11/2018 à 15:36, Benjamin Peterson a écrit :  
> > >>
> > >> I'd like to point the discussion is asymmetric here.
> > >>
> > >> On the one hand, people who don't have access to PyPI would _really_
> > >> benefit from a larger stdlib with more batteries included.
> > >>
> > >> On the other hand, people who have access to PyPI _don't_ benefit from
> > >> having a slim stdlib.  There's nothing virtuous or advantageous about
> > >> having _less_ batteries included.  Python doesn't become magically
> > >> faster or more powerful by including less in its standard
> > >> distribution: the best it does is make the distribution slightly
> > >> smaller.
> > >>
> > >> So there's really one bunch of people arguing for practical benefits,
> > >> and another bunch of people arguing for mostly aesthetical or
> > >> philosophical reasons.  
> > > 
> > > I don't think it's asymmetric. People have raised several practical 
> > > problems with a large stdlib in this thread. These include:
> > > - The evelopment of stdlib modules slows to the rate of the Python 
> > > release schedule.  
> > 
> > Can you explain why that would be the case?  As a matter of fact, the
> > Python release schedule seems to remain largely the same even though we
> > accumulate more stdlib modules and functionality.  
> 
> The problem is the length of the Python release schedule. It means, in the 
> extreme case, that stdlib modifications won't see the light of day for 2 
> years (feature freeze + 1.5 year release schedule). And that's only if people 
> update Python immediately after a release. PyPI modules can evolve much more 
> rapidly.

Ah, I realize I had misread as "the development of stdlib modules slows
the rate of the Python release schedule" (instead of "slows *to* the
rate").

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Toshio Kuratomi]

On Thu, Nov 29, 2018, 6:56 AM Benjamin Peterson 
>
> On Wed, Nov 28, 2018, at 15:27, Steven D'Aprano wrote:
> > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
> >
> > > PyPI makes getting more algorithms easy.
> >
> > Can we please stop over-generalising like this? PyPI makes getting
> > more algorithms easy for *SOME* people. (Sorry for shouting, but you
> > just pressed one of my buttons.)
> >
> > PyPI might as well not exist for those who cannot, for technical or
> > policy reasons, install addition software beyond the std lib on the
> > computers they use. (I hesitate to say "their computers".)
> >
> > In many school or corporate networks, installing unapproved software can
> > get you expelled or fired. And getting approval may be effectively
> > impossible, or take months of considerable effort navigating some
> > complex bureaucratic process.
> >
> > This is not an argument either for or against adding LZ4, I have no
> > opinion either way. But it is a reminder that "just get it from PyPI"
> > represents an extremely privileged position that not all Python users
> > are capable of taking, and we shouldn't be so blase about abandoning
> > those who can't to future std lib improvements.
>
> While I'm sympathetic to users in such situations, I'm not sure how much
> we can really help them. These are the sorts of users who are likely to
> still be stuck using Python 2.6. Any stdlib improvements we discuss and
> implement today are easily a decade away from benefiting users in
> restrictive environments. On that kind of timescale, it's very hard to know
> what to do, especially since, as Paul says, we don't hear much feedback
> from such users.
>

As a developer of software that has to run in such environments, having a
library be in the stdlib is helpful as it is easier to convince the rest of
the team to bundle a backport of something that's in a future stdlib than a
random package from pypi.  Stdlib inclusion gives the library a known
future and a (perhaps illusory, perhaps real) blessing from the core devs
that helps to sell the library as the preferred solution.

-Toshio

>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steven D'Aprano]

On Thu, Nov 29, 2018 at 09:36:51AM -0500, Benjamin Peterson wrote:

> I don't think it's asymmetric. People have raised several practical 
> problems with a large stdlib in this thread. These include:
>
> - The evelopment of stdlib modules slows to the rate of the Python 
> release schedule.

That's not a bug, that's a feature :-)

Of course that's a concern for rapidly changing libraries, but they 
won't be considered for the stdlib because they are rapidly changing.


> - stdlib modules become a permanent maintenance burden to CPython core 
> developers.

That's a concern, of course. Every proposed library needs to convince 
that the potential benefit outweighs the potential costs.

On the other hand mature, stable software can survive with little or 
no maintenance for a very long time. The burden is not necessarily high.


> - The blessed status of stdlib modules means that users might use a 
> substandard stdlib modules when a better thirdparty alternative 
> exists.

Or they might re-invent the wheel and write something worse than either. 

I don't think it is productive to try to guess what users will do and 
protect them from making the "wrong" choice. Wrong choice according to 
whose cost-benefit analysis?


-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Benjamin Peterson]

On Thu, Nov 29, 2018, at 08:45, Antoine Pitrou wrote:
> 
> Le 29/11/2018 à 15:36, Benjamin Peterson a écrit :
> >>
> >> I'd like to point the discussion is asymmetric here.
> >>
> >> On the one hand, people who don't have access to PyPI would _really_
> >> benefit from a larger stdlib with more batteries included.
> >>
> >> On the other hand, people who have access to PyPI _don't_ benefit from
> >> having a slim stdlib.  There's nothing virtuous or advantageous about
> >> having _less_ batteries included.  Python doesn't become magically
> >> faster or more powerful by including less in its standard
> >> distribution: the best it does is make the distribution slightly
> >> smaller.
> >>
> >> So there's really one bunch of people arguing for practical benefits,
> >> and another bunch of people arguing for mostly aesthetical or
> >> philosophical reasons.
> > 
> > I don't think it's asymmetric. People have raised several practical 
> > problems with a large stdlib in this thread. These include:
> > - The evelopment of stdlib modules slows to the rate of the Python release 
> > schedule.
> 
> Can you explain why that would be the case?  As a matter of fact, the
> Python release schedule seems to remain largely the same even though we
> accumulate more stdlib modules and functionality.

The problem is the length of the Python release schedule. It means, in the 
extreme case, that stdlib modifications won't see the light of day for 2 years 
(feature freeze + 1.5 year release schedule). And that's only if people update 
Python immediately after a release. PyPI modules can evolve much more rapidly.

CPython releases come in one big chunk. Even if you want to use improvements to 
one stdlib module in a new Python version, you may be hampered by breakages in 
your code from language changes or changes in a different stdlib module. 
Modules on PyPI, being decoupled, don't have this problem.

> 
> The only risk is the prematurate inclusion of an unstable or ill-written
> module which may lengthen the stabilization cycle.
> 
> > - stdlib modules become a permanent maintenance burden to CPython core 
> > developers.
> 
> This is true.  We should only accept modules that we think are useful
> for a reasonable part of the user base.

We agree with in the ultimate conclusion then.

> 
> > - The blessed status of stdlib modules means that users might use a 
> > substandard stdlib modules when a better thirdparty alternative exists.
> 
> We can always mention the better thirdparty alternative in the docs
> where desired.  But there are many cases where the stdlib module is good
> enough for what people use it.
> 
> For example, the upstream simplejson module may have slightly more
> advanced functionality, but for most purposes the stdlib json module
> fits the bill and spares you from tracking a separate dependency.

Yes, I think json was a fine addition to the stdlib. However, we also have 
stdlib modules where there is a universally better alternative. For example, 
requests is better than urllib.* both in simple and complex cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Benjamin Peterson]

On Wed, Nov 28, 2018, at 15:27, Steven D'Aprano wrote:
> On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
> 
> > PyPI makes getting more algorithms easy.
> 
> Can we please stop over-generalising like this? PyPI makes getting 
> more algorithms easy for *SOME* people. (Sorry for shouting, but you 
> just pressed one of my buttons.)
> 
> PyPI might as well not exist for those who cannot, for technical or 
> policy reasons, install addition software beyond the std lib on the 
> computers they use. (I hesitate to say "their computers".)
> 
> In many school or corporate networks, installing unapproved software can 
> get you expelled or fired. And getting approval may be effectively 
> impossible, or take months of considerable effort navigating some 
> complex bureaucratic process.
> 
> This is not an argument either for or against adding LZ4, I have no 
> opinion either way. But it is a reminder that "just get it from PyPI" 
> represents an extremely privileged position that not all Python users 
> are capable of taking, and we shouldn't be so blase about abandoning 
> those who can't to future std lib improvements.

While I'm sympathetic to users in such situations, I'm not sure how much we can 
really help them. These are the sorts of users who are likely to still be stuck 
using Python 2.6. Any stdlib improvements we discuss and implement today are 
easily a decade away from benefiting users in restrictive environments. On that 
kind of timescale, it's very hard to know what to do, especially since, as Paul 
says, we don't hear much feedback from such users.

The model these users are living in is increasingly at odds with how software 
is written and used these days. Browsers and Rust are updated every month. The 
node.js world is a frenzy of change. Are these users' environments running 5 
year old browsers? I hope not for security's sake. At some point, languorous 
corporate environments will have to catch up to how modern software development 
is done to avoid seriously hampering their employees' effectiveness (and 
security!).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

Le 29/11/2018 à 15:36, Benjamin Peterson a écrit :
>>
>> I'd like to point the discussion is asymmetric here.
>>
>> On the one hand, people who don't have access to PyPI would _really_
>> benefit from a larger stdlib with more batteries included.
>>
>> On the other hand, people who have access to PyPI _don't_ benefit from
>> having a slim stdlib.  There's nothing virtuous or advantageous about
>> having _less_ batteries included.  Python doesn't become magically
>> faster or more powerful by including less in its standard
>> distribution: the best it does is make the distribution slightly
>> smaller.
>>
>> So there's really one bunch of people arguing for practical benefits,
>> and another bunch of people arguing for mostly aesthetical or
>> philosophical reasons.
> 
> I don't think it's asymmetric. People have raised several practical problems 
> with a large stdlib in this thread. These include:
> - The evelopment of stdlib modules slows to the rate of the Python release 
> schedule.

Can you explain why that would be the case?  As a matter of fact, the
Python release schedule seems to remain largely the same even though we
accumulate more stdlib modules and functionality.

The only risk is the prematurate inclusion of an unstable or ill-written
module which may lengthen the stabilization cycle.

> - stdlib modules become a permanent maintenance burden to CPython core 
> developers.

This is true.  We should only accept modules that we think are useful
for a reasonable part of the user base.

> - The blessed status of stdlib modules means that users might use a 
> substandard stdlib modules when a better thirdparty alternative exists.

We can always mention the better thirdparty alternative in the docs
where desired.  But there are many cases where the stdlib module is good
enough for what people use it.

For example, the upstream simplejson module may have slightly more
advanced functionality, but for most purposes the stdlib json module
fits the bill and spares you from tracking a separate dependency.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Benjamin Peterson]

On Thu, Nov 29, 2018, at 04:54, Antoine Pitrou wrote:
> On Thu, 29 Nov 2018 09:52:29 +
> Paul Moore  wrote:
> > [This is getting off-topic, so I'll limit my comments to this one email]
> > 
> > On Thu, 29 Nov 2018 at 03:17, Brett Cannon  wrote:
> > > We have never really had a discussion about how we want to guide the 
> > > stdlib going forward (e.g. how much does PyPI influence things, 
> > > focus/theme, etc.). Maybe we should consider finally having that 
> > > discussion once the governance model is chosen and before we consider 
> > > adding a new module as things like people's inability to access PyPI come 
> > > up pretty consistently (e.g. I know Paul Moore also brings this up 
> > > regularly).  
> > 
> > I'm not sure a formal discussion on this matter will help much - my
> > feeling is that most people have relatively fixed views on how they
> > would like things to go (large stdlib/batteries included vs external
> > modules/PyPI/slim stdlib). The "problem" isn't so much with people
> > having different views (as a group, we're pretty good at achieving
> > workable compromises in the face of differing views) as it is about
> > people forgetting that their experience isn't the only reality, which
> > causes unnecessary frustration in discussions. That's more of a people
> > problem than a technical one.
> 
> I'd like to point the discussion is asymmetric here.
> 
> On the one hand, people who don't have access to PyPI would _really_
> benefit from a larger stdlib with more batteries included.
> 
> On the other hand, people who have access to PyPI _don't_ benefit from
> having a slim stdlib.  There's nothing virtuous or advantageous about
> having _less_ batteries included.  Python doesn't become magically
> faster or more powerful by including less in its standard
> distribution: the best it does is make the distribution slightly
> smaller.
> 
> So there's really one bunch of people arguing for practical benefits,
> and another bunch of people arguing for mostly aesthetical or
> philosophical reasons.

I don't think it's asymmetric. People have raised several practical problems 
with a large stdlib in this thread. These include:
- The evelopment of stdlib modules slows to the rate of the Python release 
schedule.
- stdlib modules become a permanent maintenance burden to CPython core 
developers.
- The blessed status of stdlib modules means that users might use a substandard 
stdlib modules when a better thirdparty alternative exists.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Jonathan Underwood]

On Thu, 29 Nov 2018 at 11:00, Andrew Svetlov  wrote:
> I worked with lz4 python binding a year ago.
> It sometimes crashed to core dump when used in multithreaded environment (we 
> used to run compressor/decompresson with asyncio by loop.run_in_executor() 
> call).
> I hope the bug is fixed now, have no update for the current state.

Andrew and I discussed this off-list. The upshot is that this was
happening in a situation where a (de)compression context was being
used simultaneously by multiple threads, which is not supported. I'll
look at making this use case not crash, though.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Thu, 29 Nov 2018 16:02:29 +1100
Steven D'Aprano  wrote:
> On Wed, Nov 28, 2018 at 07:14:03PM -0800, Brett Cannon wrote:
> > On Wed, 28 Nov 2018 at 13:29, Steven D'Aprano  wrote:
> >   
> > > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
> > >  
> > > > PyPI makes getting more algorithms easy.  
> > >
> > > Can we please stop over-generalising like this? PyPI makes getting
> > > more algorithms easy for *SOME* people. (Sorry for shouting, but you
> > > just pressed one of my buttons.)  
> > 
> > Is shouting necessary to begin with, though?  
> 
> Yes.
> 
> My apology was a polite fiction, a left over from the old Victorian 
> British "stiff upper lip" attitude that showing emotion in public is Not 
> The Done Thing. I should stop making those faux apologies, it is a bad 
> habit.
> 
> We aren't robots, we're human beings and we shouldn't apologise for 
> showing our emotions. Nothing important ever got done without people 
> having, and showing, strong emotions either for or against it.

+1.  Not that this is very important to discuss, but I don't think
putting a single word in caps is offensive or agressive behavior.
Let's cut people some slack :-)

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Andrew Svetlov]

5 cents about lz4 alternatives: Broli (mentioned above) is widely supported
by web.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
mentions it along with gzip and deflate methods.
I don't recall lz4 or Zstd metioning in this context.

Both Chrome/Chromium and Firefox accepts it by default (didn't check
Microsoft products yet).

P.S.
I worked with lz4 python binding a year ago.
It sometimes crashed to core dump when used in multithreaded environment
(we used to run compressor/decompresson with asyncio by
loop.run_in_executor() call).
I hope the bug is fixed now, have no update for the current state.

On Thu, Nov 29, 2018 at 12:04 PM INADA Naoki  wrote:

> On Thu, Nov 29, 2018 at 6:27 AM Steven D'Aprano 
> wrote:
> >
> > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
> >
> > > PyPI makes getting more algorithms easy.
> >
> > Can we please stop over-generalising like this? PyPI makes getting
> > more algorithms easy for *SOME* people. (Sorry for shouting, but you
> > just pressed one of my buttons.)
>
> I don't think this is over-generalising.
>
> If "get it from PyPI" is not easy enough, why not adding hundreds of
> famous libraries?
> Because we can't maintain all of them well.
>
> When considering adding new format (not only compression, but also
> serialization like toml), I think it should be stable, widely used, and
> will
> be used widely for a long time.  If we want to use the format in Python
> core
> or Python stdlib, it's good reasoning too.  gzip and json are good example.
>
> When we say "we can use PyPI", it means "are there enough reasons
> make the package special enough to add to stdlib?"  We don't mean
> "everyone can use PyPI."
>
> Regards,
> --
> INADA Naoki  
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>


-- 
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Thu, 29 Nov 2018 09:52:29 +
Paul Moore  wrote:
> [This is getting off-topic, so I'll limit my comments to this one email]
> 
> On Thu, 29 Nov 2018 at 03:17, Brett Cannon  wrote:
> > We have never really had a discussion about how we want to guide the stdlib 
> > going forward (e.g. how much does PyPI influence things, focus/theme, 
> > etc.). Maybe we should consider finally having that discussion once the 
> > governance model is chosen and before we consider adding a new module as 
> > things like people's inability to access PyPI come up pretty consistently 
> > (e.g. I know Paul Moore also brings this up regularly).  
> 
> I'm not sure a formal discussion on this matter will help much - my
> feeling is that most people have relatively fixed views on how they
> would like things to go (large stdlib/batteries included vs external
> modules/PyPI/slim stdlib). The "problem" isn't so much with people
> having different views (as a group, we're pretty good at achieving
> workable compromises in the face of differing views) as it is about
> people forgetting that their experience isn't the only reality, which
> causes unnecessary frustration in discussions. That's more of a people
> problem than a technical one.

I'd like to point the discussion is asymmetric here.

On the one hand, people who don't have access to PyPI would _really_
benefit from a larger stdlib with more batteries included.

On the other hand, people who have access to PyPI _don't_ benefit from
having a slim stdlib.  There's nothing virtuous or advantageous about
having _less_ batteries included.  Python doesn't become magically
faster or more powerful by including less in its standard
distribution: the best it does is make the distribution slightly
smaller.

So there's really one bunch of people arguing for practical benefits,
and another bunch of people arguing for mostly aesthetical or
philosophical reasons.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[INADA Naoki]

On Thu, Nov 29, 2018 at 6:27 AM Steven D'Aprano  wrote:
>
> On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
>
> > PyPI makes getting more algorithms easy.
>
> Can we please stop over-generalising like this? PyPI makes getting
> more algorithms easy for *SOME* people. (Sorry for shouting, but you
> just pressed one of my buttons.)

I don't think this is over-generalising.

If "get it from PyPI" is not easy enough, why not adding hundreds of
famous libraries?
Because we can't maintain all of them well.

When considering adding new format (not only compression, but also
serialization like toml), I think it should be stable, widely used, and will
be used widely for a long time.  If we want to use the format in Python core
or Python stdlib, it's good reasoning too.  gzip and json are good example.

When we say "we can use PyPI", it means "are there enough reasons
make the package special enough to add to stdlib?"  We don't mean
"everyone can use PyPI."

Regards,
-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Paul Moore]

[This is getting off-topic, so I'll limit my comments to this one email]

On Thu, 29 Nov 2018 at 03:17, Brett Cannon  wrote:
> We have never really had a discussion about how we want to guide the stdlib 
> going forward (e.g. how much does PyPI influence things, focus/theme, etc.). 
> Maybe we should consider finally having that discussion once the governance 
> model is chosen and before we consider adding a new module as things like 
> people's inability to access PyPI come up pretty consistently (e.g. I know 
> Paul Moore also brings this up regularly).

I'm not sure a formal discussion on this matter will help much - my
feeling is that most people have relatively fixed views on how they
would like things to go (large stdlib/batteries included vs external
modules/PyPI/slim stdlib). The "problem" isn't so much with people
having different views (as a group, we're pretty good at achieving
workable compromises in the face of differing views) as it is about
people forgetting that their experience isn't the only reality, which
causes unnecessary frustration in discussions. That's more of a people
problem than a technical one.

What would be nice would be if we could persuade people *not* to
assume "adding an external dependency is easy" in all cases, and frame
their arguments in a way that doesn't make that mistaken assumption.
The arguments/debates are fine, what's not so fine is having to
repeatedly explain how people are making fundamentally unsound
assumptions. Having said that, what *is* difficult to know, is how
*many* people are in situations where adding an external dependency is
hard, and how hard it is in practice. The sorts of environment where
PyPI access is hard are also the sorts of environment where
participation in open source discussions is rare, so getting good
information is hard.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Jonathan Underwood]

On Thu, 29 Nov 2018 at 09:13, Gregory P. Smith  wrote:
> Q: Are there other popular alternatives to fill that niche that we should 
> strongly consider instead or as well?
>
> 5 years ago the answer would've been Snappy.  15 years ago the answer 
> would've been LZO.
>

Today LZ4 hits a sweet spot for fast compression and decompression at
the lower compression ratio end of the spectrum, offering
significantly faster compression and decompression than zlib or bz2,
but not as high compression ratios (at usable speeds). It's also had
time to stabilize, and a standard frame format for compressed data has
been adopted by the community.

The other main contenders in town are zstd, which was mentioned
earlier in the thread, and brotli. Both are based on dictionary
compression. Zstd is very impressive, offering high compression
ratios, but is being very actively developed at present, so is a bit
more of a moving target.Brotli is in the same ballpark as Zstd. They
both cover the higher compression end of the spectrum than lz4. Some
nice visualizations are here (although the data is now a bit out of
date - lz4 has had some speed improvements at the higher compression
ratio end):

https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/

> I suggest not rabbit-holing this on whether we should adopt a top level 
> namespace for these such as "compress".  A good question to ask, but we can 
> resolve that larger topic on its own without blocking anything.
>

It's funny, but I had gone around in that loop in my head ahead of
sending my email. My thinking was: there's a real need for some
unification and simplification in the compression space, but I'll work
on integrating LZ4, and in the process look at opportunities for the
new interface design. I'm a fan of learning through iteration, rather
than spending 5 years designing the ultimate compression abstraction
and then finding a corner case that it doesn't fit.

> lz4 has claimed the global pypi lz4 module namespace today so moving it to 
> the stdlib under that name is normal - A pretty transparent transition.  If 
> we do that, the PyPI version of lz4 should remain for use on older CPython 
> versions, but effectively be frozen, never to gain new features once lz4 has 
> landed in its first actual CPython release.
>

Yes, that was what I was presuming would be the path forward.

Cheers,
Jonathan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Gregory P. Smith]

On Wed, Nov 28, 2018 at 10:43 AM Gregory P. Smith  wrote:

>
> On Wed, Nov 28, 2018 at 9:52 AM Brett Cannon  wrote:
>
>> Are we getting to the point that we want a compresslib like hashlib if we
>> are going to be adding more compression algorithms?
>>
>
> Lets avoid the lib suffix when unnecessary.  I used the name hashlib
> because the name hash was already taken by a builtin that people normally
> shouldn't be using.  zlib gets a lib suffix because a one letter name is
> evil and it matches the project name. ;)  "compress" sounds nicer.
>
> ... looking on PyPI to see if that name is taken:
> https://pypi.org/project/compress/ exists and is already effectively what
> you are describing.  (never used it or seen it used, no idea about quality)
>
> I don't think adding lz4 to the stdlib is worthwhile.  It isn't required
> for core functionality as zlib is (lowest common denominator zip support).
> I'd argue that bz2 doesn't even belong in the stdlib, but we shouldn't go
> removing things.  PyPI makes getting more algorithms easy.
>
> If anything, it'd be nice to standardize on some stdlib namespaces that
> others could plug their modules into.  Create a compress in the stdlib with
> zlib and bz2 in it, and a way for extension modules to add themselves in a
> managed manner instead of requiring a top level name?  Opening up a
> designated namespace to third party modules is not something we've done as
> a project in the past though.  It requires care.  I haven't thought that
> through.
>
> -gps
>

While my gut reaction was to say "no" to adding lz4 to the stdlib above...

I'm finding myself reconsidering and not against adding lz4 to the stdlib.

I just want us to have a good reason if we do. This type of extension
module tends to be very easy to maintain (and you are volunteering). A good
reason in the past has been the algorithm being widely used.  Obviously the
case with zlib (gzip and zipfile), bz2, and lzma (.xz).  Those are all
slower and tighter though.  lz4 is extremely fast, especially for
decompression.  It could make a nice addition as that is an area our
standard library offers nothing.

So change my -1 to a +0.5.

Q: Are there other popular alternatives to fill that niche that we should
strongly consider instead or as well?

5 years ago the answer would've been Snappy.  15 years ago the answer
would've been LZO.

I suggest not rabbit-holing this on whether we should adopt a top level
namespace for these such as "compress".  A good question to ask, but we can
resolve that larger topic on its own without blocking anything.

lz4 has claimed the global pypi lz4 module namespace today so moving it to
the stdlib under that name is normal - A pretty transparent transition.  If
we do that, the PyPI version of lz4 should remain for use on older CPython
versions, but effectively be frozen, never to gain new features once lz4
has landed in its first actual CPython release.

-gps


>
>>
>> On Wed, 28 Nov 2018 at 08:44, Antoine Pitrou  wrote:
>>
>>> On Wed, 28 Nov 2018 10:28:19 +
>>> Jonathan Underwood  wrote:
>>> > Hi,
>>> >
>>> > I have for sometime maintained the Python bindings to the LZ4
>>> > compression library[0, 1]:
>>> >
>>> > I am wondering if there is interest in having these bindings move to
>>> > the standard library to sit alongside the gzip, lzma etc bindings?
>>> > Obviously the code would need to be modified to fit the coding
>>> > guidelines etc.
>>>
>>> Personally I would find it useful indeed.  LZ4 is very attractive
>>> when (de)compression speed is a primary factor, for example when
>>> sending data over a fast network link or a fast local SSD.
>>>
>>> Another compressor worth including is Zstandard (by the same author as
>>> LZ4). Actually, Zstandard and LZ4 cover most of the (speed /
>>> compression ratio) range quite well. Informative graphs below:
>>>
>>> https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/
>>>
>>> Regards
>>>
>>> Antoine.
>>>
>>>
>>> ___
>>> Python-Dev mailing list
>>> Python-Dev@python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steven D'Aprano]

On Wed, Nov 28, 2018 at 07:14:03PM -0800, Brett Cannon wrote:
> On Wed, 28 Nov 2018 at 13:29, Steven D'Aprano  wrote:
> 
> > On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
> >
> > > PyPI makes getting more algorithms easy.
> >
> > Can we please stop over-generalising like this? PyPI makes getting
> > more algorithms easy for *SOME* people. (Sorry for shouting, but you
> > just pressed one of my buttons.)
> 
> Is shouting necessary to begin with, though?

Yes.

My apology was a polite fiction, a left over from the old Victorian 
British "stiff upper lip" attitude that showing emotion in public is Not 
The Done Thing. I should stop making those faux apologies, it is a bad 
habit.

We aren't robots, we're human beings and we shouldn't apologise for 
showing our emotions. Nothing important ever got done without people 
having, and showing, strong emotions either for or against it.

Of course I'm not genuinely sorry for showing my strength of feeling 
over this issue. Its just a figure of speech: all-caps is used to give 
emphasis in a plain text medium, it is not literal shouting.

In any case, I retract the faux apology, it was a silly thing for me to 
say that undermines my own message as well as reinforcing the pernicious 
message that expressing the strength of emotional feeling about an issue 
is a bad thing that needs to be surpressed.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Brett Cannon]

On Wed, 28 Nov 2018 at 13:29, Steven D'Aprano  wrote:

> On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:
>
> > PyPI makes getting more algorithms easy.
>
> Can we please stop over-generalising like this? PyPI makes getting
> more algorithms easy for *SOME* people. (Sorry for shouting, but you
> just pressed one of my buttons.)
>

Is shouting necessary to begin with, though? I understand people relying on
PyPI more and more can be troublesome for some and a sticking point, but if
you know it's a trigger for you then waiting until you didn't feel like
shouting seems like a reasonable course of action while still getting your
point across.


>
> PyPI might as well not exist for those who cannot, for technical or
> policy reasons, install addition software beyond the std lib on the
> computers they use. (I hesitate to say "their computers".)
>
> In many school or corporate networks, installing unapproved software can
> get you expelled or fired. And getting approval may be effectively
> impossible, or take months of considerable effort navigating some
> complex bureaucratic process.
>
> This is not an argument either for or against adding LZ4, I have no
> opinion either way.



> But it is a reminder that "just get it from PyPI"
> represents an extremely privileged position that not all Python users
> are capable of taking, and we shouldn't be so blase about abandoning
> those who can't to future std lib improvements.
>

We have never really had a discussion about how we want to guide the stdlib
going forward (e.g. how much does PyPI influence things, focus/theme,
etc.). Maybe we should consider finally having that discussion once the
governance model is chosen and before we consider adding a new module as
things like people's inability to access PyPI come up pretty consistently
(e.g. I know Paul Moore also brings this up regularly).

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Nathaniel Smith]

On Wed, Nov 28, 2018, 12:08 Antoine Pitrou  On Wed, 28 Nov 2018 19:35:31 +
> Jonathan Underwood  wrote:
> > On Wed, 28 Nov 2018 at 18:57, Antoine Pitrou 
> wrote:
> > >
> > > On Wed, 28 Nov 2018 10:43:04 -0800
> > > "Gregory P. Smith"  wrote:
> > [snip]
> > > > I don't think adding lz4 to the stdlib is worthwhile.  It isn't
> required
> > > > for core functionality as zlib is (lowest common denominator zip
> support).
> > >
> > > Actually, if some people are interested in compressing .pyc files, lz4
> > > is probably the best candidate (will yield significant compression
> > > benefits with very little CPU overhead).
> > >
> >
> > It's interesting to note that there's an outstanding feature request
> > to enable "import modules from a library.tar.lz4", justified on the
> > basis that it would be helpful to the python-for-android project:
> >
> > https://github.com/python-lz4/python-lz4/issues/45
>
> Interesting.  The tar format isn't adequate for this: the whole tar
> file will be compressed at once, so you need to uncompress it all even
> to import a single module.  The zip format is more adapted, but it
> doesn't seem to have LZ4 in its registered codecs.
>

Zip can be used without compression as a simple container for files,
though, and there's nothing that says those can't be .pyc.lz4 files.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Steven D'Aprano]

On Wed, Nov 28, 2018 at 10:43:04AM -0800, Gregory P. Smith wrote:

> PyPI makes getting more algorithms easy.

Can we please stop over-generalising like this? PyPI makes getting 
more algorithms easy for *SOME* people. (Sorry for shouting, but you 
just pressed one of my buttons.)

PyPI might as well not exist for those who cannot, for technical or 
policy reasons, install addition software beyond the std lib on the 
computers they use. (I hesitate to say "their computers".)

In many school or corporate networks, installing unapproved software can 
get you expelled or fired. And getting approval may be effectively 
impossible, or take months of considerable effort navigating some 
complex bureaucratic process.

This is not an argument either for or against adding LZ4, I have no 
opinion either way. But it is a reminder that "just get it from PyPI" 
represents an extremely privileged position that not all Python users 
are capable of taking, and we shouldn't be so blase about abandoning 
those who can't to future std lib improvements.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Wed, 28 Nov 2018 19:35:31 +
Jonathan Underwood  wrote:
> On Wed, 28 Nov 2018 at 18:57, Antoine Pitrou  wrote:
> >
> > On Wed, 28 Nov 2018 10:43:04 -0800
> > "Gregory P. Smith"  wrote:  
> [snip]
> > > I don't think adding lz4 to the stdlib is worthwhile.  It isn't required
> > > for core functionality as zlib is (lowest common denominator zip 
> > > support).  
> >
> > Actually, if some people are interested in compressing .pyc files, lz4
> > is probably the best candidate (will yield significant compression
> > benefits with very little CPU overhead).
> >  
> 
> It's interesting to note that there's an outstanding feature request
> to enable "import modules from a library.tar.lz4", justified on the
> basis that it would be helpful to the python-for-android project:
> 
> https://github.com/python-lz4/python-lz4/issues/45

Interesting.  The tar format isn't adequate for this: the whole tar
file will be compressed at once, so you need to uncompress it all even
to import a single module.  The zip format is more adapted, but it
doesn't seem to have LZ4 in its registered codecs.

At least for pyc files, though, this could be done at the marshal level
rather than at the importlib level.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Jonathan Underwood]

On Wed, 28 Nov 2018 at 18:57, Antoine Pitrou  wrote:
>
> On Wed, 28 Nov 2018 10:43:04 -0800
> "Gregory P. Smith"  wrote:
[snip]
> > I don't think adding lz4 to the stdlib is worthwhile.  It isn't required
> > for core functionality as zlib is (lowest common denominator zip support).
>
> Actually, if some people are interested in compressing .pyc files, lz4
> is probably the best candidate (will yield significant compression
> benefits with very little CPU overhead).
>

It's interesting to note that there's an outstanding feature request
to enable "import modules from a library.tar.lz4", justified on the
basis that it would be helpful to the python-for-android project:

https://github.com/python-lz4/python-lz4/issues/45

Cheers,
Jonathan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Wed, 28 Nov 2018 10:43:04 -0800
"Gregory P. Smith"  wrote:
> On Wed, Nov 28, 2018 at 9:52 AM Brett Cannon  wrote:
> 
> > Are we getting to the point that we want a compresslib like hashlib if we
> > are going to be adding more compression algorithms?
> >  
> 
> Lets avoid the lib suffix when unnecessary.  I used the name hashlib
> because the name hash was already taken by a builtin that people normally
> shouldn't be using.  zlib gets a lib suffix because a one letter name is
> evil and it matches the project name. ;)  "compress" sounds nicer.
> 
> ... looking on PyPI to see if that name is taken:
> https://pypi.org/project/compress/ exists and is already effectively what
> you are describing.  (never used it or seen it used, no idea about quality)
> 
> I don't think adding lz4 to the stdlib is worthwhile.  It isn't required
> for core functionality as zlib is (lowest common denominator zip support).

Actually, if some people are interested in compressing .pyc files, lz4
is probably the best candidate (will yield significant compression
benefits with very little CPU overhead).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Gregory P. Smith]

On Wed, Nov 28, 2018 at 9:52 AM Brett Cannon  wrote:

> Are we getting to the point that we want a compresslib like hashlib if we
> are going to be adding more compression algorithms?
>

Lets avoid the lib suffix when unnecessary.  I used the name hashlib
because the name hash was already taken by a builtin that people normally
shouldn't be using.  zlib gets a lib suffix because a one letter name is
evil and it matches the project name. ;)  "compress" sounds nicer.

... looking on PyPI to see if that name is taken:
https://pypi.org/project/compress/ exists and is already effectively what
you are describing.  (never used it or seen it used, no idea about quality)

I don't think adding lz4 to the stdlib is worthwhile.  It isn't required
for core functionality as zlib is (lowest common denominator zip support).
I'd argue that bz2 doesn't even belong in the stdlib, but we shouldn't go
removing things.  PyPI makes getting more algorithms easy.

If anything, it'd be nice to standardize on some stdlib namespaces that
others could plug their modules into.  Create a compress in the stdlib with
zlib and bz2 in it, and a way for extension modules to add themselves in a
managed manner instead of requiring a top level name?  Opening up a
designated namespace to third party modules is not something we've done as
a project in the past though.  It requires care.  I haven't thought that
through.

-gps


>
> On Wed, 28 Nov 2018 at 08:44, Antoine Pitrou  wrote:
>
>> On Wed, 28 Nov 2018 10:28:19 +
>> Jonathan Underwood  wrote:
>> > Hi,
>> >
>> > I have for sometime maintained the Python bindings to the LZ4
>> > compression library[0, 1]:
>> >
>> > I am wondering if there is interest in having these bindings move to
>> > the standard library to sit alongside the gzip, lzma etc bindings?
>> > Obviously the code would need to be modified to fit the coding
>> > guidelines etc.
>>
>> Personally I would find it useful indeed.  LZ4 is very attractive
>> when (de)compression speed is a primary factor, for example when
>> sending data over a fast network link or a fast local SSD.
>>
>> Another compressor worth including is Zstandard (by the same author as
>> LZ4). Actually, Zstandard and LZ4 cover most of the (speed /
>> compression ratio) range quite well. Informative graphs below:
>>
>> https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/
>>
>> Regards
>>
>> Antoine.
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Wed, 28 Nov 2018 09:51:57 -0800
Brett Cannon  wrote:
> Are we getting to the point that we want a compresslib like hashlib if we
> are going to be adding more compression algorithms?

It may be useful as a generic abstraction wrapper for simple usage but
some compression libraries have custom facilities that would still
require a dedicated interface.

For example, LZ4 has two formats: a raw format and a framed format.
Zstandard allows you to pass a custom dictionary to optimize
compression of small data. I believe lzma has many tunables.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[David Mertz]

+1 to Brett's idea. It's hard to have a good mental model already of which
compression algorithms are and are not in stdlib. A package to contain them
all would help a lot.

On Wed, Nov 28, 2018, 12:56 PM Brett Cannon  Are we getting to the point that we want a compresslib like hashlib if we
> are going to be adding more compression algorithms?
>
> On Wed, 28 Nov 2018 at 08:44, Antoine Pitrou  wrote:
>
>> On Wed, 28 Nov 2018 10:28:19 +
>> Jonathan Underwood  wrote:
>> > Hi,
>> >
>> > I have for sometime maintained the Python bindings to the LZ4
>> > compression library[0, 1]:
>> >
>> > I am wondering if there is interest in having these bindings move to
>> > the standard library to sit alongside the gzip, lzma etc bindings?
>> > Obviously the code would need to be modified to fit the coding
>> > guidelines etc.
>>
>> Personally I would find it useful indeed.  LZ4 is very attractive
>> when (de)compression speed is a primary factor, for example when
>> sending data over a fast network link or a fast local SSD.
>>
>> Another compressor worth including is Zstandard (by the same author as
>> LZ4). Actually, Zstandard and LZ4 cover most of the (speed /
>> compression ratio) range quite well. Informative graphs below:
>>
>> https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/
>>
>> Regards
>>
>> Antoine.
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Brett Cannon]

Are we getting to the point that we want a compresslib like hashlib if we
are going to be adding more compression algorithms?

On Wed, 28 Nov 2018 at 08:44, Antoine Pitrou  wrote:

> On Wed, 28 Nov 2018 10:28:19 +
> Jonathan Underwood  wrote:
> > Hi,
> >
> > I have for sometime maintained the Python bindings to the LZ4
> > compression library[0, 1]:
> >
> > I am wondering if there is interest in having these bindings move to
> > the standard library to sit alongside the gzip, lzma etc bindings?
> > Obviously the code would need to be modified to fit the coding
> > guidelines etc.
>
> Personally I would find it useful indeed.  LZ4 is very attractive
> when (de)compression speed is a primary factor, for example when
> sending data over a fast network link or a fast local SSD.
>
> Another compressor worth including is Zstandard (by the same author as
> LZ4). Actually, Zstandard and LZ4 cover most of the (speed /
> compression ratio) range quite well. Informative graphs below:
> https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Inclusion of lz4 bindings in stdlib?

[Antoine Pitrou]

On Wed, 28 Nov 2018 10:28:19 +
Jonathan Underwood  wrote:
> Hi,
> 
> I have for sometime maintained the Python bindings to the LZ4
> compression library[0, 1]:
> 
> I am wondering if there is interest in having these bindings move to
> the standard library to sit alongside the gzip, lzma etc bindings?
> Obviously the code would need to be modified to fit the coding
> guidelines etc.

Personally I would find it useful indeed.  LZ4 is very attractive
when (de)compression speed is a primary factor, for example when
sending data over a fast network link or a fast local SSD.

Another compressor worth including is Zstandard (by the same author as
LZ4). Actually, Zstandard and LZ4 cover most of the (speed /
compression ratio) range quite well. Informative graphs below:
https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com