Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

+1 for not adding in-pickle compression as it is already very easy to handle compression externally (for instance by passing a compressing file object as an argument to the pickler). Furthermore, as PEP 574 makes it possible to stream the buffer bytes directly to the file-object without any

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

Hi all, I agree that compression is often a good idea when moving serialized objects around on a network, but for what it's worth I as a library author would always set compress=False and then handle it myself as a separate step. There are a few reasons for this: 1. Bandwidth is often pretty

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

Antoine Pitrou schrieb am 25.05.2018 um 23:11: > On Fri, 25 May 2018 14:50:57 -0600 > Neil Schemenauer wrote: >> On 2018-05-25, Antoine Pitrou wrote: >>> Do you have something specific in mind? >> >> I think compressed by default is a good idea. My quick proposal: >> >> - Use fast compression

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On Fri, May 25, 2018 at 3:35 PM, Neil Schemenauer wrote: > This discussion can easily lead into bikeshedding (e.g. relative > merits of different compression schemes). Since I'm not > volunteering to implement anything, I will stop responding at this > point. ;-) I think

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On 2018-05-25, Antoine Pitrou wrote: > The question is what purpose does it serve for pickle to do it rather > than for the user to compress the pickle themselves. You're basically > saving one line of code. It's one line of code everywhere pickling or unpicking happens. And you probably need

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On Fri, 25 May 2018 14:50:57 -0600 Neil Schemenauer wrote: > On 2018-05-25, Antoine Pitrou wrote: > > Do you have something specific in mind? > > I think compressed by default is a good idea. My quick proposal: > > - Use fast compression like lz4 or zlib with

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On 2018-05-25, Antoine Pitrou wrote: > Do you have something specific in mind? I think compressed by default is a good idea. My quick proposal: - Use fast compression like lz4 or zlib with Z_BEST_SPEED - Add a 'compress' keyword argument with a default of None. For protocol 5, None means to

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On 25.05.2018 20:36, Raymond Hettinger wrote: On May 24, 2018, at 10:57 AM, Antoine Pitrou wrote: While PEP 574 (pickle protocol 5 with out-of-band data) is still in draft status, I've made available an implementation in branch "pickle5" in my GitHub fork of CPython:

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

On Fri, 25 May 2018 10:36:08 -0700 Raymond Hettinger wrote: > > On May 24, 2018, at 10:57 AM, Antoine Pitrou wrote: > > > > While PEP 574 (pickle protocol 5 with out-of-band data) is still in > > draft status, I've made available an

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

> On May 24, 2018, at 10:57 AM, Antoine Pitrou wrote: > > While PEP 574 (pickle protocol 5 with out-of-band data) is still in > draft status, I've made available an implementation in branch "pickle5" > in my GitHub fork of CPython: >

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

I tried this implementation to add no-copy pickling for large numpy arrays and seems to work as expected (for a simple contiguous array). I took some notes on the numpy tracker to advertise this PEP to the numpy developers: https://github.com/numpy/numpy/issues/11161 -- Olivier ​

Re: [Python-Dev] PEP 574 (pickle 5) implementation and backport available

Link to the PEP: "PEP 574 -- Pickle protocol 5 with out-of-band data" https://www.python.org/dev/peps/pep-0574/ Victor 2018-05-24 19:57 GMT+02:00 Antoine Pitrou : > > Hi, > > While PEP 574 (pickle protocol 5 with out-of-band data) is still in > draft status, I've made

[Python-Dev] PEP 574 (pickle 5) implementation and backport available

Hi, While PEP 574 (pickle protocol 5 with out-of-band data) is still in draft status, I've made available an implementation in branch "pickle5" in my GitHub fork of CPython: https://github.com/pitrou/cpython/tree/pickle5 Also I've published an experimental backport on PyPI, for Python 3.6 and