On 11/18/2013 07:48 AM, Guido van Rossum wrote:
Clearly the framing is the weakest point of the PEP (== elicits the
most bikeshedding).
Indeed--it is still ongoing:
http://bugs.python.org/issue19780
//arry/
___
Python-Dev mailing list
Python-De
Hello,
I've now pushed Alexandre's implementation, and the PEP is marked final.
Regards
Antoine.
On Sat, 16 Nov 2013 19:15:26 +0100
Antoine Pitrou wrote:
>
> Hello,
>
> Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
> the PEP 3154 implementation - pickle protocol
[Alexandre Vassalotti]
> Looking at the different options available to us:
>
> 1A. Mandatory framing
> (+) Allows the internal buffering layer of the Unpickler to rely
> on the presence of framing to simplify its implementation.
> (-) Forces all implementations of pickle to in
Yup. Agreed. Ship it!
On Wed, Nov 20, 2013 at 4:54 PM, Antoine Pitrou wrote:
> On Wed, 20 Nov 2013 18:45:53 -0600
> Tim Peters wrote:
> > [Antoine]
> > > I have made two last-minute changes to the PEP:
> > >
> > > - addition of the FRAME opcode, as discussed with Tim, and keeping a
> > > fix
On Wed, 20 Nov 2013 18:45:53 -0600
Tim Peters wrote:
> [Antoine]
> > I have made two last-minute changes to the PEP:
> >
> > - addition of the FRAME opcode, as discussed with Tim, and keeping a
> > fixed 8-byte frame size
>
> Cool!
>
>
> > - addition of the MEMOIZE opcode, courtesy of Alexand
[Antoine]
> I have made two last-minute changes to the PEP:
>
> - addition of the FRAME opcode, as discussed with Tim, and keeping a
> fixed 8-byte frame size
Cool!
> - addition of the MEMOIZE opcode, courtesy of Alexandre, which replaces
> PUT opcodes in protocol 4 and helps shrink the size
Hello,
I have made two last-minute changes to the PEP:
- addition of the FRAME opcode, as discussed with Tim, and keeping a
fixed 8-byte frame size
- addition of the MEMOIZE opcode, courtesy of Alexandre, which replaces
PUT opcodes in protocol 4 and helps shrink the size of pickles
If ther
[Tim]
>> BTW, I'm not a web guy: in what way is HTTP chunked transfer mode
>> viewed as being flawed? Everything I ever read about it seemed to
>> think it was A Good Idea.
[Martin]
> It just didn't work for some time, see e.g.
>
> http://bugs.python.org/issue1486335
> http://bugs.python.org/iss
A problem with chunked IIRC is that the frame headers are variable-length
(a CRLF followed by a hex number followed by some optional gunk followed by
CRLF) so you have to drop back into one-byte-at-a-time to read it. (Well, I
suppose you could read 5 bytes, which is the minimum: CR, LF, X, CR, LF,
Am 20.11.13 06:18, schrieb Tim Peters:
> BTW, I'm not a web guy: in what way is HTTP chunked transfer mode
> viewed as being flawed? Everything I ever read about it seemed to
> think it was A Good Idea.
It just didn't work for some time, see e.g.
http://bugs.python.org/issue1486335
http://bugs.
On Tue, Nov 19, 2013 at 2:09 PM, Antoine Pitrou wrote:
>
> Well, I don't think it's a big deal to add a FRAME opcode if it doesn't
> change the current framing logic. I'd like to defer to Alexandre on this
> one, anyway.
Looking at the different options available to us:
1A. Mandatory framing
[Martin v. Löwis]
> ...
> AFAICT, the real driving force is the desire to not read-ahead
> more than the pickle is long. This is what complicates the code.
> The easiest (and most space-efficient) solution to that problem
> would be to prefix the entire pickle with a data size field
> (possibly in
On Wed, 20 Nov 2013 00:56:13 +0100
"Martin v. Löwis" wrote:
> AFAICT, the real driving force is the desire to not read-ahead
> more than the pickle is long. This is what complicates the code.
> The easiest (and most space-efficient) solution to that problem
> would be to prefix the entire pickle w
Am 19.11.13 23:50, schrieb Antoine Pitrou:
> Ok, thanks. So now that I look at the patch I see the following problems
> with this idea:
>
> - "pickle + framing" becomes a different protocol than "pickle" alone,
> which means we lose the benefit of protocol autodetection. It's as
> though pickl
On Tue, 19 Nov 2013 23:05:07 +0100
"Martin v. Löwis" wrote:
> Am 19.11.13 21:28, schrieb Antoine Pitrou:
>
> > Well, unless you propose a patch before Saturday, I will happily ignore
> > your proposal.
>
> See
>
> http://bugs.python.org/file32709/framing.diff
Ok, thanks. So now that I look at
Am 19.11.13 21:28, schrieb Antoine Pitrou:
> Well, unless you propose a patch before Saturday, I will happily ignore
> your proposal.
See
http://bugs.python.org/file32709/framing.diff
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
On Tue, 19 Nov 2013 16:06:22 -0600
Tim Peters wrote:
> [Antoine]
> >>> Ahah, ok, I see where you're going. But how many other implementations
> >>> of unpickling are there?
>
> [Tim]
> >> That's something you should have researched when writing the PEP ;-)
> >> How many implementations of Python
[Antoine]
>>> Ahah, ok, I see where you're going. But how many other implementations
>>> of unpickling are there?
[Tim]
>> That's something you should have researched when writing the PEP ;-)
>> How many implementations of Python aren't CPython? That's probably
>> the answer. I'm not an expert o
On Tue, 19 Nov 2013 15:41:51 -0600
Tim Peters wrote:
> [Tim]
> >> ...
> >> But if some _other_ implementation of unpickling didn't give a hoot
> >> about framing, having an explicit opcode means that implementation
> >> could ignore the whole scheme very easily: just implement the FRAME
> >> opco
[Tim]
>> ...
>> But if some _other_ implementation of unpickling didn't give a hoot
>> about framing, having an explicit opcode means that implementation
>> could ignore the whole scheme very easily: just implement the FRAME
>> opcode in *its* opcode-decoding loop to consume the FRAME argument,
>>
On Tue, 19 Nov 2013 15:17:06 -0600
Tim Peters wrote:
>
> > Note some drawbacks of frame opcodes:
> > - the decoder has to sanity check the frame opcodes (what if a frame
> > opcode is encountered when already inside a frame?)
> > - a pickle-mutating function such as pickletools.optimize() may n
[Tim]
>> ...
>> better than implicit" kinds of reasons. The only way now to know that
>> you're looking at a frame size is to keep a running count of bytes
>> processed and realize you've reached a byte offset where a frame size
>> "is expected".
[Antoine]
> That's integrated to the built-in buff
19.11.13 21:59, Antoine Pitrou написав(ла):
Note some drawbacks of frame opcodes:
- the decoder has to sanity check the frame opcodes (what if a frame
opcode is encountered when already inside a frame?)
This is only one simple check when reading the frame opcode.
- a pickle-mutating functi
On Tue, 19 Nov 2013 21:25:34 +0100
"Martin v. Löwis" wrote:
> Am 19.11.13 20:59, schrieb Antoine Pitrou:
> > That's integrated to the built-in buffering. It's not really an
> > additional constraint: the frame sizes simply dictate how buffering
> > happens in practice. The main point of framing is
Am 19.11.13 20:59, schrieb Antoine Pitrou:
> That's integrated to the built-in buffering. It's not really an
> additional constraint: the frame sizes simply dictate how buffering
> happens in practice. The main point of framing is to *simplify* the
> buffering logic (of course, the old buffering lo
On Tue, 19 Nov 2013 13:22:52 -0600
Tim Peters wrote:
> [Guido]
> > So using an opcode for framing is out? (Sorry, I've lost track of the
> > back-and-forth.)
>
> It was never in ;-) I'd *prefer* one, but not enough to try to block
> the PEP. As is, framing is done at a "lower level" than opcode
[Guido]
> So using an opcode for framing is out? (Sorry, I've lost track of the
> back-and-forth.)
It was never in ;-) I'd *prefer* one, but not enough to try to block
the PEP. As is, framing is done at a "lower level" than opcode
decoding. I fear this is brittle, for all the usual "explicit is
Well, both fixed 8-byte framing and variable-size framing it introduces a
new way of representing numbers in the stream, which means that everyone
parsing and generating pickles must be able to support both styles. (But
fixed is easier since the XXX8 opcodes use the same format.)
I'm thinking of h
On Tue, 19 Nov 2013 11:05:45 -0800
Guido van Rossum wrote:
> So using an opcode for framing is out? (Sorry, I've lost track of the
> back-and-forth.)
It doesn't seem to bring anything, and it makes the overhead worse for
tiny pickles (since it will be two bytes at least, instead of one byte
with
So using an opcode for framing is out? (Sorry, I've lost track of the
back-and-forth.)
On Tue, Nov 19, 2013 at 10:57 AM, Antoine Pitrou wrote:
> On Tue, 19 Nov 2013 10:52:58 -0800
> Guido van Rossum wrote:
> > So why is framing different?
>
> Because it doesn't use opcodes, so it can't use diff
On Tue, 19 Nov 2013 10:52:58 -0800
Guido van Rossum wrote:
> So why is framing different?
Because it doesn't use opcodes, so it can't use different opcodes to
differentiate between different frame size widths :-)
Regards
Antoine.
___
Python-Dev mailin
On Tue, 19 Nov 2013 19:51:10 +0100
Antoine Pitrou wrote:
> On Mon, 18 Nov 2013 16:48:05 -0800
> Guido van Rossum wrote:
> >
> > Food for thought: maybe we should have variable-encoding lengths for all
> > opcodes, rather than the current cumbersome scheme?
>
> Well, it's not that cumbersome...
So why is framing different?
On Tue, Nov 19, 2013 at 10:51 AM, Antoine Pitrou wrote:
> On Mon, 18 Nov 2013 16:48:05 -0800
> Guido van Rossum wrote:
> >
> > Food for thought: maybe we should have variable-encoding lengths for all
> > opcodes, rather than the current cumbersome scheme?
>
> Well,
On Mon, 18 Nov 2013 16:48:05 -0800
Guido van Rossum wrote:
>
> Food for thought: maybe we should have variable-encoding lengths for all
> opcodes, rather than the current cumbersome scheme?
Well, it's not that cumbersome... If you look at CPU encodings, they
also tend to have different opcodes f
Food for thought: maybe we should have variable-encoding lengths for
all opcodes, rather than the current cumbersome scheme?
Funny, it sounds like UTF-8 :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/p
Food for thought: maybe we should have variable-encoding lengths for
all opcodes, rather than the current cumbersome scheme?
Funny, it sounds like UTF-8 :-)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/
19.11.13 01:10, Antoine Pitrou написав(ла):
- switching to the PREFETCH scheme requires some non-trivial work on the
current patch, work done by either Alexandre or me (but I already
have pathlib (PEP 428) on my plate, so it'll have to be Alexandre) -
unless you want to do it, of course?
[Antoine]
> ...
> Well, it's a question of cost / benefit: does it make sense to optimize
> something that will be dwarfed by other factors in real world
> situations?
For most of my career, a megabyte of RAM was an unthinkable luxury.
Now I'm running on an OS that needs a gigabyte of RAM just to
On 19/11/2013 12:55am, Tim Peters wrote:
[Richard Oudkerk]
I tried using multiprocessing.Pipe() and send_bytes()/recv_bytes() to send
messages between processes:
8 bytes messages -- 525,000 msgs/sec
15 bytes messages -- 556,000 msgs/sec
So the size of small messages does not seem to mak
[Antoine]
>>> - first byte: bits 7-5: N (= frame size bytes length - 1)
>>> - first byte: bits 4-0: first 5 bits of frame size
>>> - remaning N bytes: remaining bits of frame size
[Tim]
>> I'm unclear on how that would work for, e.g., encoding 40 =
>> 0b000101000. That has 6 significant bits. Wo
On Mon, 18 Nov 2013 18:50:01 -0600
Tim Peters wrote:
> [Antoine]
> > Yet another possibility: keep framing but use a variable-length
> > encoding for the frame size:
> >
> > - first byte: bits 7-5: N (= frame size bytes length - 1)
> > - first byte: bits 4-0: first 5 bits of frame size
> > - reman
[Richard Oudkerk]
> I tried using multiprocessing.Pipe() and send_bytes()/recv_bytes() to send
> messages between processes:
>
> 8 bytes messages -- 525,000 msgs/sec
> 15 bytes messages -- 556,000 msgs/sec
>
> So the size of small messages does not seem to make much difference.
To the contrar
[Guido]
> Food for thought: maybe we should have variable-encoding lengths for all
> opcodes, rather than the current cumbersome scheme?
Yes, but not for protocol 4 - time's running out fast for that. When
we "only" had the XXX1, XXX2, and XXX4 opcodes, it was kinda silly,
but after adding XXX8 f
[Antoine]
> Yet another possibility: keep framing but use a variable-length
> encoding for the frame size:
>
> - first byte: bits 7-5: N (= frame size bytes length - 1)
> - first byte: bits 4-0: first 5 bits of frame size
> - remaning N bytes: remaining bits of frame size
>
> With this scheme, very
On Mon, Nov 18, 2013 at 4:30 PM, Nick Coghlan wrote:
> On 19 November 2013 09:57, Antoine Pitrou wrote:
> > On Mon, 18 Nov 2013 16:44:59 -0600
> > Tim Peters wrote:
> >> [Tim]
> >> >> But it has a different kind of advantage: PREFETCH was optional. As
> >> >> Guido said, it's annoying to bloa
On 19 November 2013 09:57, Antoine Pitrou wrote:
> On Mon, 18 Nov 2013 16:44:59 -0600
> Tim Peters wrote:
>> [Tim]
>> >> But it has a different kind of advantage: PREFETCH was optional. As
>> >> Guido said, it's annoying to bloat the size of small pickles (which
>> >> may, although individually
On Mon, 18 Nov 2013 16:44:59 -0600
Tim Peters wrote:
> [Tim]
> >> But it has a different kind of advantage: PREFETCH was optional. As
> >> Guido said, it's annoying to bloat the size of small pickles (which
> >> may, although individually small, occur in great numbers) by 8 bytes
> >> each. The
On 19 Nov 2013 09:33, "Tim Peters" wrote:
>
> [Antoine]
> > Well, sending oceans of tiny integers will also incur many system calls
> > and additional synchronization costs, since sending data on a
> > multiprocessing Queue has to acquire a semaphore. So it generally
> > sounds like a bad idea, IM
[Antoine]
> Well, sending oceans of tiny integers will also incur many system calls
> and additional synchronization costs, since sending data on a
> multiprocessing Queue has to acquire a semaphore. So it generally
> sounds like a bad idea, IMHO.
>
> That said, I agree with:
>> Since pickle intend
On 18/11/2013 10:25pm, Tim Peters wrote:
The difference between 8 and 15 is, e.g., nearly doubling the amount
of network traffic (for apps that use pickles across processes or
machines).
I tried using multiprocessing.Pipe() and send_bytes()/recv_bytes() to
send messages between processes:
Ok, how about merging the two sub-threads :-)
On Mon, 18 Nov 2013 16:44:59 -0600
Tim Peters wrote:
> [Antoine]
> > You can't know how much space the pickle will take until the pickling
> > ends, though, which makes it difficult to decide whether you want to
> > emit a PREFETCH opcode or not.
>
On 19 Nov 2013 08:52, "Tim Peters" wrote:
>
> [Tim]
> >> But it has a different kind of advantage: PREFETCH was optional. As
> >> Guido said, it's annoying to bloat the size of small pickles (which
> >> may, although individually small, occur in great numbers) by 8 bytes
> >> each. There's real
[Tim]
>> But it has a different kind of advantage: PREFETCH was optional. As
>> Guido said, it's annoying to bloat the size of small pickles (which
>> may, although individually small, occur in great numbers) by 8 bytes
>> each. There's really no point to framing small chunks of data, right?
[A
On Mon, 18 Nov 2013 16:25:07 -0600
Tim Peters wrote:
>
> [Antoine]
> > Are very small pickles that size-sensitive? I have the impression that
> > if 8 bytes vs. e.g. 15 bytes makes a difference for your application,
> > you'd be better off with a hand-made format.
>
> The difference between 8 an
On Mon, 18 Nov 2013 16:18:21 -0600
Tim Peters wrote:
>
> But it has a different kind of advantage: PREFETCH was optional. As
> Guido said, it's annoying to bloat the size of small pickles (which
> may, although individually small, occur in great numbers) by 8 bytes
> each. There's really no po
[Tim]
>> ...
>> It was already annoying when the PROTO opcode was introduced,
>> and the size of small pickles increased by 2 bytes. That
>> added up too :-(
[Antoine]
> Are very small pickles that size-sensitive? I have the impression that
> if 8 bytes vs. e.g. 15 bytes makes a difference for yo
[Tim]
>> But I wonder why it isn't done with a new framing opcode instead (say,
>> FRAME followed by 8-byte count). I suppose that would be like the
>> "prefetch" idea, except that framing opcodes would be mandatory
>> (instead of optional) in proto 4. Why I initially like that:
>>
>> - Uniform d
[Guido]
>> Clearly the framing is the weakest point of the PEP (== elicits the most
>> bikeshedding). I am also unsure about the value of framing when pickles are
>> written to strings.
[Antoine]
> It hasn't much value in that case,
It has _no_ value in that case, yes? It doesn't appear to have
On Mon, 18 Nov 2013 16:02:31 -0600
Tim Peters wrote:
> [Guido]
> >> Clearly the framing is the weakest point of the PEP (== elicits the most
> >> bikeshedding). I am also unsure about the value of framing when pickles are
> >> written to strings.
>
> [Antoine]
> > It hasn't much value in that cas
On Mon, 18 Nov 2013 17:14:21 +0100
Antoine Pitrou wrote:
> On Sun, 17 Nov 2013 23:53:09 -0600
> Tim Peters wrote:
> >
> > But I wonder why it isn't done with a new framing opcode instead (say,
> > FRAME followed by 8-byte count). I suppose that would be like the
> > "prefetch" idea, except that
On Mon, Nov 18, 2013 at 8:10 AM, Antoine Pitrou wrote:
> On Mon, 18 Nov 2013 07:48:27 -0800
> Guido van Rossum wrote:
> > On Mon, Nov 18, 2013 at 3:28 AM, Serhiy Storchaka >wrote:
> >
> > > 18.11.13 07:53, Tim Peters написав(ла):
> > >
> > > - Some easy sanity checking due to the tiny redundan
On Sun, 17 Nov 2013 23:53:09 -0600
Tim Peters wrote:
>
> But I wonder why it isn't done with a new framing opcode instead (say,
> FRAME followed by 8-byte count). I suppose that would be like the
> "prefetch" idea, except that framing opcodes would be mandatory
> (instead of optional) in proto 4
On Mon, 18 Nov 2013 07:48:27 -0800
Guido van Rossum wrote:
> On Mon, Nov 18, 2013 at 3:28 AM, Serhiy Storchaka wrote:
>
> > 18.11.13 07:53, Tim Peters написав(ла):
> >
> > - Some easy sanity checking due to the tiny redundancy (if the byte
> >> immediately following the current frame is not a FR
On Mon, Nov 18, 2013 at 3:28 AM, Serhiy Storchaka wrote:
> 18.11.13 07:53, Tim Peters написав(ла):
>
> - Some easy sanity checking due to the tiny redundancy (if the byte
>> immediately following the current frame is not a FRAME opcode, the
>> pickle is corrupt; and also corrupt if a FRAME opcode
18.11.13 07:53, Tim Peters написав(ла):
- Some easy sanity checking due to the tiny redundancy (if the byte
immediately following the current frame is not a FRAME opcode, the
pickle is corrupt; and also corrupt if a FRAME opcode is encountered
_inside_ the current frame).
For efficient unpickli
[Antoine Pitrou]
> Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
> the PEP 3154 implementation - pickle protocol 4.
>
> I think it would be good to get the PEP and the implementation accepted
> for 3.4. As far as I can say, this has been a low-controvery proposal,
> and it
On Sat, Nov 16, 2013 at 10:15 AM, Antoine Pitrou wrote:
> Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
> the PEP 3154 implementation - pickle protocol 4.
>
> I think it would be good to get the PEP and the implementation accepted
> for 3.4. As far as I can say, this has
On 17 Nov 2013 04:45, "Eric Snow" wrote:
>
> On Sat, Nov 16, 2013 at 11:15 AM, Antoine Pitrou
wrote:
> >
> > Hello,
> >
> > Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
> > the PEP 3154 implementation - pickle protocol 4.
> >
> > I think it would be good to get the PEP
On Sat, Nov 16, 2013 at 11:15 AM, Antoine Pitrou wrote:
>
> Hello,
>
> Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
> the PEP 3154 implementation - pickle protocol 4.
>
> I think it would be good to get the PEP and the implementation accepted
> for 3.4.
+1
Once the cor
Hello,
Alexandre Vassalotti (thanks a lot!) has recently finalized his work on
the PEP 3154 implementation - pickle protocol 4.
I think it would be good to get the PEP and the implementation accepted
for 3.4. As far as I can say, this has been a low-controvery proposal,
and it brings fairly obvi
70 matches
Mail list logo
