Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread tds...@gmail.com
- bounces+kristjan=ccpgames@python.org] On Behalf Of Victor Stinner Sent: Monday, January 27, 2014 23:35 To: Wolfgang Cc: Python-Dev Subject: Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol) Hi, I'm surprised: marshal.dumps() doesn't raise an error if you pass an invalid

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Martin v. Löwis
I've debugged this a little bit. I couldn't originally see where the problem is, since I expected that the code dealing with shared references shouldn't ever trigger - none of the tuples in the example are actually shared (i.e. they all have a ref-count of 1, except for the outer list, which is

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Barry Warsaw
On Jan 28, 2014, at 09:17 AM, tds...@gmail.com wrote: yes I know the main usage is to generate pyc files. But marshal is also used for other stuff and is the fastest built in serialization method. For some use cases it makes sense to use it instead of pickle or others. And people use it not only

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Victor Stinner
2014-01-28 Martin v. Löwis mar...@v.loewis.de: Debugging reveals that it is actually the many integer objects which trigger the sharing code. So a much simplified example of Victor's benchmarking code can use data = [0]*1000 The difference between version 2 and version 3 here is that v2

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Antoine Pitrou
On Tue, 28 Jan 2014 11:22:40 +0100 Victor Stinner victor.stin...@gmail.com wrote: 2014-01-28 Martin v. Löwis mar...@v.loewis.de: Debugging reveals that it is actually the many integer objects which trigger the sharing code. So a much simplified example of Victor's benchmarking code can use

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread tds...@gmail.com
On 28.01.2014 10:23, Barry Warsaw wrote: On Jan 28, 2014, at 09:17 AM, tds...@gmail.com wrote: yes I know the main usage is to generate pyc files. But marshal is also used for other stuff and is the fastest built in serialization method. For some use cases it makes sense to use it instead of

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Kristján Valur Jónsson
3.4, marshal dumps slower (version 3 protocol) Hi, I tested the latest beta from 3.4 (b3) and noticed there is a new marshal protocol version 3. The documentation is a little silent about the new features, not going into detail. I've run a performance test with the new protocol version and noticed

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Kristján Valur Jónsson
Of Barry Warsaw Sent: Tuesday, January 28, 2014 17:23 To: python-dev@python.org Subject: Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol) marshall is not guaranteed to be backward compatible between Python versions, so it's generally not a good idea to use it for serialization

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-28 Thread Terry Reedy
On 1/28/2014 10:02 PM, Kristján Valur Jónsson wrote: marshall is not guaranteed to be backward compatible between Python versions, so it's generally not a good idea to use it for serialization. How often I hear this argument :) For many people, serialized data is not persisted. But used

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Victor Stinner
Hi, I'm surprised: marshal.dumps() doesn't raise an error if you pass an invalid version. In fact, Python 3.3 only supports versions 0, 1 and 2. If you pass 3, it will use the version 2. (Same apply for version 99.) Python 3.4 has two new versions: 3 and 4. The version 3 shares common object

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Paul Moore
On 27 January 2014 15:35, Victor Stinner victor.stin...@gmail.com wrote: Version 2 is the fastest in Python 3.3 and 3.4, but version 4 with Python 3.4 produces the smallest file. Which version is used when creating pyc files? This benchmark might suggest that version 2 is the best... Paul

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Brett Cannon
On Mon, Jan 27, 2014 at 10:42 AM, Paul Moore p.f.mo...@gmail.com wrote: On 27 January 2014 15:35, Victor Stinner victor.stin...@gmail.com wrote: Version 2 is the fastest in Python 3.3 and 3.4, but version 4 with Python 3.4 produces the smallest file. Which version is used when creating pyc

[Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Wolfgang
Hi, I tested the latest beta from 3.4 (b3) and noticed there is a new marshal protocol version 3. The documentation is a little silent about the new features, not going into detail. I've run a performance test with the new protocol version and noticed the new version is two times slower in

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Wolfgang
Thanks Victor for improving this. I also have to note, version 3 is only in the case of tuple in tuple slower. If you use a flat tuple it is faster than version 2. So I asked for this corner case and thought the recursion detection or something else has a huge cost. For pyc files, I think the

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Serhiy Storchaka
27.01.14 17:35, Victor Stinner написав(ла): Python 3.4 has two new versions: 3 and 4. The version 3 shares common object references, the version 4 adds short tuples and short strings (produce smaller files). Why we need two new versions added in one Python release?

Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol)

2014-01-27 Thread Kristján Valur Jónsson
Stinner Sent: Monday, January 27, 2014 23:35 To: Wolfgang Cc: Python-Dev Subject: Re: [Python-Dev] Python 3.4, marshal dumps slower (version 3 protocol) Hi, I'm surprised: marshal.dumps() doesn't raise an error if you pass an invalid version. In fact, Python 3.3 only supports versions 0, 1