[issue16475] Support object instancing and recursion in marshal

2014-04-04 Thread Kristján Valur Jónsson
Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___ ___

[issue16475] Support object instancing and recursion in marshal

2014-04-04 Thread Kristján Valur Jónsson
Changes by Kristján Valur Jónsson krist...@ccpgames.com: -- resolution: - fixed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___ ___

[issue16475] Support object instancing and recursion in marshal

2013-03-26 Thread Roundup Robot
Roundup Robot added the comment: New changeset 84e73ace3d7e by Kristjan Valur Jonsson in branch 'default': Issue #16475: Add a whatsnew entry for 3.4 http://hg.python.org/cpython/rev/84e73ace3d7e -- ___ Python tracker rep...@bugs.python.org

[issue16475] Support object instancing and recursion in marshal

2013-03-25 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Thanks, Martin, for clarifying this. Glad you are seeing improvements Amaury. In fact, large modules can get additional improvements by identifying identical generated constructs from the compiler, like tuples and strings. The compiler isn't very

[issue16475] Support object instancing and recursion in marshal

2013-03-25 Thread Ezio Melotti
Ezio Melotti added the comment: I am unsure about how whatsnew is handled these days. Is it incrementally updated or managed by someone? It's better to add at least a stub to the whatsnew, even if someone will eventually go through it before the release. --

[issue16475] Support object instancing and recursion in marshal

2013-03-24 Thread Amaury Forgeot d'Arc
Amaury Forgeot d'Arc added the comment: Sorry, what does instancing mean? And does this change bring interesting features? And is there an impact on regular .pyc files? -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org

[issue16475] Support object instancing and recursion in marshal

2013-03-24 Thread Martin v . Löwis
Martin v. Löwis added the comment: Sorry, what does instancing mean? He means keeping track of instance identities, so that objects that were shared before marshal continue to be shared after loading. And does this change bring interesting features? interesting to whom? And is there an

[issue16475] Support object instancing and recursion in marshal

2013-03-24 Thread Amaury Forgeot d'Arc
Amaury Forgeot d'Arc added the comment: The size of the pyc files may decrease This is very good news! Indeed, I noticed decimal.cpython-34.pyc going from 212k to 178k. 17% less! This is worth an entry in whatsnew/3.4.rst IMO. -- ___ Python

[issue16475] Support object instancing and recursion in marshal

2013-03-21 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Yes, they were fixed with #42bf74b90626 which also added unittests in test_marshal.py to make sure invalid EOFs are always caught. -- ___ Python tracker rep...@bugs.python.org

[issue16475] Support object instancing and recursion in marshal

2013-03-20 Thread Ezio Melotti
Ezio Melotti added the comment: I'm getting two failures after this: == ERROR: testRaising (test.test_exceptions.ExceptionTests) -- Traceback (most recent call

[issue16475] Support object instancing and recursion in marshal

2013-03-20 Thread Antoine Pitrou
Antoine Pitrou added the comment: Indeed, would be nice to fix the test failures. Besides, it would be extra nice if you could run the test suite *before* pushing your changes. Otherwise you're wasting everyone else's time. -- ___ Python tracker

[issue16475] Support object instancing and recursion in marshal

2013-03-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: This should not have happened and it was indeed all tested. I'll investigate why these errors are happening. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475

[issue16475] Support object instancing and recursion in marshal

2013-03-20 Thread Roundup Robot
Roundup Robot added the comment: New changeset f4c21179690b by Kristján Valur Jónsson in branch 'default': Issue #16475: Simplify the interface to r_ref_allocate and improve comments. http://hg.python.org/cpython/rev/f4c21179690b New changeset 42bf74b90626 by Kristján Valur Jónsson in branch

[issue16475] Support object instancing and recursion in marshal

2013-03-19 Thread Roundup Robot
Roundup Robot added the comment: New changeset 01372117a5b4 by Kristján Valur Jónsson in branch 'default': Issue #16475: Support object instancing, recursion and interned strings http://hg.python.org/cpython/rev/01372117a5b4 -- nosy: +python-dev ___

[issue16475] Support object instancing and recursion in marshal

2013-03-19 Thread Benjamin Peterson
Benjamin Peterson added the comment: I don't understand some of this code. Why does r_ref_reserve take a first parameter which it just returns on success without using? -- nosy: +benjamin.peterson ___ Python tracker rep...@bugs.python.org

[issue16475] Support object instancing and recursion in marshal

2013-03-19 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: I thought I had explained this already, but can't find it, so here is the explanation: It matches the pattern of r_ref(), which semantically is a combination of r_ref_register() and r_ref_insert(). It is a convenence calling pattern because these

[issue16475] Support object instancing and recursion in marshal

2013-03-19 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: I should add comments explaining this to the file. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___

[issue16475] Support object instancing and recursion in marshal

2012-11-21 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Code objects can indeed be shared. One thing that the recode module does, or allows you to do, is to strip file and line number information from code objects. This will theoretically allow them to be collapsed. Martin, I agree the .pyc size matters.

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Antoine, I understand that _you_ may not see any need for object references in marshal streams. Also, I am not going to try to convince you it is a good idea, since I have long figured out that you are against any contributions from me on some sort

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Mark Dickinson
Mark Dickinson added the comment: I have long figured out that you are against any contributions from me on some sort of principle. I suspect that Antoine's principles have very little to do with *who* the contributions originate from, and much more to do with the content of those

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Serhiy, to answer your questions: Can you please measure the time of unmarshalling? Sure. Normally, I wouldn't consider time to be important here because for code objects, unmarshaling is done only once per run However, testing it is simple: I added

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: New patch, incorporating suggested fixes from review. -- Added file: http://bugs.python.org/file28047/marshalinstance.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Thank you, Kristján, for the statistics. It makes your proposition more attractive. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: New patch with changes. -- Added file: http://bugs.python.org/file28048/marshalinstance.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Personally I don't like the use of macros inn this code. I think that without them the code would be clearer. If anyone is interested, here are the statistics for all the standard modules (Lib/__pycache__/*.pyc). UNICODE 105248 61% TUPLE

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Changes as suggested by Serhiy -- Added file: http://bugs.python.org/file28052/marshalinstance.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: The size of the .pyc files is secondary. The size that is important is the memory footprint of loaded code objects, which can be done by stripping and folding code objects. This works springs out of work for embedding python on the PS3 console where

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Antoine Pitrou
Antoine Pitrou added the comment: However, even if you cannot agree that it is a good idea, can you explain to me how it is a BAD idea? How can expanding object references to strings to all objects, using the same mechanism, be bad? It is a bad idea because features have to be supported in

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Antoine Pitrou
Antoine Pitrou added the comment: By the way, please follow PEP 8 in test_marshal.py. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Here is the statistics for all pyc-files (not only in Lib/__pycache__). This includes encoding tables and tests. I count also memory usage for some types (for tuples shared size is estimated upper limit). type count % size shared %

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Here is the statistics for all pyc-files (not only in Lib/__pycache__). This includes encoding tables and tests. I count also memory usage for some types (for tuples shared size is estimated upper limit). type count % size shared %

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Changes by Serhiy Storchaka storch...@gmail.com: -- Removed message: http://bugs.python.org/msg176012 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Antoine Pitrou
Antoine Pitrou added the comment: Here is the statistics for all pyc-files (not only in Lib/__pycache__). This includes encoding tables and tests. I count also memory usage for some types (for tuples shared size is estimated upper limit). Did you examine the sharing per file or among all

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Per file. With total sharing: UNICODE 622812 58%26105085 18262484 70% TUPLE 224214 21% 8184848 4007404 49% STRING 90992 8.4%6931342 1361618 20% INT 52087 4.8% 71540095666 13% CODE

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: Total size of all *.pyc files is 22 MB. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___ ___

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Martin v . Löwis
Martin v. Löwis added the comment: Am 20.11.12 17:32, schrieb Kristján Valur Jónsson: The size of the .pyc files is secondary. This really depends on whom you ask. When I did the string interning support, the primary concern *was* for the size of the pyc files, and there *was* a real project

[issue16475] Support object instancing and recursion in marshal

2012-11-20 Thread Martin v . Löwis
Martin v. Löwis added the comment: Am 20.11.12 18:02, schrieb Antoine Pitrou: It is a bad idea because features have to be supported in the long-term, which means more maintenance effort. So, basically, this is the same reason we don't accept every feature request + patch that gets posted to

[issue16475] Support object instancing and recursion in marshal

2012-11-19 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: If you have string sharing, adding support for general sharing falls automatically out without any effort. There is no reason _not_ to support it, in other words. Marshal may be primarily used for .pyc files but it is not the only usage. It is a

[issue16475] Support object instancing and recursion in marshal

2012-11-19 Thread Christian Heimes
Changes by Christian Heimes li...@cheimes.de: -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___ ___

[issue16475] Support object instancing and recursion in marshal

2012-11-19 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: There is no many sense to use references for TYPE_INT whose representation size not greater then a reference representation size. I doubt about references to mutable objects. -- nosy: +serhiy.storchaka ___

[issue16475] Support object instancing and recursion in marshal

2012-11-19 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Ok, I did some tests with my recode module. The following are the sizes of the marshal data: test2To3 ... 24748 24748 212430 212430 test3To3 ... 18420 17848 178969 174806 test4To3 ... 18425 18411 178969 178550 The columns: a) test_marshal.py without

[issue16475] Support object instancing and recursion in marshal

2012-11-19 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: The following are the sizes of the marshal data: Can you please measure the time of unmarshalling? It would be interesting. If you can count the statistics about marshalled types (what percent of shared and non shared integers, strings, etc), it would also

[issue16475] Support object instancing and recursion in marshal

2012-11-19 Thread Antoine Pitrou
Antoine Pitrou added the comment: I should also point out that the marshal protocol takes care to be able to serialize lists, sets and frozensets correctly, the latter being added in version 2.4. This despite the fact that code objects don't make use of these. Code objects do use

[issue16475] Support object instancing and recursion in marshal

2012-11-18 Thread Gregory P. Smith
Changes by Gregory P. Smith g...@krypto.org: -- nosy: +gregory.p.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475 ___ ___

[issue16475] Support object instancing and recursion in marshal

2012-11-17 Thread Antoine Pitrou
Antoine Pitrou added the comment: I agree that restoring the string interning behaviour would be a good thing. However, I don't agree that supporting recursive objects and instantiation is useful. marshal is specialized for code objects, and you shouldn't find any recursive constants there.

[issue16475] Support object instancing and recursion in marshal

2012-11-17 Thread Martin v . Löwis
Martin v. Löwis added the comment: When I added interning support to marshal, I specifically cared about the size of pyc. I find it sad that this support was thrown out, so I support restoring it. I'm also skeptical about general sharing, and would like to see some specific numbers pointing

[issue16475] Support object instancing and recursion in marshal

2012-11-16 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: This change is specifically aimed at code objects. As it is, it is impossible to produce code objects that share common data (e.g. filename strings, common tuples, name strings, etc) that don't unserialize to separate objects. Also, separately but

[issue16475] Support object instancing and recursion in marshal

2012-11-16 Thread Antoine Pitrou
Antoine Pitrou added the comment: This change is specifically aimed at code objects. As it is, it is impossible to produce code objects that share common data (e.g. filename strings, common tuples, name strings, etc) that don't unserialize to separate objects. Shouldn't strings be interned

[issue16475] Support object instancing and recursion in marshal

2012-11-16 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Basically, reuse of strings (and preservation of their internment status) fell by the wayside somewhere in the 3.x transition. Strings have been reused, and interned strings re-interned, since protocol version 1 in 2.x. This patch adds that feature

[issue16475] Support object instancing and recursion in marshal

2012-11-15 Thread Kristján Valur Jónsson
New submission from Kristján Valur Jónsson: The format used by the marshal module does not support instancing. This precludes certain data optimizations, such as sharing string constants, common tuples, even common code objects. Since the marshal format is used to write compiled code, this

[issue16475] Support object instancing and recursion in marshal

2012-11-15 Thread Kristján Valur Jónsson
Kristján Valur Jónsson added the comment: Second patch which adds the missing internment support for strings, including unittests. -- Added file: http://bugs.python.org/file27988/marshalinstance.patch ___ Python tracker rep...@bugs.python.org

[issue16475] Support object instancing and recursion in marshal

2012-11-15 Thread Antoine Pitrou
Antoine Pitrou added the comment: marshal is only supposed to be used to serialize code objects, not arbitrary user data. Why don't you use pickle? -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue16475