[issue16475] Support object instancing and recursion in marshal

Changes by Kristján Valur Jónsson : -- resolution: -> fixed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: ht

[issue16475] Support object instancing and recursion in marshal

Changes by Kristján Valur Jónsson : -- status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue16475] Support object instancing and recursion in marshal

Roundup Robot added the comment: New changeset 84e73ace3d7e by Kristjan Valur Jonsson in branch 'default': Issue #16475: Add a whatsnew entry for 3.4 http://hg.python.org/cpython/rev/84e73ace3d7e -- ___ Python tracker

[issue16475] Support object instancing and recursion in marshal

Ezio Melotti added the comment: > I am unsure about how whatsnew is handled these days. > Is it incrementally updated or managed by someone? It's better to add at least a stub to the whatsnew, even if someone will eventually go through it before the release. -- ___

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Thanks, Martin, for clarifying this. Glad you are seeing improvements Amaury. In fact, large modules can get additional improvements by identifying identical generated constructs from the compiler, like tuples and strings. The compiler isn't very good

[issue16475] Support object instancing and recursion in marshal

Amaury Forgeot d'Arc added the comment: > The size of the pyc files may decrease This is very good news! Indeed, I noticed decimal.cpython-34.pyc going from 212k to 178k. 17% less! This is worth an entry in whatsnew/3.4.rst IMO. -- ___ Python track

[issue16475] Support object instancing and recursion in marshal

Martin v. Löwis added the comment: > Sorry, what does "instancing" mean? He means "keeping track of instance identities", so that objects that were shared before marshal continue to be shared after loading. > And does this change bring interesting features? "interesting" to whom? > And is th

[issue16475] Support object instancing and recursion in marshal

Amaury Forgeot d'Arc added the comment: Sorry, what does "instancing" mean? And does this change bring interesting features? And is there an impact on regular .pyc files? -- nosy: +amaury.forgeotdarc ___ Python tracker

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Yes, they were fixed with #42bf74b90626 which also added unittests in test_marshal.py to make sure invalid EOFs are always caught. -- ___ Python tracker __

[issue16475] Support object instancing and recursion in marshal

Ezio Melotti added the comment: FTR the two failures I saw earlier are now gone. -- ___ Python tracker ___ ___ Python-bugs-list mailin

[issue16475] Support object instancing and recursion in marshal

Roundup Robot added the comment: New changeset f4c21179690b by Kristján Valur Jónsson in branch 'default': Issue #16475: Simplify the interface to r_ref_allocate and improve comments. http://hg.python.org/cpython/rev/f4c21179690b New changeset 42bf74b90626 by Kristján Valur Jónsson in branch 'de

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: This should not have happened and it was indeed all tested. I'll investigate why these errors are happening. -- ___ Python tracker ___

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: Indeed, would be nice to fix the test failures. Besides, it would be extra nice if you could run the test suite *before* pushing your changes. Otherwise you're wasting everyone else's time. -- ___ Python tracker

[issue16475] Support object instancing and recursion in marshal

Ezio Melotti added the comment: I'm getting two failures after this: == ERROR: testRaising (test.test_exceptions.ExceptionTests) -- Traceback (most recent call l

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: I should add comments explaining this to the file. -- ___ Python tracker ___ ___ Python-bugs

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: I thought I had explained this already, but can't find it, so here is the explanation: It matches the pattern of r_ref(), which semantically is a combination of r_ref_register() and r_ref_insert(). It is a convenence calling pattern because these funct

[issue16475] Support object instancing and recursion in marshal

Benjamin Peterson added the comment: I don't understand some of this code. Why does r_ref_reserve take a first parameter which it just returns on success without using? -- nosy: +benjamin.peterson ___ Python tracker

[issue16475] Support object instancing and recursion in marshal

Roundup Robot added the comment: New changeset 01372117a5b4 by Kristján Valur Jónsson in branch 'default': Issue #16475: Support object instancing, recursion and interned strings http://hg.python.org/cpython/rev/01372117a5b4 -- nosy: +python-dev ___ P

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Code objects can indeed be shared. One thing that the "recode" module does, or allows you to do, is to strip file and line number information from code objects. This will theoretically allow them to be collapsed. Martin, I agree the .pyc size matters.

[issue16475] Support object instancing and recursion in marshal

Martin v. Löwis added the comment: Am 20.11.12 18:02, schrieb Antoine Pitrou: > It is a bad idea because features have to be supported in the long-term, > which means more maintenance effort. So, basically, this is the same > reason we don't accept every feature request + patch that gets posted >

[issue16475] Support object instancing and recursion in marshal

Martin v. Löwis added the comment: Am 20.11.12 17:32, schrieb Kristján Valur Jónsson: > The size of the .pyc files is secondary. This really depends on whom you ask. When I did the string interning support, the primary concern *was* for the size of the pyc files, and there *was* a real project w

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: Total size of all *.pyc files is 22 MB. -- ___ Python tracker ___ ___ Python-bugs-list mailing lis

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: Per file. With total sharing: UNICODE 622812 58%26105085 18262484 70% TUPLE 224214 21% 8184848 4007404 49% STRING 90992 8.4%6931342 1361618 20% INT 52087 4.8% 71540095666 13% CODE 42147

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: > Here is the statistics for all pyc-files (not only in > Lib/__pycache__). This includes encoding tables and tests. I count > also memory usage for some types (for tuples shared size is estimated > upper limit). Did you examine the sharing per file or among all

[issue16475] Support object instancing and recursion in marshal

Changes by Serhiy Storchaka : -- Removed message: http://bugs.python.org/msg176012 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: Here is the statistics for all pyc-files (not only in Lib/__pycache__). This includes encoding tables and tests. I count also memory usage for some types (for tuples shared size is estimated upper limit). type count % size shared % UN

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: Here is the statistics for all pyc-files (not only in Lib/__pycache__). This includes encoding tables and tests. I count also memory usage for some types (for tuples shared size is estimated upper limit). type count % size shared % UN

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: By the way, please follow PEP 8 in test_marshal.py. -- ___ Python tracker ___ ___ Python-bugs-list m

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: > However, even if you cannot agree that it is a good idea, can you > explain to me how it is a BAD idea? How can expanding object > references to strings to all objects, using the same mechanism, be > bad? It is a bad idea because features have to be supported

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: The size of the .pyc files is secondary. The size that is important is the memory footprint of loaded code objects, which can be done by stripping and folding code objects. This works springs out of work for embedding python on the PS3 console where e

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Changes as suggested by Serhiy -- Added file: http://bugs.python.org/file28052/marshalinstance.patch ___ Python tracker ___

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: Personally I don't like the use of macros inn this code. I think that without them the code would be clearer. If anyone is interested, here are the statistics for all the standard modules (Lib/__pycache__/*.pyc). UNICODE 105248 61% TUPLE

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: New patch with changes. -- Added file: http://bugs.python.org/file28048/marshalinstance.patch ___ Python tracker ___ ___

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: Thank you, Kristján, for the statistics. It makes your proposition more attractive. -- ___ Python tracker ___ ___

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: New patch, incorporating suggested fixes from review. -- Added file: http://bugs.python.org/file28047/marshalinstance.patch ___ Python tracker _

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Serhiy, to answer your questions: > Can you please measure the time of unmarshalling? Sure. Normally, I wouldn't consider time to be important here because for code objects, unmarshaling is done only once per run However, testing it is simple: I added

[issue16475] Support object instancing and recursion in marshal

Mark Dickinson added the comment: > I have long figured out that you are against any contributions from > me on some sort of principle. I suspect that Antoine's principles have very little to do with *who* the contributions originate from, and much more to do with the content of those contribu

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Antoine, I understand that _you_ may not see any need for object references in marshal streams. Also, I am not going to try to convince you it is a good idea, since I have long figured out that you are against any contributions from me on some sort of

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: > I should also point out that the marshal protocol takes care to be > able to serialize lists, sets and frozensets correctly, the latter > being added in version 2.4. This despite the fact that code objects > don't make use of these. Code objects do use frozen

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: > The following are the sizes of the marshal data: Can you please measure the time of unmarshalling? It would be interesting. If you can count the statistics about marshalled types (what percent of shared and non shared integers, strings, etc), it would also

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Ok, I did some tests with my recode module. The following are the sizes of the marshal data: test2To3 ... 24748 24748 212430 212430 test3To3 ... 18420 17848 178969 174806 test4To3 ... 18425 18411 178969 178550 The columns: a) test_marshal.py without t

[issue16475] Support object instancing and recursion in marshal

Serhiy Storchaka added the comment: There is no many sense to use references for TYPE_INT whose representation size not greater then a reference representation size. I doubt about references to mutable objects. -- nosy: +serhiy.storchaka ___ Python

[issue16475] Support object instancing and recursion in marshal

Changes by Christian Heimes : -- nosy: +christian.heimes ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http:/

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: If you have string sharing, adding support for general sharing falls automatically out without any effort. There is no reason _not_ to support it, in other words. Marshal may be primarily used for .pyc files but it is not the only usage. It is a very

[issue16475] Support object instancing and recursion in marshal

Changes by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue16475] Support object instancing and recursion in marshal

Martin v. Löwis added the comment: When I added interning support to marshal, I specifically cared about the size of pyc. I find it sad that this support was thrown out, so I support restoring it. I'm also skeptical about general sharing, and would like to see some specific numbers pointing o

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: I agree that restoring the string interning behaviour would be a good thing. However, I don't agree that supporting recursive objects and instantiation is useful. marshal is specialized for code objects, and you shouldn't find any recursive constants there. As

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Basically, reuse of strings (and preservation of their internment status) fell by the wayside somewhere in the 3.x transition. Strings have been reused, and interned strings re-interned, since protocol version 1 in 2.x. This patch adds that feature b

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: > This change is specifically aimed at code objects. > As it is, it is impossible to produce code objects that share common > data (e.g. filename strings, common tuples, name strings, etc) that > don't unserialize to separate objects. Shouldn't strings be intern

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: This change is specifically aimed at code objects. As it is, it is impossible to produce code objects that share common data (e.g. filename strings, common tuples, name strings, etc) that don't unserialize to separate objects. Also, separately but rela

[issue16475] Support object instancing and recursion in marshal

Antoine Pitrou added the comment: marshal is only supposed to be used to serialize code objects, not arbitrary user data. Why don't you use pickle? -- nosy: +pitrou ___ Python tracker _

[issue16475] Support object instancing and recursion in marshal

Kristján Valur Jónsson added the comment: Second patch which adds the missing internment support for strings, including unittests. -- Added file: http://bugs.python.org/file27988/marshalinstance.patch ___ Python tracker

[issue16475] Support object instancing and recursion in marshal

New submission from Kristján Valur Jónsson: The format used by the marshal module does not support instancing. This precludes certain data optimizations, such as sharing string constants, common tuples, even common code objects. Since the marshal format is used to write compiled code, this m