[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: Closing indeed! -- resolution: -> fixed stage: patch review -> committed/rejected status: open -> closed ___ Python tracker ___ ___

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: We already lost this improvement for 3.3. :( Could we now close the issue? -- ___ Python tracker ___

[issue14596] struct.unpack memory leak

Roundup Robot added the comment: New changeset 6707637f68ca by Serhiy Storchaka in branch 'default': Issue #14596: The struct.Struct() objects now use more compact implementation. http://hg.python.org/cpython/rev/6707637f68ca -- nosy: +python-dev ___

[issue14596] struct.unpack memory leak

Meador Inge added the comment: > Are you sure the PEP 3118 changes will land in 3.4? It would be a pity > to lose a simple improvement because it was deferred to a bigger > change. No, I am not sure. That is why I said that I understand if others felt this bug was critical to fix now since the

[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: Le mardi 14 mai 2013 à 16:37 +, Meador Inge a écrit : > If we feel that this optimization is really critical, then I agree > let's not hold it up and I will just work around it with my patch for > issue3132. I don't see it as that critical, but I understand

[issue14596] struct.unpack memory leak

Meador Inge added the comment: > I don't think Serhiy's patch should be blocked by a larger issue. > I suppose you could rebase easily over his changes. Where rebase=undo, sure. The changes for issue3132 are pretty extensive (the basic data structures are changed). And as mentioned in msg16589

[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: I don't think Serhiy's patch should be blocked by a larger issue. I suppose you could rebase easily over his changes. -- versions: +Python 3.4 -Python 2.7, Python 3.2, Python 3.3 ___ Python tracker

[issue14596] struct.unpack memory leak

Meador Inge added the comment: > > Serhiy Storchaka added the comment: > > So what about more compact Struct object? I already implemented the count optimization as a part of my patch for implementing PEP 3188 in issue3132. I need to rebaseline the patch. It has gotten stale. Hopefully there

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: So what about more compact Struct object? This is not only memory issue (a Struct object can require several times larger memory than size of processed data), but performance issue (time of creating such object is proportional to it's size). Here is a patch

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: > I do think issue (3) should be fixed, but a separate issue should be opened > for it. Issue #15402. -- ___ Python tracker ___ ___

[issue14596] struct.unpack memory leak

Meador Inge added the comment: I just read through all this and see three separate points be discussed: 1. The unbounded caching behavior. 2. A more compact representation for repeat counts. 3. Correct __sizeof__ support for struct. For issue (1) I think this is unfortunate, but I don't

[issue14596] struct.unpack memory leak

Changes by Meador Inge : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: Please, can anyone do the review? -- ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: Here is Struct.__sizeof__ patch with tests. -- Added file: http://bugs.python.org/file26115/struct_sizeof-2.patch ___ Python tracker ___diff

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: The struct_sizeof patch looks fine, but lacks tests. I think it might be reasonable to call this a bugfix. -- ___ Python tracker ___

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: Now internal representation of Struct with small format string may consume unexpectedly large memory and this representation may be invisible cached. With patch you can get large internal representation only for large format strings. It is expected. And how a

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: I'm still not convinced that something like struct_repeat.patch is necessary. So unless someone else wants to own this issue and review the struct_repeat, I'd say that it's too late for 3.3. -- ___ Python tracker

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: Reduction of memory consumption of struct is a new feature. Any chance to commit struct_repeat.patch+struct_sizeof.patch today and to get this feature in Python 3.3? -- ___ Python tracker

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: Here is a patch that implements method __sizeof__ for Struct. This can be used to limit the caching. In any case, it is useful to know the real memory consumption of the object. I'm not sure of the correctness of the method implementation. -- Added f

[issue14596] struct.unpack memory leak

R. David Murray added the comment: If that is the case, then a doc footnote that a large repeat count will result in a large cache seems appropriate. Some way to control the max size of the cache would also be a reasonable enhancement request, at which point the cache size issue could be doc

[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: AFAIU, it's not unbounded, but it's the memory footprint of individual cached objects which can grow senseless. -- ___ Python tracker ___

[issue14596] struct.unpack memory leak

R. David Murray added the comment: Other of our functions that do caching have been fixed so that the cache does not grow unbounded (usually by using lrucache, I think). IMO a cache that is unbounded by default is a bug. -- nosy: +r.david.murray _

[issue14596] struct.unpack memory leak

Robert Elsner added the comment: Well then at least the docs need an update. I simply fail to see how a cache memory leak constitutes "just fine" (while the caching behavior of struct.unpack is not documented - if somebody wants caching, he ought to use struct.Struct.unpack which does cache and

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: IMO, the struct module does what it's intended to do just fine here. I don't a big need for any change. I'd propose closing this as "won't fix". -- ___ Python tracker _

[issue14596] struct.unpack memory leak

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: The proposed patch uses a more compact encoding format of large structures. -- keywords: +patch Added file: http://bugs.python.org/file25242/struct_repeat.patch ___ Python tracker

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: > Or perhaps not care at all? That's also possible. :-) IMO, Robert's use-case doesn't really match the intended use-case for struct (parsing structures of values laid out like a C-struct ). There the caching makes sense. -- _

[issue14596] struct.unpack memory leak

Robert Elsner added the comment: Well I stumbled across this leak while reading big files. And what is the point of having a fast C-level unpack when it can not be used with big files? I am not adverse to the idea of caching the format string but if the cache grows beyond a reasonable size, it s

[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: > Perhaps the best quick fix would be to only cache small > PyStructObjects, for some value of 'small'. (Total size < a few > hundred bytes, perhaps.) Or perhaps not care at all? Is there a use case for huge repeat counts? (limiting cacheability could decrease

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: Perhaps the best quick fix would be to only cache small PyStructObjects, for some value of 'small'. (Total size < a few hundred bytes, perhaps.) -- ___ Python tracker __

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: > It appears the storage of Struct instances is rather inefficient when > there's a repeat code such as "<48L" Right. Repeat counts aren't directly supported in the underlying PyStructObject; a format string containing repeat counts is effectively 'compile

[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: It appears the storage of Struct instances is rather inefficient when there's a repeat code such as "<48L". In this example, 48 almost identical structures describing the "L" format (struct _formatcode) will be created. You can guess what happens with large r

[issue14596] struct.unpack memory leak

Serhiy Storchaka added the comment: Yes, the problem is the place to be in Python 2.7, in Python 3.2 and in Python 3.3. Random size of the structure is important -- if you remove the randint, leakage will not. The memory is not released in cached structuress, which are created in module-leve

[issue14596] struct.unpack memory leak

Antoine Pitrou added the comment: > I suspect that this is due to the struct module cache, which caches > Struct instances corresponding to formats used. If that's true, > there's no real leak as such. Well, the posted code creates 30 struct instances. That shouldn't exhaust the memory of a 8

[issue14596] struct.unpack memory leak

Robert Elsner added the comment: Well the problem is, that performance is severely degraded when calling unpack multiple times. I do not know in advance the size of the files and they might vary in size from 1M to 1G. I could use some fixed-size buffer which is inefficient depending on the fi

[issue14596] struct.unpack memory leak

Robert Elsner added the comment: Well seems like 3.1 is in the Debian repos as well. Same memory leak. So it is very unlikely it has been fixed in 2.7. I modified the test case to be compatible to 3.1 and 2.6. -- versions: +Python 3.1 Added file: http://bugs.python.org/file25239/unpac

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: I suspect that this is due to the struct module cache, which caches Struct instances corresponding to formats used. If that's true, there's no real leak as such. As a test, what happens if you increase your xrange(30) to xrange(300)? (And perhaps decrease

[issue14596] struct.unpack memory leak

Robert Elsner added the comment: I would love to test but I am in a production environment atm and can't really spare the time to set up a test box. But maybe somebody with access to 2.7 on linux could test it with the supplied script (just start it and it should happily eat 8GB of memory - I

[issue14596] struct.unpack memory leak

Mark Dickinson added the comment: Do you see the same results with Python 2.7? Python 2.6 is only receiving security bugfixes at this point. -- nosy: +mark.dickinson ___ Python tracker __

[issue14596] struct.unpack memory leak

New submission from Robert Elsner : When unpacking multiple files with _variable_ length, struct unpack leaks massive amounts of memory. The corresponding functions from numpy (fromfile) or the array (fromfile) standard lib module behave as expected. I prepared a minimal testcase illustrating