[issue10261] tarfile iterator without members caching

Lars Gustäbel added the comment: Closing after six years of inactivity. -- resolution: -> wont fix stage: -> resolved status: open -> closed ___ Python tracker ___

[issue10261] tarfile iterator without members caching

Lars Gustäbel added the comment: There is no trivial or backwards-compatible solution to this problem. The way it is now, there is no alternative to storing all TarInfo objects: there is no central table of contents in an archive we could use, so we must create our own. In other words, tarfil

[issue10261] tarfile iterator without members caching

Changes by Antoine Pitrou : -- type: feature request -> resource usage versions: +Python 2.7, Python 3.1, Python 3.2 ___ Python tracker ___ __

[issue10261] tarfile iterator without members caching

Karsten Wolf added the comment: Yes, I'm on 2.6. I checked the Python 3.x tarfile just for this one line in TarFile.next(): self.members.append(tarinfo) to conclude it would have the same problem. Reducing 2.5gb memory usage as measured in my particular case by 60%, still leaves 1.5gb ram b

[issue10261] tarfile iterator without members caching

Lars Gustäbel added the comment: I assume you're using Python 2.x. because tarfile's memory footprint was significantly reduced in Python 3.0, see the patch in issue2058 and r62337. This patch was not backported to the 2.x branch back then. As the 2.x branch has been closed for new features,

[issue10261] tarfile iterator without members caching

New submission from Karsten Wolf : It would be helpful to have a tarfile iterator that does not cache every archive member encountered. This makes it nearly impossible to iterate over an archive with millions of files. -- components: Library (Lib) messages: 120041 nosy: karstenw prior