[issue19640] Drop _source attribute of namedtuple (waste memory)
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- assignee: - rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple (waste memory)
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- versions: -Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple (waste memory)
Raymond Hettinger added the comment: FWIW, the verbose option is mentioned as outdated because the _source attribute was added. Also, there are real use cases, people are using the _source as writing it to a .py file so that the dynamic namedtuple generation step can be skipped on subsequent imports. This is useful when people want to avoid the use of eval or want to run cython on the code. The attribute can be dropped. It is part of the API. Sorry, the memory use bugs you. It is bigger than typical docstrings but is not a significant memory consumer in most applications. I like the idea of dynamically generating the source upon lookup, but want to think about whether there are any unintended consequences to that space saving hack. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Eric Snow added the comment: It does not necessarily require a metaclass. You can accomplish it using a custom descriptor: class classattr: def __init__(self, getter): self.getter = getter def __get__(self, obj, cls): return self.getter(cls) FWIW, this is a descriptor that may be worth adding somewhere regardless. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple (waste memory)
STINNER Victor added the comment: namedtuple_source.patch: Replace _source attribute wasting memory with a property generating the source on demand. The patch adds also unit test for the verbose attribute (which is public and documented, even it is said to be outdated). The patch removes also repr_fmt and num_fields parameters of the class definition template, compute these values using the list of fields. I suggested to change Python 3.4.1 and 3.5. Test script: --- import email import http.client import pickle import test.regrtest import test.test_os import tracemalloc import xmlrpc.server snap = tracemalloc.take_snapshot() with open(dump.pickle, wb) as fp: pickle.dump(snap, fp, 2) --- With the patch, the memory footprint is reduced by 176 kB. -- keywords: +patch title: Drop _source attribute of namedtuple - Drop _source attribute of namedtuple (waste memory) type: - resource usage versions: +Python 3.5 Added file: http://bugs.python.org/file34484/namedtuple_source.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple (waste memory)
Eric Snow added the comment: Also be sure the have Raymond's sign-off before committing anything for this. :) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
STINNER Victor added the comment: As an alternative, how about turning _source into a property? A class or an instance property? A class property requires a metaclass. I guess that each namedtuple type requires its own metaclass, right? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Changes by Éric Araujo mer...@netwok.org: -- nosy: +eric.araujo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Éric Araujo added the comment: In a first version namedtuple had an argument (named echo or verbose) that would cause the source code to be printed out, for use at the interactive prompt. Raymond later changed it to a _source attribute, more easy to work with than printed output. About the other question you asked on the ML (why isn’t there a base NamedTuple class to inherit): this has been discussed on python-ideas IIRC, and people have written ActiveState recipes for that idea. It should be easy to find the ML archive links from the ActiveState posts. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Eric Snow added the comment: A while back, because of those python-ideas discussions, Raymond added a link at the bottom of the namedtuple section of the docs at http://docs.python.org/3.4/library/collections.html#namedtuple-factory-function-for-tuples-with-named-fields. The link points to a nice recipe by Jan Kaliszewski. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Eric Snow added the comment: As an alternative, how about turning _source into a property? -- nosy: +eric.snow ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
New submission from STINNER Victor: The definition of a new nametuple creates a large Python script to create the new type. The code stores the code in a private attribute: namespace = dict(__name__='namedtuple_%s' % typename) exec(class_definition, namespace) result = namespace[typename] result._source = class_definition This attribute wastes memory, I don't understand the purpose of the attribute. It was not discussed in an issue, so I guess that there is no real use case: changeset: 68879:bffdd7e9265c user:Raymond Hettinger pyt...@rcn.com date:Wed Mar 23 12:52:23 2011 -0700 files: Doc/library/collections.rst Lib/collections/__init__.py Lib/test/test_collections.py description: Expose the namedtuple source with a _source attribute. Can we just drop this attribute to reduce the Python memory footprint? -- messages: 203270 nosy: haypo, rhettinger priority: normal severity: normal status: open title: Drop _source attribute of namedtuple versions: Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
STINNER Victor added the comment: I found this issue while using my tracemalloc module to analyze the memory consumption of Python. On the Python test suite, the _source attribute is the 5th line allocating the memory memory: /usr/lib/python3.4/collections/__init__.py: 676.2 kB -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Changes by Christian Heimes li...@cheimes.de: -- nosy: +christian.heimes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
Changes by Eric V. Smith e...@trueblade.com: -- nosy: +eric.smith ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19640] Drop _source attribute of namedtuple
STINNER Victor added the comment: the 5th line allocating the memory memory oops, the 5th line allocating the *most* memory -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19640 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
