[issue19048] itertools.tee doesn't have a __sizeof__ method
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- resolution: - rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Martin v. Löwis added the comment: Antoine: in (my experience of) memory analysis, the size of a single object is mostly irrelevant. If you need to know how much memory something consumes, you typically want to know the memory of a set of objects. So this is the case that really must be supported. For that, users will have to use libraries that know how to count memory. It's not asked to much that the authors of such libraries know about internals of Python (such as the existence of sys.getsizeof, or gc.get_referents). The question is: can such a library reasonably implemented? For that, it is important that getsizeof behaves uniformly across objects. If you really don't like the proposed uniformity, please propose a different rule. However, don't give deviations in other places (OrderedDict) as a reason to break the rule here as well. Instead, if OrderedDict.__getsizeof__ is broken, it needs to be fixed as well. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Changes by Raymond Hettinger raymond.hettin...@gmail.com: -- assignee: - rhettinger versions: -Python 3.3 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Raymond Hettinger added the comment: getsizeof() is interesting only if it gives sensible results when used correctly, especially if you want to sum these values and get a global memory usage. If accounting for global memory usage is a goal, it needs to have a much more comprehensively thought out, implementation dependent approach. There are many issues (memory fragmentation, key-sharing dictionaries, dummy objects, list over-allocation, the minsize dictionary that is part of the dict object in addition to its variable sized portion, non-python objects held by Python objects, the extra few bytes per object consumed by the freelisting scheme in Objects/obmalloc.c etc). The thing is, Total size is generally meaningless. I concur. This is a pipe dream without a serious investment of time and without creating a new and unnecessary maintenance burden. (By the way, OrderedDict.__sizeof__ already breaks the rule you are trying to impose) FWIW, the way OrderedDict computes sizeof is probably typical of how anyone is currently using sys.getsizeof(). If you change the premise of how it operates, you're probably going to break the code written by the very few people in the world who care about sys.getsizeof(): def __sizeof__(self): sizeof = _sys.getsizeof n = len(self) + 1 # number of links including root size = sizeof(self.__dict__)# instance dictionary size += sizeof(self.__map) * 2 # internal dict and inherited dict size += sizeof(self.__hardroot) * n # link objects size += sizeof(self.__root) * n # proxy objects return size I don't have any specific recommendation for itertools.tee other than that I think it doesn't really need a __sizeof__ method. The typical uses of tee are a transient phenomena that temporarily use some memory and then disappear. I'm not sure that any mid-stream sizeof checks reveal information of any worth. Overall, this thread indicates that the entire concept of __sizeof__ has been poorly defined, unevenly implemented, and not really useful when aggregated. For those who are interested in profiling and optimizing Python's memory usage, I think we would be much better off providing a memory allocator hook that can know about every memory allocation and how those allocations have been arranged (revealing the fragmentation of the unused memory in the spaces between). Almost anything short of that will provide a grossly misleading picture of memory usage. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
New submission from Antoine Pitrou: An itertools.tee object can cache an arbitrary number of objects (pointers), but its sys.getsizeof() value will always remain the same. -- components: Library (Lib) messages: 198048 nosy: pitrou, rhettinger, serhiy.storchaka priority: normal severity: normal stage: needs patch status: open title: itertools.tee doesn't have a __sizeof__ method type: behavior versions: Python 3.3, Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: I find the implementation of itertools.tee a bit weird: why does teedataobject have to be a PyObject? It seems to complicate things and make them less optimal. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: Anywhere, here is a patch. -- keywords: +patch Added file: http://bugs.python.org/file31813/tee_sizeof.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: This is a duplicate of issue15475. -- superseder: - Correct __sizeof__ support for itertools ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: Hmm, no, I'm wrong, it's not a duplication. -- superseder: Correct __sizeof__ support for itertools - ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Changes by Serhiy Storchaka storch...@gmail.com: -- stage: needs patch - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: I'm not sure that sys.getsizeof() should recursively count all Python subobjects. That is why I had omitted tee() in my patch. sys.getsizeof([[]]) 36 sys.getsizeof([list(range(1))]) 36 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: I'm not sure that sys.getsizeof() should recursively count all Python subobjects. Those are private subobjects. They are not visible to the programmer (except perhaps by calling __reduce__ or __setstate__). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment:
Thy are visible by calling gc.get_referents(). High-level function can use this
to count recursive size of objects.
import sys, gc, itertools
def gettotalsizeof(*args, seen=None):
... if seen is None:
... seen = {}
... sum = 0
... for obj in args:
... if id(obj) not in seen:
... seen[id(obj)] = obj
... sum += sys.getsizeof(obj)
... sum += gettotalsizeof(*gc.get_referents(obj), seen=seen)
... return sum
...
a, b = tee(range(1))
sum(next(a) for i in range(1000))
499500
gettotalsizeof(a)
750
gettotalsizeof(b)
18734
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19048
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: Thy are visible by calling gc.get_referents(). That's totally besides the point. The point is that those objects are invisible in normal conditions, not that they can't be read using advanced implementation-dependent tricks. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: The point is that your patch breaks functions like gettotalsizeof(). It makes impossible to get a total size of general object. It will be better to add gettotalsizeof() to the stdlib (or add an optional parameter to sys.getsizeof() for recursive counting). -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: The point is that your patch breaks functions like gettotalsizeof(). It makes impossible to get a total size of general object. The thing is, Total size is generally meaningless. It can include things such as the object's type, or anything transitively referenced by the object, such as modules. It will be better to add gettotalsizeof() to the stdlib (or add an optional parameter to sys.getsizeof() for recursive counting). This patch has *nothing* to do with recursive counting. It counts the internal arrays of itertools.tee() as part of its memory size, which is reasonable and expected. It does *not* count memory recursively: it doesn't count the size of the itertools.tee()'s cached objects, for example. Recursive counting doesn't make sense with Python. Where do you stop counting? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Amaury Forgeot d'Arc added the comment: I like the definition of __sizeof__ that was discussed some time ago: http://bugs.python.org/issue14520#msg157798 With that definition (do we have it somewhere in the docs, by the way?) The current code works gives the correct answer. -- nosy: +amaury.forgeotdarc ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: Isn't sys.getsizeof() a low-level debugging tool? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: Isn't sys.getsizeof() a low-level debugging tool? What would it help debug exactly? :-) I would hope it gives remotely useful information about the passed object. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Amaury Forgeot d'Arc added the comment: getsizeof() is interesting only if it gives sensible results when used correctly, especially if you want to sum these values and get a global memory usage. One usage is to traverse objects through gc.get_referents(); in this case the definition above is correct. Now, are you suggesting to traverse objects differently? With dir(), or __dict__? (btw, this discussion explains why pypy still does not implement getsizeof()) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: Well. itertools._tee is one Python object and itertools._tee_dataobject is another Python object. sys.getsizeof() gives you the memory usage of this objects separately. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment:
How do you stop walking your graph if it spans the whole graph of Python
objects?
We can stop at specific types of objects (for example types and modules).
If sys.getsizeof() is only useful for people who know *already* how an
object is implemented internally, then it's actually useless, because
those people can just as well do the calculation themselves.
It's why sys.getsizeof() is a low-level tool. We need high-level tool in the
stdlib. Even imperfect recursive counting will be better than confusing for
novices sys.getsizeof().
(By the way, OrderedDict.__sizeof__ already breaks the rule you are trying to
impose)
Yes, I know, and I think it is wrong.
Here is improved version of gettotalsizeof():
def gettotalsizeof(*args, exclude_types=(type, type(sys))):
seen = {}
stack = []
for obj in args:
if id(obj) not in seen:
seen[id(obj)] = obj
stack.append(obj)
sum = 0
while stack:
obj = stack.pop()
sum += sys.getsizeof(obj)
for obj in gc.get_referents(obj):
if id(obj) not in seen and not isinstance(obj, exclude_types):
seen[id(obj)] = obj
stack.append(obj)
return sum
gettotalsizeof(sys)
206575
gettotalsizeof(gc)
2341
gettotalsizeof(sys.getsizeof)
60
gettotalsizeof(gettotalsizeof)
60854
class C: pass
...
gettotalsizeof(C)
805
gettotalsizeof(C())
28
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19048
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: Here is why using get_referents() is stupid in the general case: class C: pass ... c = C() gc.get_referents(c) [class '__main__.C'] With your method, measuring c's memory consumption also includes the memory consumption of its type object. (and of course this is only a trivial example... one can only imagine what kind of mess it is with a non-trivial object) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: (By the way, OrderedDict.__sizeof__ already breaks the rule you are trying to impose) -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: Well. itertools._tee is one Python object and itertools._tee_dataobject is another Python object. sys.getsizeof() gives you the memory usage of this objects separately. This is great... And how do I know that I need to use gc.get_referents() to get those objects in case I'm measuring the memory consumption of a teeobject (rather than, say, trusting __dict__, or simply trusting the getsizeof() output at face value)? If sys.getsizeof() is only useful for people who know *already* how an object is implemented internally, then it's actually useless, because those people can just as well do the calculation themselves. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: It's why sys.getsizeof() is a low-level tool. We need high-level tool in the stdlib. Even imperfect recursive counting will be better than confusing for novices sys.getsizeof(). Ok, but I need to see a satisfying version of gettotalsizeof before I'm convinced (see below). Here is improved version of gettotalsizeof(): [...] gettotalsizeof(gettotalsizeof) 60854 Why that big? Does it make sense? What if say, a large object is shared between many small objects? Should it count towards the memory size of any of those small objects? What if that object is actually immortal (it is also a module global, for example)? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: Optionally we can also not count objects which are referenced from outside of a graph of objects (this isn't so easy implement in Python). I.e. gettotalsizeof([1, 'abc', math.sqrt(22)], inner=True) will count only bare list and a square of 22, because 1 and 'abc' are interned. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Martin v. Löwis added the comment: The problem is that that definition isn't helpful. If we ever change itertools.tee to use non-PyObjects internally, s suddenly its sys.getsizeof() would have to return much larger numbers despite visible behaviour not having changed at all (and despite the memory overhead being actually lower). I see no problem with that. If the internal representation changes, nobody should be surprised if sizeof changes. I would hope it gives remotely useful information about the passed object. It certainly does: it reports the memory consumption of the object itself, not counting the memory of other objects. I proposed a precise definition of what an other object is. If you don't like it, please propose a different definition that still allows to automatically sum up the memory of a graph of objects. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: I see no problem with that. If the internal representation changes, nobody should be surprised if sizeof changes. Who is nobody? Users aren't aware of internal representation changes. It sounds like you want sys.getsizeof() to be a tool for language implementors anyway. I proposed a precise definition of what an other object is. If you don't like it, please propose a different definition that still allows to automatically sum up the memory of a graph of objects. What is the use case for summing up the memory of a graph of objects? How do you stop walking your graph if it spans the whole graph of Python objects? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: Optionally we can also not count objects which are referenced from outside of a graph of objects (this isn't so easy implement in Python). I.e. gettotalsizeof([1, 'abc', math.sqrt(22)], inner=True) will count only bare list and a square of 22, because 1 and 'abc' are interned. That's only part of the equation. What if I have an object which references, for example, a logging.Logger? Loggers are actually eternal (they live in a global dictionary somewhere in the logging module), but gettotalsizeof() will still count it. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: getsizeof() is interesting only if it gives sensible results when used correctly, especially if you want to sum these values and get a global memory usage. Getting a global memory usage isn't a correct use of getsizeof(), though, because it totally ignores the memory allocation overhead (not to mention fragmentation, or any memory areas that may have been allocated without being accounted for by __sizeof__). If you want global Python memory usage, use sys._debugmallocstats(), not sys.getsizeof(). One usage is to traverse objects through gc.get_referents(); in this case the definition above is correct. What are the intended semantics? get_referents() can give you references you didn't expect, such as type objects, module objects... Now, are you suggesting to traverse objects differently? With dir(), or __dict__? sys.getsizeof() gives you the memory usage of a given Python object, it doesn't guarantee that traversing objects will give you the right answer for any question. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Antoine Pitrou added the comment: I like the definition of __sizeof__ that was discussed some time ago: http://bugs.python.org/issue14520#msg157798 The problem is that that definition isn't helpful. If we ever change itertools.tee to use non-PyObjects internally, suddenly its sys.getsizeof() would have to return much larger numbers despite visible behaviour not having changed at all (and despite the memory overhead being actually lower). And gc.get_referents() is really a low-level debugging tool, certainly not a reflection API (inspect would serve that role). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19048] itertools.tee doesn't have a __sizeof__ method
Serhiy Storchaka added the comment: Here is advanced function which counts only objects on which there are no external references. import itertools a, b = itertools.tee(range(1)) max(zip(a, range(100))) (99, 99) sys.getsizeof(a) 32 gettotalinnersizeof(a) 32 gettotalinnersizeof(b) 292 gettotalinnersizeof(a, b) 608 Total size of a and b is larger than a sum of sizes of a and b. It's because it includes size of one shared between a and teedataobject and one shared range iterator. -- Added file: http://bugs.python.org/file31822/gettotalsizeof.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19048 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
