[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file26588/stringio_sizeof-2.7.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file26587/stringio_sizeof-3.2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file26810/stringio_sizeof-3.3_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file26586/stringio_sizeof-3.3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file26811/stringio_sizeof-3.2_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file26812/stringio_sizeof-2.7_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Alexandre Vassalotti alexan...@peadrop.com: Removed file: http://bugs.python.org/file27241/stringio_sizeof-3.2_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Hynek Schlawack h...@ox.cx: -- nosy: -hynek ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: -- versions: +Python 3.4 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment: Patches updated. Now the computations done in size_t. Note that the patches for the different versions differ. -- Added file: http://bugs.python.org/file27240/stringio_sizeof-3.3_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file27241/stringio_sizeof-3.2_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file27242/stringio_sizeof-2.7_3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Martin v. Löwis added the comment: I disagree that sizeof cannot work well for variable-sized types. It works very well for strings, lists, tuple, dicts, and other regular containers. I agree that it is not important that it is absolutely correct (in some sense) for every object, but it shouldn't lose big chunks of data. A bug where it misses four bytes is much less important than a bug where it misses N bytes (for an object-specific value N that can grow indefinitely). As for the specific patch, I don't think any action should be taken before the 3.3 release. I would personally prefer if the computations where done in Py_ssize_t, not PyObject* (i.e. the result of the recursive call should be unwrapped). -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment:
It can't work well if we count internal Python objects that can be shared. This
is because the work well concept is not well defined. And because the
implementation of a certain defined calculation algorithm can be completely
unmaintainable, that is not well.
If we wrote in the StringIO the same 1 MB string twice, should the __sizeof__()
return about 1) 2 MB, 2) 1 MB or 3) size of empty stream if there are external
links to this shared string? Patch implements the second strategy, it can be
simplified to the first one or complicated even more to a third one. Even more
complication will need to take into account the sharing of eol string ('\r' and
'\n' always shared, '\r\n' may be).
I would personally prefer if the computations where done in Py_ssize_t, not
PyObject*
I too. But on platforms with 64-bit pointers and 32-bit sizes we can allocate
total more than PY_SIZE_MAX bytes (hey, I remember the DOS memory models with
16-bit size_t and 32-bit pointers). Even faster we get an overflow if allow the
repeated counting of shared objects. What to do with overflow? Return
PY_SIZE_MAX or ignore the possibility of errors?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15490
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Martin v. Löwis added the comment: Am 17.09.2012 14:26, schrieb Serhiy Storchaka: I would personally prefer if the computations where done in Py_ssize_t, not PyObject* I too. But on platforms with 64-bit pointers and 32-bit sizes we can allocate total more than PY_SIZE_MAX bytes (hey, I remember the DOS memory models with 16-bit size_t and 32-bit pointers). Even faster we get an overflow if allow the repeated counting of shared objects. What to do with overflow? Return PY_SIZE_MAX or ignore the possibility of errors? It can never overflow. We cannot allocate more memory than SIZE_MAX; this is (mostly) guaranteed by the C standard. I don't know whether you deliberately brought up the obscure case of 64-bit pointers and 32-bit sizes. If there are such systems, we don't support them. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment: It would be good if someone looked at the patches. I'm not sure that they are good enough. Perhaps they are too complicated and we shouldn't worry about duplicates? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Antoine Pitrou added the comment: Well, I do think that it's not very important to make __sizeof__ strictly exact according to some specification. I'm also not sure we want to complicate maintenance when the calculation becomes tedious or difficult. To me __sizeof__ is mostly an educational tool which gives a hint at the memory impact of an object, but cannot really convey a precise information except for trivial atomic types. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Aaron Iles aaron.i...@gmail.com: -- nosy: +aliles ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: storchaka - keywords: +needs review stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment: Patch updated. Now private pyobjects (readnl, accu) counted. Note all three patches rather different. -- Added file: http://bugs.python.org/file26810/stringio_sizeof-3.3_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file26811/stringio_sizeof-3.2_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file26812/stringio_sizeof-2.7_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: -- assignee: - storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment: Do I understand correctly that for each Python subobject must be one of the following: 1) or it is visited in tp_traverse; 2) or it is accounted in __sizeof__ together with all its subobjects (if they cannot leak out)? PyAccu can contains not only Unicode objects, but at least an instance of a Unicode subtype, which can has a reference to StringIO object, creating a cycle. Does this mean that PyAccu should be visited in tp_traverse and not be accounted in __sizeof__? Or should we tighten the control and ensure that PyAccu contains only exact Unicode objects? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Antoine Pitrou added the comment: Does this mean that PyAccu should be visited in tp_traverse and not be accounted in __sizeof__? Or should we tighten the control and ensure that PyAccu contains only exact Unicode objects? IMO, the latter. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Jesús Cea Avión j...@jcea.es: -- nosy: +jcea ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment:
For the PyAccu, AFAICT, objects cannot leak out of it (except for
gc.getobjects in debug mode).
Not only in debug mode.
import io, gc
s=io.StringIO()
s.write('12345')
5
s.write('qwerty')
6
for o in gc.get_objects():
... if '123 in repr(o) and len(repr(o)) 1000:
... print(type(o), repr(o))
...
class 'list' ['12345', 'qwerty']
class 'list' ['o', 'gc', 'get_objects', 'repr', 'o', '123, 1000, 'len',
'repr', 'o', 'print', 'type', 'o', 'repr', 'o']
class 'tuple' ('123, 1000, None)
Someone can summarize sys.getsizeof() for all managed by GC objects and
therefore count internal objects twice.
I think the standard library should provide a method for recursive calculation
of memory (in some sense, perhaps using different strategies), but that should
wait up to 3.4. __sizeof__ should count non-object memory that is not available
for GC. As I understand it.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15490
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Martin v. Löwis added the comment: For the PyAccu, AFAICT, objects cannot leak out of it (except for gc.getobjects in debug mode). Not only in debug mode. I see. I meant sys.getobjects, which is in debug mode only, but I now see that gc.get_objects will get the list (but not the strings) of the Accu. That still leaves readnl and writenl. I think the standard library should provide a method for recursive calculation of memory (in some sense, perhaps using different strategies), but that should wait up to 3.4. Actually, this is (and should be) a separate project: http://guppy-pe.sourceforge.net/ __sizeof__ should count non-object memory that is not available for GC. As I understand it. I think you misunderstand slightly; the GC relevance is only a side effect. __sizeof__ should only account for memory that isn't separately accessible. The notion of separately accessible is somewhat weak, since it may depend on the container. Non-PyObject blocks clearly need to be accounted for. PyObject blocks normally don't need to be accounted for, as they are typically accessible separately, by the following means: - gc.get_objects (for GC objects) - gc.get_referents (for contained non-GC objects) There are also irregular ways to get objects: - in debug mode only: sys.getobjects - customer access functions or attributes For memory accounting, it would really be best if the latter two categories wouldn't exist, i.e. if all non-GC objects were still visible through tp_traverse. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file26588/stringio_sizeof-2.7.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
New submission from Serhiy Storchaka: Here is a patch that implements correct __sizeof__ for C implementation of io.StringIO. I haven't tested the patch on narrow build. -- components: IO, Library (Lib) files: stringio_sizeof-3.3.patch keywords: patch messages: 166807 nosy: benjamin.peterson, hynek, pitrou, storchaka, stutzbach priority: normal severity: normal status: open title: Correct __sizeof__ support for StringIO type: behavior versions: Python 2.7, Python 3.2, Python 3.3 Added file: http://bugs.python.org/file26586/stringio_sizeof-3.3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Changes by Serhiy Storchaka storch...@gmail.com: Added file: http://bugs.python.org/file26587/stringio_sizeof-3.2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Antoine Pitrou added the comment: Here is a patch that implements correct __sizeof__ for C implementation of io.StringIO. For some value of correct, since the internal accumulator could hold alive some unicode strings. I haven't tested the patch on narrow build. There's no narrow build anymore on 3.3. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Serhiy Storchaka added the comment: For some value of correct, since the internal accumulator could hold alive some unicode strings. This is not a concern of __sizeof__, because these lists and strings are managed by GC. There's no narrow build anymore on 3.3. I mean 2.7 and 3.2. I have written test for such case, but can't check if it correct. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Antoine Pitrou added the comment: For some value of correct, since the internal accumulator could hold alive some unicode strings. This is not a concern of __sizeof__, because these lists and strings are managed by GC. It is, since they are private and not known by the user. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15490] Correct __sizeof__ support for StringIO
Martin v. Löwis added the comment: The question really is what memory blocks can leak out of the object, and those don't really belong to the container. Those shouldn't be accounted for, since somebody else may get hold of them, and pass them to sys.sizeof. For the PyAccu, AFAICT, objects cannot leak out of it (except for gc.getobjects in debug mode). So I think the memory allocated by the accu really belongs to the StringIO, and needs to be accounted there. The same goes for readnl, writenl, and weakreflist. On a related note, I wonder why the tp_clear and tp_traverse functions for stringio are so short. Shouldn't it also visit decoder? -- nosy: +loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue15490 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
