Changes by Jesús Cea Avión j...@jcea.es:
--
nosy: +jcea
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
___
Python-bugs-list mailing list
Jim Jewett added the comment:
These comments refer to
http://hg.python.org/cpython/file/5c9af8194d3b/Doc/library/tracemalloc.rst
which is newer than the current patches.
I believe I have consolidated the still-open issues that I see (including those
that aren't mine) for this file:
Jim Jewett added the comment:
Drat: forgot one at line 277
.. function:: get_traced_memory()
Get the current size and maximum size of memory blocks traced by the
:mod:`tracemalloc` module as a tuple: ``(size: int, max_size: int)``.
I have a tendency to read maximum size as the most it
STINNER Victor added the comment:
@Jim: This issue has been closed, please don't comment closed issues. I created
a new issue for your new comments: #19818.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
STINNER Victor added the comment:
The code has been merged. I didn't see any test_tracemalloc on buildbots. I
tried to address all remarks on the Rietveld reviews. So I'm now closing the
issue.
Please open new issue if you have more remarks. For example, I opened:
- #19798: tracemalloc:
STINNER Victor added the comment:
Here is the patch to tidy up the Lib/test_tracemalloc.py.
It fixed the typo, and removed unused import and unused variables.
Thanks, I applied your patch.
changeset: 87547:841dec769a04
tag: tip
user:Victor Stinner victor.stin...@gmail.com
STINNER Victor added the comment:
Here is the patch to tidy up the Lib/test_tracemalloc.py.
It fixed the typo, and removed unused import and unused variables.
Thanks, I applied your patch.
Oh, except:
-data = [allocate_bytes(123) for count in range(1000)]
+
STINNER Victor added the comment:
@neologix: I run test_tracemalloc on IRIX64 silicon 6.5 07202013 IP35 (our
IRIX buildbot) which uses a 32-bit MIPS CPU, and the test pass. The code was
compiled with gcc 3.4.6. So packed structures works also on 32-bit MIPS. Well,
it is not surprising, int
Roundup Robot added the comment:
New changeset 2da28004dfac by Victor Stinner in branch 'default':
Issue #18874: tracemalloc: explain the purpose of get_traces.tracebacks in a
comment
http://hg.python.org/cpython/rev/2da28004dfac
New changeset b6414aa8cf77 by Victor Stinner in branch
Roundup Robot added the comment:
New changeset a2811425dbde by Victor Stinner in branch 'default':
Issue #18874: allow to call tracemalloc.Snapshot.statistics(cumulative=True)
http://hg.python.org/cpython/rev/a2811425dbde
--
___
Python tracker
Roundup Robot added the comment:
New changeset cf2c4cd08dd9 by Victor Stinner in branch 'default':
Issue #18874: tracemalloc: Comment the trace_t structure
http://hg.python.org/cpython/rev/cf2c4cd08dd9
New changeset f06a50c2bf85 by Victor Stinner in branch 'default':
Issue #18874: make it more
Vajrasky Kok added the comment:
Here is the patch to tidy up the Lib/test_tracemalloc.py.
It fixed the typo, and removed unused import and unused variables.
--
nosy: +vajrasky
Added file: http://bugs.python.org/file32828/minor_makeup_test_tracemalloc.patch
Roundup Robot added the comment:
New changeset 6e2089dbc5ad by Victor Stinner in branch 'default':
Issue #18874: Implement the PEP 454 (tracemalloc)
http://hg.python.org/cpython/rev/6e2089dbc5ad
--
___
Python tracker rep...@bugs.python.org
Roundup Robot added the comment:
New changeset 66db0c66a6ee by Victor Stinner in branch 'default':
Issue #18874: Remove tracemalloc.set_traceback_limit()
http://hg.python.org/cpython/rev/66db0c66a6ee
--
___
Python tracker rep...@bugs.python.org
Charles-François Natali added the comment:
Victor, is the attached patch up-to-date?
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
Yes, it is up to date. The only difference in the mercurial repository are
the new location of the two hashtable files.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
STINNER Victor added the comment:
Oh, Jim Jewett found a bug: if set_traceback_limit() is called while
tracemalloc is tracing, two tracebacks are seen different because their length
is different, whereas the full traceback would be the same.
To avoid this issue, I propose to add an optional
Charles-François Natali added the comment:
If you really want to use packing, keep it.
But please remove this:
+/* ensure that the frame_t structure is packed */
+assert(sizeof(frame_t) == (sizeof(PyObject*) + sizeof(int)));
--
___
Python
STINNER Victor added the comment:
If you really want to use packing, keep it.
But please remove this:
+/* ensure that the frame_t structure is packed */
+assert(sizeof(frame_t) == (sizeof(PyObject*) + sizeof(int)));
I added this assertion to ensure that I used correct GCC
STINNER Victor added the comment:
3de17d13002d.patch:
- take in account Antoine Pitrou's and vajrasky's comments
- pack again frame_t structure (without the assertion and with a comment)
--
Added file: http://bugs.python.org/file32753/3de17d13002d.patch
STINNER Victor added the comment:
Oh yes, Antoine also asked if Python/ and Include/ directories are the right
place for hashtable.c and hashtable.h files. _Py_slist and _Py_hashtable are
only used in Modules/_tracemalloc.c yet.
It was discussed to reuse _Py_slist somewhere (I don't remember
STINNER Victor added the comment:
It was discussed to reuse _Py_slist somewhere (I don't remember where) (...)
I did a quick search in Python code based using link.*list regex.
Structures linked in a single-linked list (candidate for _Py_slist):
* Modules/_curses_panel.c: list_of_panels
*
STINNER Victor added the comment:
Oh, I just realized _Py_slist_init(), _Py_slist_prepend(), and
_Py_slist_remove() functions are currently private (static in hashtable.c).
These functions should be declared in hashtable.h if _Py_slist API is reused.
--
STINNER Victor added the comment:
Reuse _Py_hashtable for the methods cache is not trivial, the API doesn't fit
exactly. I didn't spend much time to try to adapt _Py_hashtable API or the
methods cache.
I don't understand which part is the key and which part is the enetry data in
Eric Snow added the comment:
FYI, the C OrderedDict implementation in #16991 implements its own
doubly-linked list, built around the needs of OrderedDict. I looked into BSD's
queue.h [1], but it ended up simpler to roll my own.
[1]
Charles-François Natali added the comment:
STINNER Victor added the comment:
I'm not convinced myself that hashtable.c/h can be reused immediatly, so I
prefer to move these two files to Modules. The files may be moved later if
the containers are reused.
Please do so. I'd also like to
STINNER Victor added the comment:
FYI, the C OrderedDict implementation in #16991 implements its own
doubly-linked list, built around the needs of OrderedDict. I looked into
BSD's queue.h [1], but it ended up simpler to roll my own.
Oh nice, it's probably possible to factorize code and
STINNER Victor added the comment:
I'm not convinced myself that hashtable.c/h can be reused immediatly, so I
prefer to move these two files to Modules. The files may be moved later if the
containers are reused.
--
___
Python tracker
Charles-François Natali added the comment:
I made a review at http://bugs.python.org/review/18874/#ps9860 (not sure you
got notified).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file32549/3bf73dcd0b42.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file32547/69fd2d766005.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
Patch updated for Charles François's comments.
--
Added file: http://bugs.python.org/file32724/4430e893d89f.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
STINNER Victor added the comment:
Charles-François doesn't like packed structure (frame_t) because he fears crash
on architectures not supporting non-aligned memory access or bad performances.
Antoine and me want them to reduce the memory footprint of the tracemalloc
module
STINNER Victor added the comment:
pack.patch: Patch to pack frame_t structure.
I tested tracemalloc with packed structure on Linux/AMD64, FreeBSD/AMD64,
OpenBSD/AMD64, OpenIndiana/AMD64, Windows/x86 (Windows 7 64-bit with Python
compiled in 32-bit). I don't have access to SPARC.
--
Charles-François Natali added the comment:
Would it be possible to generate a clean patch?
The latest one contains many unrelated commits.
--
nosy: +neologix
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
STINNER Victor added the comment:
2013-11-09 06:53:03 neologixset files: + 3bf73dcd0b42.diff
I guess that you clicked on [Create Patch]. This button doesn't work, it
generates a patch with unrelated changes. Maybe I did something wrong on
features/tracemalloc repository.
Here
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file32494/022955935ba3.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
Patch updated to the latest version of the PEP 454.
--
Added file: http://bugs.python.org/file32547/69fd2d766005.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Changes by Charles-François Natali cf.nat...@gmail.com:
Added file: http://bugs.python.org/file32549/3bf73dcd0b42.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Added file: http://bugs.python.org/file32494/022955935ba3.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file32478/65e72bf01246.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file32211/57ae01bf96cb.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
Updated patch to the latest implementation: 65e72bf01246.patch
The developement is now done in the start branch of
http://hg.python.org/features/tracemalloc
--
Added file: http://bugs.python.org/file32478/65e72bf01246.patch
STINNER Victor added the comment:
Update the patch to the last implementation.
--
Added file: http://bugs.python.org/file32211/57ae01bf96cb.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file31976/ec121a72e848.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Roundup Robot added the comment:
New changeset bea4447c22bf by Victor Stinner in branch 'default':
Issue #18874: PyCode_New() now ensures that the filename is a ready Unicode
http://hg.python.org/cpython/rev/bea4447c22bf
New changeset ba27cba3ae20 by Victor Stinner in branch 'default':
Issue
Changes by Antoine Pitrou pit...@free.fr:
--
nosy: +tim.peters
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
___
Python-bugs-list
Raymond Hettinger added the comment:
I recommend that tracemalloc focus exclusively on real allocations and ignore
freelisting. The former are interesting because they directly affect
performance. Freelists on the other hand are close to being free (as in beer).
Measuring the freelist
STINNER Victor added the comment:
2013/10/6 Raymond Hettinger rep...@bugs.python.org:
I recommend that tracemalloc focus exclusively on real allocations and ignore
freelisting. The former are interesting because they directly affect
performance. Freelists on the other hand are close to
STINNER Victor added the comment:
If you use hash table code from another project, be sure to check its
licensing.
Done: see
https://mail.python.org/pipermail/python-legal-sig/2013-September/44.html
I contacted the original author and python-legal mailing list. The BSD
license asks to
STINNER Victor added the comment:
ec121a72e848.patch: updated patch, based on revision ec121a72e848 of the
tracemalloc repository.
--
Added file: http://bugs.python.org/file31976/ec121a72e848.patch
___
Python tracker rep...@bugs.python.org
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file31806/21f7c3df0f15.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
TODO list:
* unit test for task reschedule
* hash_destroy(): use a double-linked list to avoid the O(n) complexity?
* Snapshot.add_process_memory_metrics():
* rename metrics on Windows?
* more metrics on Linux?
* implement get_process_memory() on BSD
*
STINNER Victor added the comment:
ec121a72e848.patch:
- ignore changes on Lib/test/support/__init__.py: it's my own fix for issue
#18948 which will be fixed differently
- ignore changes on Lib/test/regrtest.py: they should not be commited, it's
just a convinient way to test tracemalloc
STINNER Victor added the comment:
The issue #16742 must be fixed to be able to trace memory blocks allocated by
PyMem_RawMalloc().
--
dependencies: +PyOS_Readline drops GIL and calls PyOS_StdioReadline, which
isn't thread safe
___
Python tracker
STINNER Victor added the comment:
For the development, it would also be nice to fix #18948.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
Updated patch.
--
Added file: http://bugs.python.org/file31805/5d8817cc9e69.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file31545/tracemalloc-2.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Added file: http://bugs.python.org/file31806/21f7c3df0f15.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file31805/5d8817cc9e69.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
Changes by STINNER Victor victor.stin...@gmail.com:
Removed file: http://bugs.python.org/file31517/tracemalloc.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
Include/tracemalloc.h: PyTraceMalloc_DisableTemporary() and
PyTraceMalloc_RestoreTemporary() should be removed, they were tests to try to
fix issues with subinterpreter when tracing PyMem_RawMalloc().
Lib/test/regrtest.py: These changes should not be
STINNER Victor added the comment:
TODO list:
* remove dependency to psutil
* enable PYMEM_RAW: fix subinterpreter issue, check which variables are
protected by which lock
* test the command line interface?
* don't trace objects created by get_stats(), reentrant flag should be a
thread-local
Changes by Alexander Belopolsky alexander.belopol...@gmail.com:
--
nosy: +belopolsky
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
___
STINNER Victor added the comment:
Oh, another TODO: drop the (optional) dependency to psutil. Implement
get_process_memory() in C (in the _tracemalloc module).
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
STINNER Victor added the comment:
New patch (version 2):
- an hash table entry can now contain the data directly instead of a pointer to
the data
- _tracemalloc._get_stats() copies the hash table to not have to lock the hash
table too long, which will be especially useful for TRACE_RAW_MALLOC
STINNER Victor added the comment:
Ok, let's start with a first patch. It works in the common cases, because they
are some corner cases like subinterpreter which might still crash.
The hook on PyMem_RawMalloc() takes the GIL. It is disabled because it has
still bugs (it introduces a deadlock
Amaury Forgeot d'Arc added the comment:
Is it really impossible to use a standard Python dict instead of cfuhash?
--
nosy: +amaury.forgeotdarc
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
STINNER Victor added the comment:
Is it really impossible to use a standard Python dict instead of cfuhash?
I don't know if it is possible or not, I didn't try. Using PyDict has many
requirements:
* the GIL must be held: I plan to hook also PyMem_Raw (which is called without
the GIL being
New submission from STINNER Victor:
Thanks to the PEP 445, it becomes possible to trace easily memory allocations.
I propose to add a new tracemalloc module computing the memory usage per file
and per line number.
I implemented the module as a third party module for Python 2.5-3.4, but it
STINNER Victor added the comment:
Example of output:
$ PYTHONTRACEMALLOC=1 ./python -m test
...
== CPython 3.4.0a1+ (default:2ce9e5f6b47c+, Aug 29 2013, 02:03:02) [GCC 4.7.2
20121109 (Red Hat 4.7.2-8)]
== Linux-3.9.4-200.fc18.x86_64-x86_64-with-fedora-18-Spherical_Cow
little-endian
==
STINNER Victor added the comment:
tracemalloc.get_process_memory() should be moved to the shutil module.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue18874
___
STINNER Victor added the comment:
I'm not sure that the DisplayGarbage class is still useful. The PEP 442 has
been implemented in Python 3.4, so objects with __del__() method are no more
uncollectable. I don't know which kind of objects can enter the gc.garbage
list (by default, not using
73 matches
Mail list logo
