[Python-Dev] Re: Custom memory profiler and PyTraceMalloc_Track
> In other words, the allocators are concerned about > memory, not tracing or anything else that can be done by overriding them. Ok I now understand that `tracemalloc`'s use of allocator APIs is just an implementation detail. Allocator APIs were used for tracing but they are not designed for that: which makes perfect sense. However, I still wonder: is there anyway to support `PyTraceMalloc_Track` API without being dependent to `tracemalloc`? I know there is not many memory tracing tools but I mean I still feel like there should be a generic way of doing this: A very vague example for demonstration: PyMemHooks_GetAllocators/PyMemHooks_SetAllocators/PyMemHooks_TrackAlloc which are public APIs and tracemalloc using them internally instead of allocator APIs? >Regards from rainy London, Cheers from a cold Istanbul, :) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BBHAJQGIUIBRTW7ZWUYHSP3JACLCTIHS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Custom memory profiler and PyTraceMalloc_Track
Hi everyone, I would like to ask a question about an issue that we faced regarding profiling memory usage: We have implemented a custom memory profiler using `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now, we are facing an issue with numpy: numpy seems to have its own memory allocator and they use `PyTraceMalloc_Track` APIs to notify tracemalloc about the allocation. A simple search on GitHub reveals there are more projects using this approach: https://github.com/search?q=PyTraceMalloc_Track=code which is fine. I believe it is common to use custom memory allocators for scientific computation which makes perfect sense. However, I would have expected to have at least some form of `PyMem_NotifyAllocator` type of API instead of a one that is specific to `tracemalloc`? I might be missing some context here. WDYT? Best, ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BHOIDGRUWPM5WEOB3EIDPOJLDMU4WQ4F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Hold multiple tracebacks per-object in tracemalloc
Thanks Victor, I have a better understanding of tracemalloc internals. On Tue, Nov 19, 2019 at 10:07 PM Victor Stinner wrote: > Hi, > > Le mar. 19 nov. 2019 à 18:33, Sümer Cip a écrit : > > First of all, I would like to thank for such a useful tool for debugging > memory issues. I am pretty proud as a Python lover that we have such a > tool: tracemalloc:) > > As the module author: you're welcome ;-) > > > > AFAIU, tracemalloc holds a single traceback per-object: > > "tracemalloc" name means that a "trace" is stored in mapping > associated to an allocated memory block. A trace is just: (size, > traceback): memory block in bytes, and the Python traceback where the > memory block has been allocated. > > tracemalloc is not aware of Python objects at all. For example, it is > unable to associate an object __dict__ to the object. It works at > malloc() and free() level, not at the _PyObject_GC_Malloc() level. > > One advantage is that memory which is not tracked by the GC is tracked > by tracemalloc. Another is that the implementation is "simple" (from > my point of view :-)). > > One drawback is that tracemalloc traces are harder to analyze. > > > > """ > > Output: > > ... > > 5.py:38: size=9264 B, count=1, average=9264 B > > [' File "5.py", line 38', 'a.alloc_more()', ' File "5.py", line > 32', 'self._d += [1] * 512'] > > """ > > When the internal storage of a Python list grows, the old memory block > is released and a new memory block is allocated. From the tracemalloc > point of view, the old trace is removed, a new trace is created. There > is no relationship between the two traces, even if realloc() has been > used. In tracemalloc, realloc() traces work as two isolated > operations: free() + malloc(). > > > > Now, allocation of alloc and alloc_more functions seems to be merged > into a single traceback. Is it possible we add (maybe a constant number of > tracebacks) per object? This might be pretty useful to find the responsible > leak in situations where same object is modified in many places? > > You might try to take snapshots frequently and write an heuristic to > link traces which look similar... good luck with that :-) > > -- > > There are other tools to track Python memory usage which may help you > to provide a view closer to what you are looking for. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > -- Sümer Cip ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7ECDFRQBT2ZHVCEOTNP6XPZ7NERIXZJF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Hold multiple tracebacks per-object in tracemalloc
Hi everyone,
First of all, I would like to thank for such a useful tool for debugging
memory issues. I am pretty proud as a Python lover that we have such a
tool: tracemalloc:)
I would like to ask if something is feasible/practical regarding
tracemalloc:
AFAIU, tracemalloc holds a single traceback per-object:
Example:
class A:
def __init__(self):
self._d = []
def alloc(self):
self._d += [1] * 512
def alloc_more(self):
self._d += [1] * 512
tracemalloc.start(25)
a = A()
a.alloc()
a.alloc_more()
stats = tracemalloc.take_snapshot().statistics('traceback')
for t in stats:
print(t)
print(t.traceback.format())
"""
Output:
...
5.py:38: size=9264 B, count=1, average=9264 B
[' File "5.py", line 38', 'a.alloc_more()', ' File "5.py", line 32',
'self._d += [1] * 512']
"""
Now, allocation of alloc and alloc_more functions seems to be merged into a
single traceback. Is it possible we add (maybe a constant number of
tracebacks) per object? This might be pretty useful to find the responsible
leak in situations where same object is modified in many places?
I have been playing with this module only lately on tracemalloc, so please
correct if a major piece is missing in my example or my understanding is
wrong in some way.
Thanks as always!
--
Sümer Cip
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/GLTOV6CV5AGM465CFSRSZHIVU7R4436G/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Getting function arguments of PyCFunctionObject
Hi everyone, I have been working on some feature of a deterministic profiler( github.com/sumerc/yappi). The feature is about getting arguments for a given set of function names. For example: you can define something like foo 1,3,arg_name and when foo function is called, profiler will simply collect the first, third from *args and a named argument arg_name from *kwargs. For Python functions I am using following approach: Please note that the code below is executing in the PyTrace_CALL event of the profiler in C side: Look co_argcount and co_varnames to determine the names of arguments and then use these names to retrieve values from f_locals. It seems to be working fine for now. My first question is: Is there a better way to do this? And for C functions, I am totally in dark. I have played with f_valuestack and retrieve some values from there but the indexes seem to change from Python version to version and also I think there is no way of getting named arguments... I have been dealing with this for a while, so there might be unclear points in my explanation. I would really appreciate if anyone can help me on a correct direction on this. Thanks, Best Regards. -- Sümer Cip ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/L65BJKQU5IK6WEERASRYJ2FBWL7WW62N/ Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] [RFC] PEP 418: Add monotonic time, performance counter and process time functions
On Sun, Apr 15, 2012 at 6:18 PM, Victor Stinner victor.stin...@gmail.comwrote: Here is a simplified version of the first draft of the PEP 418. The full version can be read online. http://www.python.org/dev/peps/pep-0418/ FYI there is no time.thread_time() function. It would only be available on Windows and Linux. It does not use seconds but CPU cycles. No module or program of the Python source code need such function, Just FYI: in MACOSx, you can use thread_info() to get that information. Also you can get that information in Solaris,too. In yappi profiler I use all of these approaches together to have an OS independent thread_times() functionality. Here is the relevant code: http://bitbucket.org/sumerc/yappi/src/7c7dc11e8728/timing.chttps://bitbucket.org/sumerc/yappi/src/7c7dc11e8728/timing.c I also think that you are right about Python really not have any use case for this functionality, ... -- Sümer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] CPU vs Wall time Profiling
Hi all, Is there a reason behind the fact that the Python profilers work with Wall time by default? There are OS-dependent ways to get the CPU time of a thread, and giving that choice to the user _somehow_ ( to use wall vs cpu time) might be a good feature? -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] CPU vs Wall time Profiling
The original reason was that the Unix wall clock was more accurate than its CPU clock. If that's changed we should probably (perhaps in a platform-dependent way) change the default to the most accurate clock available. Currently it seems clock_gettime() APIs have nanosecond resolution and OTOH gettimeofday() have microsecond. Other than that, clock_gettime() has a significant advantage: it has per-process timer available which will increase the accuracy of the timing information of the profiled application. -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Improvements for Porting C Extension from 2 to 3
Hi, While porting a C extension from 2 to 3, I realized that there are some general cases which can be automated. For example, for my specific application (yappi - http://code.google.com/p/yappi/), all I need to do is following things: 1) define PyModuleDef 2) change PyString_AS_STRING calls to _PyUnicode_AsString 3) change module init code a little. It occurred to me all these kind of standart changes can be automated via a script. Not sure on the usability of this however, because of my limited knowledge on the area. Does such a tool worth being implemented? -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
1) define PyModuleDef 2) change PyString_AS_STRING calls to _PyUnicode_AsString Aside: Please don't use private APIs in Python extensions. Esp. the above Unicode API is likely going to be phased out. You're better off, using PyUnicode_AsUTF8String() instead and then leaving the PyString_AS_STRING() macro in place. In the standart Python 3.2 source tree, Modules/_lsprof.c uses that internal function _PyUnicode_AsString. Internal means internal to the whole distribution here I think?. But IMHO, this should not be the case, C API modules in the standart dist. should not use internal functions of other areas. Like in the example: cProfile code has nothing to do with the Unicode internals. New developers like me, are in need a consistent examples of usage of Python C API, especially on Python 3.2. Thanks, -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
I'm not sure why what C-API is used in Python's extension modules needs to be anyway to you. I just tought it is a better programming practice to decrease/narrow inter modular usage of functions inside the whole dist. and also a good example for the other 3rd party C Extension developers. I may be wrong. Anyway thanks for the answers. This is just a suggestion. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] yappi - any thoughts?
Hi there, yappi(Yet Another Python Profiler) is a multithreaded profiler for 2.x series(do not know if it will work on 3k, not tested yet). I have done my best to make it efficient as possible.(details are under the technical bullet in the website). It is aimed for long-running programs, to attach/detach profiler and retrieve stats on the fly. The current cProfile module needs substantial amount of work to accomplish this task and also have some problems with recursive functions, basically that is why I have written a profiler from scratch. yappi is currently(v0.2) modifying the profilefunc variable fo the ThreadState object to profile an exsiting thread. The newly created threads are catched via threading.setprofile() call(from VM to our extension). And for the deleted threads, we rely on the recycling of the ThreadState objects.(Please see the website for details.) I have tested it under a game server (100k players per-day) for about a month for any issues and now releasing it. Please see: http://code.google.com/p/yappi/ Any thoughts/comments/ideas, anything?:) -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
