New update of the PEP combining various remarks: * Remove GroupedStats class and Snapshot.group_by(): replaced with a new Snapshot.statistics() method which combines all features * Rename reset() to clear_traces() and explain how to get traces before clearing traces * Snapshot.apply_filters() now returns a new Snapshot instance * Rename Filter.include to Filter.inclusive * Rename Filter.traceback to Filter.all_frames * Add a section "Log calls to the memory allocator"
Thanks Jim, Charles-François and Kristjan for your feedback! Here is the new section, tell me if it sounds good. I didn't implement logging just to compare performances. "slower than" is just a previous experience with a very basic logging code. Tell me if you disagree. @Kristjan: I understood that you implemented a tool to log calls on a Playstation3 and send them over the network. How do you process so much data (I computed 29 GB/hour)? Do you log all calls, or only a few of them? Rejected Alternatives ===================== Log calls to the memory allocator --------------------------------- A different approach is to log calls to ``malloc()``, ``realloc()`` and ``free()`` functions. Calls can be logged into a file or send to another computer through the network. Example of a log entry: name of the function, size of the memory block, address of the memory block, Python traceback where the allocation occurred, timestamp. Logs cannot be used directly, getting the current status of the memory requires to parse previous logs. For example, it is not possible to get directly the traceback of a Python object, like ``get_object_traceback(obj)`` does with traces. Python uses objects with a very short lifetime and so makes an extensive use of memory allocators. It has an allocator optimized for small objects (less than 512 bytes) with a short lifetime. For example, the Python test suites calls ``malloc()``, ``realloc()`` or ``free()`` 270,000 times per second in average. If the size of log entry is 32 bytes, logging produces 8.2 MB per second or 29.0 GB per hour. The alternative was rejected because it is less efficient and has less features. Parsing logs in a different process or a different computer is slower than maintaining traces on allocated memory blocks in the same process. Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
