[issue20218] Add methods to `pathlib.Path`: `write_text`, `read_text`, `write_bytes`, `read_bytes`
Matthias Troffaes added the comment: Chiming in here: Sphinx's testing framework does include a feature that allows easily read/write files into/from text/bytes directly from path-like objects. There is thus a demand out there. If this feature were to make it into stdlib, it would be loved at least by Sphinx testers and sphinx extension module testers. Current implementation in Sphinx: https://bitbucket.org/birkenfeld/sphinx/src/f87ae5c0272e7384dc976414d091aa8b175827cd/tests/path.py?at=default#cl-129 Discussion to move to pathlib on Sphinx tracker: https://bitbucket.org/birkenfeld/sphinx/issue/1241/move-test-utilities-into-a-sphinx#comment-12645576 Some code examples of how this is typically used in Sphinx: https://bitbucket.org/tk0miya/sphinx-testing/src/ee89298fa8f848b7c3ca93656df5330db85b4291/README.rst?at=default https://github.com/mcmtroffaes/sphinxcontrib-bibtex/commit/0a0bf07a34e6f7e3c66ddacc0da02b4a2caba794 -- nosy: +matthiastroffaes ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20218 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20218] Add methods to `pathlib.Path`: `write_text`, `read_text`, `write_bytes`, `read_bytes`
Matthias Troffaes added the comment: Thanks for the quick response. I agree with Georg on all points, i.e. longer function names and no extra options. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue20218 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Matthias Troffaes matthias.troff...@gmail.com added the comment: Patch against current tip attached. I can no longer reproduce the large memory leaks with the current tip (which is of course wonderful!), so I guess the second part of the patch (freeing the freelists during gc.collect) makes no longer sense. -- Added file: http://bugs.python.org/file22833/py3k-04082011-clearfreelist-dict_set_list.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Changes by Matthias Troffaes matthias.troff...@gmail.com: Added file: http://bugs.python.org/file17427/py3k-rev81387-clearfreelist-dict_set_list.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Changes by Matthias Troffaes matthias.troff...@gmail.com: Added file: http://bugs.python.org/file17428/py3k-rev81387-clearfreelist-gc_collect.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Changes by Matthias Troffaes matthias.troff...@gmail.com: Added file: http://bugs.python.org/file17429/py3k-rev81387-clearfreelist-time_gc_collect.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Matthias Troffaes matthias.troff...@gmail.com added the comment: I uploaded updates of the three relevant patches against the current revision of the py3k branch, as the old patches no longer applied cleanly due to whitespace changes. To summarize: * The first patch, py3k-rev81387-clearfreelist-dict_set_list.patch, simply adds freelist methods to the public API for dict, list, and set. No opposition has been expressed against this, so I hope this can be accepted. * The second patch, py3k-rev81387-clearfreelist-gc_collect.patch, adds calls to these methods to gc.collect() - some opposition was expressed against the (already present before my patch!!) method of freeing lists during highest generation garbage collection. I attempted to measure the actual time spent on freeing the freelists in a simply python program which does a lot of allocation (attached as py3k-freelist_test.py). This apparently shows that clearing the freelists does not affect timing much at all. * The third patch, file py3k-rev81387-clearfreelist-time_gc_collect.patch, causes estimates of the time spent on freeing the freelists to be printed to the console, and is obviously for testing/benchmarking purpose only. * The tp_free_list patch is no longer relevant (see comment by Guido). Hoping for a conclusion of this issue, Matthias -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Matthias Troffaes matthias.troff...@gmail.com added the comment: To aid the discussion, I attach another quick patch which reports the time spent on PyXXX_ClearFreeList calls during highest generation garbage collection (including gc.collect()). For simplicity, the timer uses clock() so the resolution is quite limited (appears to be 10ms on my machine) and I don't claim that this is the best way of measuring execution speed, but at least it gives some indication. The patch also gives an indication at how frequently the highest generation is collected. Below is the result of the patch on the py3k-freelist_test.py test script, on my machine (again, debug build). For reference, I've measured the total time spent by the script as well, with the time command. Summarizing, (30+70+30+30+50+70+20+110)/48420.0 = 0.0085 = 0.85% of time is spent on freeing the freelists, in my test. Another way to look at the data is that it roughly takes 10ms for each 100MB allocated (at least for the types of data in the script). Floats seem to be an exception and take at least twice as long (not sure why). Keep in mind that the test merely allocates and deallocates memory, without doing much else, so it isn't a typical Python application. $ time ./python py3k-freelist_test.py Memory used (begin): 121Mb memtest 200 int Memory used (peak): 297Mb cleared free lists in 0 clock ticks (0.00ms) Memory used (end): 121Mb Unfreed memory: 0Mb memtest 200 str Memory used (peak): 455Mb cleared free lists in 3 clock ticks (30.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 float Memory used (peak): 236Mb cleared free lists in 7 clock ticks (70.00ms) Memory used (end): 127Mb Unfreed memory: 6Mb memtest 200 int Memory used (peak): 312Mb cleared free lists in 0 clock ticks (0.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) Memory used (peak): 372Mb cleared free lists in 3 clock ticks (30.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test2 cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) Memory used (peak): 361Mb cleared free lists in 3 clock ticks (30.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _tuple Memory used (peak): 529Mb cleared free lists in 5 clock ticks (50.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _set cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) Memory used (peak): 846Mb cleared free lists in 7 clock ticks (70.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 20 _dict Memory used (peak): 233Mb cleared free lists in 2 clock ticks (20.00ms) Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test3 cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks (0.00ms) cleared free lists in 0 clock ticks
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Matthias Troffaes matthias.troff...@gmail.com added the comment: Thanks for the feedback! Attaching a new patch which implements tp_free_list slot as suggested - I hope I did it correctly. I've only implemented the new slot for dict so far, but I'm happy to tp_free_list-ify the other freelist types as well, in a future patch, if this gets the green light. Description of the patch: * added tp_free_list slot to PyTypeObject (definitely for review: is the location of tp_free_list right after tp_free sensible?) * added PyType_ClearFreeList(PyTypeObject *) to C API, which calls the tp_free_list function if not NULL * inserted the new slot where necessary (e.g. in PyGen_Type) to sync type definitions with the updated PyTypeObject * created dict_free_list function and added it to PyDict_Type * call PyType_ClearFreeList(PyDict_Type) from gc.collect() -- Added file: http://bugs.python.org/file14725/py3k-tp_free_list-dict.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
New submission from Matthias Troffaes matthias.troff...@gmail.com: The Python C API provides PyXXX_ClearFreeList functions to allow the float, int, etc... freelists to be freed, potentially releasing memory to the OS earlier. Currently, there is no such API for the dict, set, and list freelists. The attached patch adds PyXXX_ClearFreeList functions to the C API, so the dict, set, and list freelists can be freed as well. -- components: Interpreter Core files: py3k-clearfreelist-dict_set_list.patch keywords: patch messages: 91520 nosy: matthiastroffaes severity: normal status: open title: PyXXX_ClearFreeList for dict, set, and list type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file14708/py3k-clearfreelist-dict_set_list.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Matthias Troffaes matthias.troff...@gmail.com added the comment: I attach a second patch which also calls the new PyXXX_ClearFreeList functions on garbage collection, during gc.collect(). -- Added file: http://bugs.python.org/file14709/py3k-clearfreelist-gc_collect.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6695] PyXXX_ClearFreeList for dict, set, and list
Matthias Troffaes matthias.troff...@gmail.com added the comment: I'm also attaching a test script to check the effect of the two patches on gc.collect(). If many objects are allocated, space savings appear to be relevant. Before applying the patch (debug build on linux 64 bit): Memory used (begin): 121Mb memtest 200 int Memory used (peak): 297Mb Memory used (end): 122Mb Unfreed memory: 1Mb memtest 200 str Memory used (peak): 455Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 float Memory used (peak): 236Mb Memory used (end): 127Mb Unfreed memory: 6Mb memtest 200 int Memory used (peak): 313Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test Memory used (peak): 372Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test2 Memory used (peak): 361Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _tuple Memory used (peak): 529Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _set Memory used (peak): 846Mb Memory used (end): 765Mb Unfreed memory: 644Mb memtest 200 _dict Memory used (peak): 1241Mb Memory used (end): 1241Mb Unfreed memory: 1120Mb memtest 200 Test3 Memory used (peak): 1241Mb Memory used (end): 765Mb Unfreed memory: 644Mb [40720 refs] After applying the patch (same build system): Memory used (begin): 121Mb memtest 200 int Memory used (peak): 298Mb Memory used (end): 121Mb Unfreed memory: 0Mb memtest 200 str Memory used (peak): 455Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 float Memory used (peak): 236Mb Memory used (end): 127Mb Unfreed memory: 6Mb memtest 200 int Memory used (peak): 312Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test Memory used (peak): 374Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test2 Memory used (peak): 361Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _tuple Memory used (peak): 528Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _set Memory used (peak): 846Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 _dict Memory used (peak): 1240Mb Memory used (end): 123Mb Unfreed memory: 2Mb memtest 200 Test3 Memory used (peak): 999Mb Memory used (end): 123Mb Unfreed memory: 2Mb [40740 refs] -- Added file: http://bugs.python.org/file14710/py3k-freelist_test.py ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue6695 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
