[issue7113] ConfigParser load speedup
Łukasz Langa luk...@langa.pl added the comment: In msg111399 I remarked the new itertools dependency. It seems it is in fact problematic because when building Python from scratch `setup.py` which is used to build C extensions is using configparser. And one of the C-only modules is itertools :) Attached a new patch that doesn't include itertools dependencies. The only difference with the last one is: 96d95 import itertools 549,550c548,550 all_sections = itertools.chain([self._defaults], self._sections.values()) --- all_sections = [self._defaults] all_sections.extend(self._sections.values()) -- nosy: +michael.foord Added file: http://bugs.python.org/file18200/issue7113_without_itertools.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Changes by Łukasz Langa luk...@langa.pl: Removed file: http://bugs.python.org/file18200/issue7113_without_itertools.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Łukasz Langa luk...@langa.pl added the comment: Patch from py3k root. Some minor formatting changes suggested by Ezio Melotti included. -- Added file: http://bugs.python.org/file18204/issue7113_without_itertools.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Brian Curtin cur...@acm.org added the comment: Committed to py3k in r83154 and release27-maint in r83155. -- nosy: +brian.curtin resolution: - fixed stage: patch review - committed/rejected status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Mark Lawrence breamore...@yahoo.co.uk added the comment: @Łukasz: I patched the unit test file only and ran it and all tests passed, I assume that this behaviour is incorrect. -- nosy: +BreamoreBoy stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Łukasz Langa luk...@langa.pl added the comment: Thanks, Mark. The tests should pass for both the old version and the patched one. The change in the code involves performance, not functionality. I didn't want to set assertions that are time based so naturally the new test also runs on the old implementation. The thing is that it runs slower. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Mark Lawrence breamore...@yahoo.co.uk added the comment: Łukasz in that case I think we should get this committed as the patch is small and looks clean. I assume that this will be backported as 2.7 and 3.1 are likely to be around for some years yet. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
Łukasz Langa luk...@langa.pl added the comment: Won't happen for Python 2.x but the idea is good. The original patch won't apply for Py3k so I implemented this approach from scratch for Python 3.2. Patch included. Some microbenchmarks of mine show: - a 3-5% slowdown for very small files (we can ignore this since it's still a blink of an eye) - a 10% speedup in usual cases (files the size of a typical Pylons app configuration or a typical buildout) - more so in unrealistic ones (many values with 100 lines each) Brett, a quick summary of changes: - the idea to introduce a list of lines instead of gluing '%s\n%s' during file reading was introduced - this approach needs an additional phase of joining the lists to strings after reading is done. - I used itertools.chain to add the default section for iteration purposes without having to create a separate list. I hope the additional dependency on itertools is no problem. - I reformatted a bit the changes by fred.drake in r78233 (he added support for valueless keys) - some other slight formatting changes apply as well - in the tests I've added a testcase for a file that uses multi-line values heavily. The file is generated temporarily since it's 127kB in size. Creation takes a fraction of a second so shouldn't be noticeable. -- nosy: +brett.cannon, lukasz.langa versions: +Python 3.2 -Python 2.6 Added file: http://bugs.python.org/file18166/issue7113.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7113] ConfigParser load speedup
New submission from albert hofkamp a.t.hofk...@tue.nl: Current implementation (r71564) uses '%s\n%s' % (old_val, new_line) to merge multi-line options into one string. For options with many lines, this wastes a lot of CPU power. Attached patch against r71564 fixes this problem by first building a list of lines for each loaded option, and after reading the whole file, merging them with the already loaded data. In that way, the '\n'.join() can be performed once. Patched ConfigParser.py works against test/test_cfgparser.py (and Python 2.5) We have witnessed a reduction from 4 hours to 3 seconds loading time with Python 2.6 and an option of 80 lines. -- components: Library (Lib) files: speedfix_71564.patch keywords: patch messages: 93895 nosy: aioryi severity: normal status: open title: ConfigParser load speedup type: performance versions: Python 2.6 Added file: http://bugs.python.org/file15109/speedfix_71564.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue7113 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
