[issue5671] Speed up pickling of lists in cPickle
Alexandre Vassalotti alexan...@peadrop.com added the comment: It is too late now for the 2.x version. And, the huge patch in issue 9410 includes an updated version of this patch for 3.x. -- resolution: - duplicate stage: - committed/rejected status: open - closed superseder: - Add Unladen Swallow's optimizations to Python 3's pickle. ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Changes by Skip Montanaro s...@pobox.com: -- nosy: -skip.montanaro ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Sean Reifschneider j...@tummy.com added the comment: pickle_batch_list_exact_py3k.diff applies cleanly on current py3k trunk and passes tests. cpickle_list.patch applies cleanly against 2.x trunk and passes make test. I don't see any objections brought up about this set of patches, so we can get these applied? -- assignee: - alexandre.vassalotti nosy: +jafo priority: - normal ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Skip Montanaro s...@pobox.com added the comment: Still applies cleanly (with a little fuzz) to the trunk after applying the issue 5683 patch. Tests all still pass (including xpickle w/ 2.4, 2.5, 2.6 available). -- nosy: +skip.montanaro ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Antoine Pitrou pit...@free.fr added the comment: A micro-benchmark of Collin's patch: python -m timeit -s import cPickle; l=range(150) cPickle.dumps(l, protocol=-1) * before: 12.1 usec per loop * after: 10.1 usec per loop = 15% faster on a favorable case -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Collin Winter coll...@gmail.com added the comment: I've added a microbenchmark to perf.py called pickle_list. Running that on this change (perf.py -r -b pickle_list): pickle_list: Min: 1.126 - 0.888: 26.86% faster Avg: 1.154 - 0.906: 27.43% faster Significant (t=115.404547, a=0.95) That's probably the upper bound on the performance improvement to be realized from this patch. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
New submission from Collin Winter coll...@gmail.com: The attached patch adds another version of cPickle.c's batch_list(), batch_list_exact(), which is specialized for type(x) is list. This provides a nice performance boost when pickling objects that use lists. This is similar to the approach taken in issue 5670, though the performance boost on our benchmark is smaller: Pickle: Min: 2.231 - 2.200: 1.39% faster Avg: 2.266 - 2.227: 1.72% faster Significant (t=10.994064, a=0.95) Benchmark is at http://code.google.com/p/unladen-swallow/source/browse/tests/performance/macro_pickle.py (driver is ../perf.py; perf.py was run with --rigorous -b pickle). Workloads involving more lists will benefit more. This patch passes all the tests added in issue 5665. I would recommend reviewing that patch first. I'll port to py3k once this is reviewed for trunk. -- components: Extension Modules files: cpickle_list.patch keywords: needs review, patch messages: 85250 nosy: collinwinter severity: normal status: open title: Speed up pickling of lists in cPickle type: performance versions: Python 2.7 Added file: http://bugs.python.org/file13585/cpickle_list.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Raymond Hettinger rhettin...@users.sourceforge.net added the comment: Out of curiousity, are you also benchmarking against marshal and json? ISTM, that there is always one of them that will be the fastest and that the others should mimic that approach. -- nosy: +rhettinger ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Collin Winter coll...@gmail.com added the comment: No, we haven't started looking at other serialization formats yet. Marshal will probably be my next target, with json being a lower priority. There were enough instances of low-hanging fruit in cPickle that I didn't go looking at the other implementations for ideas. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Alexandre Vassalotti alexan...@peadrop.com added the comment: Here's a patch for py3k. I also added a special-case for 1-item lists. -- nosy: +alexandre.vassalotti Added file: http://bugs.python.org/file13595/pickle_batch_list_exact_py3k.diff ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5671] Speed up pickling of lists in cPickle
Changes by Alexandre Vassalotti alexan...@peadrop.com: -- versions: +Python 3.1 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue5671 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
