[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Serhiy Storchaka added the comment: Here is a patch which addresses Antoine's comments. Also added dis() output for binary example in comments. -- Added file: http://bugs.python.org/file37469/pickle_optimize_memoize_2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Roundup Robot added the comment: New changeset c49b7acba06f by Serhiy Storchaka in branch '3.4': Issue #19858: pickletools.optimize() now aware of the MEMOIZE opcode, can https://hg.python.org/cpython/rev/c49b7acba06f New changeset e7dd739b4b4e by Serhiy Storchaka in branch 'default': Issue #19858: pickletools.optimize() now aware of the MEMOIZE opcode, can https://hg.python.org/cpython/rev/e7dd739b4b4e -- nosy: +python-dev ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Serhiy Storchaka added the comment: Many thanks for all your reviews Antoine! -- assignee: - serhiy.storchaka resolution: - fixed stage: patch review - resolved status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Serhiy Storchaka added the comment: Ping. -- keywords: +needs review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Serhiy Storchaka added the comment: Here is a patch which makes pickletools.optimize() to work with the MEMOIZE opcode. As side effect it also renumbers memoized values, this allows to use short BINPUT and BINGET instead of LONG_BINPUT and LONG_BINGET. -- keywords: +patch stage: needs patch - patch review versions: +Python 3.4 Added file: http://bugs.python.org/file37317/pickle_optimize_memoize.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Alexandre Vassalotti added the comment: Ah, I almost forgot! I did implement the verification in pickletools.dis() for MEMOIZE: http://hg.python.org/cpython/file/2612ea573ff7/Lib/pickletools.py#l2420 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Serhiy Storchaka added the comment: I afraid this is not a feature, but a bug. pickletools.optimize() can produce invalid pickled data when *PUT and MEMOIZE opcodes are used. import pickle, pickletools p = b'\x80\x04]\x94(\x8c\x04spamq\x01\x8c\x03ham\x94h\x02e.' pickle.loads(p) ['spam', 'ham', 'ham'] pickletools.dis(p) 0: \x80 PROTO 4 2: ]EMPTY_LIST 3: \x94 MEMOIZE 4: (MARK 5: \x8c SHORT_BINUNICODE 'spam' 11: qBINPUT 1 13: \x8c SHORT_BINUNICODE 'ham' 18: \x94 MEMOIZE 19: hBINGET 2 21: eAPPENDS(MARK at 4) 22: .STOP highest protocol among opcodes = 4 p2 = pickletools.optimize(p) p2 b'\x80\x04\x95\x13\x00\x00\x00\x00\x00\x00\x00]\x94(\x8c\x04spam\x8c\x03ham\x94h\x02e.' pickle.loads(p2) Traceback (most recent call last): File stdin, line 1, in module KeyError: 2 pickletools.dis(p2) 0: \x80 PROTO 4 2: \x95 FRAME 19 11: ]EMPTY_LIST 12: \x94 MEMOIZE 13: (MARK 14: \x8c SHORT_BINUNICODE 'spam' 20: \x8c SHORT_BINUNICODE 'ham' 25: \x94 MEMOIZE 26: hBINGET 2 Traceback (most recent call last): File stdin, line 1, in module File /home/serhiy/py/cpython/Lib/pickletools.py, line 2458, in dis raise ValueError(errormsg) ValueError: memo key 2 has never been stored into -- nosy: +serhiy.storchaka ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Alexandre Vassalotti added the comment: Well, that can only happen if MEMOIZE and PUT are both used together, which won't happen with the Pickler classes we support. The easiest thing to do here is to disable pickletools.optimize on protocol 4. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Changes by Antoine Pitrou pit...@free.fr: -- nosy: +tim.peters ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Antoine Pitrou added the comment: By the way, I was curious if there were many users of pickletools.optimize(), so I did a search and most matches seem to be copies of the stdlib source tree: http://code.ohloh.net/search?s=pickletools%20optimizep=0pp=0fl=Pythonmp=1ml=1me=1md=1ff=1filterChecked=true -- nosy: +pitrou ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Tim Peters added the comment: Is it _documented_ that MEMOIZE and PUT can't be used together? If not, it should be documented; and pickletools dis() and optimize() should verify that this restriction is honored in their inputs. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
Alexandre Vassalotti added the comment: MEMOIZE and PUT can be used together. They just need to not step on each other toes when they write to the memo table. As specified by PEP 3154, the memo index used by MEMOIZE is the number of elements currently in the memo table. This obviously means functions like, pickletools.optimize, have to be more careful when they rewrite pickles. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19858] Make pickletools.optimize aware of the MEMOIZE opcode.
New submission from Alexandre Vassalotti: PEP 3154 introduced the MEMOIZE opcode which lowered the overhead of memoization compared to the PUT opcodes which were previously used. We should update pickletools.optimize to remove superfluous uses of this new opcode. -- components: Library (Lib) messages: 204985 nosy: alexandre.vassalotti priority: normal severity: normal stage: needs patch status: open title: Make pickletools.optimize aware of the MEMOIZE opcode. type: behavior versions: Python 3.5 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue19858 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
