[issue33578] cjkcodecs missing getstate and setstate implementations

Change by INADA Naoki : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue33578] cjkcodecs missing getstate and setstate implementations

INADA Naoki added the comment: New changeset 488c0a6cdf09e21774e63c2a430ecc0de804d147 by INADA Naoki (Christopher Thorne) in branch 'master': bpo-33578: Fix getstate/setstate for CJK decoder (GH-10290) https://github.com/python/cpython/commit/488c0a6cdf09e21774e63c2a430ecc0de804d147

[issue33578] cjkcodecs missing getstate and setstate implementations

Change by Christopher Thorne : -- pull_requests: +9600 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue33578] cjkcodecs missing getstate and setstate implementations

miss-islington added the comment: New changeset ac22f6aa989f18c33c12615af1c66c73cf75d5e7 by Miss Islington (bot) (Christopher Thorne) in branch 'master': bpo-33578: Add getstate/setstate for CJK codec (GH-6984) https://github.com/python/cpython/commit/ac22f6aa989f18c33c12615af1c66c73cf75d5e7

[issue33578] cjkcodecs missing getstate and setstate implementations

Change by INADA Naoki : -- type: behavior -> enhancement versions: -Python 2.7, Python 3.6, Python 3.7 ___ Python tracker ___ ___

[issue33578] cjkcodecs missing getstate and setstate implementations

Christopher Thorne added the comment: Thanks Naoki, that simplifies things a lot. I've updated the PR with a new test case for ISO-2022-JP and a corresponding implementation for the encoder state methods. -- ___ Python tracker

[issue33578] cjkcodecs missing getstate and setstate implementations

INADA Naoki added the comment: > `MultibyteCodec_State` can occupy 8 bytes, and `pending` can occupy 2 bytes > (MAXENCPENDING), we get a total of 10 bytes which I think exceeds what a > PyLong can represent. PyLong is "long integer", aka "big integer", not C's long type.

[issue33578] cjkcodecs missing getstate and setstate implementations

Christopher Thorne added the comment: Ah, good find. I suppose that means `MultibyteCodec_State` and `pending` are both needed to fully capture state, as is done in `decoder.getstate`/`setstate` by returning a tuple of both. Unfortunately `encoder.getstate` is defined to return an integer,

[issue33578] cjkcodecs missing getstate and setstate implementations

INADA Naoki added the comment: ISO-2022-* is "stateful" encoding. It uses escape sequence to change state. So keeping only `pending` is not enough. >>> enc.encode("abcあいう") b'abc\x1b$B$"$$$&' >>> enc.getstate() 0 >>> enc.encode("abc") b'\x1b(Babc' >>> enc.encode("abcあいう") b'abc\x1b$B$"$$$&'

[issue33578] cjkcodecs missing getstate and setstate implementations

Change by Roundup Robot : -- keywords: +patch pull_requests: +6638 stage: -> patch review ___ Python tracker ___

[issue33578] cjkcodecs missing getstate and setstate implementations

New submission from Christopher Thorne : The cjkcodecs module provides support for various Chinese/Japanese/Korean codecs through its MultibyteIncrementalDecoder and MultibyteIncrementalEncoder classes. While working with one of these cjkcodecs (euc-jp), I came across an