New issue 3012: io.IncrementalNewlineDecoder.newlines behaves differently than
CPython's
https://bitbucket.org/pypy/pypy/issues/3012/ioincrementalnewlinedecodernewlines
Joshua Oreman:
When io.IncrementalNewlineDecoder in translate mode is fed input one character
at a time, it does correctly translate “\\r\\n” sequences to “\\n”, but does
not record “\\r\\n” in its newlines attribute \(which I understand is intended
to represent the newline sequences that were encountered in the input\).
Reproducer:
```
import io
import codecs
info = codecs.lookup("utf-8")
inner_decoder = info.incrementaldecoder("strict")
outer_decoder = io.IncrementalNewlineDecoder(inner_decoder, True)
msg = b"The quick brown fox jumps over the lazy dog.\r\n\n\r\r\n\n"
decoded = ""
for ch in msg:
decoded += outer_decoder.decode(bytes([ch]))
decoded += outer_decoder.decode(b"", final=True)
assert decoded == "The quick brown fox jumps over the lazy dog.\n\n\n\n\n"
assert set(outer_decoder.newlines) == {"\r", "\n", "\r\n"}
```
This passes on CPython but fails on PyPy-3.6 v7.1.1: on the last line,
outer\_decoder.newlines contains “\\r” and “\\n” but not “\\r\\n”.
There is no issue in non-translating mode; the newlines property does wind up
containing all three sequences in that case.
_______________________________________________
pypy-issue mailing list
pypy-issue@python.org
https://mail.python.org/mailman/listinfo/pypy-issue
