[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Changes by Martin Panter vadmium...@gmail.com: -- dependencies: +Fix codecs.iterencode/decode() by allowing data parameter to be omitted stage: - patch review ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: Here is patch v4. The stream writer is now automatically generated by default by the CodecInfo constructor if no custom stream writer parameter is supplied. The base64 and quopri codecs are adjusted to also use this default stream writer to help with Issue 20132. -- Added file: http://bugs.python.org/file37847/zlib-bz2-writer.v4.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: Sorry, I changed the name of the attribute and forgot to update the doc string. Its new name was _Encoder. Your description was fairly accurate. I am adding patch v3, with an expanded the doc string. Hopefully that explains it a bit better. Since it is just implementing the documented StreamWriter API, I only added brief descriptions of the methods pointing back to the StreamWriter methods. -- Added file: http://bugs.python.org/file37719/zlib-bz2-writer.v3.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: [issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
On 15.01.2015 05:43, Martin Panter wrote: New patch that also fixes StreamWriter.writelines() in general for the byte codecs Could you explain this new undocumented class ? +class _IncrementalBasedWriter(StreamWriter): +Generic StreamWriter implementation. + +The _EncoderClass attribute must be set to an IncrementalEncoder +class to use. + + +def __init__(self, stream, errors='strict'): +super().__init__(stream, errors) +self._encoder = self._Encoder(errors) + +def write(self, object): +self.stream.write(self._encoder.encode(object)) + +def reset(self): +self.stream.write(self._encoder.encode(final=True)) + Note that the doc-string mentions a non-existing attribute and there are doc-string missing for the other methods. The purpose appears to be a StreamWriter which works with an IncrementalEncoder. A proper name would thus be IncrementalStreamWriter which provides an .encode() method which adapts the signature of the incremental encoder to the one expected for StreamWriters and Codecs. -- Marc-Andre Lemburg eGenix.com ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: New patch that also fixes StreamWriter.writelines() in general for the byte codecs -- Added file: http://bugs.python.org/file37710/zlib-bz2-writer.v2.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: Here is a patch to implement the zlib-codec and bz2-codec StreamWriter classes based on their IncrementalEncoder classes. It depends on my patch for Issue 23231, though I guess it could be tweaked to work around that if desired. -- keywords: +patch versions: +Python 3.5 Added file: http://bugs.python.org/file37702/zlib-bz2-writer.patch ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: See Issue 23231 for a proposal which should make the incremental codec API compatible with a generic StreamReader/Writer class. I discovered that many of the codec files are generated by gencodec.py, not hand-written. However when I tried regenerating them, I found a handful had diverged from the generated code, but others were more or less true to the generated code. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: The corresponding stream reader has a related issue which I mentioned in Issue 20132 -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Martin Panter added the comment: Even if all these issues aren’t worth fixing, I think the documentation should at least say which codecs work fully (e.g. most text encodings), which ones work suboptimally (e.g. UTF-7), and which ones only work for single-shot encoding and decoding. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Walter Dörwald added the comment: The stream part of the codecs isn't used that much in Python 3 any more, so I'm not sure if this is worth fixing. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Changes by Martin Panter vadmium...@gmail.com: -- nosy: +vadmium ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Changes by Serhiy Storchaka storch...@gmail.com: -- nosy: +doerwalter, lemburg, loewis ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Andrew McNabb amcn...@mcnabbs.org added the comment: It looks like encodings/zlib_codec.py defines a custom IncrementalEncoder and IncrementalDecoder, but its StreamWriter and StreamReader rely on the standard implementation of codecs.StreamWriter and codecs.StreamReader. One solution might be to have zlib_codec.StreamWriter inherit from zlib_codec.IncrementalEncoder instead of from zlib_encoder.Codec. I'm not familiar enough with the codecs library to know whether this is the best approach. Unfortunately, there are 120 codec files in the encodings directory, and it's unclear how many of them would need to be modified. Based on the number of them that implement StreamWriter as class StreamWriter(Codec,codecs.StreamWriter), it looks like it might be a lot of them. Was each of these 120 files hand-written? -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Changes by R. David Murray rdmur...@bitdance.com: -- nosy: +haypo ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
STINNER Victor victor.stin...@gmail.com added the comment: See also issue #7475. -- ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
Jesús Cea Avión j...@jcea.es added the comment: Andrew, could you possibly write a patch and a test for 3.3? -- nosy: +jcea versions: +Python 3.3 -Python 3.2 ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue13881 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13881] Stream encoder for zlib_codec doesn't use the incremental encoder
New submission from Andrew McNabb amcn...@mcnabbs.org:
The stream encoder for the zlib_codec doesn't use the incremental encoder, so
it has limited usefulness in practice. This is easiest to show with an example.
Here is the behavior with the stream encoder:
filelike = io.BytesIO()
wrapped = codecs.getwriter('zlib_codec')(filelike)
wrapped.write(b'hello')
filelike.getvalue()
b'x\x9c\xab\x00\x00\x00y\x00y'
wrapped.write(b'x')
filelike.getvalue()
b'x\x9c\xab\x00\x00\x00y\x00yx\x9c\xab\x00\x00\x00y\x00y'
However, this is the behavior of the incremental encoder:
ienc = codecs.getincrementalencoder('zlib_codec')()
ienc.encode(b'x')
b'x\x9c'
ienc.encode(b'x', final=True)
b'\xab\xa8\x00\x00\x01j\x00\xf1'
The stream encoder is apparently encoding each write as an individual block,
but the incremental encoder buffers until it gets a block that's large enough
to be meaningfully compressed.
Fixing this may require addressing a separate issue with stream encoders.
Unlike with the GzipFile module, closing a stream encoder closes the underlying
file. If this underlying file is a BytesIO, then closing makes it free its
buffer, making it impossible to get at the completed file.
--
components: IO
messages: 152029
nosy: amcnabb
priority: normal
severity: normal
status: open
title: Stream encoder for zlib_codec doesn't use the incremental encoder
type: behavior
versions: Python 3.2
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13881
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
