[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Antoine posted a simpler reconfiguration RFE over in https://bugs.python.org/issue30526 (for setting line buffering). While technically orthogonal to this RFE, we're thinking it might be possible to expose them to users as a common "reconfigure()" API, rather th

[issue15216] Support setting the encoding on a text stream after creation

Changes by INADA Naoki : -- pull_requests: +166 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.py

[issue15216] Support setting the encoding on a text stream after creation

INADA Naoki added the comment: set_encoding-8.patch dropped support of changing encoding after read. It is based on set_encoding-newline.patch -- Added file: http://bugs.python.org/file46259/set_encoding-8.patch ___ Python tracker

[issue15216] Support setting the encoding on a text stream after creation

INADA Naoki added the comment: > Inada, I think you messed up the positioning of bits of the patch. E.g. there > are now test methods declared > inside a helper function (rather than a test > class). I'm sorry. `patch -p1` merged previous patch into wrong place, and test passed accidently.

[issue15216] Support setting the encoding on a text stream after creation

Martin Panter added the comment: Inada, I think you messed up the positioning of bits of the patch. E.g. there are now test methods declared inside a helper function (rather than a test class). Since it seems other people are in favour of this API, I would like to expand it a bit to cover two

[issue15216] Support setting the encoding on a text stream after creation

INADA Naoki added the comment: This patch doesn't change current behavior of "decode by chunk". It's difficult to support changing encoding via header, like Python's encoding cookie. >>> import io >>> buff = io.BytesIO("hello\nこんにちは".encode('utf-8')) >>> f = io.TextIOWrapper(buff, encoding='asci

[issue15216] Support setting the encoding on a text stream after creation

INADA Naoki added the comment: I had not read the patch carefully when updating. Now I'm reading pure Python part of the patch. For writing side, this patch is very simple. For reading side, this patch is very complex, because TextIOWrapper.tell() and .seek() is complex already. How about start

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Reviewing Inada-san's latest version of the patch, we seem to be in a somewhat hybrid state where: 1. The restriction to only being used with seekable() streams if there is currently unread data in the read buffer is in place 2. We don't actually call seek() an

[issue15216] Support setting the encoding on a text stream after creation

INADA Naoki added the comment: Updated the patch for default branch. -- versions: +Python 3.7 -Python 3.6 Added file: http://bugs.python.org/file46204/set_encoding-7.patch ___ Python tracker ___

[issue15216] Support setting the encoding on a text stream after creation

Changes by Zachary Ware : -- hgrepos: -334 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python

[issue15216] Support setting the encoding on a text stream after creation

Serhiy Storchaka added the comment: quad, please don't change issue attributes to unrelated values. -- type: security -> enhancement versions: +Python 3.6 ___ Python tracker ___

[issue15216] Support setting the encoding on a text stream after creation

Changes by Serhiy Storchaka : -- components: -2to3 (2.x to 3.x conversion tool) ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue15216] Support setting the encoding on a text stream after creation

Changes by Kwamenichols : -- components: +2to3 (2.x to 3.x conversion tool) hgrepos: +334 versions: -Python 3.6 ___ Python tracker ___ __

[issue15216] Support setting the encoding on a text stream after creation

Changes by Kwamenichols : -- nosy: +quad type: enhancement -> security ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubs

[issue15216] Support setting the encoding on a text stream after creation

Changes by Andrei Dorian Duma : -- nosy: -andrei.duma ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Revisiting the idea Nikolaus raised last year regarding whether or not this could be done using a dedicated API in the sys module, I realised today that even if we decided to use a separate public API, *that API* would still need a defined way to modify the stre

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Reviewing the items I had flagged as dependencies of issue 22555 for personal tracking purposes, I suggest we defer further consideration of this idea to 3.6 after folks have had a chance to get some experience with the interpreter defaulting to using "surrogate

[issue15216] Support setting the encoding on a text stream after creation

Changes by Berker Peksag : -- nosy: +berker.peksag stage: needs patch -> patch review ___ Python tracker ___ ___ Python-bugs-list mail

[issue15216] Support setting the encoding on a text stream after creation

Martin Panter added the comment: Some more use cases for temporarily switching error handler in the middle of writing to a stream: * Possibly simplify the implementation of sys.displayhook() * I have done a similar hack at

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Metaclasses and monkeypatching encourage bad API design, too, but we still provide them because they're essential tools for some use cases. In Python, the system integrator is always right, and we provide them with power tools to deal with the cases where reality d

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: On 02/04/2014 03:28 AM, Nick Coghlan wrote: > > Nick Coghlan added the comment: > > If operating systems always exposed accurate metadata and configuration > settings, I'd agree with you. They don't though, so sometimes developers need > to be able to override

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: If operating systems always exposed accurate metadata and configuration settings, I'd agree with you. They don't though, so sometimes developers need to be able to override whatever the interpreter figured out automatically. In addition, needing to cope with bad

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: Newest version of the patch attached, the reference leak problem has been fixed. -- Added file: http://bugs.python.org/file33893/set_encoding-6.patch ___ Python tracker

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: I think that any API that gives you a TextIOWrapper object with undesired properties is broken and needs to be fixed, rather than a workaround added to TextIOWrapper. That said, I defer to the wisdow of the core developers, and I'll be happy to implement what

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: It's a blunt instrument to be sure, but a necessary one, I think - otherwise we end with a scattering of API specific solutions rather than a single consistent approach. Here's a thought, though: what if we went with Serhiy's "reconfigure" idea, limited that to s

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: The attached patch now passes all testcases. However, every invocation of set_encoding() when there is buffered data leaks one reference. I haven't been able to find the error yet. As for adding a reopen() or configure() method: I don't like it very much, but

[issue15216] Support setting the encoding on a text stream after creation

Serhiy Storchaka added the comment: What about other TextIOWrapper parameters? I think "newline" should be changed in same cases as "encoding" and "errors". And changing "buffering" and "line_buffering" can be useful too. What if add the "configure" or "reopen" method which accepts all this p

[issue15216] Support setting the encoding on a text stream after creation

Changes by Nikolaus Rath : Added file: http://bugs.python.org/file33857/set_encoding-4.patch ___ Python tracker ___ ___ Python-bugs-list maili

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: Wow, I didn't realize that programming Python using the C interface was that tedious and verbose. I have attached a work-in-progress patch. It is not complete yet, but maybe somebody could already take a look to make sure that I'm not heading completely in the

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: The specific motivating use cases I'm aware of involve the standard streams (for example, "How would you implement the equivalent of iconv in Python 3?"). There's actually the workaround for the missing feature right now: replace the standard streams with new strea

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: I'm about 40% done with translating Victor's patch into C. However, in the process I got the feeling that this approach may not be so good after all. Note that: * The only use-case for set_encoding that I have found was changing the encoding of sys.{stdin,std

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: That's entirely plausible - Victor's current patch is definitely a proof-of-concept rather than a battle-tested implementation. I'm not sure you could actually *provoke* that bug with any of our existing codecs, though. -- _

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: Thanks Nick! Will take this into account. I've stumbled over another question in the meantime: It seems to me that after the call to set_encoding(), self._snapshot contains the decoder flags from the state of the *old* decoder. On the next call of eg. tell(),

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: A given encoding may have multiple aliases, and also variant spellings that are normalized before doing the codec lookup. Doing the lookup first means we run through all of the normalisation and aliasing machinery and then compare the *canonical* names. For exam

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: Second question: The following looks as if a BOM might be written for writeable, non-seekable streams: # don't write a BOM in the middle of a file if self._seekable and self.writable(): position = self.buffer.tell() if po

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: Question: What is the point of the old_encoding = codecs.lookup(self._encoding).name encoding = codecs.lookup(encoding).name if encoding == old_encoding and errors == self._errors: # no change return dance? Isn't

[issue15216] Support setting the encoding on a text stream after creation

Nikolaus Rath added the comment: I am working on this now. -- nosy: +nikratio ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue15216] Support setting the encoding on a text stream after creation

Changes by Antoine Pitrou : -- versions: +Python 3.5 -Python 3.4 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue15216] Support setting the encoding on a text stream after creation

Changes by Jakub Wilk : -- nosy: +jwilk ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org

[issue15216] Support setting the encoding on a text stream after creation

Andrei Dorian Duma added the comment: I have to postpone this for a few weeks. Sorry if I stayed in anyone's way. -- ___ Python tracker ___ __

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Thanks, Andrei, that would be great. The tests and the Python version in Victor's patch show the desired API and behaviour. In theory, the C version should just be a matter of adding an equivalent textiowrapper_set_encoding method as a static function in hg.py

[issue15216] Support setting the encoding on a text stream after creation

Andrei Dorian Duma added the comment: I am interested in working on this, but I might need guidance at times. Is that acceptable? If yes, I'm willing to start as soon as possible. -- nosy: +andrei.duma ___ Python tracker

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Just reviewed Victor's patch - aside from a trivial typo and not covering the C implementation or documentation yet, it looks good to me. Most importantly, the tests in that patch could be used to validate a C implementation as well. I'll see if I can find anyon

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: > Is that a reasonable possibility before beta 1, or do we need to bump this > one to 3.5? My patch was not reviewed yet and must be reimplemented in C. I will not have time before the beta1 to finish the work. --

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: It would be good to have this addressed in Python 3.4 (I'm following up on a few other encoding related issues at the moment). Is that a reasonable possibility before beta 1, or do we need to bump this one to 3.5? -- ___

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Oops, meant to link to my post in the thread about a buffer manipulation API: http://mail.python.org/pipermail/python-ideas/2013-March/019769.html -- ___ Python tracker

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: That's a fair point - I think it's acceptable to throw an error in the case of *already decoded* characters that haven't been read. There's also a discussion on python-ideas about an explicit API for clearing the internal buffers, and pushing data back into a st

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: Note: it is not possible to reencode the buffer of decoded characters to compute the offset in bytes. Some codecs are not bijective. Examples: * b'\x00'.decode('utf7').encode('utf7') == b'+AAA-' * b'\xff'.decode('ascii', 'replace').encode('ascii', 'replace')

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: Oh, set_encoding.patch is wrong: +offset = self._decoded_chars_used - len(next_input) self._decoded_chars_used is a number of Unicode characters, len(next_input) is a number of bytes. It only works with 7 and 8 bit encodings like ascii or latin1,

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Ah, you're right - peeking into the underlying buffer would be enough to handle encoding detection. -- ___ Python tracker ___

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: > Victor proposes that it's acceptable to simply disallow changing the encoding > of a stream that isn't seekable. It is no what I said. My patch raises an exception if you already started to read stdin. It should work fine if stdin is a pipe but the read buffe

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: To bring back Victor's comments from the list: - stdout/stderr are fairly easy to handle, since the underlying buffers can be flushed before switching the encoding and error settings. Yes, there's a risk of creating mojibake, but that's unavoidable and, for this

[issue15216] Support setting the encoding on a text stream after creation

Changes by rurpy the second : -- nosy: +rurpy2 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: The reason I marked this as a release blocker for 3.4 is because it's a key piece of functionality for writing command line apps which accept an encoding argument. I'll use "high" instead. An interesting proposal was posted to the python-dev thread [1]: using s

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: > My implementation permits to change both (encoding, errors, encoding and > errors). We may also add a set_errors() method: def set_errors(self, errors): self.set_encoding(self.encoding, errors) -- ___ Python t

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: > That will be fragile. A bit of prematurate input or output > (for whatever reason) and your program breaks. I agree that it is not the most pure solution, but sometimes practicality beats purity (rule #9) ;-) We can add an ugly big red warning in the doc ;-)

[issue15216] Support setting the encoding on a text stream after creation

STINNER Victor added the comment: Here is a Python implementation of TextIOWrapper.set_encoding(). The main limitation is that it is not possible to set the encoding on a non-seekable stream after the first read (if the read buffer is not empty, ie. if there are pending decoded characters). +

[issue15216] Support setting the encoding on a text stream after creation

Changes by INADA Naoki : -- nosy: +naoki ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org

[issue15216] Support setting the encoding on a text stream after creation

Martin v. Löwis added the comment: I fail to see why this is a release blocker; no rationale is given in the original message, nor in the quoted message. So unblocking. -- nosy: +loewis priority: release blocker -> normal ___ Python tracker

[issue15216] Support setting the encoding on a text stream after creation

Changes by Atsuo Ishimoto : -- nosy: +ishimoto ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue15216] Support setting the encoding on a text stream after creation

Antoine Pitrou added the comment: > Would a "set_encoding" method be Pythonic? I would've preferred an > "encoding" property which flushes the output when it's changed. I would prefer to have a method. The side-effect is too violent to be hidden behind a property. Besides, you want to set encod

[issue15216] Support setting the encoding on a text stream after creation

Matthew Barnett added the comment: Would a "set_encoding" method be Pythonic? I would've preferred an "encoding" property which flushes the output when it's changed. -- nosy: +mrabarnett ___ Python tracker __

[issue15216] Support setting the encoding on a text stream after creation

Nick Coghlan added the comment: Indeed. However, the current alternatives (based on detach() and fileNo()) are also problematic - using detach() breaks the corresponding sys.__std*__ entry, while using fileNo() means you now have two independent IO stacks using the same underlying descriptor.

[issue15216] Support setting the encoding on a text stream after creation

Antoine Pitrou added the comment: > The proposal [1] is that TextIOWrapper support a set_encoding() method > that is only supported between creation of the stream and the first > read or write operation. That will be fragile. A bit of prematurate input or output (for whatever reason) and your

[issue15216] Support setting the encoding on a text stream after creation

Changes by Arfrever Frehtes Taifersar Arahesis : -- nosy: +Arfrever ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue15216] Support setting the encoding on a text stream after creation

Serhiy Storchaka added the comment: > The proposal [1] is that TextIOWrapper support a set_encoding() method that > is only supported between creation of the stream and the first read or write > operation. IMHO "encoding", "errors", "buffering", "line_buffering" and "newline" should be writa

[issue15216] Support setting the encoding on a text stream after creation

New submission from Nick Coghlan : As discussed on python-ideas, swapping out the sys.std* streams can be fraught with peril. This can make writing code that wants to support an "--encoding" option for pipeline processing difficult. The proposal [1] is that TextIOWrapper support a set_encoding