[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[STINNER Victor]

STINNER Victor  added the comment:

> Is subprocess affected by PYTHONIOENCODING?

Yes, as any Python process.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Éric Araujo]

Éric Araujo  added the comment:

Is subprocess affected by PYTHONIOENCODING?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Arfrever Frehtes Taifersar Arahesis]

Changes by Arfrever Frehtes Taifersar Arahesis :


--
nosy: +Arfrever

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Éric Araujo]

Changes by Éric Araujo :


--
nosy: +eric.araujo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> Firstly, I don't think it makes any sense to set encoding information
> globally for the Popen object. As a simple example, consider using
> Python to write a test suite for the iconv command line tool: there's
> only one Popen instance (for the iconv call), but different encodings
> for stdin and stdout.

Isn't that the exception rather than the rule? I think it actually makes
sense, in at least 99.83% of cases ;-), to have a common encoding
setting for all streams.
(I'm not sure about the "errors" setting, though: should we use strict
for stdin/stdout and backslashreplace for stderr, as the interpreter
does?)

Perhaps the common case should be made extra easy.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Nick Coghlan]

Nick Coghlan  added the comment:

I discovered this same problem recently when updating the subprocess docs, and 
also in working on the improved shell invocation support I am proposing for 3.3 
(#13238).

I initially posted an earlier variant this suggestion as a new issue (#13442), 
but Victor redirected me here.

Firstly, I don't think it makes any sense to set encoding information globally 
for the Popen object. As a simple example, consider using Python to write a 
test suite for the iconv command line tool: there's only one Popen instance 
(for the iconv call), but different encodings for stdin and stdout.

Really, we want to be able to make full use of Python 3's layered I/O model, 
but we want the subprocess pipe instances to be slotted in at the lowest layer 
rather than creating them ourselves.

The easiest way to do that is to have a separate class that specifies the 
additional options for pipe creation and does the wrapping:

class TextPipe:
def __init__(self, *args, **kwds):
self.args = args
self.kwds = kwds
def wrap_pipe(self, pipe):
return io.TextIOWrapper(pipe, *self.args, **self.kwds)

The stream creation process would then include a new "wrap = 
getattr(stream_arg, 'wrap_pipe', None)" check that is similar to the existing 
check for subprocess.PIPE, but invokes the method to wrap the pipe after 
creating it.

So to read UTF-8 encoded data from a subprocess, you could just do:

data = check_stdout(cmd, stdout=TextPipe('utf-8'), stderr=STDOUT)

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[STINNER Victor]

STINNER Victor  added the comment:

> ... it always seems to use the local 8-bit encoding

The locale encoding is not necessary a 8-bit encoding, it can by a multibyte 
like... UTF-8 :-)

--

subprocess.patch: You should maybe use io.open(process.stdout.fileno(), 
encoding=..., errors=...) instead of codecs.getreader/getwriter. The code will 
be closer to Python 3. I think that the io module has a better support of 
unicode than codec reader/writer objects and a nicer API. See:
http://bugs.python.org/issue8796#msg106339

--

> ... allowing [encoding and errors] to accept either a single string
> (as now), or a dict with keys 'stdin', 'stdout', 'stderr'

I like this idea. But what about other TextIOWrapper (or other file classes) 
options: buffer size, newline, line_buffering, etc.?

Why not using a dict for existing stdin, stdout and stderr arguments? Dummy 
example:

process = Popen(
   command,
   stdin={'file': PIPE, 'encoding': 'iso-8859-1', 'newline': False},
   stdout={'file': PIPE', 'encoding': 'utf-8', 'buffering': 0, 
'line_buffering': False},
   ...)

If stdin, stdout or stderr is a dict: the default value of its 'file' key can 
be set to PIPE. I don't think that it's possible to choose the encoding, buffer 
size, or anything else if stdin, stdout or stderr is not a pipe.

With this solution, you cannot specify the encoding for stdin, stdout and 
stderr at once. You have at least to repeat the encoding for stdin and stdout 
(and use stderr=STDOUT).

--

I still hesitate to accept this feature request. Is it really needed to add 
extra arguments for TextIOWrapper? Can't the developer create its own 
TextIOWrapper object with all interesting options?

In Python 3, be able to specify stdout encoding is an interesting feature. 
Control the buffers size is also an interesting option.

My problem is maybe the usage of a dict to specify various options. I'm not 
sure that it is extensible to support future needs.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[STINNER Victor]

STINNER Victor  added the comment:

About the topic:
> subprocess seems to use local 8-bit encoding and gives no choice
I don't understand that: by default, Python 2 and Python 3 use byte strings, so 
there is no encoding (nor error handler).

I don't see how you can get unicode from a process only using subprocess. But 
with Python 3, you can get unicode if you set universal_newlines option to True.

So for Python 2, it's a new feature (get unicode), and for Python 3, it's a new 
option to specify the encoding. The title should be changed to something like 
"subprocess: add an option to specify stdin, stdout and/or stderr encoding and 
errors" and the type should be changed to "feature request".

Or am I completly wrong?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Ned Deily]

Changes by Ned Deily :


--
nosy: +haypo -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Mark Lawrence]

Mark Lawrence  added the comment:

In 2.7 and py3k test_subprocess has a class BaseTestCase which has a 
assertStderrEqual method.  These don't exist in 3.1.  I believe that the py3k 
code will need to be backported to 3.1.  Can this be done on this issue, or do 
we need a new one to keep things clean?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Mark Lawrence]

Mark Lawrence  added the comment:

Ran new unit test before and after patching subprocess on Windows Vista against 
3.1 debug maintenance release, all ok apart from this at end of latter.

  File "test\test_subprocess.py", line 568, in test_encoded_stderr
self.assertEqual(p.stderr.read(), send)
AssertionError: 'ï[32943 refs]\r\n' != 'ï'

I'm sure I've seen a ref to this somewhere, can anyone remember where?

--
nosy: +BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Mark Summerfield]

Mark Summerfield  added the comment:

On Thu, Dec 31, 2009 at 1:30 PM, Amaury Forgeot d'Arc
 wrote:
>
> Amaury Forgeot d'Arc  added the comment:
>
> I don't understand. How is the subprocess stdout related to the main
> program output?
> Stream-specific encoding could be useful for subprocesses that expect
> latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
> support this.

Yes, you're right.

(What I had in mind was a scenario where you read one process's stdout
and wrote to another process's stdin; but of course using your errors
& encoding arguments this will work because there'll be two separate
process objects each of which can have its encoding and errors set
separately.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

I don't understand. How is the subprocess stdout related to the main
program output?
Stream-specific encoding could be useful for subprocesses that expect
latin-1 from stdin but write utf-8 to stdout. I'm not sure we should
support this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Mark Summerfield]

Mark Summerfield  added the comment:

I agree with Florian Mayer that the encoding handling should be
stream-specific. You could easily be reading the stdout of some third
party program that uses, say, latin1, but want to do your own output in,
say, utf-8.

One solution that builds on what Amaury Forgeot d'Arc has done (i.e.,
the encoding and errors parameters) by allowing those parameters to
accept either a single string (as now), or a dict with keys 'stdin',
'stdout', 'stderr'. Of course it is possible that the client might not
specify all the dict's keys in which case those would use the normal
default (local 8-bit etc.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Shahar Or]

Changes by Shahar Or :


--
nosy: +DawnLight

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Added file: http://bugs.python.org/file14295/test_subprocess3.py.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Florian Mayer  added the comment:

Should we also cover the unusual case where stdout, stderr and stdin
have different encodings, because now we are assuming the are all the same.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Added file: http://bugs.python.org/file14294/subprocess3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Added file: http://bugs.python.org/file14293/subprocess.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Removed file: http://bugs.python.org/file14291/subprocess3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Removed file: http://bugs.python.org/file14289/subprocess.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Two things:
1. The argument should be called `errors` for consistency with open()
and TextIOWrapper(), not `error`
2. You should add some unit tests.

--
nosy: +pitrou
stage: needs patch -> patch review
versions: +Python 2.7, Python 3.2 -Python 2.6, Python 3.0

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Removed file: http://bugs.python.org/file14290/subprocess3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Added file: http://bugs.python.org/file14291/subprocess3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Added file: http://bugs.python.org/file14290/subprocess3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Changes by Florian Mayer :


Removed file: http://bugs.python.org/file14286/subprocess.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Florian Mayer  added the comment:

Cosmetic update.

--
Added file: http://bugs.python.org/file14289/subprocess.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Florian Mayer]

Florian Mayer  added the comment:

I wrote a patch to add encoding and error to subprocess.Popen in Python
2.7 (trunk).

--
keywords: +patch
nosy: +segfaulthunter
Added file: http://bugs.python.org/file14286/subprocess.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc  added the comment:

I propose to add two parameters (encoding, error) to the
subprocess.Popen function.
- python 2.x could build and return codecs.StreamReader objects
- python 3.x would just pass these parameters to io.TextIOWrapper

I'll try to come with a patch.

--
nosy: +amaury.forgeotdarc
stage:  -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Sridhar Ratnakumar]

Sridhar Ratnakumar  added the comment:

Related discussion thread: https://answers.launchpad.net/bzr/+question/63601

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Sridhar Ratnakumar]

Changes by Sridhar Ratnakumar :


--
versions: +Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Sridhar Ratnakumar]

Changes by Sridhar Ratnakumar :


--
nosy: +srid

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6135] subprocess seems to use local 8-bit encoding and gives no choice

[Mark Summerfield]

New submission from Mark Summerfield :

When I start a process with subprocess.Popen() and pipe the stdin and
stdout, it always seems to use the local 8-bit encoding.

I tried setting process.stdin.encoding = "utf8" and the same for stdout
(where process is the subprocess object), but to no avail.

I also tried using shell=True since on Mac, Terminal.app is fine with
Unicode, but that didn't work.

So basically, I have programs that output Unicode and run fine on the
Mac terminal, but that cannot be executed by subprocess because
subprocess uses the mac_roman encoding instead of Unicode.

I wish it were possible to specify the stdin and stdout encoding that is
used; then I could use the same one on all platforms. (But perhaps it is
possible, and I just haven't figured out how?)

--
components: Library (Lib)
messages: 88466
nosy: mark
severity: normal
status: open
title: subprocess seems to use local 8-bit encoding and gives no choice
type: behavior
versions: Python 3.0

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com