[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[miss-islington]

miss-islington  added the comment:


New changeset de2c91a89aa0e89d5d01d71697d0f27a9f60c11d by Miss Islington (bot) 
in branch '3.9':
bpo-26897: Clarify Popen stdin, stdout, stderr file object docs (GH-30231)
https://github.com/python/cpython/commit/de2c91a89aa0e89d5d01d71697d0f27a9f60c11d


--

___
Python tracker 
<https://bugs.python.org/issue26897>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[miss-islington]

miss-islington  added the comment:


New changeset 1f956d11dbd952fed9b214a0c6e5fd9f91a1f4ca by Miss Islington (bot) 
in branch '3.10':
bpo-26897: Clarify Popen stdin, stdout, stderr file object docs (GH-30231)
https://github.com/python/cpython/commit/1f956d11dbd952fed9b214a0c6e5fd9f91a1f4ca


--

___
Python tracker 
<https://bugs.python.org/issue26897>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[miss-islington]

Change by miss-islington :


--
pull_requests: +29711
pull_request: https://github.com/python/cpython/pull/31588

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[Jelle Zijlstra]

Change by Jelle Zijlstra :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[Jelle Zijlstra]

Jelle Zijlstra  added the comment:


New changeset dd69f734218ac5d3a551227069ac53ee09b0cd3e by Kumar Aditya in 
branch 'main':
bpo-26897: Clarify Popen stdin, stdout, stderr file object docs (GH-30231)
https://github.com/python/cpython/commit/dd69f734218ac5d3a551227069ac53ee09b0cd3e


--
nosy: +Jelle Zijlstra

___
Python tracker 
<https://bugs.python.org/issue26897>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 4.0 -> 5.0
pull_requests: +29710
pull_request: https://github.com/python/cpython/pull/31587

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Min RK]

Min RK  added the comment:

It appears that connect_read_pipe also doesn't accept pipes returned by 
`os.pipe`. If that's the case, what _does_ ProactorEventLoop.connect_read_pipe 
accept? I haven't been able to find any examples of `connect_read_pipe` that 
work on Windows, and every connect_read_pipe call in the cpython test suite 
appears to be skipped on win32. Should it still be raising NotImplementedError 
on ProactorEventLoop?

I think the error handling could be better (I only get logged errors, nothing I 
can catch/handle). It seems like `connect_read_pipe` itself should raise when 
it fails to register the pipe with IOCP. If that's not feasible, 
connection_lost/transport.close should probably be triggered, but it isn't with 
Python 3.9, at least.

Example that works on posix, but seems to fail with non-catchable errors with 
ProactorEventLoop:

```
import asyncio
import os
import sys

class PipeProtocol(asyncio.Protocol):
def __init__(self):
self.finished = asyncio.Future()

def connection_made(self, transport):
print("connection made", file=sys.stderr)
self.transport = transport

def connection_lost(self, exc):
print("connection lost", exc, file=sys.stderr)
self.finished.set_result(None)

def data_received(self, data):
print("data received", data, file=sys.stderr)
self.handler(data)

def eof_received(self):
print("eof received", file=sys.stderr)
self.finished.set_result(None)

async def test():
r, w = os.pipe()
rf = os.fdopen(r, 'r')
x, p = await asyncio.get_running_loop().connect_read_pipe(PipeProtocol, rf)
await asyncio.sleep(1)
print("writing")
os.write(w, b'asdf')
await asyncio.sleep(2)
print("closing")
os.close(w)
await asyncio.wait([p.finished], timeout=3)
x.close()

if __name__ == "__main__":
asyncio.run(test())
```

--
nosy: +minrk
versions: +Python 3.10, Python 3.11, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[miss-islington]

miss-islington  added the comment:


New changeset 91e888904478271c27c52c773863b41f5a8f7f30 by Miss Islington (bot) 
in branch '3.10':
bpo-14916: use specified tokenizer fd for file input (GH-31006)
https://github.com/python/cpython/commit/91e888904478271c27c52c773863b41f5a8f7f30


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[miss-islington]

Change by miss-islington :


--
pull_requests: +29250
pull_request: https://github.com/python/cpython/pull/31065

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[miss-islington]

miss-islington  added the comment:


New changeset 89b13042fcfc95bae21a49806a205ef62f1cdd73 by Paul m. p. P in 
branch 'main':
bpo-14916: use specified tokenizer fd for file input (GH-31006)
https://github.com/python/cpython/commit/89b13042fcfc95bae21a49806a205ef62f1cdd73


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

Change by pmp-p :


--
pull_requests: +29187
pull_request: https://github.com/python/cpython/pull/31006

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[Kumar Aditya]

Change by Kumar Aditya :


--
keywords: +patch
nosy: +kumaraditya303
nosy_count: 3.0 -> 4.0
pull_requests: +28459
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/30231

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] [doc] Clarify Popen stdin, stdout, stderr

[Irit Katriel]

Change by Irit Katriel :


--
keywords: +easy
title: Clarify Popen stdin, stdout, stderr -> [doc] Clarify Popen stdin, 
stdout, stderr
versions: +Python 3.10, Python 3.11, Python 3.9 -Python 2.7, Python 3.5, Python 
3.6

___
Python tracker 
<https://bugs.python.org/issue26897>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43308] subprocess.Popen leaks file descriptors opened for DEVNULL or PIPE stdin/stdout/stderr arguments

[Alexey Izbyshev]

Change by Alexey Izbyshev :


--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43308] subprocess.Popen leaks file descriptors opened for DEVNULL or PIPE stdin/stdout/stderr arguments

[cptpcrd]

New submission from cptpcrd :

TL;DR: subprocess.Popen's handling of file descriptors opened for DEVNULL or 
PIPE inputs/outputs has serious problems, and it can be coerced into leaking 
file descriptors in several ways. This can cause issues related to resource 
exhaustion.

# The basic problem

As part of its setup, Popen.__init__() calls Popen._get_handles(), which looks 
at the given stdin/stdout/stderr arguments and returns a tuple of 6 file 
descriptors (on Windows, file handles) indicating how stdin/stdout/stderr 
should be redirected. However, these file descriptors aren't properly closed if 
exceptions occur in certain cases.

# Variant 1: Bad argument errors (introduced in 3.9)

The first variant of this bug is shockingly easy to reproduce (note that this 
only works on platforms with /proc/self/fd, like Linux):

```
import os, subprocess

def show_fds():
for entry in os.scandir("/proc/self/fd"):
print(entry.name, "->", os.readlink(entry.path))

print("Before:")
show_fds()

try:
subprocess.Popen(["ls"], stdin=subprocess.PIPE, user=1.0)
except TypeError as e:  # "User must be a string or an integer"
print(e)

print("After:")
show_fds()
```

This produces something like:

```
Before:
0 -> /dev/pts/1
1 -> /dev/pts/1
2 -> /dev/pts/1
3 -> /proc/12345/fd
User must be a string or an integer
After:
0 -> /dev/pts/1
1 -> /dev/pts/1
2 -> /dev/pts/1
3 -> pipe:[1234567]
3 -> pipe:[1234567]
5 -> /proc/12345/fd
```

The process never got launched (because of the invalid `user` argument), but 
the (unused) pipe created for piping to stdin is left open! Substituting 
DEVNULL for PIPE instead leaves a single file descriptor open to `/dev/null`.

This happens because the code that validates the `user`, `group`, and 
`extra_groups` arguments [1] was added to Popen.__init__() *after* the call to 
Popen._get_handles() [2], and there isn't a try/except that closes the file 
descriptors if an exception gets raised during validation (which can easily 
happen).

# Variant 2: Error opening file descriptors (seems to have been around in 
`subprocess` forever)

Within Popen._get_handles() (on Windows [3] or POSIX [4]), previously opened 
file descriptors are not closed if an error occurs while opening later file 
descriptors.

For example, take the case where only one more file descriptor can be opened 
without hitting the limit on the number of file descriptors, and 
`subprocess.Popen(["ls"], stdin=subprocess.DEVNULL, stdout=supbrocess.PIPE)` is 
called. subprocess will be able to open `/dev/null` for stdin, but trying to 
creating a `pipe()` for stdout will fail with EMFILE or ENFILE. Since 
Popen._get_handles() doesn't handle exceptions from `pipe()` (or when opening 
`/dev/null`), the `/dev/null` file descriptor opened for stdin will be be left 
open.

This variant is most easily triggered by file descriptor exhaustion, and it 
makes that problem worse by leaking even *more* file descriptors.

Here's an example that reproduces this by monkey-patching `os` to force an 
error:

```
import os, subprocess

def show_fds():
for entry in os.scandir("/proc/self/fd"):
print(entry.name, "->", os.readlink(entry.path))

print("Before:")
show_fds()

# Trigger an error when trying to open /dev/null
os.devnull = "/NOEXIST"

try:
subprocess.Popen(["ls"], stdin=subprocess.PIPE, stdout=subprocess.DEVNULL)
except FileNotFoundError as e:  # "User must be a string or an integer"
print(e)

print("After:")
show_fds()
```

Output:

```
Before:
0 -> /dev/pts/1
1 -> /dev/pts/1
2 -> /dev/pts/1
3 -> /proc/12345/fd
[Errno 2] No such file or directory: '/dev/null'
After:
0 -> /dev/pts/1
1 -> /dev/pts/1
2 -> /dev/pts/1
3 -> pipe:[1234567]
4 -> pipe:[1234567]
5 -> /proc/12345/fd
```

Again, the pipe is left open.

# Paths to fix.

Variant 1 can be fixed by simply reordering code in Popen.__init__() (and 
leaving comments warning about the importance of maintaining the order!). I've 
attached a basic patch that does this.

Variant 2 might take some more work -- especially given the shared 
Popen._devnull file descriptor that needs to be accounted for separately -- and 
may require significant changes to both Popen.__init__() and 
Popen._get_handles() to fix.

[1]: https://github.com/python/cpython/blob/master/Lib/subprocess.py#L872
[2]: https://github.com/python/cpython/blob/master/Lib/subprocess.py#L840
[3]: https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1251
[4]: https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1581

--
components: Library (Lib)
files: subprocess-validation-fd-leak.patch
keywords: patch
messages: 387589
nosy: cptpcrd
priority: normal
severity: normal
status: open
title: subprocess.Popen leaks file descriptors o

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

Change by pmp-p :


--
pull_requests: +21251
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/22190

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

Change by pmp-p :


--
versions: +Python 3.6, Python 3.7, Python 3.8, Python 3.9 -Python 2.6, Python 
2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[pmp-p]

pmp-p  added the comment:

all PyRun_InteractiveOne* functions are also affected
it is really annoying when implementing repl behaviour when embedding( use 
cases :  pyodide, android, wasi )

But I think the correct patch is : 
-char *newtok = PyOS_Readline(stdin, stdout, tok->prompt);
+char *newtok = PyOS_Readline(tok->fp? tok->fp : stdin, stdout, 
tok->prompt);

--
nosy: +pmpp

___
Python tracker 
<https://bugs.python.org/issue14916>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37275] GetConsole(Output)CP is used even when stdin/stdout is redirected

[Inada Naoki]

Inada Naoki  added the comment:

On Sat, Jun 15, 2019 at 2:43 AM Eryk Sun  wrote:
>
> Eryk Sun  added the comment:
>
> > # Power Shell 6 (use cp65001 by default)
> > PS C:¥> python3 -c "print('おはよう')" > ps.txt
>
> PowerShell standard I/O redirection is different from any shell I've ever 
> used. In this case, it runs Python with StandardOutput set to a handle for a 
> pipe instead of a handle for the file. It decodes Python's output using 
> whatever encoding is configured for input and re-encodes it with whatever 
> encoding is configured for output.

I'm sorry,  I mixed my assumption.  I checked `os.device_encoding()` in cmd,
but forgot to check it on Power Shell.  All I said about Power Shell was just
my assumption and it was wrong.  And thank you for clarifying.

I confirmed writing UTF-8 to pipe cause mojibake, because Power Shell decodes
it using cp932.

```
PS C:\Users\inada-n> python3 -Xutf8 -c "import os,sys;
print(os.device_encoding(1), sys.stdout.encoding, file=sys.stderr);
print('こんにちは')" >x
None utf-8
PS C:\Users\inada-n> type x
```

Hmm, how can I teach to Power Shell about python3 is using
UTF-8 for stdout?
It seems cmd.exe with chcp 65001 and PYTHONUTF8=1 is better
than PowerShell when I want to use UTF-8 on Windows.

Anyway, nothing is wrong about python.  I just didn't understand
PowerShell at all.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37275] GetConsole(Output)CP is used even when stdin/stdout is redirected

[Inada Naoki]

Change by Inada Naoki :


--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37275] GetConsole(Output)CP is used even when stdin/stdout is redirected

[Eryk Sun]

Eryk Sun  added the comment:

> # Power Shell 6 (use cp65001 by default)
> PS C:¥> python3 -c "print('おはよう')" > ps.txt

PowerShell standard I/O redirection is different from any shell I've ever used. 
In this case, it runs Python with StandardOutput set to a handle for a pipe 
instead of a handle for the file. It decodes Python's output using whatever 
encoding is configured for input and re-encodes it with whatever encoding is 
configured for output. 

To see what Python is actually writing, try using Start-Process with 
StandardOutput redirected to the file. For example:

Start-Process -FilePath python3.exe -ArgumentList "-c `"print('おはよう')`"" 
-NoNewWindow -Wait -RedirectStandardOutput ps.txt

> # cmd.exe
> C:¥> chcp 65001
> C:¥> python3 -c "print('おはよう')" > cmd.txt

CMD uses simple redirection, like every other shell I've ever used. It runs 
python3.exe with a handle for the file as its StandardOutput. So "cmd.txt" 
contains exactly what Python writes.

> * TextIOWrapper tries `os.device_encoding(1)`
> * `os.device_encoding(1)` use GetConsoleOutputCP() without checking stdout is 
> console

No, _Py_device_encoding returns None if the isatty(fd) is false, i.e. for a 
pipe or disk file. In this case, TextIOWrapper defaults to the encoding from 
locale.getpreferredencoding().

The current preferred encoding is the system ANSI codepage from GetACP(). 
Changing the default to UTF-8 would be disruptive. You can use UTF-8 mode (i.e. 
-X utf8). Or, to override just stdin, stdout, and stderr, set the environment 
variable "PYTHONIOENCODING=utf-8".

> In the example above, a console is attached when python is called from 
> Power Shell 6, but it is not attached when python is called from 
> cmd.exe.

In both cases the Python process inherits the console of the parent shell. The 
only way to run python.exe without a console is to use the CreateProcess 
creation flag DETACHED_PROCESS.

> There is a relating issue: UTF-8 mode doesn't override 
> stdin,stdout,stderr encoding when console is attached.

It works for me. For example, testing with stdout redirected to a pipe:

C:\>python -c "import sys;print(sys.stdout.encoding)" | more
cp1252

C:\>python -X utf8 -c "import sys;print(sys.stdout.encoding)" | more
utf-8

--

___
Python tracker 
<https://bugs.python.org/issue37275>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37275] GetConsole(Output)CP is used even when stdin/stdout is redirected

[Steve Dower]

Steve Dower  added the comment:

Isn't the point that device_encoding(FD) gets the encoding of the specified 
file? In this case stdout?

It seems odd that chcp doesn't actually update the console code page here, as 
that is its entire purpose. Perhaps TextIOWrapper is actually getting ACP 
rather than the console encoding through some other path?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37275] GetConsole(Output)CP is used even when stdin/stdout is redirected

[Karthikeyan Singaravelan]

Change by Karthikeyan Singaravelan :


--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37275] GetConsole(Output)CP is used even when stdin/stdout is redirected

[Inada Naoki]

New submission from Inada Naoki :

When stdout is redirected to file and cp65001 is used, stdout encoding is 
unexpectable:

  # Power Shell 6 (use cp65001 by default)
  PS C:¥> python3 -c "print('おはよう')" > ps.txt

  # cmd.exe
  C:¥> chcp 65001
  C:¥> python3 -c "print('おはよう')" > cmd.txt

Now, ps.txt is encoded by UTF-8, but cmd.txt is encoded by cp932 (ACP).


This is because:

* TextIOWrapper tries `os.device_encoding(1)`
* `os.device_encoding(1)` use GetConsoleOutputCP() without checking stdout is 
console

In the example above, a console is attached when python is called from Power 
Shell 6, but it is not attached when python is called from cmd.exe.

I think using GetConsoleOutputCP() for non console is abusing.

---

There is a relating issue: UTF-8 mode doesn't override stdin,stdout,stderr 
encoding when console is attached.

On Unix, os.device_encoding() uses locale encoding and UTF-8 mode overrides 
locale encoding.  Good.

But on Windows, os.device_encoding() uses GetConsole(Output)CP().  UTF-8 mode 
doesn't override it.

If we stop abusing GetConsoleOutputCP(), this issue is fixed automatically.
But if we keep using GetConsoleOutputCP() for stdout which is not a console, 
UTF-8 mode should override it.

--
components: Windows
messages: 345551
nosy: inada.naoki, paul.moore, steve.dower, tim.golden, vstinner, zach.ware
priority: normal
severity: normal
status: open
title: GetConsole(Output)CP is used even when stdin/stdout is redirected
type: behavior
versions: Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue37275>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[Andrew Svetlov]

Change by Andrew Svetlov :


--
resolution:  -> fixed
versions:  -Python 3.5, Python 3.6, Python 3.7, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[Simon Bernier St-Pierre]

Change by Simon Bernier St-Pierre :


--
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[miss-islington]

miss-islington  added the comment:


New changeset f0d4c64019ecf8a5f362aa5a478786241613e5c3 by Miss Islington (bot) 
(sbstp) in branch 'master':
bpo-36686: Improve the documentation of the std* params in loop.subprocess_exec 
(GH-13586)
https://github.com/python/cpython/commit/f0d4c64019ecf8a5f362aa5a478786241613e5c3


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[Simon Bernier St-Pierre]

Change by Simon Bernier St-Pierre :


--
keywords: +patch
pull_requests: +13493
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/13586

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Pull Request is welcome!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[Simon Bernier St-Pierre]

Simon Bernier St-Pierre  added the comment:

Could be cool to also mention that `encoding` / `errors` does not work yet. 
https://bugs.python.org/issue31087

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36686] Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not clear how to inherit stdin, stdout or stderr in the subprocess

[Simon Bernier St-Pierre]

New submission from Simon Bernier St-Pierre :

I had trouble figuring out how to simply inherit stdin, stdout, or stderr in 
the asyncio.create_subprocess_exec / asyncio.subprocess_exec docs. My 
experiments show that passing either None or `sys.std*` works but the way the 
docs are written make it hard to figure that out in my opinion.

> stdout: either a file-like object representing the pipe to be connected to 
> the subprocess’s standard output stream using connect_read_pipe(), or the 
> subprocess.PIPE constant (default). By default a new pipe will be created and 
> connected.

I would add a mention that using None makes the subprocess inherit the file 
descriptor.

--
components: asyncio
messages: 340593
nosy: asvetlov, sbstp, yselivanov
priority: normal
severity: normal
status: open
title: Docs: asyncio.loop.subprocess_exec documentation is confusing, it's not 
clear how to inherit stdin, stdout or stderr in the subprocess
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue36686>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[STINNER Victor]

Change by STINNER Victor :


--
nosy:  -vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Martijn Pieters]

Martijn Pieters  added the comment:

I'm trying to figure out why Windows won't let us do this. I think the reason 
is that sys.std(in|out) filehandles are not opened as pipes, and do not have 
the required OVERLAPPED flag set (see the CreateIoCompletionPort documentation 
at 
https://docs.microsoft.com/en-us/windows/desktop/fileio/createiocompletionport; 
it's that function that is used to handle pipes (via IocpProactor.recv -> 
IocpProactor._register_with_iocp -> overlapped.CreateIoCompletionPort).

The solution then would be to create a pipe for a stdio filehandle with the 
flag set.

And that's where my Windows-fu ends, and where I lack the VM and motivation to 
go try that out.

--
nosy: +mjpieters

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] Clarify Popen stdin, stdout, stderr

[Martin Panter]

Martin Panter added the comment:

Yes, in all cases I understand it is just the file descriptor that matters. 
Perhaps any documentation update could also take Issue 24358 into account, 
which points out that compressed and buffered file object wrappers also cause 
inconsistencies in the data at the high-level file object and the OS-level file 
descriptor.

--
nosy: +martin.panter
stage:  -> needs patch
versions: +Python 2.7, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Gabriel Mesquita Cangussu]

Gabriel Mesquita Cangussu added the comment:

Terry,

About your question, "Isn't there some other way to asynchronously read/file 
files, as opposed to sockets and pipes, on Windows?", that is exactly the 
problem I was trying to overcome, specifically for reading the stdin.

On my research I found both this stackoverflow questions with no valid answers:

http://stackoverflow.com/questions/31510190/aysncio-cannot-read-stdin-on-windows

http://stackoverflow.com/questions/31511563/file-to-socket-adapter-in-python

And I understood that 'pipe' wouldn't mean a 'file', but why can't I use 
'sys.stdin' with 'ProactorEventLoop.connect_read_pipe()', can't 'sys.stdin' be 
a pipe on Windows like it can on Posix?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26897] Clarify Popen stdin, stdout, stderr

[Yclept Nemo]

New submission from Yclept Nemo:

From: https://docs.python.org/dev/library/subprocess.html#popen-constructor

"..., an existing file descriptor (a positive integer), an existing file 
object, and None."

It should be made clear that the file object must be backed by an operating 
system file, i.e. must have a valid file descriptor.

As it stands, the documentation suggests implicit concurrency: that any file 
object can be used, and on read/write from the subprocess/OS arbitrary python 
code will be executed, i.e. the python interpreter is pre-empted or python 
spawns a hidden I/O thread.

Anyway, I'm pretty sure subprocess.Popen() just redirects the forked process' 
stdout/stdin/stderr to the file descriptors from the provided file object.

--
assignee: docs@python
components: Documentation
messages: 264579
nosy: Yclept.Nemo, docs@python
priority: normal
severity: normal
status: open
title: Clarify Popen stdin, stdout, stderr
versions: Python 3.6

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26897>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Steve Dower]

Steve Dower added the comment:

Pipes and file handles are equivalent in Windows, but socket handles are their 
own namespace and have their own functions. Some Python code will switch 
between them automatically to make socket functions work with file descriptors, 
but generally I'd expect stream pipes (as opposed to, IIRC, datagram pipes) to 
be compatible with files but not sockets.

Not entirely sure how that plays into this issue, but it's a bit more 
background.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Terry J. Reedy]

Terry J. Reedy added the comment:

The surprise to me, being on Windows, is that the pipe connection methods 
sometimes work with non-pipes.  The limitations of Windows event loops are 
given in https://docs.python.org/3/library/asyncio-eventloops.html#windows.  
The pipe connection functions are discussed in 
https://docs.python.org/3/library/asyncio-eventloop.html#connect-pipes.  Both 
say that the methods do not work with Windows' SelectorEventLoop.

My understanding is that this is because Windows' select() call does not work 
with pipes -- meaning honest-to-goodness OS pipes.  So I understood "*pipe* is 
file-like object." more as a un-surprising statement of fact than as a 
permissive "'pipe' can be any file-like object and not necessarily a pipe".

If 'pipe' were intended to mean 'file-like', then why use 'pipe'?  But I can 
see how a current unix user would understand that sentence the other way.  
Perhaps the sentence should read "For SelectorEventLoops (not on Windows), 
*pipe* can also be any file-like object with the appropriate methods." -- 
assuming that this is true on all non-Windows systems.

Isn't there some other way to asynchronously read/file files, as opposed to 
sockets and pipes, on Windows?

--
nosy: +terry.reedy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Steve Dower]

Steve Dower added the comment:

> File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in 
> _register_with_iocp
>   _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)

I don't think there's any case where this is going to work on an actual file - 
CreateIoCompletionPort needs a handle and not a file descriptor.

Presumably this line of code normally gets used with something else that 
defines fileno()?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26832] ProactorEventLoop doesn't support stdin/stdout nor files with connect_read_pipe/connect_write_pipe

[Gabriel Mesquita Cangussu]

New submission from Gabriel Mesquita Cangussu:

The documentation of asyncio specifies that the methods connect_read_pipe and 
connect_write_pipe are available on Windows with the ProactorEventLoop. The 
documentation then says that those methods accept file-like objects on the pipe 
parameter. However, the methods doesn't seem to work with stdio or any disk 
file under Windows. The following example catches this problem:

import asyncio
import sys

class MyProtocol(asyncio.Protocol):
def connection_made(self, transport):
print('connection established')

def data_received(self, data):
print('received: {!r}'.format(data.decode()))

def connection_lost(self, exc):
print('lost connection')

if sys.platform.startswith('win32'):
loop = asyncio.ProactorEventLoop()
else:
loop = asyncio.SelectorEventLoop()
coro = loop.connect_read_pipe(MyProtocol, sys.stdin)
loop.run_until_complete(coro)
loop.run_forever()
loop.close()


This code when executed on Ubuntu have the desired behavior, but under Windows 
10 it gives OSError: [WinError 6] The handle is invalid. The complete output is 
this:

c:\Users\Gabriel\Documents\Python Scripts>python async_pipe.py
connection established
Fatal read error on pipe transport
protocol: <__main__.MyProtocol object at 0x01970EB2FAC8>
transport: <_ProactorReadPipeTransport fd=0>
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\asyncio\proactor_events.py", line 195, in 
_loop_reading
self._read_fut = self._loop._proactor.recv(self._sock, 4096)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 425, in 
recv
self._register_with_iocp(conn)
  File "C:\Program Files\Python35\lib\asyncio\windows_events.py", line 606, in 
_register_with_iocp
_overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 6] Identificador inválido
lost connection


I think that the documentation should state that there is no support for disk 
files and stdio with the methods in question and also state what exactly they 
support (an example would be nice). And, of course, better support for watching 
file descriptors on Windows on future Python releases would be nice.

Thank you,
Gabriel

--
components: Windows, asyncio
messages: 264043
nosy: Gabriel Mesquita Cangussu, gvanrossum, haypo, paul.moore, steve.dower, 
tim.golden, yselivanov, zach.ware
priority: normal
severity: normal
status: open
title: ProactorEventLoop doesn't support stdin/stdout nor files with 
connect_read_pipe/connect_write_pipe
type: behavior
versions: Python 3.5

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26832>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Yauheni Kaliuta]

Yauheni Kaliuta added the comment:

Any progress with the problem? I just wanted to use the feature, but it looks 
like the bug.sh is still reproduces the bug.

--
nosy: +Yauheni Kaliuta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Python3 pickle vs. stdin/stdout - unable to get clean byte streams in Python 3

[John Nagle]

On 3/12/2015 5:18 PM, John Nagle wrote:
 On 3/12/2015 2:56 PM, Cameron Simpson wrote:
 On 12Mar2015 12:55, John Nagle na...@animats.com wrote:
 I have working code from Python 2 which uses pickle to talk to a
 subprocess via stdin/stdio.  I'm trying to make that work in Python
 3.

   I'm starting to think that the cpickle module, which Python 3
uses by default, has a problem. After the program has been
running for a while, I start seeing errors such as

  File C:\projects\sitetruth\InfoSiteRating.py, line 200, in scansite
if len(self.badbusinessinfo)  0 :  # if bad stuff
NameError: name 'len' is not defined

which ought to be impossible in Python, and

  File C:\projects\sitetruth\subprocesscall.py, line 129, in send
self.writer.dump(args)  # send data
OSError: [Errno 22] Invalid argument

from somewhere deep inside CPickle.

I got

  File C:\projects\sitetruth\InfoSiteRating.py, line 223, in
get_rating_text
(ratingsmalliconurl, ratinglargiconurl, ratingalttext) =
DetailsPageBuilder.getratingiconinfo(rating)
NameError: name 'DetailsPageBuilder' is not defined
(That's an imported module.  It worked earlier in the run.)

and finally, even after I deleted all .pyc files and all Python
cache directories:  

Fatal Python error: GC object already tracked

Current thread 0x1a14 (most recent call first):
  File C:\python34\lib\site-packages\pymysql\connections.py, line 411
in description
  File C:\python34\lib\site-packages\pymysql\connections.py, line 1248
in _get_descriptions
  File C:\python34\lib\site-packages\pymysql\connections.py, line 1182
in _read_result_packet
  File C:\python34\lib\site-packages\pymysql\connections.py, line 1132
in read
  File C:\python34\lib\site-packages\pymysql\connections.py, line 929
in _read_query_result
  File C:\python34\lib\site-packages\pymysql\connections.py, line 768
in query
  File C:\python34\lib\site-packages\pymysql\cursors.py, line 282 in
_query
  File C:\python34\lib\site-packages\pymysql\cursors.py, line 134 in
execute
  File C:\projects\sitetruth\domaincacheitem.py, line 128 in select
  File C:\projects\sitetruth\domaincache.py, line 30 in search
  File C:\projects\sitetruth\ratesite.py, line 31 in ratedomain
  File C:\projects\sitetruth\RatingProcess.py, line 68 in call
  File C:\projects\sitetruth\subprocesscall.py, line 140 in docall
  File C:\projects\sitetruth\subprocesscall.py, line 158 in run
  File C:\projects\sitetruth\RatingProcess.py, line 89 in main
  File C:\projects\sitetruth\RatingProcess.py, line 95 in module

That's a definite memory error.

So something is corrupting memory.  Probably CPickle.

All my code is in Python. Every library module came in via pip, into a
clean Python 3.4.3 (32 bit) installation on Win7/x86-64.
Currently installed packages:

beautifulsoup4 (4.3.2)
dnspython3 (1.12.0)
html5lib (0.999)
pip (6.0.8)
PyMySQL (0.6.6)
pyparsing (2.0.3)
setuptools (12.0.5)
six (1.9.0)

And it works fine with Python 2.7.9.

Is there some way to force the use of the pure Python pickle module?
My guess is that there's something about reusing pickle instances
that botches memory uses in CPython 3's C code for cpickle.

John Nagle  

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 pickle vs. stdin/stdout - unable to get clean byte streams in Python 3

[Steven D'Aprano]

John Nagle wrote:

I'm starting to think that the cpickle module, which Python 3
 uses by default, has a problem. After the program has been
 running for a while, I start seeing errors such as
 
   File C:\projects\sitetruth\InfoSiteRating.py, line 200, in scansite
 if len(self.badbusinessinfo)  0 :  # if bad stuff
 NameError: name 'len' is not defined
 
 which ought to be impossible in Python, and

Impossible?

py len
built-in function len
py import __builtin__  # use builtins in Python 3
py del __builtin__.len
py len
Traceback (most recent call last):
  File stdin, line 1, in module
NameError: name 'len' is not defined


Why something is deleting builtins len is a mystery. Sounds to me that your
Python installation is borked.


   File C:\projects\sitetruth\subprocesscall.py, line 129, in send
 self.writer.dump(args)  # send data
 OSError: [Errno 22] Invalid argument
 
 from somewhere deep inside CPickle.

Why do you say deep inside CPickle? The traceback says 
C:\projects\sitetruth\subprocesscall.py

Is it possible you have accidentally shadowed the CPickle module with
something? What does this say?

import cPickle
print cPickle.__file__

Use _pickle in Python 3.


 I got
 
   File C:\projects\sitetruth\InfoSiteRating.py, line 223, in
 get_rating_text
 (ratingsmalliconurl, ratinglargiconurl, ratingalttext) =
 DetailsPageBuilder.getratingiconinfo(rating)
 NameError: name 'DetailsPageBuilder' is not defined
 (That's an imported module.  It worked earlier in the run.)
 
 and finally, even after I deleted all .pyc files and all Python
 cache directories:
 
 Fatal Python error: GC object already tracked
 
 Current thread 0x1a14 (most recent call first):
   File C:\python34\lib\site-packages\pymysql\connections.py, line 411
 in description
   File C:\python34\lib\site-packages\pymysql\connections.py, line 1248
 in _get_descriptions
   File C:\python34\lib\site-packages\pymysql\connections.py, line 1182
 in _read_result_packet
   File C:\python34\lib\site-packages\pymysql\connections.py, line 1132
 in read
   File C:\python34\lib\site-packages\pymysql\connections.py, line 929
 in _read_query_result
   File C:\python34\lib\site-packages\pymysql\connections.py, line 768
 in query
   File C:\python34\lib\site-packages\pymysql\cursors.py, line 282 in
 _query
   File C:\python34\lib\site-packages\pymysql\cursors.py, line 134 in
 execute
   File C:\projects\sitetruth\domaincacheitem.py, line 128 in select
   File C:\projects\sitetruth\domaincache.py, line 30 in search
   File C:\projects\sitetruth\ratesite.py, line 31 in ratedomain
   File C:\projects\sitetruth\RatingProcess.py, line 68 in call
   File C:\projects\sitetruth\subprocesscall.py, line 140 in docall
   File C:\projects\sitetruth\subprocesscall.py, line 158 in run
   File C:\projects\sitetruth\RatingProcess.py, line 89 in main
   File C:\projects\sitetruth\RatingProcess.py, line 95 in module
 
 That's a definite memory error.

 So something is corrupting memory.  Probably CPickle.


 All my code is in Python. Every library module came in via pip, into a
 clean Python 3.4.3 (32 bit) installation on Win7/x86-64.
 Currently installed packages:
 
 beautifulsoup4 (4.3.2)
 dnspython3 (1.12.0)
 html5lib (0.999)
 pip (6.0.8)
 PyMySQL (0.6.6)
 pyparsing (2.0.3)
 setuptools (12.0.5)
 six (1.9.0)
 
 And it works fine with Python 2.7.9.
 
 Is there some way to force the use of the pure Python pickle module?

Try renaming the _pickle module. This works on Linux:

mv /usr/local/lib/python3.3/lib-dynload/_pickle.cpython-33m.so 
/usr/local/lib/python3.3/lib-dynload/_pickle.cpython-33m.so~


 My guess is that there's something about reusing pickle instances
 that botches memory uses in CPython 3's C code for cpickle.

How are you reusing instances?


-- 
Steven

-- 
https://mail.python.org/mailman/listinfo/python-list

Python3 pickle vs. stdin/stdout - unable to get clean byte streams in Python 3

[John Nagle]

  I have working code from Python 2 which uses pickle
to talk to a subprocess via stdin/stdio.  I'm trying to
make that work in Python 3.

  First, the subprocess Python is invoked with the -d' option,
so stdin and stdio are supposed to be unbuffered binary streams.
That was enough in Python 2, but it's not enough in Python 3.

The subprocess and its connections are set up with

  proc = subprocess.Popen(launchargs,stdin=subprocess.PIPE,
stdout=subprocess.PIPE, env=env)

  ...
  self.reader = pickle.Unpickler(self.proc.stdout)
  self.writer = pickle.Pickler(self.proc.stdin, 2)

after which I get

  result = self.reader.load()
TypeError: 'str' does not support the buffer interface

That's as far as traceback goes, so I assume this is
disappearing into C code.

OK, I know I need a byte stream.  I tried

  self.reader = pickle.Unpickler(self.proc.stdout.buffer)
  self.writer = pickle.Pickler(self.proc.stdin.buffer, 2)

That's not allowed.  The stdin and stdout that are
fields of proc do not have buffer.  So I can't do that
in the parent process.  In the child, though, where
stdin and stdout come from sys, sys.stdin.buffer is valid.
That fixes the str does not support the buffer interface
error.  But now I get the pickle error Ran out of input
on the process child side.  Probably because there's a
str/bytes incompatibility somewhere.

So how do I get clean binary byte streams between parent
and child process?

John Nagle
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 pickle vs. stdin/stdout - unable to get clean byte streams in Python 3

[John Nagle]

On 3/12/2015 2:56 PM, Cameron Simpson wrote:
 On 12Mar2015 12:55, John Nagle na...@animats.com wrote:
 I have working code from Python 2 which uses pickle to talk to a
 subprocess via stdin/stdio.  I'm trying to make that work in Python
 3. First, the subprocess Python is invoked with the -d' option, so
 stdin and stdio are supposed to be unbuffered binary streams.
 
 You shouldn't need to use unbuffered streams specifically. It should
 be enough to .flush() the output stream (at whichever end) after you
 have written the pickle data.

Doing that.

It's a repeat-transaction thing.  Main process sends pickeled
item to subprocess, subprocess reads item, subprocess does work,
subprocess writes picked item to parent.  This repeats.

I call writer.clear_memo() and set reader.memo = {} at the
end of each cycle, to clear Pickle's cache.  That all worked
fine in Python 2.  Are there any known problems with reusing
Python 3 pickles streams?

The identical code works with Python 2.7.9; it's converted to Python
3 using six so I can run on both Python versions and look for
differences.  I'm using Pickle format 2, for compatibility.
(Tried 0, the ASCII format; it didn't help.)

 I'm skipping some of your discussion; I can see nothing wrong. I
 don't use pickle itself so aside from saying that your use seems to
 conform to the python 3 docs I can't comment more deeply. That said,
 I do use subprocess a fair bit.

 I'll have to put in more logging and see exactly what's going
over the pipes.

John Nagle

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 pickle vs. stdin/stdout - unable to get clean byte streams in Python 3

[Cameron Simpson]

On 12Mar2015 12:55, John Nagle na...@animats.com wrote:

 I have working code from Python 2 which uses pickle
to talk to a subprocess via stdin/stdio.  I'm trying to
make that work in Python 3.
 First, the subprocess Python is invoked with the -d' option,
so stdin and stdio are supposed to be unbuffered binary streams.


You shouldn't need to use unbuffered streams specificly. It should be enough to 
.flush() the output stream (at whichever end) after you have written the pickle 
data.


I'm skipping some of your discussion; I can see nothing wong. I don't use 
pickle itself so aside from saying that your use seems to conform to the python 
3 doco I can't comment more deeply. That said, I do use subprocess a fair bit.


[...]

 result = self.reader.load()
TypeError: 'str' does not support the buffer interface
That's as far as traceback goes, so I assume this is
disappearing into C code.


No line numbers at all? Or, I suppose, just the line number from your program 
and nothing from the pickle module?



OK, I know I need a byte stream.  I tried
 self.reader = pickle.Unpickler(self.proc.stdout.buffer)
 self.writer = pickle.Pickler(self.proc.stdin.buffer, 2)


You should not need to care about these. They're not required.


That's not allowed.  The stdin and stdout that are
fields of proc do not have buffer.  So I can't do that
in the parent process.  In the child, though, where
stdin and stdout come from sys, sys.stdin.buffer is valid.


But irrelevant. Besides, the stream buffer may not contain the whole pickle 
data anyway; it will be empty before a read and quite possibly incomplete 
afterwards. It is just a buffer.



That fixes the str does not support the buffer interface
error.


I'm not sure fix is the right characterisation here.


But now I get the pickle error Ran out of input
on the process child side.  Probably because there's a
str/bytes incompatibility somewhere.


No, probably because the buffer is only ever a snapshot of part of the stream.

str/bytes errors are more glaringly obviously so.


So how do I get clean binary byte streams between parent
and child process?


This is where I'm confused: my experience is that Popen.subprocess gives you 
binary streams; I always need to put an encoder/decoder on them to use text.  
Did that just the other day.


BTW, this is on some UNIX variant? Should not be very relevant...

Further questions:

What does self.proc.stdout.__class__ say? And for stdin?

Cheers,
Cameron Simpson c...@zip.com.au

My opinions are borrowed from someone who no longer needs them.
   -- katma...@uga.cc.uga.edu
--
https://mail.python.org/mailman/listinfo/python-list

Re: Python3 pickle vs. stdin/stdout - unable to get clean byte streams in Python 3

[Peter Otten]

John Nagle wrote:

   I have working code from Python 2 which uses pickle
 to talk to a subprocess via stdin/stdio.  I'm trying to
 make that work in Python 3.
 
   First, the subprocess Python is invoked with the -d' option,
 so stdin and stdio are supposed to be unbuffered binary streams.
 That was enough in Python 2, but it's not enough in Python 3.
 
 The subprocess and its connections are set up with
 
   proc = subprocess.Popen(launchargs,stdin=subprocess.PIPE,
 stdout=subprocess.PIPE, env=env)
 
   ...
   self.reader = pickle.Unpickler(self.proc.stdout)
   self.writer = pickle.Pickler(self.proc.stdin, 2)
 
 after which I get
 
   result = self.reader.load()
 TypeError: 'str' does not support the buffer interface
 
 That's as far as traceback goes, so I assume this is
 disappearing into C code.
 
 OK, I know I need a byte stream.  I tried
 
   self.reader = pickle.Unpickler(self.proc.stdout.buffer)
   self.writer = pickle.Pickler(self.proc.stdin.buffer, 2)
 
 That's not allowed.  The stdin and stdout that are
 fields of proc do not have buffer.  So I can't do that
 in the parent process.  In the child, though, where
 stdin and stdout come from sys, sys.stdin.buffer is valid.
 That fixes the str does not support the buffer interface
 error.  But now I get the pickle error Ran out of input
 on the process child side.  Probably because there's a
 str/bytes incompatibility somewhere.
 
 So how do I get clean binary byte streams between parent
 and child process?

I don't know what you have to do to rule out deadlocks when you use pipes 
for both stdin and stdout, but binary streams are the default for 
subprocess. Can you provide a complete example?

Anyway, here is a demo for two-way communication using the communicate() 
method:

$ cat parent.py
import pickle
import subprocess

data = (5, 4.3, üblich ähnlich nötig)

p = subprocess.Popen(
[python3, child.py],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

result = p.communicate(pickle.dumps(data, protocol=2))[0]
print(pickle.loads(result))

$ cat child.py
import sys
import pickle

a, b, c = pickle.load(sys.stdin.buffer)
pickle.dump((a, b, c.upper()), sys.stdout.buffer)

$ python3 parent.py 
(5, 4.3, 'ÜBLICH ÄHNLICH NÖTIG')

This is likely not what you want because here everything is buffered so that 
continuous interaction is not possible.

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Mark Lawrence]

Mark Lawrence added the comment:

This can be closed as out of date as extensive changes made to the Windows 
FAQ have removed the wording referenced in the patch.

--
nosy: +BreamoreBoy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Brian Curtin]

Changes by Brian Curtin br...@python.org:


--
nosy:  -brian.curtin

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Terry J. Reedy]

Changes by Terry J. Reedy tjre...@udel.edu:


--
resolution:  - out of date
stage: needs patch - resolved
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Unicode stdin/stdout

[Robin Becker]

On 15/11/2013 18:16, random...@fastmail.us wrote:

Of course, the real solution to this issue is to replace sys.stdout on
windows with an object that can handle Unicode directly with the
WriteConsoleW function - the problem there is that it will break code
that expects to be able to use sys.stdout.buffer for binary I/O. I also
wasn't able to get the analogous stdin replacement class to work with
input() in my attempts.


I started to use this on my windows installation


#c:\python33\lib\site-packages\sitecustomize.py
import sys, codecs
sys.stdout = codecs.getwriter(utf-8)(sys.stdout.detach())
sys.stderr = codecs.getwriter(utf-8)(sys.stderr.detach())

which makes them writable with any unicode; after many years I am quite used to 
garbage appearing in the windows console.


Unfortunately the above doesn't go into virtual environments, but I assume a 
hacked site.py could do that.

--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list

Re: Unicode stdin/stdout

[Robin Becker]

On 18/11/2013 11:47, Robin Becker wrote:
...

#c:\python33\lib\site-packages\sitecustomize.py
import sys, codecs
sys.stdout = codecs.getwriter(utf-8)(sys.stdout.detach())
sys.stderr = codecs.getwriter(utf-8)(sys.stderr.detach())


it seems that the above needs extra stuff to make some distutils logging work 
etc etc; so now I'm using sitecustomize.py containing


import sys, codecs
sys.stdout = codecs.getwriter(utf-8)(sys.stdout.detach())
sys.stdout.encoding = 'utf8'
sys.stderr = codecs.getwriter(utf-8)(sys.stderr.detach())
sys.stderr.encoding = 'utf8'

--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Unicode stdin/stdout

[Nick Coghlan]

On 18 Nov 2013 22:36, Robin Becker ro...@reportlab.com wrote:

 On 18/11/2013 11:47, Robin Becker wrote:
 ...

 #c:\python33\lib\site-packages\sitecustomize.py
 import sys, codecs
 sys.stdout = codecs.getwriter(utf-8)(sys.stdout.detach())
 sys.stderr = codecs.getwriter(utf-8)(sys.stderr.detach())

 
 it seems that the above needs extra stuff to make some distutils logging
work etc etc; so now I'm using sitecustomize.py containing

 import sys, codecs
 sys.stdout = codecs.getwriter(utf-8)(sys.stdout.detach())
 sys.stdout.encoding = 'utf8'
 sys.stderr = codecs.getwriter(utf-8)(sys.stderr.detach())
 sys.stderr.encoding = 'utf8'

Note that calling detach() on the standard streams isn't officially
supported, since it breaks the shadow streams saved in sys.__stderr__, etc.

Cheers,
Nick.


 --
 Robin Becker

 ___
 Python-ideas mailing list
 python-id...@python.org
 https://mail.python.org/mailman/listinfo/python-ideas
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Unicode stdin/stdout

[Victor Stinner]

Why do you need to force the UTF-8 encoding? Your locale is not
correctly configured?

It's better to set PYTHONIOENCODING rather than replacing
sys.stdout/stderr at runtime.

There is an open issue to add a TextIOWrapper.set_encoding() method:
http://bugs.python.org/issue15216

Victor
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unicode stdin/stdout

[Robin Becker]

On 18/11/2013 15:25, Victor Stinner wrote:

Why do you need to force the UTF-8 encoding? Your locale is not
correctly configured?

It's better to set PYTHONIOENCODING rather than replacing
sys.stdout/stderr at runtime.

There is an open issue to add a TextIOWrapper.set_encoding() method:
http://bugs.python.org/issue15216

Victor

well reportlab does all sorts of character sets and languages; if I put in a 
quick print to try and debug stuff I prefer that it create some output rather 
than create an error of its own. In the real world it's not possible always to 
know what the output contains (especially in error cases) so having any 
restriction on the allowed textual outputs is a bit constraining.


The utf8 encoding should allow any unicode to be properly encoded, rendering is 
another issue and I expect some garbage when things are going wrong.


I think you are right and I should use PYTHONIOENCODING to set this up. In the 
codec writer approach I think it's harder to get interactive behaviour working 
properly (the output seems to be buffered differently). My attempts to make 
windows xp use code page 65001 everywhere have been fairly catastrophic eg 
non-booting :(

--
Robin Becker

--
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Unicode stdin/stdout

[random832]

On Mon, Nov 18, 2013, at 7:33, Robin Becker wrote:
 UTF-8 stuff

This doesn't really solve the issue I was referring to, which is that
windows _console_ (i.e. not redirected file or pipe) I/O can only
support unicode via wide character (UTF-16) I/O with a special function,
not via using byte-based I/O with the normal write function.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: [Python-ideas] Unicode stdin/stdout

[Andrew Barnert]

From: random...@fastmail.us random...@fastmail.us



 On Mon, Nov 18, 2013, at 7:33, Robin Becker wrote:
  UTF-8 stuff
 
 This doesn't really solve the issue I was referring to, which is that
 windows _console_ (i.e. not redirected file or pipe) I/O can only
 support unicode via wide character (UTF-16) I/O with a special function,
 not via using byte-based I/O with the normal write function.


The problem is that Windows 16-bit I/O doesn't fit into the usual io module 
hierarchy. Not because it uses an encoding of UTF-16 (although anyone familiar 
with ReadConsoleW/WriteConsoleW from other languages may be a bit confused that 
Python's lowest-level wrappers around them deal in byte counts instead of WCHAR 
counts), but because you have to use HANDLEs instead of fds. So, there are 
going to be some compromises and some complexity.

One possibility is to use as much of the io hierarchy as possible, but not try 
to make it flexible enough to be reusable for arbitrary HANDLEs: Add 
WindowsFileIO and WindowsConsoleIO classes that implement RawIOBase with a 
native HANDLE and ReadFile/WriteFile and ReadConsoleW/WriteConsoleW 
respectively. Both work in terms of bytes (which means WindowsConsoleIO.read 
has to //2 its argument, and write has to *2 the result). You also need a 
create_windows_io function that wraps a HANDLE by calling GetConsoleMode and 
constructing a WindowsConsoleIO or WindowsFileIO as appropriate, then creates a 
BufferedReader/Writer around that, then constructs a TextIOWrapper with UTF-16 
or the default encoding around that. At startup, you just do that for the three 
GetStdHandle handles, and that's your stdin, stdout, and stderr.

Besides not being reusable enough for people who want to wrap HANDLEs from 
other libraries or attach to new consoles from Python, it's not clear what 
fileno() should return. You could fake it and return the MSVCRT fds that 
correspond to the same files as the HANDLEs, but it's possible to end up with 
one redirected and not the other (e.g., if you detach the console), and I'm not 
sure what happens if you mix and match the two. A more correct solution would 
be to call _open_osfhandle on the HANDLE (and then keep track of the fact that 
os.close closes the HANDLE, or leave it up to the user to deal with bad handle 
errors?), but I'm not sure that's any better in practice. Also, should a 
console HANDLE use _O_WTEXT for its fd (in which case the user has to know that 
he has a _O_WTEXT handle even though there's no way to see that from Python), 
or not (in which case he's mixing 8-bit and 16-bit I/O on the same file)?

It might be reasonable to just not expose fileno(); most code that wants the 
fileno() for stdin is just going to do something Unix-y that's not going to 
work anyway (select it, tcsetattr it, pass it over a socket to another file, …).

A different approach would be to reuse as _little_ of io as possible, instead 
of as much: Windows stdin/stdout/stderr could each be custom TextIOBase 
implementations that work straight on HANDLEs and don't even support buffer (or 
detach), much less fileno. That exposes even less functionality to users, of 
course. It also means we need a parallel implementation of all the buffering 
logic. (On the other hand, it also leaves the door open to expose some Windows 
functionality, like async ReadFileEx/WriteFileEx, in a way that would be very 
hard through the normal layers…)


It shouldn't be too hard to write most of these via an extension module or 
ctypes to experiment with it. As long as you're careful not to mix 
winsys.stdout and sys.stdout (the module could even set sys.stdin, sys.stdout, 
sys.stderr=stdin, stdout, stderr at import time, or just del them, for a bit of 
protection), it should work.

It might be worth implementing a few different designs to play with, and 
putting them through their paces with some modules and scripts that do 
different things with stdio (including running the scripts with cmd.exe 
redirected I/O and with subprocess PIPEs) to see which ones have problems or 
limitations that are hard to foresee in advance.

If you have a design that you think sounds good, and are willing to experiment 
the hell out of it, and don't know how to get started but would be willing to 
debug and finish a mostly-written/almost-working implementation, I could slap 
something together with ctypes to get you started.
-- 
https://mail.python.org/mailman/listinfo/python-list

Unicode stdin/stdout (was: Re: python 3.3 repr)

[random832]

Of course, the real solution to this issue is to replace sys.stdout on
windows with an object that can handle Unicode directly with the
WriteConsoleW function - the problem there is that it will break code
that expects to be able to use sys.stdout.buffer for binary I/O. I also
wasn't able to get the analogous stdin replacement class to work with
input() in my attempts.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue19326] asyncio: child process exit isn't detected if its stdin/stdout/stderr FDs have been inherited by a child process

[Charles-François Natali]

New submission from Charles-François Natali:

(This is a spinoff from http://bugs.python.org/issue19293#msg200598)

When SIGCHLD is received, _sig_chld() is executed:

def _sig_chld(self):
[...]
transp = self._subprocesses.get(pid)
if transp is not None:
transp._process_exited(returncode)

Then, here's _process_exited():

def _process_exited(self, returncode):
assert returncode is not None, returncode
assert self._returncode is None, self._returncode
self._returncode = returncode
self._loop._subprocess_closed(self)
self._call(self._protocol.process_exited)
self._try_finish()

And here's _try_finish():

def _try_finish(self):
assert not self._finished
if self._returncode is None:
return
if all(p is not None and p.disconnected
   for p in self._pipes.values()):
self._finished = True
self._loop.call_soon(self._call_connection_lost, None)

Thus, _UnixSubprocessTransport protocol's connection_lost is only
called if the all() expression is true:
and it's true only if all the subprocess pipes have been disconnected
(or if we didn't setup any pipe).

Unfortunately, this might very well never happen: imagine that the
subprocess forks a process: this grand-child process inherits the
child process's pipes, so when the child process exits, we won't
receive any notification, since this grand-child process still has
open FDs pointing to the original child's stdin/stdout/stderr.

The following simple test will hang until the background 'sleep' exits:

diff -r 47618b00405b Lib/test/test_asyncio/test_events.py
--- a/Lib/test/test_asyncio/test_events.py  Sat Oct 19 10:45:48 2013 +0300
+++ b/Lib/test/test_asyncio/test_events.py  Sun Oct 20 19:32:37 2013 +0200
@@ -1059,6 +1059,23 @@

 @unittest.skipIf(sys.platform == 'win32',
  Don't support subprocess for Windows yet)
+def test_subprocess_inherit_fds(self):
+proto = None
+
+@tasks.coroutine
+def connect():
+nonlocal proto
+transp, proto = yield from self.loop.subprocess_shell(
+functools.partial(MySubprocessProtocol, self.loop),
+'sleep 60 ')
+self.assertIsInstance(proto, MySubprocessProtocol)
+
+self.loop.run_until_complete(connect())
+self.loop.run_until_complete(proto.completed)
+self.assertEqual(0, proto.returncode)
+
+@unittest.skipIf(sys.platform == 'win32',
+ Don't support subprocess for Windows yet)
 def test_subprocess_close_after_finish(self):
 proto = None
 transp = None


If waitpid() returns a process's PID, then the process is done,
there's no reason to further wait for pipe's disconnection: they can
be used as a hint that the process terminated, but there's definitely
not required...

--
messages: 200744
nosy: neologix
priority: normal
severity: normal
stage: needs patch
status: open
title: asyncio: child process exit isn't detected if its stdin/stdout/stderr 
FDs have been inherited by a child process
type: behavior
versions: Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19326
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19326] asyncio: child process exit isn't detected if its stdin/stdout/stderr FDs have been inherited by a child process

[Guido van Rossum]

Changes by Guido van Rossum gu...@python.org:


--
nosy: +gvanrossum

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19326
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19326] asyncio: child process exit isn't detected if its stdin/stdout/stderr FDs have been inherited by a child process

[Guido van Rossum]

Guido van Rossum added the comment:

This is by design. Let me try to defend the design.

As long as one of the pipes is still open the parent might be interested in it. 
The Protocol object does get notified of the process's exit, via 
process_exited(), and if at that point it wants to be done, it can close the 
pipes itself. (To do that, call transport.get_pipe_transport(FD).close().) Once 
that's done the subprocess protocol's connection_lost() method will be called.

I suppose an alternative approach might be to assume that when the subprocess 
exist the parent should close the pipes and be done, but there's a race 
condition there: there may still be data in one of the pipes (stdout or stderr) 
that should be processed before calling connection_lost(), similar to how we 
delay the socket connection_lost() call until we have processed all data read 
from it.

So I don't think that alternative is viable.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19326
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue19326] asyncio: child process exit isn't detected if its stdin/stdout/stderr FDs have been inherited by a child process

[Charles-François Natali]

Charles-François Natali added the comment:

 This is by design. Let me try to defend the design.

OK, if that's a know limitation, then that's fine.
It would be nice to add maybe a note somewhere in the documentation, so that 
people don't get bitten by this (and also probably add a test for 
process_exited).

--
resolution:  - rejected
stage: needs patch - committed/rejected
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19326
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,
I now fully agree with you. Regarding points 2  3 I dismissed modifying 
sys.stdin/out in python out of hand because it still would not allow to have a 
proper behaviour with two concurrent consoles on the same interpreter. Anyway 
this is not a very meaningful use-case, so modifying sys.stdin/out looks like 
the best solution.
As for point 1, it can be worked around in a twisted and probably non-portable 
way (setting PyOS_ReadlineFunctionPointer to a custom function that forks a 
child which runs GNU readline or whatnot on its fds 0,1,2, and which 
communicates with the parent through pipes) but it's a pity that from 
PyOS_Readline() on, argument sys_stdin is correctly passed down, and that the 
chain has only this gap in tok_nextc() which dismisses the caller's argument 
and uses plain stdin.
If the user arguments are not used, why does PyRun_InteractiveLoop() take any 
arguments at all?
It would be nice to know the opinion of the python development team and to work 
out a complete fix (considering also stdout as you suggest) under their 
authority.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

emmanuel,

The Python interpreter isn't reentrant, so you could only run two interactive 
sessions connected to the same Python environment if you implemented your own 
REPL function that unlocked the GIL when waiting for input, then lock it just 
long enough to interpret the input.

Also, the readline module already does what you suggest (sets 
PyOS_ReadlineFunctionPointer to a GNU libreadline wrapper.) My readline hack 
(working.c) forces it to behave as it's supposed to. Rather, it *undoes* what 
PyRun_InteractiveLoop does every iteration, which is pass stdin/stdout for I/O, 
which libreadline in turn uses.

I agree that it would be nice to get a Python developer involved, mostly 
because I expect things to break when this problem is fixed. Bad things happen 
when you think you've tested a lot of different cases that actually turn out to 
be the exact same case.

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

emmanuel,

Regarding your points: All three can be taken care of with a combination of my 
patch and setting sys.stdin, sys.stdout, and sys.stderr to the pty. (That 
should really be done internally with another patch, since os.fdopen is 
OS-specific. Also, sys.stdin, sys.stdout, and sys.stderr should each have 
distinct underlying file descriptors, which I didn't do in working.c.) Those 
can safely be replaced since they're just the effective standard files, and 
sys.__stdin__ et al. refer to the actual C stdin et al. The remaining issue 
would be that the same descriptor shouldn't be used for both input and output 
in the interpreter loop, especially if the FILE* passed is only open for 
reading (since standard input technically doesn't have to be writable.)

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,
I've read more carefully your messages and investigated some more.
It seems that there are several issues:
1/ To take input from a defined tty without interfering with standard file 
descriptors
2/ To have the result (object) of evaluation printed to a defined tty without 
interfering with standard file descriptors
3/ (optionally) To direct to the tty (or not) the output that is a side effect 
of the evaluation
Provided that no one messes with PyOS_ReadlineFunctionPointer (as with import 
readline) it should be possible to solve 1 without modifying Python, and 
approximately solve 2 (with 3 implied out of necessity).
On the other hand, modifying Python as you suggest could solve 1, but issues 2 
and 3 would still remain and probably require some other modifications.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,

These are good points.

I had a cursory look at the python source code and observed the following:
- There may also be a concern with stderr (used to print the prompt in 
PyOS_Readline)
- PyOS_Readline has two different definitions in files pgenmain.c and 
myreadline.c
- There is this interesting comment in myreadline.c:
/* By initializing this function pointer, systems embedding Python can
   override the readline function.
   Note: Python expects in return a buffer allocated with PyMem_Malloc. */
char *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *);

This pointer is actually used (set it to (void*)1 and the interpreter crashes) 
so it could offer a means to redirect stdin as we want. For stdout/stderr 
further investigation is needed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

Changes by emmanuel garcia6.emman...@wanadoo.fr:


--
nosy: +emmanuel

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

run the attached shell script to observe the bug
./bug.sh 0 - shows the bug
./bug.sh 1 - shows the expected behaviour (using a workaround)
tested on linux with python 2.7

--
Added file: http://bugs.python.org/file29431/bug.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

Changes by emmanuel garcia6.emman...@wanadoo.fr:


Added file: http://bugs.python.org/file29432/bug.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

Changes by emmanuel garcia6.emman...@wanadoo.fr:


Removed file: http://bugs.python.org/file29431/bug.sh

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

emmanuel,

Thanks for the suggestion. Your workaround is exactly the same as using dup2 
(in C) to replace stdin/stdout/stderr with the pty, however. If you added the 
following lines to your C code, it would have the same effect as the 
command-line redirection in the workaround:

dup2(fileno(file), STDIN_FILENO);
dup2(fileno(file), STDOUT_FILENO);
dup2(fileno(file), STDERR_FILENO);

In fact, that's exactly what bash does after forking, just before executing 
exe. In most cases, developers who use PyRun_InteractiveLoop in a pty 
probably also do exactly that, which is why I'm the only one who's reported 
this as a bug. For applications like mine, however, where the interactive 
Python session needs to be an unobtrusive add-on to an otherwise-complete 
program, this solution won't work. The standard file descriptors aren't 
disposable in most of the programs I work on.

Thanks again!

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[emmanuel]

emmanuel added the comment:

Kevin,

Indeed the code I submitted can be written entirely in C using pipe fork execl 
dup2 etc. as you suggest. The only purpose of mixing bash and C is to have a 
short self-contained file showing the problem.

Anyway, whether in C or bash the workaround is less than satisfying in that it 
uses up fds 0 1 2, and I have exactly the same goal and constraints as you.

Finally, thanks for identifying the limitation in the python implementation and 
submitting a patch. Like you I hope it will be eventually applied.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

One additional issue, which my patch doesn't address, is that 
PyRun_InteractiveLoop should really take *two* FILE* arguments, with the second 
one being optional. This is because on Linux (and presumably on other *nixes) 
if a read operation is blocked on a file descriptor then write operations (from 
other threads) to the same file descriptor will also block. That doesn't happen 
in the current Python implementations because PyOS_Readline is always called 
with two FILE* objects, anyway (stdin and stdout.) I would, however, expect 
such a problem to appear if a user created a Python thread in the interactive 
session that periodically printed to the terminal, then read input from the 
terminal. In that case, I would expect to see no output from the thread while 
the read operations were blocked, but I haven't tested it. (I don't remember if 
this came up after I applied my patch locally.)

I actually considered this when I created the patch; however, I didn't feel 
like going to all the trouble of adding a member to tok and propagating the 
change throughout the entire core. I had hoped this bug would get more 
attention and I'd be able to discuss it with a developer involved in the Python 
project, but ultimately that didn't happen and I ended up forgetting about it.

Kevin Barry

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Nacsa Kristóf]

Nacsa Kristóf added the comment:

@dhgmgn I think the change is ok. That said, I add two things. 

It maybe should be noted which Windows version has this corrected.
The article contains this string: This problem was first corrected in 
Microsoft Windows 2000 Service Pack 4. This problem was first corrected in 
Windows XP Service Pack 1.

The other thing is that I prefer if the relevant information is copied due to 
various reasons. One reason is that it serves as a highlight (nice time-saver 
and is also against TL;DR situations). Another is that Microsoft may simply 
cease support, or discontinue this link (they tend to do this).

I'm not familiar with how the python docs is constructed, if there is a style 
guide, etc., but I can tell that I'd be happy if I could see this information 
there directly. This is also how I've found out about that cmd header 
line/'trick'.

@everyone
Another thing, can anyone else confirm that this works for [s]he?
I've tested this on an 32-bit xp pro sp3.

When _not_ using the fix, something like `foo | myscript.py | bar` results in:
IOError: [Errno 9] Bad file descriptor
With _using_ the registry fix from Microsoft, the command from above just works 
for me on the XP. (The cmd.exe/console needs to be restarted.)

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Jan Duzinkiewicz]

Jan Duzinkiewicz added the comment:

I've referenced the kb article in the docs. Please let me know if the fix is ok 
- maybe the whole registry change process should be included in the docs 
directly?

--
keywords: +patch
nosy: +dhgmgn
Added file: http://bugs.python.org/file27839/15962.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry added the comment:

I still see the potential cause addressed by my patch in the 2.7, 3.3, and 
default branches, so I'm assuming that all versions from 2.6 on have this 
problem.

I also see that I can elect to change the Status and Resolution of this 
report. Does that mean I need to do something besides wait for someone involved 
in the project to look at my patch?

Kevin Barry

--
versions: +Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 
3.5

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Antoine Pitrou]

Changes by Antoine Pitrou pit...@free.fr:


--
nosy: +haypo

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Nacsa Kristóf]

New submission from Nacsa Kristóf:

The Python docs faq says that due to a bug in Windows NT's cmd.exe, the output 
redirection and piping won't work when started from file extension.

http://docs.python.org/faq/windows.html#how-do-i-make-python-scripts-executable
http://docs.python.org/py3k/faq/windows.html#how-do-i-make-python-scripts-executable

This is actually fixed in Windows, but the fix is disabled by default, and you 
need to enable it in regedit, the keyword is InheritConsoleHandles.

Proof  Details:
http://support.microsoft.com/default.aspx?kbid=321788

It would be great if the docs could refer to this fix, as the docs is the major 
source of information for many.

--
assignee: docs@python
components: Documentation, Installation, Windows
messages: 170641
nosy: Nacsa.Kristóf, docs@python
priority: normal
severity: normal
status: open
title: Windows STDIN/STDOUT Redirection is actually FIXED
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue15962] Windows STDIN/STDOUT Redirection is actually FIXED

[Ezio Melotti]

Changes by Ezio Melotti ezio.melo...@gmail.com:


--
keywords: +easy
nosy: +brian.curtin, terry.reedy, tim.golden
stage:  - needs patch
type:  - enhancement
versions:  -Python 2.6, Python 3.1

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue15962
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

The patch from before needed a slight modification for when Python actually 
defaults to an interactive session on stdin. Since I rebuild this for my 
current distro (Slackware64 13.37,) I switched to the Python 2.6.6 source. This 
might not be the proper way to handle the default case (e.g. 'Py_Main'), but 
it's a start.


The patch (also attached):

--- ./Parser/tokenizer.c.orig   2012-07-23 22:24:56.513992301 -0400
+++ ./Parser/tokenizer.c2012-07-23 22:23:24.329992167 -0400
@@ -805,7 +805,7 @@
 return Py_CHARMASK(*tok-cur++);
 }
 if (tok-prompt != NULL) {
-char *newtok = PyOS_Readline(stdin, stdout, tok-prompt);
+char *newtok = PyOS_Readline(tok-fp? tok-fp : stdin, (tok-fp  
tok-fp != stdin)? tok-fp : stdout, tok-prompt);
 if (tok-nextprompt != NULL)
 tok-prompt = tok-nextprompt;
 if (newtok == NULL)


Kevin Barry

--
Added file: 
http://bugs.python.org/file26501/Python-2.6.6-Run_Interactive-fix.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

I've attached a simplified example program (working3.c) that demonstrates both 
the original problem and that the patch 
(Python-2.6.6-Run_Interactive-fix.patch) works. It eliminates the need for a 
pty, 'xterm', and redirection.


Compile the attached program with:

 gcc `python-config --cflags` working3.c -o working3 `python-config --ldflags`

and run it with (before and after patching):

 ./working3


Also, for verification, run 'python' with no arguments to show that default 
interactivity is preserved.

Kevin Barry

--
Added file: http://bugs.python.org/file26503/working3.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

Here is a patch that corrects the problem (quoted below and attached.) This 
only corrects the problem when 'PyOS_ReadlineFunctionPointer' is set, e.g. you 
must 'import readline', otherwise Python will defer to stdin/stdout with 
'PyOS_StdioReadline'.


The patch:


--- Python-2.6.8/Parser/tokenizer.c 2012-04-10 11:32:11.0 -0400
+++ Python-2.6.8-patched/Parser/tokenizer.c 2012-07-23 19:56:39.645992101 
-0400
@@ -805,7 +805,7 @@
 return Py_CHARMASK(*tok-cur++);
 }
 if (tok-prompt != NULL) {
-char *newtok = PyOS_Readline(stdin, stdout, tok-prompt);
+char *newtok = PyOS_Readline(tok-fp? tok-fp : stdin, tok-fp? 
tok-fp : stdout, tok-prompt);
 if (tok-nextprompt != NULL)
 tok-prompt = tok-nextprompt;
 if (newtok == NULL)


Kevin Barry

--
keywords: +patch
status: open - pending
Added file: 
http://bugs.python.org/file26491/Python-2.6.8-Run_Interactive-fix.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

Kevin Barry ta0k...@gmail.com added the comment:

I've attached a new example source file to demonstrate the fix.


Compile the attached program with (*after* patching and installing Python):

 gcc `python-config --cflags` working2.c -o working2 `python-config --ldflags`

and run it with:

 ./working2 xterm -S/0  /dev/null  /dev/null

(The redirection shows that it works when stdin/stdout aren't a tty.)


I looked at the most-recent revision of tokenizer.c 
(http://hg.python.org/cpython/file/52032b13243e/Parser/tokenizer.c) and see 
that the change in my patch above hasn't been made already.

Kevin Barry

--
status: pending - open
Added file: http://bugs.python.org/file26492/working2.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14916] PyRun_InteractiveLoop fails to run interactively when using a Linux pty that's not tied to stdin/stdout

[Kevin Barry]

New submission from Kevin Barry ta0k...@gmail.com:

I have been trying to get PyRun_InteractiveLoop to run on a pty (Linux) without 
replacing stdin and stdout with that pty; however, it seems like Python (2.6.6) 
is hard-coded to only run interactively on stdin and stdout.


Compile the attached program with:

 gcc `python-config --cflags` working.c -o working `python-config --ldflags`

and run it with:

 ./working xterm -S/0

and you should see that there is no interactivity in the xterm that's opened.


Compile the attached file with:

 gcc -DREADLINE_HACK `python-config --cflags` working.c -o working 
 `python-config --ldflags` -lreadline -lcurses

and run it with:

 ./working xterm -S/0

to see how it runs with my best attempt to get it to function properly with a 
readline hack. Additionally, try running:

 ./working xterm -S/0  /dev/null
 ./working xterm -S/0  /dev/null

both of which should cause interactivity in the xterm to fail, indicating that 
Python is checking stdin/stdout for tty status when determining if it should 
run interactively (i.e. it's not checking the tty status of the file passed to 
PyRun_InteractiveLoop.)


Am I somehow using this function wrong? I've been trying to work around this 
problem for a while, and I don't think I should be using readline hacks 
(especially since they don't port to other OSes with ptys, e.g. OS X.) I even 
tried to patch the call to PyOS_Readline in tok_nextc (Parser/tokenizer.c) to 
use tok-fp instead of stdin/stdout, which caused I/O to use the pty but it 
still failed to make interactivity work.


Thanks!

Kevin Barry

--
components: Interpreter Core, Library (Lib)
files: working.c
messages: 161586
nosy: Kevin.Barry
priority: normal
severity: normal
status: open
title: PyRun_InteractiveLoop fails to run interactively when using a Linux pty 
that's not tied to stdin/stdout
type: behavior
versions: Python 2.6
Added file: http://bugs.python.org/file25706/working.c

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14916
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

 so there really was a bug.
 If fixed in 3.2.1, this issue could be closed.

This issue is a duplicate of #11272: upgrade to Python 3.2.1.

--
resolution:  - duplicate
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

WinXP, 3.2.0
 type(sys.stdin)
class 'idlelib.rpc.RPCProxy'
 sys.stdin.readline()
a
'a\n'

--
nosy: +terry.reedy

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[R. David Murray]

R. David Murray rdmur...@bitdance.com added the comment:

I'm not sure what the point of your example is, Terry.  Is it not fixed in 
3.2.1?

--
nosy: +r.david.murray

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[Terry J. Reedy]

Terry J. Reedy tjre...@udel.edu added the comment:

That, like Antoine, I also could not reproduce, even in 3.2.0, when running 
under IDLE. However, with regular command line Python:
Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit (Intel)]
 import sys
 sys.stdin.readline()
a
'a\r\n'
so there really was a bug. If fixed in 3.2.1, this issue could be closed.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[Antoine Pitrou]

Antoine Pitrou pit...@free.fr added the comment:

I can't reproduce. Victor?


Z:\defaultPCbuild\amd64\python_d.exe
Python 3.3a0 (default, Jun  8 2011, 17:49:13) [MSC v.1500 64 bit (AMD64)] on 
win32
Type help, copyright, credits or license for more information.
 import sys
 sys.stdin.readline()
a
'a\n'
 ^Z

Z:\defaultPCbuild\amd64\python_d.exe -u
Python 3.3a0 (default, Jun  8 2011, 17:49:13) [MSC v.1500 64 bit (AMD64)] on 
win32
Type help, copyright, credits or license for more information.
 import sys
 sys.stdin.readline()
a
'a\n'

--
nosy: +amaury.forgeotdarc, haypo, pitrou
versions: +Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

I think that this issue is a duplicate of the issue #11272: Python 3.2.1 has 
been released recently and contains the fix. Can you try this version Fan?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[STINNER Victor]

STINNER Victor victor.stin...@haypocalc.com added the comment:

Note: Python 3.2 has another regression related to the Windows console (issue 
#11395), bug fixed in 3.2.1.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue12337] Need real TextIOWrapper for stdin/stdout

[Fan Decheng]

New submission from Fan Decheng fandech...@gmail.com:

Since -u is made default and binary stdio implemented in 3.2, many of my 
scripts cannot run directly in Python 3.2, because they expect \n from stdin, 
but on Windows \r\n is got.

Since that binary stdio being default is necessary for features like CGI, what 
I am expecting is to get a real TextIOWrapper for stdin, so that I can still 
use my old code. Besides, type(sys.stdio) should no longer say TextIOWrapper, 
because TextIOWrapper implies the conversion from \r\n to \n.

Steps to reproduce:
In Python 3.2:
 import sys
 type(sys.stdin)
class '_io.TextIOWrapper'
 sys.stdin.readline()
a
'a\r\n'
 type(sys.stdout)
class '_io.TextIOWrapper'

Expected result:
There should be some form of text support, such as 
textstream(sys.stdin).readline(). Also type(sys.stdin) should not say something 
like TextIOWrapper. Same for sys.stdout.

--
components: IO
messages: 138356
nosy: r_mosaic
priority: normal
severity: normal
status: open
title: Need real TextIOWrapper for stdin/stdout
type: behavior
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12337
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10480] cgi.py should document the need for binary stdin/stdout

[Glenn Linderman]

Glenn Linderman v+pyt...@g.nevcal.com added the comment:

Fixed by issue 10841 and issue 4953.

--
status: open - closed

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10480
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10480] cgi.py should document the need for binary stdin/stdout

[Glenn Linderman]

New submission from Glenn Linderman v+pyt...@g.nevcal.com:

CGI is a bytestream protocol.  Python assumes a text mode encoding for stdin 
and stdout, this is inappropriate for the CGI interface.

CGI should provide an API to do the right thing to make stdin and stout 
binary mode interfaces (including mscvrt setting to binary on Windows).  
Failing that, it should document the need to do so in CGI applications.

Failing that, it should be documented somewhere, CGI seems the most appropriate 
place to me.

--
components: Library (Lib)
messages: 121868
nosy: v+python
priority: normal
severity: normal
status: open
title: cgi.py should document the need for binary stdin/stdout
versions: Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10480
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10480] cgi.py should document the need for binary stdin/stdout

[Glenn Linderman]

Changes by Glenn Linderman v+pyt...@g.nevcal.com:


--
assignee:  - d...@python
components: +Documentation
nosy: +d...@python
type:  - behavior

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue10480
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: merge stdin, stdout?

[Anssi Saari]

jonny lowe jonny.lowe.12...@gmail.com writes:

 The result is the same as before. I've tested in fedora11.

I don't think script is the answer here, since it only stores what's
displayed on a terminal and your program's input comes from a file and
is not displayed on the terminal.

Simplest solution is probably that you have your program echo every
line of input. Maybe some hairy terminal trickery could be done?
Really more of a Linux question than python.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: merge stdin, stdout?

[jonny lowe]

On Feb 5, 11:10 pm, Gabriel Genellina gagsl-...@yahoo.com.ar
wrote:
 En Fri, 05 Feb 2010 17:39:07 -0300, jonny lowe
 jonny.lowe.12...@gmail.com escribió:





  On Feb 4, 8:20 pm, exar...@twistedmatrix.com wrote:
  On 01:56 am, jonny.lowe.12...@gmail.com wrote:
  What I want is to have an easy way tomergeinput.txt and thestdout
  so that output.txt look like:

  Enter a number: 42
  You entered 42.

  Here, the first 42 is of course from input.txt.

  It sounds like you might be looking forscript(1)?

  $ script -c ./y  input.txt output.txt
  Script started, file is output.txt
  gimme x:you entered hello
  Script done, file is output.txt

 Try moving the redirection out of the command:

 $ script -c ./y output.txt  input.txt

 --
 Gabriel Genellina- Hide quoted text -

 - Show quoted text -

The result is the same as before. I've tested in fedora11.

-jon
-- 
http://mail.python.org/mailman/listinfo/python-list