[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Researching 'nested mainloop': the one we are concerned with is in 
pyshell.PyShell.readline, currently line 1078.

self.top.mainloop()  # nested mainloop()

This was in David Scherer's 'Initial Revision' of 2000 Aug 14 without the 
comment.  It has since been touched once to add the comment.

2004 Dec 22, KBK, 5c3df35b6b1f48cb48c91b0c7a8754590a694171.
The GUI was hanging if the shell window was closed while a raw_input() was 
pending.  Restored the quit() of the readline() mainloop().
http://mail.python.org/pipermail/idle-dev/2004-December/002307.html

So no hint of reason.
---

https://stackoverflow.com/questions/17252056/tkinter-nested-mainloop
OP asked about using a nested mainloop for a video player.  There is discussion 
(dispute) of whether or not this blocked the mainloop.  Tkinter expert Bryan 
Oakley opined "While it's possible, there is very rarely ever a need to call 
it. reason to do it, and what you are trying to do probably won't work the way 
you think it will. –" 

OP Raoul quoted FL "Event loops can be nested; it's ok to call mainloop from 
within an event handler." (dead link to draft version). It is now in mainloop 
entry of https://effbot.org/tkinterbook/widget.htm.  There is no explanation of 
why or what effect.
---

Mark Lutz, Programming Python, discusses recursive mainloop calls as an 
alternative way to make modal dialogs, without 'wait' and other setup calls.  
In the 3rd edition, displayed by Google Books, the half page is expanded to a 
full page (442-443, Example 9-14, PP3E/Gui/Tour/dlg-recursive.py) and he adds 
that calling quit() is mandatory (see KBK patch above) and that using 'wait', 
etc, is 'probably better'.  The latter is what IDLE does.  Since 'modal' means 
'disable event handling outside the dialog', the intent must be to suspend the 
outer event loop, as we see for this issue.
---

While I still think that IDLE should protect user code input in response to 
*IDLE's* '>>>' prompt, even more than it does now, I now agree that IDLE should 
not do the same with try user *input()* calls.  The usability of the latter is 
the responsibility of users who write them.

Not blocking thread prints may just be a matter of removing mainloop() and all 
corresponding quit() calls.  Proper sequencing may be trickier.  Tem4.py calls 
input() and print() from both main and thread.  Run directly with 3.9.0b1, it 
results in

thread start
main input: m  # wait before entering 'm\n'.
main got:  m
thread input: t
thread got:  t

What surprised me is 'thread input:' being held up until 'main got' was 
printed.  Run from IDLE, the result is

thread startmain input: 
m
thread input: main got: t
 thread got: m 
t

This is more jumbled, not less.  'main input:' is printed before the '\n' after 
'thread start'.  etc.  Separating print(...input()) into two statements had no 
effect.  Neither did removing sleep(.1).  We will have to retest after blocking 
is removed.

--
Added file: https://bugs.python.org/file49194/tem4.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Tal Einat]

Tal Einat  added the comment:

Can someone talk about the reasoning behind the nested Tk mainloop? I haven't 
managed to find anything useful in code comments and such. Guido?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Tal Einat]

Tal Einat  added the comment:

This seems to be at least partly intentional: When waiting for user input, IDLE 
starts a nested Tk mainloop (!), which is stopped by the end-of-line handler. 
It seems to me that this is what is causing displaying other outputs to be 
delayed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Terry J. Reedy]

Terry J. Reedy  added the comment:

I found a stackoverflow question (and my then answer) about the same issue:
https://stackoverflow.com/questions/50108354/python-idle-running-code-differently-than-other-ides

Although the code posted does not run, it gives a very legitimate use case 
( is the signal) in which stream mixing is the lesser evil compared to 
the clearly wrong blocking of the output needed to decide when to give the 
signal. I now agree we should fix blocking now, if possible.

Tal, I posted my observations partly in the hope that you might be able to 
extend them.  So please take a look.

I am thinking about how to test a fix.  Test code needs to be able to 1. send 
code to be executed by run.py, 2. send a response to input() calls in the code, 
and 3. retrieve all output from the code.  New test machinery, needed not just 
for this issue, should be a separate issue.

A possibility including the minimum of transport machinery would be to replace 
the StdInputFile and StdOutputFile in run.py with test classes giving the 
needed test access.  Without knowing where the block is, it is unclear how much 
of the run machinery has to be included.

Another possibility, including all of the transport machinery except for the 
gui display, would be a test interpreter connected to an unaltered run process 
through a socket, as usual.  pyshell.PyShell (the shell window itself) 
initializespyshell.ModifiedInterpreter with itself.  A no-gui TestShell could 
do the same.  The TestShell would also need TestIn/OutputFile replacements.

--
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Guido van Rossum]

Guido van Rossum  added the comment:

It sounds like either an old implementation restriction or an old misguided 
attempt at keeping the IDLE console clean. I’m with Tal, let’s fix it and make 
it more like regular Python. Thanks John for taking the time to report this so 
clearly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Tal Einat]

Tal Einat  added the comment:

I tend to think that we should avoid discrepancies with a simple Python 
command-line shell by default, and in this case I am not convinced that there 
is a good reason for a divergence in behavior.

I'd be happy to look into debugging this issue in the next few days.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[John Smith]

John Smith  added the comment:

Interesting finding. Just some thoughs: I have to disagree on the "the 
experience with input() in IDLE is more useful and less confusing, especially 
to beginners" part.

Here are some reasons:

1. If a beginner starts with threading and expect a concurrent execution the 
confusion will be high if there is non in IDLE => implicitly changes of the 
runtime behavior siliencly enforced by an IDE can lead to misconcepts.

2. It took me (and my team) more than one hour to track a strange issue down to 
this IDLE related behavior - mainly because I assumed that IDLE would not 
change the runtime behavior of a program in that way => the deviation in 
behavior can even confuse experienced developers.

3. During my search I came across several reports of similar issues and most 
people were told to simply quit IDLE => "quirks" have impact on the reputation.

Personally I like and mainly use IDLE and would like to continue with it.

--
versions:  -Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Summary: With IDLE running normally, with 2 processes, "input('prompt')" in the 
main thread blocks execution of "print('whatever')" in a separate thread.  
(Verified on Windows with 3.9 and macOS with 3.8.)  The user can respond to the 
prompt without being disrupted by continued output.

For standard single process python or IDLE (started with -n), the thread output 
continues and can be mixed in with both the input prompt and user response and 
in the example, scroll prompt and even initial response off the screen.

I presume the difference is related to IDLE routing user-code use of the std 
streams through a 3rd thread that manages the inter-process socket connection.  
But I did not see why after more than an hour of looking.

Overall, I think the experience with input() in IDLE is more useful and less 
confusing, especially to beginners.  (I am having trouble imagining when the 
intermixing and interference would be useful.)

Guido, do you have any recollection of whether the IDLE difference, when using 
an execution process, is intended?  Do you regard intermixing input prompts and 
responses and thread output as a required feature of Python, or as an 
(unfortunate, IMHO) side-effect of using a dumb text interface?

Note: in IDLE, '>>>' come from IDLE, not the usercode execution process.  Hence 
a) asynchonous outputs from the execution process can appear on the screen, and 
b) IDLE, unlike the standard REPL, can and does prevents such text from mixing 
into and spoiling user code input.  (It may appear after '>>>' but is put 
before the input area.  This is an intentional interface difference.

My *immediate* inclination is to regard the IDLE difference as overall being a 
positive feature and leave it alone.  But it should be documented in "Running 
user code", which is already mostly about stdio differences.  

I don't deny that blocking output while a user responds is a negative.  I just 
see mixing streams as worse.  To work towards not blocking while not mixing, I 
replaced the thread loop with
for i in range(10): sys.stdout.write(f'some text {i}\n')
and separated input(prompt) into sys.stdout.write(prompt) and 
sys.stdin.readline().

Observations:
1. .write is better as it is atomic, while print() prints '\n' separately, with 
sometimes undesirable results.
2. The block is from readline.
3. While output to Shell is blocked on readline, typing a continuation-invoking 
'.' or calltip-invoking '(' in either Shell or editor allows one line of 
output.  Both send a non-stdio message from Shell to the run process and back.  
This might be a clue as to how read/readline blocks.

In the longer term, after some changes to Shell, I could imagine replacing 
__builtins__.input with a function that would check if sys.stdin/out were 
IDLE's replacements and if so, send the prompt to Shell tagged as 'prompt' 
rather than as 'stdout', so that Shell could keep prompt and user response  
separate from regular output, similar to the way now done for code input.  If 
.readline cannot be made to not block output, it might work for Shell to send  
the response back tagged as 'response' rather than as 'stdin' and not use 
readline.

--
nosy: +gvanrossum, taleinat
stage:  -> needs patch
versions: +Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[John Smith]

John Smith  added the comment:

P.S. here (other computer but with same observed behavior) I have a 3.6 32bit 
as one can see from the output.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[John Smith]

Change by John Smith :


Added file: 
https://bugs.python.org/file48901/IDLE_threading_issue_min_example1.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[John Smith]

John Smith  added the comment:

py -m idlelib -n path output:

Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:04:45) [MSC v.1900 32 bit 
(Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
 No Subprocess 

WARNING: Running IDLE without a Subprocess is deprecated
and will be removed in a later version. See Help/IDLE Help
for details.

>>> 
waiting for inputalive
alive
alive
alive
alive
alive

 -> so its working as intended

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[Terry J. Reedy]

Terry J. Reedy  added the comment:

If I understand, input() in one thread blocks print() in another thread. 
This sound like a new issue for this tracker, and possibly a bug in IDLE's 
implementation of std streams through a socket, but I need to see your code.  
Please try attaching it again.

Also, try running >py -m idlelib -n path

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39724] IDLE threading + stdout/stdin observed blocking behavior

[John Smith]

New submission from John Smith :

preamble: I am aware that I am not the first to encounter this issue but 
neither I could identify a preexisting ticket which fully matches nor is the 
commonly recommended "solution" (stay away from IDLE) satisfying.

environment: win10, python 3.7 (tested with 32 and 64 bit version)

description: If the attached script is started from IDLE the "alive" only shows 
up once for every input, while the script output "alive" frequently if ran from 
the terminal with python. So there is a discrepancy between the behavior of 
IDLE and "plain" python, which can lead to serious "irritations". If the print 
is replaced with logging.info and the logging is setup to write into a file 
everything works as expected and equal in both environments.

thoughts: the input call seems to block access to stdout(?) in "IDLE mode". I 
noticed that there are several topics/post regarding IDLE's stdout/in behavior 
but I was unabled to find a (convinient) solution besides "just quit using 
IDLE". It feels strange that the editor bundled with python has such a 
reputation and features such a deviation in behavior from "plain" python.

--
assignee: terry.reedy
components: IDLE
messages: 362456
nosy: John Smith, terry.reedy
priority: normal
severity: normal
status: open
title: IDLE threading + stdout/stdin observed blocking behavior
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com