[issue40487] Unexpected exception handler behavior in Jupyter when returning task objects created with create_task

[Andrew Svetlov]

Andrew Svetlov  added the comment:

tasks are not awaited, this is the problem.
The reproducer is not correct.
Closing.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40487] Unexpected exception handler behavior in Jupyter when returning task objects created with create_task

[Kyle Stanley]

Kyle Stanley  added the comment:

John Smith wrote:
> The code works normally again if I remove the return statement from the 
> `run_coro` function.

I am a bit surprised that the code works as intended simply by removing the 
return statement. Perhaps Yury or Andrew can clarify on that point.

>From looking at the code above, the main issue seems to be that the tasks are 
>not awaited or cancelled and cleaned up at any point. When creating a task 
>without ever awaiting it, there's no guarantee that it will be completed 
>within the duration of the event loop. This is handled for the user when using 
>asyncio.run() by cancelling the tasks and propagating any exceptions during 
>event loop finalization, but should be done manually or by the event loop when 
>not using it. For an example, see _cancel_all_tasks() in 
>https://github.com/python/cpython/blob/d699d5e6178adca785a8701c32daf5e18fad0bf1/Lib/asyncio/runners.py#L54.

So, my best guess as to what's happening is that the tasks are still in 
progress while the event loop is finalizing, and the exception handler isn't 
called. But, I can't say that for sure without knowing the implementation of 
the Jupyter event loop.

For now, my recommended solution would be to include something like the above 
_cancel_all_tasks() or simply await your tasks at the end to ensure they are 
completed before the event loop is finalized. Alternatively, Jupyter could 
include something like that in their event loop finalization process, so that 
users don't have to include it on their own when using asyncio.

I would also consider using asyncio.run(), but I'm not certain if it works 
correctly in a Jupyter notebook. I'm aware that it's not always a viable option 
when it is desired to use an existing event loop instead of creating a separate 
one; that's why I'm not explicitly recommending it for this situation.

--
nosy: +aeros

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40487] Unexpected exception handler behavior in Jupyter when returning task objects created with create_task

[John Smith]

John Smith  added the comment:

Additional note:

In a almost identical set-up, the simple fact of assigning the task object to a 
variable:

`task = loop.create_task(coroutine())`

instead of just calling:

`loop.create_task(coroutine())`

...results in the same unexpected behavior in exception handling, without even 
attempting to return the task object.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40487] Unexpected exception handler behavior in Jupyter when returning task objects created with create_task

[John Smith]

Change by John Smith :


--
type:  -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40487] Unexpected exception handler behavior in Jupyter when returning task objects created with create_task

[John Smith]

New submission from John Smith :

Hi,

I'm running the following code in a Jupyter notebook (so there is already a 
running loop).

`run_coro` (non-async function) adds given coroutines to the loop with 
`loop.create_task`. I want to return the task objects created by 
`loop.create_task` to check for result later in the code.

Adding the task objects to a list and returning the list has the exception 
handler behave differently than expected.


```
async def test1():
''' Enumerate numbers from 1 to 9'''
n = 0
while (n := n + 1) < 10:
print(f'Test 1: {n}')
await asyncio.sleep(0.3)
return 'Test 1: DONE'


async def test2():
''' Enumerate numbers starting 1, raising exception at 5'''
n = 0
while (n := n + 1) < 10:
if n == 5:
raise ValueError('Test Exception')
print(f'Test 2: {n}')
await asyncio.sleep(0.2)


def handle_exception(loop, context):
if context['exception']:
print(f'Caught exception <{context["exception"]}> while executing 
{context["future"]}')


def run_coro(loop, coros):
loop.set_exception_handler(handle_exception)
tasks = deque()
for coro in coros:
task = loop.create_task(coro)
#tasks.append(task)
# ^--- if I uncomment, exc handler behaves strangely
return tasks
# ^-- BUT if I remove the "return" statement, it behaves correctly again


loop = asyncio.get_running_loop()   # retrieves Jupyter's running loop

tasks = run_coro(loop, [test1(), test2()])
```

Normal (expected output):

```
Test 1: 1
Test 2: 1
Test 2: 2
Test 1: 2
Test 2: 3
Test 1: 3
Test 2: 4
Caught exception  while executing :10> exception=ValueError('Test Exception')>
Test 1: 4
Test 1: 5
Test 1: 6
Test 1: 7
Test 1: 8
Test 1: 9
```

If I uncomment `tasks.append(task)` and return the list of task objects, output 
is:

```
Test 1: 1
Test 2: 1
Test 2: 2
Test 1: 2
Test 2: 3
Test 1: 3
Test 2: 4
Test 1: 4
Test 1: 5
Test 1: 6
Test 1: 7
Test 1: 8
Test 1: 9
```

If I re-run the code (re-run the Jupyter cell) the exception message would 
appear at the beginning of the output (as if it had been waiting in a queue 
somewhere), so for some reason it seems to get eaten-up somewhere.

The code works normally again if I remove the return statement from the 
`run_coro` function.

Is this expected behaviour? Am I missing something?

--
components: asyncio
messages: 367988
nosy: asvetlov, jeanmonet, yselivanov
priority: normal
severity: normal
status: open
title: Unexpected exception handler behavior in Jupyter when returning task 
objects created with create_task
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com