dabla commented on code in PR #76545:
URL: https://github.com/apache/airflow/pull/76545#discussion_r4561292426


##########
task-sdk/src/airflow/sdk/execution_time/comms.py:
##########


Review Comment:
   Tried it in b5473a47fa, and it holds up: one worker thread owned by the 
decoder replaces `_LoopQueue`, `_LoopQueues` and the weak references 
(`comms.py` +37 −90). This code is on the path of every SDK call, so I measured 
it before switching:
   
   | | per-loop queues | one worker |
   |---|---:|---:|
   | `asend` round trip (callgrind Ir) | 398.1k – 399.4k | 380.4k (−4.7 %) |
   | 50 concurrent `asend`s (Ir) | 16.34M – 16.37M | 14.85M (−9.2 %) |
   | `send` (Ir, untouched) | 104k | 104k |
   | 4 loops × 50 concurrent, µs per request | 426 – 468 | 295 – 320 |
   | one `asyncio.run()` per call, µs per request | 918 – 952 | 419 – 471 |
   
   Two things differ from the description:
   
   - The cancel marker stays. A cancelled request the worker has not picked up 
never runs, but one it has picked up and that waits for the channel behind a 
sync `send()` on another thread can no longer be cancelled by the pool and 
would still be written. 
`test_asend_cancelled_while_waiting_for_the_thread_lock_leaves_it_free` catches 
that.
   - Fork: confirmed. With the worker created up front, an `asend()` in a 
forked child waited for a thread that no longer existed. The worker is now 
started by the first `asend()` of a process, and 
`test_asend_in_a_forked_child_is_served` forks to check it.
   
   The PR description and its benchmark table are updated to the new design.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to