Andrew Lamb created ARROW-11802:
-----------------------------------
Summary: [Rust][DataFusion] Mixing of crossbeam channel and async
tasks can lead to deadlock
Key: ARROW-11802
URL: https://issues.apache.org/jira/browse/ARROW-11802
Project: Apache Arrow
Issue Type: Bug
Components: Rust - DataFusion
Reporter: Andrew Lamb
[~edrevo] noticed, on
https://github.com/apache/arrow/pull/9523#issuecomment-786237494, that the use
of crossbeam channels can potentially deadlock datafusion
The use of crossbeam channel is left over from earlier, non `async`
implementations and get been fingered in some hangs that [~MikeSeddonAU] has
observed in DataFusion ). Specifically the crossbeam channel can block a thread
when the channel is full or empty, which can result in blocking all the tokio
executor threads and deadlocking the system
The proposal is is to use tokio's mpsc channels instead of crossbeam which can
properly yield back to tokio to run another task when the channel is either
full or empty. .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
