Hi all,
I am currently working on writing a manual back-pressure mechanism for
AsOfJoin. We are trying a simple version where we maintain a buffer of
batches from one of the input sources (our left table source). We want to
pause production when we reach a certain number of batches, and resume
production from the source otherwise.
After some debugging, we observed that a call to mark the
backpressure_future_ as finished causes blocking, and we were wondering if
this is an expected behavior.
To clarify, we have a thread that handles our reading input from the source
(input received thread), and one that does the join processing (processing
thread). Our exact scenario is the following:
1) production begins normally
2) the input received thread observes that the buffer for the left
table is reached, and activates backpressure by calling 'PauseProduction'
on the left source node.
3) after processing the left table batch and clearing space in the
buffer, the processing thread deactivates backpressure by calling
'ResumeProduction' on the left source node.
4) control is not returned to the processing thread, and instead
blocks when
marking the backpressure_future_ as finished.
<https://github.com/iChauster/arrow/blob/asof_join2/cpp/src/arrow/compute/exec/source_node.cc#L181>
.
We were wondering what may cause a future to block like this, and if this
is a bug or an implementation error on our end. I've also attached my
progress so far below for the asof join node's backpressure in case you'd
like to take a look at more specifics.
Ivan
https://github.com/iChauster/arrow/blob/asof_join2/cpp/src/arrow/compute/exec/asof_join_node.cc#L792-L812
