alamb commented on a change in pull request #8503:
URL: https://github.com/apache/arrow/pull/8503#discussion_r510851478
##########
File path: rust/datafusion/src/physical_plan/merge.rs
##########
@@ -103,37 +105,56 @@ impl ExecutionPlan for MergeExec {
self.input.execute(0).await
}
_ => {
- let tasks = (0..input_partitions).map(|part_i| {
+ let (sender, receiver) =
mpsc::unbounded::<ArrowResult<RecordBatch>>();
Review comment:
FWIW I am looking at this and it is definitely not doing what I
expected.I spent a while adding instrumentation (`println!`) and observing what
was happening.
From what I can tell, problem is that nothing ever arranges for poll_next to
get invoked again -- so if the data isn't available on the first invocation,
then no results are ever produced.
I tried several different things to get this working (most notably invoking
`Waker::wake` but I couldn't get the runtime to call `poll_next` again.
My current (un proven theory) is that it has something to do with the
interaction of `futures::Stream` and `tokio::stream::Stream` -
https://docs.rs/tokio/0.2.22/tokio/stream/trait.Stream.html which i think are
similar but not quite the same.
And the mixing of `futures` and `tokio` may be a problem... Maybe
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
