alamb commented on a change in pull request #8503:
URL: https://github.com/apache/arrow/pull/8503#discussion_r513681082
##########
File path: rust/datafusion/src/physical_plan/merge.rs
##########
@@ -103,37 +105,56 @@ impl ExecutionPlan for MergeExec {
self.input.execute(0).await
}
_ => {
- let tasks = (0..input_partitions).map(|part_i| {
+ let (sender, receiver) =
mpsc::unbounded::<ArrowResult<RecordBatch>>();
Review comment:
It is fairly astonishing that Rust has made streaming so hard.
However, I got it to compile and mostly work. Here is a commit (changes to
this branch) with the approach that passes most of the the arrow and datafusion
tests and also doesn't require waiting for all the input threads to complete:
https://github.com/apache/arrow/commit/28075970d8d59fa87359ad38470669485f029578
It still does not quite work in all cases (I need to port the normal
aggregate over) but I'll finish that up.
I am curious if you have any thoughts about the approach. I'll try and write
it up coherently tomorrow morning, now I need to go take care of some other
things.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
