alamb commented on code in PR #5308:
URL: https://github.com/apache/arrow-datafusion/pull/5308#discussion_r1108929173
##########
datafusion/core/benches/merge.rs:
##########
@@ -214,6 +266,26 @@ impl MergeBenchCase {
}
}
+ fn new_with_sort_input(partitions: &[Vec<RecordBatch>]) -> Self {
+ let runtime =
tokio::runtime::Builder::new_multi_thread().build().unwrap();
+ let session_ctx = SessionContext::new();
+ let task_ctx = session_ctx.task_ctx();
+
+ let schema = partitions[0][0].schema();
+ let sort = make_sort_exprs(schema.as_ref());
+
+ let projection = None;
+ let exec = Arc::new(MemoryExec::try_new(partitions, schema,
projection).unwrap());
Review Comment:
I double checked that the sort exec does actually make multiple output
streams. 👍
FYI @mustafasrepo and @ozankabak and @mingmwang -- not sure if you are
following this work from @jaylmiller but I think his work may make
repartitioned sorts (in parallel) a more compelling option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
