alamb commented on code in PR #3386:
URL: https://github.com/apache/arrow-datafusion/pull/3386#discussion_r979394285
##########
datafusion/core/src/physical_plan/sorts/sort_preserving_merge.rs:
##########
@@ -543,22 +559,22 @@ impl SortPreservingMergeStream {
}
}
- loop {
- // NB timer records time taken on drop, so there are no
- // calls to `timer.done()` below.
- let elapsed_compute =
self.tracking_metrics.elapsed_compute().clone();
- let _timer = elapsed_compute.timer();
+ // NB timer records time taken on drop, so there are no
+ // calls to `timer.done()` below.
+ let elapsed_compute = self.tracking_metrics.elapsed_compute().clone();
Review Comment:
this simply reduces the overhead of timing , right?
##########
datafusion/core/src/physical_plan/sorts/sort_preserving_merge.rs:
##########
@@ -388,22 +389,23 @@ impl SortPreservingMergeStream {
}
Some(Ok(batch)) => {
if batch.num_rows() > 0 {
- let cursor = match SortKeyCursor::new(
+ let cols = self
+ .column_expressions
+ .iter()
+ .map(|expr| {
+
Ok(expr.evaluate(&batch)?.into_array(batch.num_rows()))
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let rows = self.row_converter.convert(&cols);
+
+ let cursor = SortKeyCursor::new(
Review Comment:
I agree that memory usage is a potential concern (as we are effectively
copying data into the `Rows` format.
A follow on PR would be good I think. I filed
https://github.com/apache/arrow-datafusion/issues/3609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
