alamb commented on code in PR #17105:
URL: https://github.com/apache/datafusion/pull/17105#discussion_r2274591575
##########
datafusion/physical-plan/src/coalesce/mod.rs:
##########
@@ -113,9 +117,16 @@ impl LimitedBatchCoalescer {
}
}
+ let num_rows = batch.num_rows();
// Limit not reached, push the entire batch
self.total_rows += batch.num_rows();
self.inner.push_batch(batch)?;
+
+ // If the number of rows in the current batch exceeds the coalesce
size,
Review Comment:
if this turns out to work well, maybe it would be a good heuristic to add to
the underlying coalescer too in non strict mode 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
