zhuqi-lucas commented on code in PR #8146:
URL: https://github.com/apache/arrow-rs/pull/8146#discussion_r2280418075
##########
arrow-select/src/coalesce.rs:
##########
@@ -166,9 +171,30 @@ impl BatchCoalescer {
// We will for sure store at least one completed batch
completed: VecDeque::with_capacity(1),
buffered_rows: 0,
+ biggest_coalesce_batch_size: None,
}
}
+ /// Set the biggest coalesce batch size limit
+ ///
+ /// If set to Some(limit), batches larger than this limit will bypass
+ /// coalescing and be passed through directly. If None, all batches
+ /// will be coalesced according to the target_batch_size.
+ pub fn with_biggest_coalesce_batch_size(mut self, limit: Option<usize>) ->
Self {
+ self.biggest_coalesce_batch_size = limit;
+ self
+ }
+
+ /// Get the current biggest coalesce batch size limit
+ pub fn get_biggest_coalesce_batch_size(&self) -> Option<usize> {
Review Comment:
Thank you @alamb for good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
