ranflarion commented on code in PR #24820:
URL: https://github.com/apache/datafusion/pull/24820#discussion_r4015713603
##########
datafusion/physical-plan/src/joins/nested_loop_join.rs:
##########
@@ -1945,8 +2042,9 @@ impl FallbackCoordinator {
}
let _build_timer = build_time.timer();
- let merged_batch = concat_batches(&left_schema, &pending_batches)?;
- let n_rows = merged_batch.num_rows();
+ // Every batch of the pass is reserved above, so compacting it here
stays within budget.
+ let chunks = coalesce_chunks(pending_batches, &left_schema,
target_batch_size)?;
Review Comment:
You're right. I made some updates and added relevant testing. @2010YOUY01
feel free to use this as a reference if the decision is to do a replacement PR.
variant | peak live allocation
-- | --
PR head (both copies) | 40.05 MiB
head with replay kept raw (transition copy only) | 40.02 MiB
fixed with replay coalesced again (replay copy only) | 13.74 MiB
fixed | 8.14 MiB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
