alamb commented on code in PR #4820:
URL: https://github.com/apache/arrow-datafusion/pull/4820#discussion_r1062893220
##########
datafusion/core/src/physical_plan/repartition.rs:
##########
@@ -467,11 +488,16 @@ impl RepartitionExec {
};
partitioner.partition(batch, |partition, partitioned| {
+ let size = partitioned.get_array_memory_size();
Review Comment:
This doesn't account for sliced data (so if a batch of 1M gets cut up into
1000 pieces, each of the 1000 pieces will be charged the entire underlying
size). that being said it is a conservative estimate so that is good 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
