andygrove commented on code in PR #7096:
URL: https://github.com/apache/datafusion-comet/pull/7096#discussion_r4338749022
##########
native/core/src/execution/delta_dv.rs:
##########
@@ -1837,21 +1808,16 @@ mod tests {
}
/// One `attach_access_plans` call must add exactly one consumer to the
task's pool no
- /// matter how many DV'd files it attaches. `CometFairMemoryPool` divides
the pool by the
- /// number of registered consumers, and the reservations live in the
returned files until
- /// the task ends, so one consumer per file would lower every other native
operator's fair
- /// limit for the whole task even when the DV bytes themselves are tiny.
Three DV'd files
- /// must leave a later consumer (a hash join build, say) its half of the
pool. Each file's
- /// bytes must still return to the pool when that file alone drops, while
the shared
- /// registration stays until the last file is gone.
+ /// matter how many DV'd files it attaches. Each file's bytes must still
return to the pool
Review Comment:
Yes, there is still one. The pool and `TrackConsumersPool` each keep an
entry per registered consumer, and its out-of-memory report lists the ten
consumers holding the most. A partition can have thousands of DV'd files, so
one consumer per file would grow both maps with the file count and fill that
report with this one operator. The comments on `attach_access_plans` and on the
test say so now (6a1c4dffca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
