Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10988 )
Change subject: IMPALA-7297: soft memory limit for reservation increases ...................................................................... Patch Set 5: Haven't looked at the code yet, just trying to understand the overall design. Mind if I repeat it back to you and you can tell me if I got it right? Current state of the world: --------------------- - Each query has a user-specified mem limit - We calculate a "reservation limit" as 80% of that mem limit (except for in some corner cases for tiny mem limits where there is a 100MB threshold) - Operators that support spilling all compete to try to snag chunks from the reserved pool. If such an allocation fails, those operators are expected to spill, etc. - Other operators like scans aren't reservation-aware, so they just allocate when they feel like it. -- This normally comes out of the 20% pool made up of (mem_limit - reservation_limit) -- If these operators fail to acquire their memory, they'll just fail the query. - If the reserved operators collectively haven't used up their allocated 80%, the other operators are allowed to slop into the reserved pool so long as mem_limit isn't exceeded. - If that "slopping" has happened, the reserved operators will still keep allocating chunks and likely put the query exactly at the mem limit, at which point any further allocation from the non-reserved operators will fail the query. This patch: ----------- Same as above, except: - if the reserved operators try to allocate but the overall mem_limit is close to being exceeded (because naughty operators slopped over into their allocation or close to doing so) they'll just pretend that the reservation amount was smaller than it actually is. Put another way, is this patch basically saying "even though we reserved 80%, we'll actually dynamically reduce that 80% allocation if it looks like our other operators use more than we had hoped"? I guess the piece I'm not 100% understanding is why this is a separate "soft" limit vs just some kind of dynamic adjustment of the reservation pool size based on observed "slop" size? Also, doesn't this somewhat rely on the scanner threads being quicker to claim the memory than the other threads? If the other threads already expanded to their full 80%, this won't help, right? Or is the hope that those other threads are cycling their memory often enough that, as scanner threads encroach on the boundary, they'll have a chance to free some memory and not re-allocate? -- To view, visit http://gerrit.cloudera.org:8080/10988 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39dcf2dd870a59c8b8cda4488fe41ce936d715ac Gerrit-Change-Number: 10988 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Thu, 26 Jul 2018 01:34:04 +0000 Gerrit-HasComments: No
