Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10988 )
Change subject: IMPALA-7297: soft memory limit for reservation increases ...................................................................... Patch Set 5: > Also, doesn't this somewhat rely on the scanner threads being quicker to > claim the memory than the other threads? If the other threads already > expanded to their full 80%, this won't help, right? Or is the hope that those > other threads are cycling their memory often enough that, as scanner threads > encroach on the boundary, they'll have a chance to free some memory and not > re-allocate? There is a follow-on patch I'm working on to make the scanner threads a bit smarter about whether they start based on available memory and reduce the raciness. It still will be somewhat racy. It is typical though that the scans grab memory first - e.g. the scans all start immediately and the joins, aggregations and sorts gradually balloon as they accumulate memory from the scans. I think one fundamental design decision we made that impacts this is that there's no kind of pre-emption - one operator can't force another operator to scale down memory consumption if it consumed too much. -- To view, visit http://gerrit.cloudera.org:8080/10988 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39dcf2dd870a59c8b8cda4488fe41ce936d715ac Gerrit-Change-Number: 10988 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]> Gerrit-Comment-Date: Thu, 26 Jul 2018 23:04:17 +0000 Gerrit-HasComments: No
