Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12262 )
Change subject: IMPALA-7941: part 2/2: use cgroups memory limit ...................................................................... Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc@233 PS2, Line 233: // Depending on the system configuration, we will have to calculate the process : // memory limit either based on the available physical memory, or if overcommitting : // is turned off, we use the memory commit limit from /proc/meminfo (see : // IMPALA-1690). > Can you update this comment to describe how cgroup mem limit fits in? Done http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc@358 PS2, Line 358: if (bytes_limit > MemInfo::physical_mem()) { : LOG(WARNING) << "Memory limit " : << PrettyPrinter::Print(bytes_limit, TUnit::BYTES) : << " exceeds physical memory of " : << PrettyPrinter::Print(MemInfo::physical_mem(), TUnit::BYTES); : } > If there is a cgroup limit, an equivalent of this logging would be useful ( Done http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc@364 PS2, Line 364: LOG(INFO) << "Using global memory limit: " : << PrettyPrinter::Print(bytes_limit, TUnit::BYTES); > We might consider adding some information to indicate how the value was set I pulled out this logic into a function since it was getting unwieldy and added some logging of the decisions. It's maybe not as nice as your suggested output but I think it's sufficient to understand the decision and the logic to produce the output is relatively simple. http://gerrit.cloudera.org:8080/#/c/12262/2/docker/daemon_entrypoint.sh File docker/daemon_entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/12262/2/docker/daemon_entrypoint.sh@45 PS2, Line 45: export JAVA_TOOL_OPTIONS="-Xmx2g $JAVA_TOOL_OPTIONS" > You might find I did see that, but I didn't want to embed test-specific logic into the containers. I have a JIRA to do the minicluster change - it's a subtask of IMPALA-691 - but I think for it to work reasonably we need to fix some of the tests with unreasonably high memory requirements. -- To view, visit http://gerrit.cloudera.org:8080/12262 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9fb4fb936a46fc194a204391d03c07c8c7fba21 Gerrit-Change-Number: 12262 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Bikramjeet Vig <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-Comment-Date: Fri, 25 Jan 2019 07:38:26 +0000 Gerrit-HasComments: Yes
