[
https://issues.apache.org/jira/browse/IMPALA-12643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804430#comment-17804430
]
ASF subversion and git services commented on IMPALA-12643:
----------------------------------------------------------
Commit c0a015fdac677c51b6d0e0f8511cc29eb2f8f304 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c0a015fda ]
IMPALA-12643 (part 1): Limit memory consumption for resolve_minidumps.py
On some platforms (Centos 7), resolve_minidumps.py's call to
minidump_stackwalk goes haywire and uses all the system memory
until it gets OOM killed. Some library must have corrupt
symbols, etc. As a workaround, this detects whether the
prlimit utility is present and uses this to run minidump_stackwalk
with a 4GB limit on virtual memory. This kills the process
earlier and avoids using all system memory.
Testing:
- Verified that bin/jenkins/finalize.sh uses resolve_minidumps.py
on a Redhat 8 Jenkins job (and it works)
- Verified that bin/jenkins/finalize.sh works properly on
my Ubuntu 20 box
- Ran a Jenkins job on Centos 7 and verified that the prlimit
code kills minidump_stackwalk when it uses 4GB of memory.
Change-Id: I4db8facb8a037327228c3714e047e0d1f0fe1d94
Reviewed-on: http://gerrit.cloudera.org:8080/20862
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
> resolve_minidumps.py's minidump_stackwalk call hits OOM on Centos7
> ------------------------------------------------------------------
>
> Key: IMPALA-12643
> URL: https://issues.apache.org/jira/browse/IMPALA-12643
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.4.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
>
> When looking into fixing IMPALA-11909 to enable running resolve_minidumps.py
> from finalize.sh, testing on Centos 7 showed that the minidump_stackwalk
> invocation from resolve_minidumps.py went haywire and used all the system's
> memory.
> resolve_minidumps.py is unique in processing all applicable system libraries.
> If some system library had corrupt symbols, it could cause minidump_stackwalk
> to get into an infinite loop.
> We should track down the offending library, but we should also run
> minidump_stackwalk with a limit on its memory usage to avoid using all the
> system memory.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
