Adar Dembo has submitted this change and it was merged. (
Change subject: pstack_watcher: blacklist older versions of gdb
pstack_watcher: blacklist older versions of gdb
Despite commit 6ed4690, I was still seeing timeouts in pstack_watcher-test
on CentOS 6.6. After going down a rabbit hole, I think I found the root
cause: a pretty printing bug that causes gdb to read from uninitialized
memory. In theory the symptom varies, but in my case gdb spent a very very
long time reading the memory backing 'std::string exit_info' in
PstackWatcher::RunStackDump. Because 'exit_info' was uninitialized at the
time of the read (i.e. while the test was blocked in Subprocess::Wait()),
its length contained garbage, and sometimes (varied with each execution of
the test binary) that garbage was a very large number. In such cases, gdb
would faithfully read the traced process' memory word by word. This looked
like an infinite loop but attaching strace to gdb revealed the truth.
Eventually the test would time out.
I couldn't pinpoint the bug with precision, nor could I figure out exactly
when it was fixed, so I ended up blacklisting versions older than a
"oldest known good" version. But I'm somewhat confident that this is the
root cause behind all of pstack_watcher-test's gdb-related issues.
Here are the combinations I manually tested:
- el6: gdb was too old and pstack was used.
- el6 with devtoolset-3: gdb was used.
- Ubuntu 14.04: gdb was used.
- Ubuntu 16.04: gdb was used.
In every case I couldn't repro the original timeout.
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <danburk...@apache.org>
2 files changed, 69 insertions(+), 19 deletions(-)
Kudu Jenkins: Verified
Dan Burkert: Looks good to me, approved
To view, visit http://gerrit.cloudera.org:8080/9962
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>