Hello Dan Burkert, Todd Lipcon,

I'd like you to do a code review. Please visit

    http://gerrit.cloudera.org:8080/9962

to review the following change.


Change subject: pstack_watcher: blacklist older versions of gdb
......................................................................

pstack_watcher: blacklist older versions of gdb

Despite commit 6ed4690, I was still seeing timeouts in pstack_watcher-test
on CentOS 6.6. After going down a rabbit hole, I think I found the root
cause: a pretty printing bug that causes gdb to read from uninitialized
memory. In theory the symptom varies, but in my case gdb spent a very very
long time reading the memory backing 'std::string exit_info' in
PstackWatcher::RunStackDump. Because 'exit_info' was uninitialized at the
time of the read (i.e. while the test was blocked in Subprocess::Wait()),
its length contained garbage, and sometimes (varied with each execution of
the test binary) that garbage was a very large number. In such cases, gdb
would faithfully read the traced process' memory word by word. This looked
like an infinite loop but attaching strace to gdb revealed the truth.
Eventually the test would time out.

I couldn't pinpoint the bug with precision, nor could I figure out exactly
when it was fixed, so I ended up blacklisting versions older than a
"oldest known good" version. But I'm somewhat confident that this is the
root cause behind all of pstack_watcher-test's gdb-related issues.

I tested on el6 and verified that pstack was used instead of gdb, unless I
ran the test in devtoolset-3, in which case its gdb was used. I also
verified that gdb is used on Ubuntu 16.04. In every case, I couldn't repro
the original timeout.

Change-Id: I8486a22462c12cbb88f6fe49cd54e82543f066bb
---
M src/kudu/util/pstack_watcher.cc
M src/kudu/util/pstack_watcher.h
2 files changed, 68 insertions(+), 17 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/9962/1
--
To view, visit http://gerrit.cloudera.org:8080/9962
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8486a22462c12cbb88f6fe49cd54e82543f066bb
Gerrit-Change-Number: 9962
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>

Reply via email to