Hello Will Berkeley, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12112
to look at the new patch set (#6).
Change subject: [util] use lighter locking scheme for ThreadMgr
......................................................................
[util] use lighter locking scheme for ThreadMgr
Use the rw_spinlock primitive to guard the ThreadMgr's registry
threads instead of Mutex. Also, use the shared lock pattern to deal
with write and read access to the guarded entities of the ThreadMgr.
In addition, do not hold the lock for the whole duration of generating
the /threadz page for the embedded webserver.
In addition, ThreadMgr now uses std::unordered_map as a container
for category-specific thread information and enforces stricter
consistency rules while adding/removing thread information from
its registry.
The stricter consistecy around the ThreadMgr's registry caused issues
with multiprocessing in Python, so I updated the code in
test_scantoken.py to address that:
a) For python2.7 and later, start worker processes of the
multiprocessing pool _before_ a Kudu client object is instantiated.
Also, every worker process will terminate after executing a single
task, so the pool will spawn a new process with fresh address
space for follow-up tasks, if any.
b) For python2.6, switch to the serial execution of scan operations
originated from serialized tokens.
NOTE:
If spawning worker processes after the setUpClass() method, each
worker in multiprocessing pool gets a copy of the ThreadMgr's thread
registry in its address space with information on threads spawned by
the client from the main test process. Later on, when a newly
spawned multiprocessing worker creates an instance of a Kudu client
class on its own, the OS might reuse the formerly used thread
identifiers in the new (i.e. worker) process. That would trigger
the CHECKs on the consistency of the thread registry.
BTW, in Python 3 the problem with the duplicated address space
in worker processes has been resolved by using context and additional
worker spawn modes (e.g., 'forkserver').
Change-Id: I4d49c1c39392e01c45019844430a4fe3d116c277
---
M python/kudu/tests/test_scantoken.py
M src/kudu/util/os-util.cc
M src/kudu/util/thread.cc
3 files changed, 162 insertions(+), 120 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/12112/6
--
To view, visit http://gerrit.cloudera.org:8080/12112
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4d49c1c39392e01c45019844430a4fe3d116c277
Gerrit-Change-Number: 12112
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wdberke...@gmail.com>
