[
https://issues.apache.org/jira/browse/IMPALA-6197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Volker updated IMPALA-6197:
--------------------------------
Target Version: Product Backlog
> Instrument our locks to track contention
> ----------------------------------------
>
> Key: IMPALA-6197
> URL: https://issues.apache.org/jira/browse/IMPALA-6197
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Lars Volker
> Priority: Major
> Labels: observability
>
> We should add instrumentation to our locks (or switch to an alternative like
> [absl::mutex|https://abseil.io/about/design/mutex]) to enable profiling of
> lock contention (see
> [absl::RegisterMutexProfiler()|https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.h#L933].
> This isn't a fully baked idea but could be very helpful with diagnosing
> issues around lock contention and lock order.
> [~tarmstrong] suggested, that it would be nice to be able to identify the
> following things:
> * Highly contended locks, based on how often threads get blocked and how long
> they get blocked for. E.g. Vincent Tran was looking at an issue that looked
> like IMPALA-4456 but we had no way to confirm.
> * Validation of lock order by defining a static lock order and then enforcing
> it with DCHECKs
> * Detection of deadlocks or livelocks involving multiple locks. This requires
> a graph of which locks are held or waited for by the different threads.
> * Some kind of per-query metrics for the profile, e.g. the number of times
> threads got blocked on locks.
> * The difficulty is doing this with low overhead and integrating with the
> lock implementations that we currently use (boost::mutex, boost::shared_mutex
> and SpinLock). Maybe it's best to do it incrementally or selectively only for
> locks that we suspect will be contention points.
> Abseil (Apache licensed) has some of this functionality:
> https://github.com/abseil/abseil-cpp/blob/master/absl/synchronization/mutex.cc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
