This is an automated email from the ASF dual-hosted git repository.
manirajv06 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new b00884e6af [YUNIKORN-3348] Fix and extend the concurrency
documentation (#571)
b00884e6af is described below
commit b00884e6afeb7d4dd33725695586360fcaa1e49b
Author: Tigerquoll <[email protected]>
AuthorDate: Thu Aug 13 16:30:27 2026 +0530
[YUNIKORN-3348] Fix and extend the concurrency documentation (#571)
Extended the concurrency documentaion, mainly focussing on deadlock
detection and locking order.
Fixed typo's as well
Signed-off-by: Tigerquoll <[email protected]>
Closes: #571
Signed-off-by: Manikandan R <[email protected]>
---
docs/design/cache_removal.md | 8 +++++--
docs/developer_guide/build.md | 6 +++++
docs/user_guide/troubleshooting.md | 45 ++++++++++++++++++++++++++++++++++++++
3 files changed, 57 insertions(+), 2 deletions(-)
diff --git a/docs/design/cache_removal.md b/docs/design/cache_removal.md
index 91bd14e8e0..6a89a7db4f 100644
--- a/docs/design/cache_removal.md
+++ b/docs/design/cache_removal.md
@@ -423,7 +423,7 @@ A state change is thus immediate and this should prevent an
issue like [YUNIKORN
### Direction of lock
It is possible to acquire another lock while holding a lock, but we need to
make sure that we do not allow:
- Holding A.lock and acquire B's lock.
-- Holding B.lock and acquire B's lock.
+- Holding B.lock and acquire A's lock.
The current code in the scheduler takes a lock as late as possible and only
for the time period needed.
Some actions are not locked on the scheduler side just on the cache side as
each object has its own lock.
@@ -453,4 +453,8 @@ The partition should be locked while retrieving for
instance the node that needs
This approach fits in with the current locking approach and will keep the
locking changes to a minimum.
Testing, specifically end-to-end testing, should catch these deadlocks.
-There are no known tools that could be used to detect or describe lock order.
+
+Since this design was written, runtime detection has been added.
+The `pkg/locking` package wraps
[go-deadlock](https://github.com/sasha-s/go-deadlock) and provides both lock
order detection and lock wait timeout detection for all locks that use it.
+It is disabled by default and is turned on using the environment variables
described in [deadlock
detection](../user_guide/troubleshooting.md#deadlock-detection).
+The unit test run, `make test`, enables it in both the core and the k8shim.
diff --git a/docs/developer_guide/build.md b/docs/developer_guide/build.md
index 88cca29676..d916b0c283 100644
--- a/docs/developer_guide/build.md
+++ b/docs/developer_guide/build.md
@@ -163,6 +163,12 @@ To make sure that the local changes will not break other
parts of the
build you should run:
- A full build `make` (build target depends on the repository)
- A full unit test run `make test`
+ This runs the tests with the race detector (`-race`) and with deadlock
detection
+ enabled: `DEADLOCK_DETECTION_ENABLED=true`, `DEADLOCK_TIMEOUT_SECONDS=10` and
+ `DEADLOCK_EXIT=true`. A run that ends with a `POTENTIAL DEADLOCK` report and
a
+ non-zero exit code is a detected lock problem, not a flaky test. See
+ [deadlock detection](../user_guide/troubleshooting.md#deadlock-detection) for
+ the settings and how to read the report.
- For diagnosing flaky tests, which are challenging due to their infrequent
failures, use a looping command to repeatedly run the test. For instance, to
diagnose `TestNoFillWithoutEventPluginRegistered` in
`yunikorn-core/pkg/events/event_publisher_test.go`, you can use the following
command:
```sh
diff --git a/docs/user_guide/troubleshooting.md
b/docs/user_guide/troubleshooting.md
index 81b6fc1b83..5a410d7798 100644
--- a/docs/user_guide/troubleshooting.md
+++ b/docs/user_guide/troubleshooting.md
@@ -167,6 +167,51 @@ With the below scheduler REST API returns information
about full state dump used
For more details around the content of the state dump, please refer to the
documentation on [retrieve-full-state-dump](api/system.md#retrieve-state-dump)
+## Deadlock detection
+
+If the scheduler stops making progress but the process is still running and
responding to the
+REST API, a lock problem is one possible cause. YuniKorn can detect this at
runtime.
+
+All locks in the scheduler core and the Kubernetes shim are created through
the `pkg/locking`
+package, which wraps [go-deadlock](https://github.com/sasha-s/go-deadlock).
The shim delegates
+its configuration to the core, so a single set of settings covers both.
Detection is **disabled
+by default** and is turned on with environment variables on the scheduler
container:
+
+| Variable | Default | Effect |
+|---|---|---|
+| `DEADLOCK_DETECTION_ENABLED` | `false` | Master switch. go-deadlock is
always compiled in, this only enables it. |
+| `DEADLOCK_TIMEOUT_SECONDS` | `60` | How long a goroutine may wait for a lock
before a potential deadlock is reported. |
+| `DEADLOCK_EXIT` | `false` | Terminate the process with exit code 1 when a
potential deadlock is detected. |
+| `DEADLOCK_DISABLE_LOCK_ORDER` | `false` | Disable lock order (ABBA)
detection and keep only the wait timeout check. |
+
+Two independent checks are performed. The **wait timeout** check reports a
goroutine that has
+been waiting for a single lock for longer than `DEADLOCK_TIMEOUT_SECONDS`. The
**lock order**
+check reports two locks that have been acquired in opposite orders by
different goroutines,
+which can deadlock even if it has not happened yet.
+
+Reports are written to the log under the `core.diagnostics` subsystem at
`ERROR` level, and
+begin with `POTENTIAL DEADLOCK`. Each report names the goroutine holding the
lock and the
+goroutine waiting for it, with a stack trace for both.
+
+Whether detection is currently active can be read back from the scheduler:
+
+```shell script
+curl -X 'GET' http://localhost:9080/ws/v1/config -H 'accept: application/json'
+```
+
+The response includes `DeadlockDetectionEnabled` and `DeadlockTimeoutSeconds`.
+
+:::note
+`DEADLOCK_EXIT=true` will terminate a running scheduler as soon as a potential
deadlock is
+seen. That is useful when reproducing a problem, because it captures the state
at the point of
+detection instead of leaving a hung process, but consider the effect on a
production cluster
+before enabling it there.
+:::
+
+Detection is enabled for unit tests in both repositories. The `make test`
target sets
+`DEADLOCK_DETECTION_ENABLED=true`, `DEADLOCK_TIMEOUT_SECONDS=10` and
`DEADLOCK_EXIT=true`, so a
+test run that ends with a `POTENTIAL DEADLOCK` report has found a real lock
problem.
+
## Restart the scheduler
:::note
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
