tigerquoll opened a new pull request, #572: URL: https://github.com/apache/yunikorn-site/pull/572
### Description The site documents what the scheduler does — states, configuration, features — but very little about how to change it safely. The conventions that avoid introducing a deadlock or a data race are followed consistently in the code, but are recorded only as comments on individual functions, so they are hard to find and easy to miss. The partial exception is `docs/design/cache_removal.md`, which is a design proposal for a migration completed under YUNIKORN-317 rather than a maintained reference. This adds `docs/developer_guide/concurrency.md` and links it from the three pages a contributor is most likely to arrive from. #### Contents - **Locking** — the lock order, and the two prohibitions that follow from it: the partition lock must not be held while manipulating an application, queue or node; and a queue must not hold its own lock while calling its parent or children. Also the convention for breaking lock nesting with a goroutine when an upward call is unavoidable, the snapshot / notify-after-unlock / clone-on-return idioms, the meaning of the read-only field blocks, and a table of the locking comment banners used throughout the code. - **State machines** — the contract that `scheduler_object_states.md` does not cover: the caller holds the object lock so callbacks must be lock free, the object lock must not be released inside a callback, `SetState` bypasses every callback, and `"no transition"` is not an error. - **Resources** — nil versus zero versus absent, mutability and copying, what `Prune()` changes, which comparison variant to use, and that usage can legitimately exceed capacity. - **Threading model** — the core's single scheduling goroutine, event channels and periodic services; the shim's context lock as a serialisation lock and its single dispatcher goroutine. - **Detecting problems**, and a **reviewer checklist**. #### Links added - `src/pages/community/coding_guidelines.md` — that page covers formatting and linting only, so a contributor reading it end to end currently learns nothing about the lock order. - `docs/developer_guide/scheduler_object_states.md` — points at the state machine section for the callback rules. - `docs/design/architecture.md` — points at the threading model. #### Notes for reviewers **This describes existing conventions, it does not propose new ones.** Everything in it was taken from the current code in `yunikorn-core` and `yunikorn-k8shim` on master. Where the code already states a rule in a comment, the page says the same thing in the same terms. **It depends on #571.** The "Detecting problems" section links to `troubleshooting.md#deadlock-detection`, which is added by that PR for YUNIKORN-3348. Until it merges the build reports one broken anchor for this page; merging #571 first, or rebasing this branch afterwards, resolves it. Everything else in this PR is independent. **Scope.** The Jira suggests agreeing the outline before opening a pull request. This is offered as a concrete starting point rather than a finished document — it is easier to argue with a draft than with a proposal. If the structure is wrong, or sections should be split, moved or dropped, please say so and I will rework it. ### Type of change - [x] Documentation ### Jira issue Jira ID : https://issues.apache.org/jira/browse/YUNIKORN-3349 - [x] I have created a Jira issue for this pull request - [x] The Jira ID is part of the title of this pull request ### AI Tooling Generated by the Author with assistance from Claude Code. - [x] The PR includes the phrase "Generated by \<tool>", where \<tool> is the name of the AI tool used. - [x] My use of AI contributions follows the ASF legal policy. Check https://www.apache.org/legal/generative-tooling.html for details. ### How Has This Been Tested? - [x] `check_license.sh` returned an "all OK" result - [ ] `local_build.sh run` generated a usable website Instead of the Docker wrapper, the site was built directly with `pnpm build`, which runs the same Docusaurus build. Result: `[SUCCESS] Generated static files in "build"`. Checks performed on the generated output: - The new page is emitted at `build/docs/next/developer_guide/concurrency/index.html` and appears in the Developer Guide sidebar. - All three inbound links resolve, including the `#state-machines` deep link from the scheduler object states page. - The only new build warning is the broken anchor described above, which is the dependency on #571. No other warning names any page touched by this PR. ### Screenshots or other details Sources in the code for the conventions described, so they can be checked: | Section | Source | |---|---| | Partition lock prohibition | The comment on the `PartitionContext` struct in `yunikorn-core/pkg/scheduler/partition.go` | | Queue parent-before-self pattern | `getHeadRoom`, `incPendingResource`, `IncAllocatedResource`, `createPreemptionSnapshot` in `yunikorn-core/pkg/scheduler/objects/queue.go` | | Breaking lock nesting with a goroutine | `executeTerminatedCallback` in `yunikorn-core/pkg/scheduler/objects/application.go` | | Notify after unlocking | `SetCapacity`, `RemoveAllocation`, `ReplaceAllocation` in `yunikorn-core/pkg/scheduler/objects/node.go` | | Comment banners | Used throughout `partition.go`, `queue.go`, `application.go` and `ugm/queue_tracker.go` | | State machine lock contract | `HandleApplicationEvent` in `yunikorn-core/pkg/scheduler/objects/application.go`, and the "Locking mechanism" comment on `Application.handle` in `yunikorn-k8shim/pkg/cache/application.go` | | Resource semantics | The package comment and operations in `yunikorn-core/pkg/common/resources/resources.go` | | Core threading model | `yunikorn-core/pkg/scheduler/scheduler.go` and `pkg/rmproxy/rmproxy.go` | | Shim context lock and dispatcher | The `Context` struct comment in `yunikorn-k8shim/pkg/cache/context.go`, and `yunikorn-k8shim/pkg/dispatcher/dispatcher.go` | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
