This is an automated email from the ASF dual-hosted git repository.
Startrekzky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/devlake-website.git
The following commit(s) were added to refs/heads/main by this push:
new 838e3e125e0 docs: describe the single binary and plugin sideloading
(#837)
838e3e125e0 is described below
commit 838e3e125e0febc0d9935b19b3df86593926c0a4
Author: Klesh Wong <[email protected]>
AuthorDate: Sun Nov 1 19:56:13 2026 +0800
docs: describe the single binary and plugin sideloading (#837)
* docs: describe the single binary and plugin sideloading
`make build` now produces one `bin/lake` with the framework, the API and
every official plugin compiled in, and a deployed binary still loads
extra `.so` plugins from PLUGIN_DIR.
- Rewrite the "Compiling" section, which was rendered as a code block and
documented two things that do not exist: `PLUGIN=<name>` (the variable is
DEVLAKE_PLUGINS) and `make build-worker`.
- Add a "Sideloading plugins" section: where the .so files go, why they
must be built with the matching toolchain, and that a plugin which fails
to load is skipped rather than taking the server down.
- Document PLUGIN_DIR in the environment variables reference, including the
container mount path and the override warning.
* docs: show how to sideload a plugin into the Docker image
The image ships one binary with every official plugin, so adding a plugin
means mounting a directory over /app/bin/plugins rather than rebuilding it.
Adds an FAQ entry with the compose snippet, the naming/override rule and the
"build it in the matching lake-builder image" constraint.
* docs: explain how plugins are compiled into the server binary
The build no longer produces one .so per plugin for the release: a
generated registry links every official plugin into bin/lake, and the
image ships that binary alone. Document the model where plugin authors
and operators will look for it.
- Architecture: state that official plugins are compiled into the lake
binary, and add a "Plugin packaging" section covering the generated
registry, DEVLAKE_PLUGINS/GO_PLUGINS selection, Python plugins, and
how PLUGIN_DIR sideloading fits on top.
- Plugin implementation: describe how a plugin gets into the registry,
how to regenerate it, the core-only builds, and when to sideload
instead.
- Developer setup: note the checked-in registry and the CI drift check,
the debug build flag, and document how to build a core-only server.
---
docs/DeveloperManuals/DeveloperSetup.md | 52 ++++++++++++++++++++++++---
docs/DeveloperManuals/PluginImplementation.md | 16 +++++++++
docs/GettingStarted/DockerComposeSetup.md | 15 ++++++++
docs/GettingStarted/Environment.md | 14 ++++++++
docs/Overview/Architecture.md | 10 +++++-
5 files changed, 102 insertions(+), 5 deletions(-)
diff --git a/docs/DeveloperManuals/DeveloperSetup.md
b/docs/DeveloperManuals/DeveloperSetup.md
index 6d0bbd4bac8..471adf1b488 100644
--- a/docs/DeveloperManuals/DeveloperSetup.md
+++ b/docs/DeveloperManuals/DeveloperSetup.md
@@ -121,10 +121,54 @@ For provisioning, customizing, and creating dashboards,
please refer to our [Gra
## Compiling
- - Compile all plugins: `make build-plugin`
- - Compile specific plugins: `PLUGIN=<PLUGIN_NAME> make build-plugin`
- - Compile server: `make build`
- - Compile worker: `make build-worker`
+`make build` produces a single `bin/lake` binary: the framework, the API and
every official plugin are compiled into it.
+
+```sh
+cd backend
+make build
+```
+
+While iterating you usually do not want all of them. `DEVLAKE_PLUGINS` takes a
comma-separated list:
+
+```sh
+DEVLAKE_PLUGINS=jira,gitlab make build
+```
+
+That variable also decides what the registry generator writes to
`backend/plugins/all/all.go`, which is checked in. Run `make plugins-registry`
without `DEVLAKE_PLUGINS` before committing, or CI's drift check will fail.
+
+Release builds strip symbols. Set `DEVLAKE_DEBUG=1` to keep them, and use
`make debug` to start the server under `dlv`.
+
+To build plugins as shared objects instead — for sideloading, or for a server
without any plugin compiled in — use:
+
+```sh
+make build-plugin # every plugin
+DEVLAKE_PLUGINS=jira,gitlab make build-plugin # a subset
+```
+
+The output lands in `bin/plugins/<name>/<name>.so`; set `PLUGIN_DIR` to write
it somewhere else.
+
+A server with no plugin compiled in is built with an empty registry:
+
+```sh
+DEVLAKE_PLUGINS=none make build
+```
+
+The `devlake_external_plugins` build tag selects the same empty registry and
makes the intent explicit:
+
+```sh
+go build -tags devlake_external_plugins -o bin/lake ./server
+```
+
+## Sideloading plugins
+
+A released `lake` binary already contains every official plugin, and it still
accepts extras. Any `.so` found under `PLUGIN_DIR` (default `bin/plugins`) is
opened at startup and registered on top of the compiled-in set, so you can add
a custom plugin — or replace an official one — without rebuilding DevLake.
+
+Two things to know before you try it:
+
+- The `.so` has to be built with `go build -buildmode=plugin` against the
**same DevLake version, Go toolchain, build flags and dependency versions** as
the binary that loads it. Go rejects a plugin whose copy of a shared package
differs, so build it in the matching `lake-builder` image rather than on your
host.
+- A plugin that fails to load is logged and skipped. The server keeps running
without it, so check the log after adding one.
+
+The plugin entry point (`PluginEntry`, implementing `plugin.PluginMeta`) and
the interfaces a plugin can implement are described in the [plugin development
manual](https://github.com/apache/devlake/blob/main/backend/DevelopmentManual.md).
## References
diff --git a/docs/DeveloperManuals/PluginImplementation.md
b/docs/DeveloperManuals/PluginImplementation.md
index 2c3e1023062..df2835631e0 100644
--- a/docs/DeveloperManuals/PluginImplementation.md
+++ b/docs/DeveloperManuals/PluginImplementation.md
@@ -12,3 +12,19 @@ support developers who prefer it using it. PyDevLake is
relatively brand new, an
The Go development manual can be found
[here](https://github.com/apache/devlake/blob/main/backend/DevelopmentManual.md).
This manual also covers the framework in detail.
The Python development manual can be found
[here](https://github.com/apache/devlake/blob/main/backend/python/README.md).
+
+## How a plugin becomes part of DevLake
+
+DevLake ships as a single `lake` binary: the framework, the API server and the
official plugins are linked at build time. A Go plugin joins that binary when
its package lives under `backend/plugins/<name>` and its `impl` package exports
a type implementing
[`plugin.PluginMeta`](https://github.com/apache/devlake/blob/main/backend/core/plugin/plugin_meta.go).
+
+Registration is generated rather than hand-written. `make plugins-registry`
scans `plugins/*/impl`, writes `backend/plugins/all/all.go`, and the server
blank-imports that package so each listed plugin registers itself on startup.
+
+- `make build` compiles in every plugin and writes `bin/lake`.
+- `DEVLAKE_PLUGINS=jira,gitlab make build` compiles in a subset. The variable
also drives the generator, so run `make plugins-registry` without it before you
commit.
+- `DEVLAKE_PLUGINS=none make build` produces a binary with no plugin compiled
in.
+
+When a plugin is added, removed or renamed, regenerate the registry and commit
the result; CI fails when `plugins/all/all.go` is out of date.
+
+A plugin developed outside this repository is not compiled in. Build it as a
Go shared object — `make build-plugin`, or `go build -buildmode=plugin` — and
place it under `PLUGIN_DIR`. The server loads it at startup on top of the
compiled-in plugins. See [Sideloading
plugins](./DeveloperSetup.md#sideloading-plugins) for the toolchain and version
requirements.
+
+Python plugins do not go through this registry. They are discovered at runtime
under `REMOTE_PLUGIN_DIR`.
diff --git a/docs/GettingStarted/DockerComposeSetup.md
b/docs/GettingStarted/DockerComposeSetup.md
index f349a660374..476ae20b9bd 100644
--- a/docs/GettingStarted/DockerComposeSetup.md
+++ b/docs/GettingStarted/DockerComposeSetup.md
@@ -174,6 +174,21 @@ config-ui:
6. Final step: `docker compose up -d`
+### Can I add my own plugin to the DevLake image?
+
+The `apache/devlake` image ships a single binary that already contains every
official plugin, so there is nothing to install for the supported data sources.
If you need one that is not included, build it as a shared object and mount it
into the container:
+
+```yaml
+devlake:
+ ...
+ volumes:
+ - ./plugins:/app/bin/plugins
+```
+
+The directory is scanned at startup and each `<name>.so` is registered on top
of the built-in plugins; one that reuses an existing plugin's name replaces it.
A plugin that fails to load is logged and skipped, and the server keeps running
without it. `PLUGIN_DIR` already defaults to `/app/bin/plugins` in the image,
so the mount is all you need.
+
+The plugin has to be built against the same DevLake release and Go toolchain
as the running image, or it will be rejected at load time — build it in the
matching `lake-builder` image rather than on your host. See [Sideloading
plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for the
details.
+
## Troubleshooting
If you run into any problem, please check the
[Troubleshooting](/Troubleshooting/Installation.md) or [create an
issue](https://github.com/apache/devlake/issues)
diff --git a/docs/GettingStarted/Environment.md
b/docs/GettingStarted/Environment.md
index cfc13ad879c..7b88d85ec1f 100644
--- a/docs/GettingStarted/Environment.md
+++ b/docs/GettingStarted/Environment.md
@@ -8,6 +8,20 @@ This document explains how to set environment variables for
Apache DevLake and w
## Environment Variables
+### PLUGIN_DIR
+
+Directory scanned at startup for shared-object plugins (`.so`). The released
binary already contains every official plugin; anything found here is loaded on
top of that set, which makes this the directory to use for a custom or
third-party plugin.
+
+| Environment Variable | Description
| Default Value |
+| -------------------- |
--------------------------------------------------------------------------------------
| ------------- |
+| `PLUGIN_DIR` | Directory scanned for sideloaded `.so` plugins. A
missing directory is not an error. | `bin/plugins` |
+
+- In the Docker image, mount your plugins over `/app/bin/plugins`.
+- A plugin that fails to load — usually a toolchain or dependency mismatch —
is logged and skipped; the server keeps running without it.
+- A sideloaded plugin whose name matches an official plugin overrides it, and
the override is logged as a warning.
+
+See [Sideloading
plugins](../DeveloperManuals/DeveloperSetup.md#sideloading-plugins) for how to
build one.
+
### ENABLE_SUBTASKS_BY_DEFAULT
This environment variable is used to enable or disable the execution of
subtasks.
diff --git a/docs/Overview/Architecture.md b/docs/Overview/Architecture.md
index b0eb280c296..630b629cb66 100755
--- a/docs/Overview/Architecture.md
+++ b/docs/Overview/Architecture.md
@@ -16,9 +16,17 @@ A DevLake installation typically consists of the following
components:
- API Server: The main programmatic interface of DevLake.
- Runner: The runner does all the heavy-lifting for executing tasks. In the
default DevLake installation, it runs within the API Server, but DevLake
provides a temporal-based runner (beta) for production environments.
- Database: The database stores both DevLake's metadata and user data
collected by data pipelines. DevLake supports MySQL and PostgreSQL as of v0.11.
-- Plugins: Plugins enable DevLake to collect and analyze dev data from any
DevOps tools with an accessible API. DevLake community is actively adding
plugins for popular DevOps tools, but if your preferred tool is not covered
yet, feel free to open a GitHub issue to let us know or check out our doc on
how to build a new plugin by yourself.
+- Plugins: Plugins enable DevLake to collect and analyze dev data from any
DevOps tools with an accessible API. The official plugins are compiled into the
`lake` binary alongside the framework, the API server and the runner, so a
standard installation runs a single executable. DevLake community is actively
adding plugins for popular DevOps tools, but if your preferred tool is not
covered yet, feel free to open a GitHub issue to let us know or check out our
doc on how to build a new plugi [...]
- Dashboards: Dashboards deliver data and insights to DevLake users. A
dashboard is simply a collection of SQL queries along with corresponding
visualization configurations. DevLake's official dashboard tool is Grafana and
pre-built dashboards are shipped in Grafana's JSON format. Users are welcome to
swap for their own choice of dashboard/BI tool if desired.
+## Plugin packaging
+
+DevLake ships as one `lake` binary. At build time a generated registry in
`backend/plugins/all` imports every plugin under `backend/plugins` and
registers it with the framework, so the framework, the API and all official
plugins are linked into that single executable. The container image therefore
contains no plugin `.so` files.
+
+`DEVLAKE_PLUGINS` selects which plugins are compiled in: leave it unset for
all of them, pass a comma-separated list for a subset, or `none` for a
core-only binary. When the container image is built, the same selection is
exposed as the `GO_PLUGINS` build argument. Python plugins are not part of this
mechanism; they are discovered at runtime under `REMOTE_PLUGIN_DIR`.
+
+A plugin that is not part of the release can still be sideloaded. `PLUGIN_DIR`
is scanned at startup and every shared object found there is registered on top
of the compiled-in plugins: a `.so` with a new name adds a plugin, and one that
reuses an official plugin's name replaces it. A file that fails to load —
usually because it was built with a different toolchain or DevLake version — is
logged and skipped instead of stopping the server. See [Sideloading
plugins](../DeveloperManuals/Dev [...]
+
## Dataflow
<p align="center">
