klesh opened a new issue, #9276: URL: https://github.com/apache/devlake/issues/9276
## What and why to refactor DevLake currently compiles every plugin with `go build -buildmode=plugin` into its own `.so` (`backend/scripts/build-plugins.sh`), and the server discovers them at startup by walking `PLUGIN_DIR` (`backend/core/runner/loader.go`). The Go plugin mechanism cannot share a linked image. Each `.so` embeds its own copy of the core, the helpers, and every third-party dependency it needs. At runtime the loader dedupes packages that the host also links, so RSS is not as bad as disk — but on disk (and therefore in the published container image) the same ~40 MB of shared code is duplicated 47 times. **Measured on the currently published images:** | Artifact | amd64 | arm64 | |---|---|---| | `apache/devlake:main_261028_1424_21af7cd` | 3.06 GB | 2.84 GB | | `apache/devlake:base` (same base, no Go binaries) | 306 MB | 293 MB | | **Go payload** | **≈ 2.75 GB** | **≈ 2.55 GB** | **Measured locally** (`backend/bin`): - `lake` (server, no plugins): 63 MB - 4 plugins: 201 MB total - a single plugin `.so` (CGO on, unstripped): 65–68 MB (`jira` 65.2 MB, `github` 67.7 MB, `gitlab` 66.4 MB) - 47 plugins × ~65 MB ≈ 3.0 GB, consistent with the published image **Proof of concept for the alternative** — a throwaway `main` that blank-imports all 47 `plugins/<name>/impl` packages: | Build | Size | |---|---| | all 47 plugins in one binary, `CGO_ENABLED=1 -ldflags "-s -w"`, incl. `gitextractor` | **47 MB** | | all 47 except `gitextractor`, `-s -w` | 31 MB | | all 47 except `gitextractor`, unstripped | 46 MB | | single plugin alone, `CGO_ENABLED=0` (shows the shared ~33 MB floor) | 33–38 MB | So the Go payload goes from ≈ 2.75 GB to ≈ 47 MB, and the image from ≈ 3.06 GB to ≈ 0.36 GB — roughly an 8× smaller image and a ~58× smaller Go payload. Build time also improves: linking 47 plugins costs ~4–5 s each (~3.5 min at the script's parallelism cap of 4), while linking all 47 into one binary takes ~1.6 s with a warm cache. Compilation is shared either way. Go itself does not offer a better dynamic plugin story, and this is unlikely to change: - The [`plugin` package docs](https://pkg.go.dev/plugin) list the drawbacks and explicitly recommend the alternative: *"it may be simpler for that person or component to generate Go source files that blank-import the desired set of plugins and then compile a static executable in the usual way."* - `-buildmode=plugin` requires external (cgo) linking, so it forces `CGO_ENABLED=1` for the whole server build. - Host and plugins must match exactly in toolchain version, build tags, flags and dependency versions, otherwise the process dies at load time. Nothing in the repo enforces this. - Plugins cannot be unloaded, are poorly supported by the race detector, and are unsupported on Windows. - Go 1.27.0/1.27.1 shipped a runtime regression where loading multiple plugins corrupted stdlib types (golang/go#81303). It is fixed in a patch release, but it is exactly the failure mode this design keeps exposing us to. So the proposal is: **make the single binary the default distribution artifact, and keep `-buildmode=plugin` as an opt-in mode.** ## Describe the solution you'd like Compile-time registration (the Caddy / Grafana / Prometheus pattern). The public plugin interfaces (`PluginMeta`, `PluginTask`, `PluginModel`, …) do not change. ### 1. Registration helper Add to `core/plugin`: ```go // MustRegister registers p under p.Name(). func MustRegister(p PluginMeta) { if p.Name() == "" { panic("plugin: empty plugin name") } _ = RegisterPlugin(p.Name(), p) } ``` `RegisterPlugin` already has overwrite semantics, so double registration (see §5) is harmless. ### 2. Each plugin self-registers Every `backend/plugins/<name>/impl` package already imports `core/plugin`, and all 47 `Name()` values are unique (verified). Add one `init()` per plugin: ```go func init() { plugin.MustRegister(Gitlab{}) } ``` No file moves are needed: `plugins/<name>/<name>.go` stays `package main` so `-buildmode=plugin` and standalone `go run ./plugins/<name>` keep working untouched. ### 3. Generated registry package New `backend/plugins/all/`: ```go //go:build !devlake_external_plugins // Code generated by tools/pluginsgen. DO NOT EDIT. package all import ( _ "github.com/apache/devlake/plugins/ae/impl" _ "github.com/apache/devlake/plugins/argocd/impl" // ... one line per plugin ) ``` plus `plugins/all/external.go` with `//go:build devlake_external_plugins` for the slim-core + `.so` build. - Generator: `backend/tools/pluginsgen` (a small Go program driven by `go list ./plugins/*/impl`), wired as `make plugins-registry` and `//go:generate`. - `DEVLAKE_PLUGINS=a,b` is honoured by the generator so the existing "build a subset" workflow keeps working. - CI gains a "generated registry is up to date" check, same pattern already used for `swag` and `mock`. ### 4. Link the registry into the server `server/main.go` blank-imports `plugins/all`. Because the import is unconditional (unless the build tag is set), the linker cannot drop the plugin packages, and `plugin.AllPlugins()` is populated before `runner.LoadPlugins` runs. ### 5. Keep `.so` loading as an option `LoadGoPlugins` stays as-is, with one fix: tolerate a missing or empty `PLUGIN_DIR` instead of failing `errors.Must`. Today `filepath.WalkDir` on a non-existent directory returns an error, so a single-binary image would have to ship an empty `bin/plugins` just to boot. Resulting modes: | Mode | Build | Runtime | |---|---|---| | default (new) | `go build ./server` → one `bin/lake` | all plugins built in | | external plugins (new opt-in) | `-tags devlake_external_plugins` + `make build-plugin` | `bin/lake` + `.so` files in `PLUGIN_DIR` | | hybrid | default build | built-ins are overwritten by `.so` files with the same name | ### 6. Build and packaging - `make build` → `go build -trimpath -ldflags "-s -w" -o bin/lake ./server`; `make build-plugin` stays as the opt-in legacy path. - `backend/Dockerfile`: replace the `build-plugin` + `build-server` steps with a single build, and stop copying `/app/bin/plugins` into the runtime stage (`/app/bin` only). - `backend/Dockerfile.local`: same. - `make dev` / `make godev`: run `go run ./server` after `make plugins-registry`; no plugin build, no `PLUGIN_DIR` needed. - Docs: update the plugin layout section of `AGENTS.md`, the developer instructions in `README.md`, and the `PLUGIN_DIR` note in `env.example`. ## Compatibility - Plugin authors keep the same interfaces and the same directory layout; the only addition is a three-line `init()` in `plugins/<name>/impl`. - `-buildmode=plugin` remains fully supported and is exercised by the opt-in build. - The e2e harness already uses exactly this registration path: `backend/test/helper/client.go` imports `dora`, `org`, `refdiff` and calls `plugin.RegisterPlugin(p.Name(), p)` directly. The framework side is therefore already proven today. - `RootPkgPath()` — currently documented as *"PkgPath information lost when compiled as plugin(.so)"* — becomes accurate for static builds. Cleaning that up is deliberately out of scope here. ## Task list - [ ] `core/plugin`: add `MustRegister` - [ ] 47 × `plugins/<name>/impl`: add `init()` registration - [ ] `tools/pluginsgen` + generated `plugins/all` package (+ `external.go` build-tag stub) - [ ] `server/main.go`: blank-import `plugins/all` - [ ] `core/runner/loader.go`: tolerate missing/empty `PLUGIN_DIR` - [ ] `scripts/`: unified build; keep `build-plugins.sh` for the opt-in mode - [ ] `Makefile` (root + `backend`): `build`, `dev`, `godev`, `plugins-registry` - [ ] `backend/Dockerfile` and `backend/Dockerfile.local`: single binary, drop `/app/bin/plugins` - [ ] CI: check the generated registry is up to date; assert the image size - [ ] Docs: `AGENTS.md`, `README.md`, `env.example` ## Risks and open questions 1. **Startup cost** — every plugin's `init()` (including the subtask dependency sorter validation in each `impl`) now runs at process start instead of at plugin load. It runs at plugin load today too, so the total work is the same, just moved earlier and serialized; worth measuring. 2. **Swagger generation** — `scripts/swag.sh` uses `--parseDependency --parseInternal` over `./server` and `./plugins/*`. Adding a registry package should not change the output, but `make swag` must be re-run and diffed. 3. **Migration ordering** — `registerPluginsMigrationScripts()` iterates the plugin map, and registration order changes from "filesystem walk order" to "generated file order". Migration scripts are plugin-namespaced, so this should be safe, but it needs verifying. 4. **Double registration in `.so` mode** — `impl.init()` plus the loader's `PluginEntry` registration. `RegisterPlugin` overwrites, so this is benign; if we would rather keep `MustRegister` strict we need an explicit "already registered, skipping" path. 5. **Subset builds** — `DEVLAKE_PLUGINS` currently drives `build-plugins.sh` and `swag.sh`. The generator has to support the same variable, otherwise `GO_PLUGINS` in `backend/Dockerfile` silently becomes a no-op. ## Related issues - #9092 — [Proposal] Deprecate Python plugin framework and consolidate on Go. Orthogonal, but both change how plugins are packaged and should be sequenced against each other. ## Additional context Raw measurements, if anyone wants to reproduce them: ```console # published image sizes $ curl -s https://hub.docker.com/v2/repositories/apache/devlake/tags/main_261028_1424_21af7cd | jq '.full_size' 3057398785 $ curl -s https://hub.docker.com/v2/repositories/apache/devlake/tags/base | jq '.full_size' 306008557 # one plugin as a Go plugin (CGO on, unstripped) $ CGO_ENABLED=1 go build -buildmode=plugin -o /tmp/jira.so ./backend/plugins/jira $ stat -c %s /tmp/jira.so 65179384 # all 47 plugins in one binary $ CGO_ENABLED=1 go build -ldflags "-s -w" -o /tmp/all ./probe_all $ stat -c %s /tmp/all 46976352 ``` The 47 buildable plugins are exactly the directories picked by the current filter in `build-plugins.sh` (`plugins/*` minus `core`, `helper`, `logs`, `schema_e2e`), and all of them have an `impl` package. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
