hubcio opened a new issue, #3614: URL: https://github.com/apache/iggy/issues/3614
the `apache/iggy-connect` image ships only the `iggy-connectors` runtime binary. no plugin `.so` files inside. so anyone who wants to actually try a connector has to get the plugins from somewhere else first: 9 of our 15 plugins are prebuilt in the edge github pre-release tarballs (download, unpack, mount into the container), and the remaining 6 (delta, doris, http, influxdb sink + source, mongodb) are not prebuilt anywhere - those require cloning the repo and a full rust toolchain. either way it is a lot of friction for what should be a five minute "docker run + config file" experience. ## proposal two flavors of the image, distinguished by tag suffix (like `python:3.12-slim`): | tag | contents | | --- | --- | | `edge`, `x.y.z`, `latest` | runtime + all connector plugins baked in (new default) | | `edge-slim`, `x.y.z-slim`, `latest-slim` | runtime binary only (what the image is today) | the fat image becomes the default because that is what a new user expects: pull, point a config at a connector, go. size-sensitive deployments that bring their own plugins switch to `-slim`. plugins land in `/usr/local/lib`, which is already on the runtime's plugin search path - so a connector config can reference just `libiggy_connector_postgres_sink` with no absolute path. zero-config out of the box. ## what gets bundled all 15 cdylib plugins currently in the workspace: - sinks: delta, doris, elasticsearch, http, iceberg, influxdb, mongodb, postgres, quickwit, s3, stdout - sources: elasticsearch, influxdb, postgres, random side note: the edge tarball plugin list in `_build_rust_artifacts.yml` is a hardcoded default with only 9 of these, and the packaging step silently skips a missing `.so` instead of failing. the plugin list should come from one place so new connectors stop missing release artifacts - pre-merge CI already derives bin + cdylib packages from `cargo metadata`, and the cdylib filter alone is exactly this set, so there is prior art to reuse. ## how (roughly) - `core/connectors/runtime/Dockerfile`: the cargo-chef cook and build steps currently target `-p iggy-connectors` only, so the heavy plugin dependencies (deltalake, arrow, aws sdk, mongodb driver...) are not in the cached recipe today. both steps grow the plugin packages, plus a second final stage that COPYs the `.so` files into `/usr/local/lib`. slim stage stays exactly what it is today. - `.github/actions/utils/docker-buildx` action: needs to learn to build a specific dockerfile stage (`target:` is not plumbed through today), or `.github/config/publish.yml` grows a second component entry per flavor (which also means touching the hardcoded docker component allowlist in the publish plan job). - `docker-manifests` job: pushes only `edge` / `x.y.z` / `latest` today, so the `-slim` variants are new. the real work is the digest plumbing: per-arch digests travel as `docker-digest-<component>-<arch>` artifacts and the manifest is created by globbing every digest in one directory, so the two flavors must keep separate digest sets or they merge into one manifest. - edge gate: `rust-connectors` currently gates on `crates: [iggy-connectors]` only (plugins are deliberately excluded since the image does not ship them - the config comment says so). with plugins bundled, the gate expands to the full connector crate set, and that comment gets updated. - licensing: the image generates `LICENSE-binary` from `core/connectors/runtime/Cargo.toml` only. `third-party-licenses.sh` already accepts multiple `--manifest` flags and merges rust closures, so the generate step in the dockerfile and the "validate iggy-connect bundle" step in the validate action both grow the 15 plugin manifests. heads up: the plugin dependency closures have never been license-validated anywhere, so the first run may surface ASF category-x findings that block bundling a plugin until resolved. ## tradeoffs - image size grows by 15 statically-linked `.so` files (arrow, deltalake, aws sdk closures - and the release profile does not strip symbols today, only default debuginfo stripping). needs an actual measurement before merging, plus a strip step if the numbers are silly. `-slim` exists for anyone who cares. - docker build time goes up materially: the plugin deps compile cold until the chef recipe includes them, CI caching is layer-level (any workspace dependency change re-cooks everything, per arch, with lto + codegen-units=1), and the publish job has a 60 minute timeout. worth watching on the first runs. - tag semantics change: `:edge` starts pulling fat on the next master push after merge, `:latest` and versioned tags only from the next release onward. existing users get a bigger image with identical behavior (plugins are inert files until a config loads them), but release notes and the docker hub description must call it out. - bigger dependency surface on the default image means CVE scanners will flag plugin closures for every user, including ones who never load a plugin. `-slim` is the answer, but expect the reports. ## Tasks - [ ] dockerfile: extend chef cook + build to plugin crates, fat final stage with `.so` in `/usr/local/lib` - [ ] publish pipeline: `-slim` flavor end to end - buildx action (stage target or second component entry + plan-job allowlist), per-flavor digest artifacts, `-slim` manifests - [ ] expand edge gate to plugin crates, update the stale config comment - [ ] unify the plugin list (docker image, edge tarballs, release notes) via the existing `cargo metadata` cdylib derivation; make missing `.so` a hard failure in tarball packaging - [ ] extend `LICENSE-binary` generation and the license validation gate to plugin manifests; resolve whatever category-x findings show up - [ ] measure fat image size, add strip if needed; verify the heavy plugin closures cross-compile under the alpine + zig builder - [ ] docs: connectors README, docker hub description, quickstart example using the bundled plugins -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
