mmmmxa opened a new pull request, #4533: URL: https://github.com/apache/iggy/pull/4533
## Which issue does this PR address? Closes #3614 Continues #3658 by @kriti-sc, which went stale. Co-authored by @kriti-sc. ## Rationale Trying out a connector meant compiling the plugin from source and wiring its path by hand, because the `apache/iggy-connect` image shipped only the runtime. Shipping the plugins in the image, together with a compose example that works out of the box, removes that friction. ## What changed? The `iggy-connect` image contained only the runtime binary, so every plugin had to be built and mounted separately. It is now published in two flavors: a fat default image with the runtime and every connector plugin in `/usr/local/lib`, loadable by file name, and a `-slim` image with the runtime only. The plugin set is derived from `cargo metadata` in `scripts/ci/connector-plugins.sh`, and the image build, edge tarball, release notes and license checks all use it. The `sink-data-producer` example is now a Delta Lake sink demo. `docker compose up -d --build` starts the server, the connectors runtime and floci (an S3 emulator) and creates the stream, topic, bucket and Delta table. `cargo run --example sink-data-producer` produces the data, and two commands show it in the table. It replaces the Quickwit quick start in the connectors README and runs correctly end to end, which removes the main friction in developing connectors. ### Changes from the review of #3658 Fixed: - The slim image no longer compiles plugins, a failed plugin copy now fails the build, and the platform mapping is defined once. - `reqwest`, `tracing-subscriber` and `serde_json` features are pinned at the workspace root, so `-p` builds keep the features of the old `--bin` build. - The fat `LICENSE-binary` and the CI license check scan the runtime and each plugin separately against the real `Cargo.lock` and merge the results. Before, the runtime's own dependencies were missing. - The build cache is scoped per flavor. One manifest job per component pushes every flavor and creates the git tag last, so a failed slim push cannot leave a tag behind. Flavor config is validated, the Dockerfile stage field is called `stage`, `.dockerignore` is part of the `use_latest_ci` overlay, and job names include the flavor. - Images carry an `org.apache.iggy.flavor` label. - The release notes list the plugins found in the shipped tarball, the unused `connector_plugins` input is gone, and the edge image variants track the plugin crates and `connector-plugins.sh`. - A `docker-connectors-fat` pre-merge task builds the fat image when the Dockerfile, plugin manifests or their build scripts change. - `connector-plugins.sh` filters on the `iggy_connector_` prefix and warns about skipped cdylibs, and the README plugin lists include the connectors that were missing. Not taken as suggested: - `--package-flags` stays in `connector-plugins.sh` because the Dockerfile planner still uses it. - The label key is `org.apache.iggy.flavor`, not `org.opencontainers.image.variant`. The `org.opencontainers.image` namespace is reserved by the OCI spec, and `variant` already means the CPU variant. - Excluding docs-only edits from the edge gate no longer applies, because master replaced the path gate with `edge-image-variants.json`. ## Local Execution - Passed. Built the fat image locally and ran the compose example end to end. - Pre-commit hooks ran (`prek`). ## AI Usage 1. Claude (Claude Code). 2. I worked through the review comments of #3658 with it. It implemented the approved fixes, the compose example, the example configs and helper scripts, and the README changes. I chose the designs and reviewed every diff. 3. Ran the example end to end locally. Format, lint and config checks ran through `prek`. 4. Yes. Co-authored-by: kriti-sc <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
