mattcasters commented on PR #7577: URL: https://github.com/apache/hop/pull/7577#issuecomment-5033007491
## Marketplace plugin publishing — validation & testing overview High-level steps taken to validate that optional marketplace plugins are packaged and can be published (including confidence for Jenkins SNAPSHOT deploy to `repository.apache.org` after merge to `main`). ### 1. Product packaging - Optional plugins are declared only in `optional-plugins.yaml` (catalog / full-client-env / publish lists). - Each module builds an installable `artifactId-version.zip` via the Maven `assembly` profile (`appendAssemblyId=false`). - Local package of registry modules produces all expected zips under `plugins/**/target/`. ### 2. Private / sandbox publish (data-hopper) - Deployed marketplace plugin SNAPSHOT zips to `https://repository.data-hopper.com/repository/apache-hop-plugins/` via `tools/deploy-snapshots-data-hopper.sh marketplace` (credentials local only; not in git). - Fixed env precedence so local Docker Nexus `.env` does not override an explicit remote URL. - Confirmed **HTTP 400** when the hosted repo was **Release-only** and SNAPSHOT was rejected; resolved by setting the repo version policy to **Mixed**. - Subsequent uploads of all optional plugins succeeded. - Marketplace install smoke against a Maven repo (install / list / validate / uninstall / apply) via `docker/marketplace-nexus/dummy-staging.sh` and related smoke tooling. ### 3. Jenkins-shaped SNAPSHOT layout (ASF CI confidence) Apache Jenkins on `main` does **not** use the data-hopper script. It: 1. `clean deploy` into `local-snapshots-dir` (file-based alt deployment repository), then 2. on `main`, `mvn -P deploy-snapshots wagon:upload` to ASF snapshots. Validation we ran for step (1) and the upload filter for step (2): - Root `pom.xml` **`deploy-snapshots`** wagon excludes: no longer drop plugin zips; still exclude hop-assemblies and core/engine/ui zips (so SNAPSHOT plugin zips can reach `repository.apache.org`). - **`maven-release-plugin`**: keeps per-plugin `assembly` (zips for releases/staging); still skips the hop-client `assemblies` reactor module. - Reproduced Jenkins **deploy layout** locally: ```bash ./mvnw -T 2 -B -DskipTests \ -DaltDeploymentRepository=snapshot-repo::default::file:$(pwd)/local-snapshots-dir \ clean deploy ``` → **BUILD SUCCESS**. - Ran `tools/verify-ci-snapshot-zips.sh` against that tree → **PASSED** (`ok=29 missing=0` for every artifact in `optional-plugins.yaml`). This supports that a successful main CI **Test & Build** will place marketplace plugin `.zip` files where wagon upload can pick them up. It does **not** replace watching Jenkins **Deploy** after merge, or a post-merge check that a SNAPSHOT zip appears under e.g. `https://repository.apache.org/content/repositories/snapshots/org/apache/hop/hop-tech-parquet/`. ### 4. Tooling added for repeatable checks - `tools/list-marketplace-plugins.sh` — registry → artifactId/modulePath - `docker/marketplace-nexus/publish-marketplace-plugins.sh` — package/deploy registry zips - `docker/marketplace-nexus/dummy-staging.sh` — publish + HTTP verify + install smoke - `tools/deploy-snapshots-data-hopper.sh` — private SNAPSHOT deploy helper - `tools/verify-ci-snapshot-zips.sh` — assert all optional plugins have zips in `local-snapshots-dir` ### Residual risk (called out on purpose) - Full Jenkins test suite / agent environment was not fully re-run here (`-DskipTests` used for the file:// deploy layout check). - Actual ASF snapshot **credentials and network upload** only happen on Jenkins main Deploy. - Official **release** staging remains a separate RM path (`release:perform` + staging repo UI). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
