mattcasters opened a new issue, #8722: URL: https://github.com/apache/hop/issues/8722
### What would you like to happen? Ran command: `find . -name "load-and-execute.sh" -o -name "Dockerfile*"` Viewed load-and-execute.sh:1-200 Ran command: `git grep -i "marketplace" | grep -i "apply"` Ran command: `find plugins/misc/marketplace -type f -name "*Command*.java" -o -name "*Cli*.java"` Viewed MarketplaceCommand.java:1-150 Viewed MarketplaceCommand.java:151-300 Viewed MarketplaceCommand.java:301-450 Viewed MarketplaceCommand.java:451-600 Viewed MarketplaceCommand.java:630-750 Viewed load-and-execute.sh:200-368 Viewed HopInstallSpec.java:1-95 Viewed HopInstallSpecLoader.java:1-100 Ran command: `ls -la /home/matt/git/ProjectDataHopper/hopper-edw/` Viewed hop-marketplace-repo.yaml:1-65 Ran command: `git grep -n "driver install"` Viewed run-web.sh:30-85 Viewed docker-container.adoc:400-460 Ran command: `find plugins/misc/marketplace/ -name "PluginInstaller.java" -o -name "PluginDiscovery.java" -o -name "MarketplaceConfig.java"` Viewed PluginInstaller.java:50-120 Viewed PluginInstaller.java:121-200 Viewed MarketplaceConfig.java:1-100 Viewed MarketplaceRepository.java:1-100 Ran command: `git grep "HOP_MARKETPLACE"` Viewed MarketplaceRepositoryDefinition.java:1-100 Viewed PluginInstaller.java:195-250 Viewed DriverInstallCommand.java:1-100 Viewed PluginDiscovery.java:50-120 Viewed PluginDiscovery.java:121-200 Viewed MavenRepositoryClient.java:1-100 Viewed MarketplaceCommand.java:725-760 Viewed MarketplaceConfig.java:110-160 Viewed MarketplaceConfig.java:161-210 Viewed MarketplaceConfig.java:211-260 Here is a structured feature request proposal to make installing marketplace plugins in the `apache/hop` Docker container seamless and declarative via environment variables. --- # Feature Request: Add Marketplace Plugin Auto-Installation to `apache/hop` Docker Container ## 1. Problem Statement & Motivation When deploying Apache Hop in containerized environments (Kubernetes, Docker Compose, Airflow, CI/CD runners), pipelines frequently require plugins hosted in corporate artifactories (JFrog Artifactory, Sonatype Nexus, AWS CodeArtifact) or community repositories (such as Project Data Hopper’s Nexus at `https://repository.data-hopper.com/repository/hop-community-plugins/`). Currently, installing a custom plugin in the standard `apache/hop` container requires one of the following: 1. **Building a custom Docker image** inheriting from `apache/hop`, defeating the convenience of using standard official images with mounted projects. 2. **Writing a custom entrypoint script** (`HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH`). 3. **Using indirect repository specs**: e.g., running `hop marketplace apply -f <spec-url>` or `hop marketplace repo import <yaml-url>`. As seen with specifications like `hop-marketplace-repo.yaml`, these files contain extensive catalog and GUI metadata (`kind`, `browse: true`, `categories`, `description`, `homepage`, `minHopVersion`, `installPath`). In the context of a headless runtime container, **all that extra metadata is unnecessary**. The container runtime only needs to know: - **Which plugin(s)** to install (coordinates / artifact IDs). - **Where** to fetch them (the Maven/Artifactory repository base URL). - **Credentials** (if the artifactory requires authentication). Apache Hop already has an established precedent for JDBC drivers: [`HOP_DRIVERS_DOWNLOAD`](file:///home/matt/git/mattcasters/hop/docker/resources/load-and-execute.sh#L32-L76) and `HOP_DRIVERS_MAVEN_REPO`. We should provide an equivalent mechanism for marketplace plugins. --- ## 2. Proposed Container Environment Variables Following the convention established by `HOP_DRIVERS_*`: | Environment Variable | Description | Example | | :--- | :--- | :--- | | **`HOP_PLUGINS_DOWNLOAD`** | Comma-separated list of plugin coordinates to install before Hop starts. Supports short names, `artifactId`, `artifactId:version`, or `groupId:artifactId:version`. | `hopper-edw:0.10.0`<br>`org.hopper:hopper-edw:0.10.0,hop-tech-parquet` | | **`HOP_PLUGINS_MAVEN_REPO`** | Optional corporate Artifactory / Nexus base Maven repository URL. If omitted, uses the default Hop marketplace repositories (ASF & Central). | `https://repository.data-hopper.com/repository/hop-community-plugins/`<br>`https://artifactory.corp/artifactory/hop-plugins/` | | **`HOP_PLUGINS_REPO_ID`** | Optional identifier for the repository in Hop configuration (default: `corporate-repo`). | `data-hopper-community` | | **`HOP_PLUGINS_REPO_AUTH_TYPE`** | Optional authentication type: `auto` (default), `basic`, or `token`. | `basic`, `token` | | **`HOP_PLUGINS_REPO_USERNAME`** | Optional username for authenticated repositories (can also fall back to `HOP_MARKETPLACE_USERNAME`). | `svc-hop-runner` | | **`HOP_PLUGINS_REPO_PASSWORD`** | Optional password or bearer token (can also fall back to `HOP_MARKETPLACE_PASSWORD`). | `${ARTIFACTORY_TOKEN}` | *(Optional declarative fallback)*: - **`HOP_PLUGINS_ENV_FILE`**: If a team *does* already have a `hop-env.yaml` or `hop-marketplace-repo.yaml` available via HTTP(S) or local volume, setting this variable could trigger `hop marketplace apply -f "${HOP_PLUGINS_ENV_FILE}"`. --- ## 3. Implementation in Container Scripts Both [`load-and-execute.sh`](file:///home/matt/git/mattcasters/hop/docker/resources/load-and-execute.sh) and [`run-web.sh`](file:///home/matt/git/mattcasters/hop/docker/resources/run-web.sh) would add a helper function `install_marketplace_plugins` that executes before Hop starts (alongside `install_jdbc_drivers`). ### Proposed Function: `install_marketplace_plugins()` ```bash # Download Marketplace plugins on container start, before Hop is launched. # Driven by environment variables: # HOP_PLUGINS_DOWNLOAD comma-separated plugin coordinates: # e.g. "hopper-edw:0.10.0,org.apache.hop:hop-tech-parquet:2.19.0" # HOP_PLUGINS_MAVEN_REPO optional Maven/Artifactory repository base URL # HOP_PLUGINS_REPO_ID optional repository ID (default: "corporate-repo") # HOP_PLUGINS_REPO_USERNAME optional username for the repository # HOP_PLUGINS_REPO_PASSWORD optional password / token for the repository # HOP_PLUGINS_REPO_AUTH_TYPE optional auth type: auto (default), basic, token # HOP_PLUGINS_ENV_FILE optional URL or path to a hop-env.yaml / hop-marketplace-repo.yaml install_marketplace_plugins() { local plugins="${HOP_PLUGINS_DOWNLOAD:-}" local env_file="${HOP_PLUGINS_ENV_FILE:-}" # 1. Alternative: Apply full install spec if specified if [ -n "${env_file}" ]; then log "Applying Hop marketplace environment file: ${env_file}" if ! "${DEPLOYMENT_PATH}"/hop marketplace apply -f "${env_file}"; then log "Error: failed to apply marketplace environment file '${env_file}'" exitWithCode 8 fi fi if [ -z "${plugins}" ]; then return 0 fi local repo_url="${HOP_PLUGINS_MAVEN_REPO:-}" local repo_id="${HOP_PLUGINS_REPO_ID:-corporate-repo}" # 2. Register repository if a custom URL is provided if [ -n "${repo_url}" ]; then log "Configuring marketplace repository '${repo_id}' (${repo_url})" local repo_args=("--id=${repo_id}" "--url=${repo_url}" "--primary") [ -n "${HOP_PLUGINS_REPO_USERNAME:-}" ] && repo_args+=("--username=${HOP_PLUGINS_REPO_USERNAME}") [ -n "${HOP_PLUGINS_REPO_PASSWORD:-}" ] && repo_args+=("--password=${HOP_PLUGINS_REPO_PASSWORD}") [ -n "${HOP_PLUGINS_REPO_AUTH_TYPE:-}" ] && repo_args+=("--auth-type=${HOP_PLUGINS_REPO_AUTH_TYPE}") # Register in hop-config.json (or update if already exists) if ! "${DEPLOYMENT_PATH}"/hop marketplace repo add "${repo_args[@]}"; then log "Warning: failed to add repository '${repo_id}' (may already exist, proceeding to install)" fi fi # 3. Install each requested plugin log "Installing marketplace plugins: ${plugins}" local spec for spec in ${plugins//,/ }; do spec="$(echo "${spec}" | tr -d '[:space:]')" [ -z "${spec}" ] && continue log "Installing marketplace plugin '${spec}'" local install_args=("${spec}") if [ -n "${repo_url}" ]; then install_args+=("--repo=${repo_id}") fi # shellcheck disable=SC2086 if ! "${DEPLOYMENT_PATH}"/hop marketplace install "${install_args[@]}"; then log "Error: failed to install marketplace plugin '${spec}'" exitWithCode 8 fi done } ``` ### Execution Location In [`load-and-execute.sh`](file:///home/matt/git/mattcasters/hop/docker/resources/load-and-execute.sh#L158-L160): ```bash # Download requested JDBC drivers (HOP_DRIVERS_DOWNLOAD) before Hop starts. install_jdbc_drivers # Download requested Marketplace plugins (HOP_PLUGINS_DOWNLOAD) before Hop starts. install_marketplace_plugins ``` --- ## 4. Complementary CLI Improvement: Add `--repo-url` to `hop marketplace install` In [`MarketplaceCommand.java`](file:///home/matt/git/mattcasters/hop/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/command/MarketplaceCommand.java#L156-L161): - Currently, `hop marketplace install` only accepts `--repo <repo-id>`, which requires the repository to be registered beforehand in `hop-config.json`. - In contrast, [`DriverInstallCommand.java`](file:///home/matt/git/mattcasters/hop/engine/src/main/java/org/apache/hop/driver/DriverInstallCommand.java#L69-L71) accepts `--repo <url>` (a direct repository URL). **Suggestion for Marketplace CLI**: Add an option `--repo-url <url>` to `hop marketplace install` (or permit passing a full URL to `--repo`). When `--repo-url` is passed: ```shell hop marketplace install org.hopper:hopper-edw:0.10.0 --repo-url https://repository.data-hopper.com/repository/hop-community-plugins/ ``` The installer uses an ad-hoc, in-memory `MarketplaceRepository` directly without needing a separate `repo add` step in `hop-config.json`. This makes marketplace plugin installation completely stateless. --- ## 5. Usage Examples ### Example 1: Installing Data Hopper EDW from community repository ```bash docker run -it --rm \ --env HOP_PLUGINS_DOWNLOAD="org.hopper:hopper-edw:0.10.0" \ --env HOP_PLUGINS_MAVEN_REPO="https://repository.data-hopper.com/repository/hop-community-plugins/"; \ --env HOP_PROJECT_FOLDER=/project \ --env HOP_PROJECT_NAME=edw-project \ --env HOP_RUN_CONFIG=local \ --env HOP_FILE_PATH='${PROJECT_HOME}/load_data_vault.hpl' \ --mount type=bind,source=/path/to/project,target=/project \ apache/hop:latest ``` ### Example 2: Installing multiple plugins from an authenticated corporate Artifactory ```bash docker run -it --rm \ --env HOP_PLUGINS_DOWNLOAD="com.mycorp:custom-pipeline-transform:1.4.0,org.hopper:hopper-edw:0.10.0" \ --env HOP_PLUGINS_MAVEN_REPO="https://artifactory.mycompany.corp/artifactory/hop-plugins-release/"; \ --env HOP_PLUGINS_REPO_USERNAME="ci-runner" \ --env HOP_PLUGINS_REPO_PASSWORD="${ARTIFACTORY_TOKEN}" \ --env HOP_PLUGINS_REPO_AUTH_TYPE="token" \ --env HOP_PROJECT_FOLDER=/project \ --env HOP_PROJECT_NAME=etl \ --env HOP_FILE_PATH='${PROJECT_HOME}/main.hwf' \ --env HOP_RUN_CONFIG=local \ --mount type=bind,source=/opt/etl,target=/project \ apache/hop:latest ``` --- ## 6. Documentation Section Add a new section `Downloading Marketplace Plugins` in [`docker-container.adoc`](file:///home/matt/git/mattcasters/hop/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc) directly following the `Downloading JDBC drivers` section, documenting: 1. The `HOP_PLUGINS_DOWNLOAD` and `HOP_PLUGINS_MAVEN_REPO` variables. 2. Coordinate formats (`groupId:artifactId:version`, `artifactId:version`, and short names). 3. Authentication parameters for private repositories. ### Issue Priority Priority: 2 ### Issue Component Component: Containers, Component: Marketplace -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
