mattcasters commented on issue #7547: URL: https://github.com/apache/hop/issues/7547#issuecomment-4995339800
### Summary of Proposed Plugin Repository Architecture To address the growing size of the Hop assembly, we should transition toward a decentralized, on-demand "Plugin Repository" architecture. Because Apache Hop is an ASF project and already has key architectural foundations in place, we can implement a zero-maintenance, automated marketplace without needing external hosting infrastructure. #### 1. Leveraging Existing ASF Infrastructure (Maven Central / Nexus) * **Zero Infrastructure Costs:** We do not need to host or manage our own artifact repository. Official Hop plugins can be published as standalone `.zip` or `.jar` artifacts to the official **ASF Nexus** (`repository.apache.org`) during the standard release process, which automatically syncs to **Maven Central**. * **Predictable Resolution:** Because Maven Central uses a strict directory structure, the Hop client can predictably deduce the exact download URL for any given plugin using its standard GAV (Group, Artifact, Version) coordinates: `[https://repo1.maven.org/maven2/](https://repo1.maven.org/maven2/)[group_path]/[artifactId]/[version]/[artifactId]-[version].zip` #### 2. Fully Automated Discovery (Fixing the Kettle/Pentaho Flaw) * **The Problem with Old Models:** The old Kettle/Pentaho marketplace relied on a static central XML/JSON file in a Git repository. This required manual upkeep, frequently broke, and led to stale or obsolete entries. * **The Dynamic Solution:** We can leverage our **existing strict naming conventions** (`org.apache.hop:hop-plugin-[type]-[name]`) to treat Maven Central as a decentralized database. When a user opens the marketplace, the Hop GUI can query the public Maven Central Solr Search API directly for the user's specific running version: `GET [https://search.maven.org/solrsearch/select?q=g](https://search.maven.org/solrsearch/select?q=g):"org.apache.hop"+AND+v:"2.18.1"&wt=json` * **Self-Cleaning:** This ensures perfect expiration management. If a plugin is removed or fails a specific release build, it simply won't appear in the API query results for that version. Dead plugins vanish gracefully without manual intervention. #### 3. Installation Workflow & UX * **Isolated Classloaders:** Since Hop already utilizes isolated classloaders per plugin, we are protected against "dependency hell" from dynamically added libraries. * **The Staging & Restart Pattern:** Instead of implementing risky runtime hot-reloading (which introduces memory leaks and file-locking issues, especially on Windows), we should lean into a simple GUI restart. 1. The user clicks "Install". 2. Hop downloads and extracts the plugin into a `plugins/.staging/` directory. 3. Hop prompts the user to restart. 4. On reboot, the core launcher clears the old plugin directory (if updating), moves the staged files to active `plugins/`, and loads the application cleanly in a few seconds. #### 4. Enterprise-Ready Considerations * **Air-Gapped Environment Support:** Many enterprise users run Hop in secure environments without internet access. We should abstract the search provider behind an interface and make the repository base URL a configurable property in `hop-config.json`. This allows corporate IT departments to point the marketplace to their internal **JFrog Artifactory** or **Sonatype Nexus** mirrors. * **API Caching:** Implement a local 24-hour cache for marketplace search queries to keep the UI snappy and avoid spamming public repository APIs on every startup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
