This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 29bfceef39 import marketplace repository definitions from a URL. fixes
#7830 (#7851)
29bfceef39 is described below
commit 29bfceef39979fb8e0450b80dfc0a5b82119634d
Author: Bart Maertens <[email protected]>
AuthorDate: Wed Aug 12 08:37:02 2026 +0200
import marketplace repository definitions from a URL. fixes #7830 (#7851)
---
.../ROOT/pages/hop-tools/hop-marketplace.adoc | 20 ++-
plugins/misc/marketplace/README.md | 21 +++
.../marketplace/command/MarketplaceCommand.java | 19 +++
.../config/MarketplaceRepositoryDefinition.java | 148 ++++++++++++++++-
.../gui/MarketplaceRepositoriesPanel.java | 134 +++++++++++++--
.../gui/messages/messages_en_US.properties | 11 +-
.../marketplace/config/PublicUrlImportTest.java | 182 +++++++++++++++++++++
7 files changed, 519 insertions(+), 16 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-marketplace.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-marketplace.adoc
index 93af602592..6640cb1641 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-marketplace.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-marketplace.adoc
@@ -210,12 +210,30 @@ Configure the global marketplace repository list (stored
under `marketplace` in
* *Set primary* — install tries this repository first
* *Up* / *Down* — order fallbacks after the primary
* *Reset defaults* — Apache Repository (primary) + Maven Central
-* *Save* — write hop-config (required to keep changes)
+* *Import…* / *Export…* — read or write a shareable
`hop-marketplace-repo.yaml` definition on disk
+* *Import from URL…* — download a definition published at an https address
(see <<marketplace-import-from-url>>)
+* *Save* — write hop-config (required to keep Add / Edit / Remove changes; the
two import buttons save on their own)
Add or edit a repository with id, name, URL, optional credentials, and
primary/enabled flags:
image::marketplace/marketplace-dialog-repository-editor.png[Add or edit
marketplace repository,60%,align="left"]
+[[marketplace-import-from-url]]
+=== Import from URL
+
+*Import from URL…* adds a repository from a definition published on the web,
so a marketplace operator can hand out one link instead of a file and a set of
instructions.
+
+Because the address is typically pasted from an email, a chat message or a web
page, the import is deliberately narrow:
+
+* *https only.* The definition names the hosts that plugin code is later
downloaded from, so it must not be modifiable in transit. For a host without
TLS, download the file and import it from disk.
+* *The download is anonymous.* No credentials are sent, so an address that
turns out to be hostile cannot collect the `HOP_MARKETPLACE_*` credentials that
apply to every repository. This means the definition itself has to be publicly
readable; the artifacts it points at do not.
+* *Credentials in the file are ignored.* A definition fetched over the network
describes *where* a repository is, never *who* connects to it. Add the username
and password on the repository entry afterwards, or set
`HOP_MARKETPLACE_<ID>_USERNAME` / `_PASSWORD`.
+* *Nothing is added until you confirm.* Hop shows the id, name, URL, download
template, catalog URL and plugin count first. Adding a repository means
trusting it as a source of code, so check that the addresses are ones you
expect.
+
+One field is not merely descriptive: a definition that sets `primary: true`
becomes the primary repository on import and demotes the current one, so every
install tries it first. If that repository is private and the credentials are
not set yet, installs that used to work will fail until they are. Check that
field before importing a definition you did not write, and use *Set primary* to
put the order back.
+
+Importing the same file *from disk* keeps any credentials it contains. That is
what makes an administrator-provisioned definition work for an internal
rollout: the file was placed there deliberately, where a URL is only ever a
pointer.
+
=== Projects lifecycle environments
When the *Projects* plugin is installed, each lifecycle environment can store
marketplace-specific *attributes* (namespaced under the `marketplace` group).
On the environment properties dialog, open the *Marketplace plugins* tab:
diff --git a/plugins/misc/marketplace/README.md
b/plugins/misc/marketplace/README.md
index 78849ab41c..a12c68b20a 100644
--- a/plugins/misc/marketplace/README.md
+++ b/plugins/misc/marketplace/README.md
@@ -249,6 +249,27 @@ Three steps:
# or explicit: ./hop marketplace install
org.apache.hop:hop-datavault:0.4.0-SNAPSHOT
```
+In the GUI, **Import from URL…** on the Repositories tab does the same from a
+published link. That path is narrower than a file or the CLI import, because
+the address is usually pasted from somewhere else:
+
+- **https only** — the definition names the hosts plugin code comes from, so it
+ must not be rewritable in transit
+- **anonymous download** — no credentials are sent, so a hostile address cannot
+ collect the global `HOP_MARKETPLACE_*` pair. The definition must be publicly
+ readable; the artifacts it points at need not be
+- **credentials in the file are ignored** — set them on the repository entry
+ afterwards, or via `HOP_MARKETPLACE_<ID>_USERNAME` / `_PASSWORD`
+- **confirmation first** — id, URL, download template, catalog URL and plugin
+ count are shown before anything is added
+
+A definition that sets `primary: true` becomes the primary repository on
import,
+which means every install tries it first. Check that field before importing a
+definition you did not write.
+
+Importing the same file **from disk** keeps its credentials, so an
+administrator can still provision one for an internal rollout.
+
**How listing works**
diff --git
a/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/command/MarketplaceCommand.java
b/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/command/MarketplaceCommand.java
index 2b12aab654..cdb0236047 100644
---
a/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/command/MarketplaceCommand.java
+++
b/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/command/MarketplaceCommand.java
@@ -797,6 +797,10 @@ public class MarketplaceCommand implements Runnable,
IHopCommand, IHasHopMetadat
MarketplaceRepository imported =
MarketplaceRepositoryDefinition.loadFromUri(fileOrUrl);
MarketplaceConfig config = MarketplaceConfig.load();
boolean existed = config.findRepository(imported.getId()) != null;
+ // Assess before applying: afterwards the demoted repository is
indistinguishable from any
+ // other fallback.
+ MarketplaceRepositoryDefinition.ImportRisk risk =
+ MarketplaceRepositoryDefinition.assess(config, imported);
MarketplaceRepositoryDefinition.applyToConfig(config, imported,
primary);
config.save();
System.out.println(
@@ -806,6 +810,21 @@ public class MarketplaceCommand implements Runnable,
IHopCommand, IHasHopMetadat
+ "' → "
+ imported.normalizedUrl()
+ (imported.isBrowse() ? " (browse enabled)" : ""));
+ // Importing one repository should not silently change which one
installs try first.
+ if (risk.takesOverPrimary() && !primary) {
+ System.out.println(
+ "WARNING: the definition set '"
+ + imported.getId()
+ + "' as the primary repository, replacing '"
+ + risk.currentPrimaryName()
+ + "'. Installs now try it first. Use 'marketplace repo
set-primary' to change"
+ + " that back.");
+ }
+ if (risk.noPublicFallback()) {
+ System.out.println(
+ "WARNING: neither the Apache nor the Maven Central repository is
configured and"
+ + " enabled, so there is no public repository to fall back
on.");
+ }
} catch (Exception e) {
System.err.println("ERROR: " + e.getMessage());
throw new CommandLine.ExecutionException(
diff --git
a/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/config/MarketplaceRepositoryDefinition.java
b/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/config/MarketplaceRepositoryDefinition.java
index 950f0a320b..a0ee552e7d 100644
---
a/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/config/MarketplaceRepositoryDefinition.java
+++
b/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/config/MarketplaceRepositoryDefinition.java
@@ -30,6 +30,7 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
+import java.util.Objects;
import org.apache.commons.lang3.StringUtils;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.json.HopJson;
@@ -49,6 +50,8 @@ public final class MarketplaceRepositoryDefinition {
public static final String KIND = "hop-marketplace-repository";
public static final String SCHEMA_VERSION = "1.0";
+ private static final String HTTPS_PREFIX = "https://";;
+
private MarketplaceRepositoryDefinition() {}
public static MarketplaceRepository load(Path file) throws HopException {
@@ -73,6 +76,60 @@ public final class MarketplaceRepositoryDefinition {
return load(Path.of(s));
}
+ /**
+ * Download and parse a definition published at a public URL, for import
from an address the user
+ * typed or pasted.
+ *
+ * <p>Differs from {@link #loadFromHttp(String)} on two points, both because
the address is not
+ * necessarily trusted:
+ *
+ * <ul>
+ * <li>The request is anonymous. Credentials are never offered to a pasted
address, so a hostile
+ * URL cannot collect the {@code HOP_MARKETPLACE_*} credentials that
apply to every
+ * repository.
+ * <li>Any {@code username} / {@code password} in the downloaded file is
dropped. A definition
+ * fetched over the network describes <em>where</em> a repository is,
never <em>who</em>
+ * connects to it; credentials are supplied locally, keyed by
repository id.
+ * </ul>
+ *
+ * <p>Plain HTTP is refused: the definition names the hosts that plugin code
is downloaded from,
+ * so it must not be modifiable in transit.
+ *
+ * <p>Only the download is anonymous. Once imported, the repository is an
ordinary configuration
+ * entry and is contacted with whatever credentials are configured for it.
+ */
+ public static MarketplaceRepository loadFromPublicUrl(String url) throws
HopException {
+ String trimmed = requireHttps(url);
+ return withoutCredentials(download(trimmed, null));
+ }
+
+ /** Trimmed URL, or a failure explaining why only https is accepted. */
+ static String requireHttps(String url) throws HopException {
+ if (StringUtils.isBlank(url)) {
+ throw new HopException("Repository definition URL is required");
+ }
+ String trimmed = url.trim();
+ if (!trimmed.regionMatches(true, 0, HTTPS_PREFIX, 0,
HTTPS_PREFIX.length())) {
+ throw new HopException(
+ "Repository definitions can only be imported over https, so that
they cannot be modified"
+ + " in transit. Download the file and import it from disk if the
host has no TLS: "
+ + trimmed);
+ }
+ return trimmed;
+ }
+
+ /**
+ * Drop credentials from a definition that came off the network. Everything
describing where the
+ * repository is survives; only who connects to it is removed.
+ */
+ static MarketplaceRepository withoutCredentials(MarketplaceRepository
imported) {
+ if (imported != null) {
+ imported.setUsername(null);
+ imported.setPassword(null);
+ }
+ return imported;
+ }
+
/**
* Download and parse a shared repository definition.
*
@@ -81,8 +138,16 @@ public final class MarketplaceRepositoryDefinition {
* a private repository could not be imported by URL at all.
*/
public static MarketplaceRepository loadFromHttp(String url) throws
HopException {
+ return download(url, new MarketplaceRepository("import", url));
+ }
+
+ /**
+ * Fetch and parse a definition. A null {@code credentials} repository makes
the request
+ * anonymous.
+ */
+ private static MarketplaceRepository download(String url,
MarketplaceRepository credentials)
+ throws HopException {
try {
- MarketplaceRepository credentials = new MarketplaceRepository("import",
url);
HttpResponse<InputStream> response =
MarketplaceHttp.send(
MarketplaceHttp.newClient(),
@@ -97,7 +162,7 @@ public final class MarketplaceRepositoryDefinition {
+ " (HTTP "
+ response.statusCode()
+ ")"
- + MarketplaceHttp.authHint(response.statusCode(),
credentials));
+ + downloadHint(response.statusCode(), credentials));
}
try (InputStream in = response.body()) {
return parse(in, url);
@@ -112,6 +177,23 @@ public final class MarketplaceRepositoryDefinition {
}
}
+ /**
+ * Explain a failed download. The anonymous path cannot act on the
environment credentials {@link
+ * MarketplaceHttp#authHint} suggests, so it says what actually applies: the
definition itself has
+ * to be readable without credentials.
+ */
+ private static String downloadHint(int status, MarketplaceRepository
credentials) {
+ if (credentials != null) {
+ return MarketplaceHttp.authHint(status, credentials);
+ }
+ if (MarketplaceHttp.isAuthFailure(status)) {
+ return ". The definition was requested without credentials, which is
deliberate for an"
+ + " address that was typed or pasted. Publish the definition so it
can be read without"
+ + " authentication, or download the file and import it from disk.";
+ }
+ return "";
+ }
+
static MarketplaceRepository parse(InputStream in, String source) throws
HopException {
try {
String name = source == null ? "" : source.toLowerCase(Locale.ROOT);
@@ -336,6 +418,68 @@ public final class MarketplaceRepositoryDefinition {
return root;
}
+ /**
+ * What importing {@code imported} would change beyond adding one repository.
+ *
+ * @param takesOverPrimary the definition claims {@code primary: true} while
a different
+ * repository currently holds it, so every install would try the new one
first
+ * @param currentPrimaryName display name of the repository that would be
demoted, or null
+ * @param noPublicFallback neither the Apache nor the Maven Central
repository is configured and
+ * enabled, so nothing public is left to fall back on
+ */
+ public record ImportRisk(
+ boolean takesOverPrimary, String currentPrimaryName, boolean
noPublicFallback) {
+
+ public boolean isSafe() {
+ return !takesOverPrimary && !noPublicFallback;
+ }
+ }
+
+ /**
+ * Inspect an import before applying it. Importing adds or updates a single
repository, with one
+ * exception: a definition may declare itself primary and demote the
repository that holds that
+ * role today. That is worth surfacing when the definition came from
somewhere else.
+ */
+ public static ImportRisk assess(MarketplaceConfig config,
MarketplaceRepository imported) {
+ if (config == null || imported == null) {
+ return new ImportRisk(false, null, false);
+ }
+ MarketplaceRepository currentPrimary = null;
+ boolean publicFallback = false;
+ for (MarketplaceRepository repo : nullSafe(config.getRepositories())) {
+ if (repo == null || !repo.isEnabled()) {
+ continue;
+ }
+ if (repo.isPrimary() && currentPrimary == null) {
+ currentPrimary = repo;
+ }
+ publicFallback |= isPublicDefault(repo);
+ }
+ // Re-importing the definition of the repository that is already primary
changes nothing.
+ boolean takesOver =
+ imported.isPrimary()
+ && currentPrimary != null
+ && !Objects.equals(currentPrimary.getId(), imported.getId());
+ return new ImportRisk(
+ takesOver, takesOver ? currentPrimary.displayName() : null,
!publicFallback);
+ }
+
+ /**
+ * The shipped public repositories, matched on id or URL: an id can be
renamed, and a URL can be
+ * pointed at a mirror, so either identifies one.
+ */
+ private static boolean isPublicDefault(MarketplaceRepository repo) {
+ String url = repo.normalizedUrl();
+ return MarketplaceConfig.DEFAULT_ASF_ID.equals(repo.getId())
+ || MarketplaceConfig.DEFAULT_CENTRAL_ID.equals(repo.getId())
+ || MarketplaceConfig.DEFAULT_ASF_URL.equals(url)
+ || MarketplaceConfig.DEFAULT_CENTRAL_URL.equals(url);
+ }
+
+ private static <T> List<T> nullSafe(List<T> list) {
+ return list == null ? List.of() : list;
+ }
+
public static void applyToConfig(
MarketplaceConfig config, MarketplaceRepository imported, boolean
makePrimary)
throws HopException {
diff --git
a/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/gui/MarketplaceRepositoriesPanel.java
b/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/gui/MarketplaceRepositoriesPanel.java
index 4fa7f4ceb2..388cc60bca 100644
---
a/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/gui/MarketplaceRepositoriesPanel.java
+++
b/plugins/misc/marketplace/src/main/java/org/apache/hop/marketplace/gui/MarketplaceRepositoriesPanel.java
@@ -22,6 +22,7 @@ import java.util.ArrayList;
import java.util.List;
import org.apache.commons.lang3.StringUtils;
import org.apache.hop.core.Const;
+import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.marketplace.catalog.OptionalPluginInfo;
@@ -30,6 +31,7 @@ import
org.apache.hop.marketplace.config.MarketplaceRepository;
import org.apache.hop.marketplace.config.MarketplaceRepositoryDefinition;
import org.apache.hop.ui.core.PropsUi;
import org.apache.hop.ui.core.dialog.BaseDialog;
+import org.apache.hop.ui.core.dialog.EnterStringDialog;
import org.apache.hop.ui.core.dialog.ErrorDialog;
import org.apache.hop.ui.core.dialog.MessageBox;
import org.apache.hop.ui.core.gui.WindowProperty;
@@ -102,6 +104,8 @@ public class MarketplaceRepositoriesPanel {
wExport.addListener(SWT.Selection, e -> exportDefinition());
Button wImport = createRightButton(parent,
"ManageRepositoriesDialog.Button.Import");
wImport.addListener(SWT.Selection, e -> importDefinition());
+ Button wImportUrl = createRightButton(parent,
"ManageRepositoriesDialog.Button.ImportUrl");
+ wImportUrl.addListener(SWT.Selection, e -> importDefinitionFromUrl());
Button wRemove = createRightButton(parent,
"ManageRepositoriesDialog.Button.Remove");
wRemove.addListener(SWT.Selection, e -> removeSelected());
Button wEdit = createRightButton(parent,
"ManageRepositoriesDialog.Button.Edit");
@@ -110,7 +114,7 @@ public class MarketplaceRepositoriesPanel {
wAdd.addListener(SWT.Selection, e -> addRepository());
Button[] rightButtons = {
- wAdd, wEdit, wRemove, wImport, wExport, wPrimary, wUp, wDown, wReset,
wSave
+ wAdd, wEdit, wRemove, wImport, wImportUrl, wExport, wPrimary, wUp,
wDown, wReset, wSave
};
layoutRightButtons(rightButtons, wlHelp);
@@ -830,17 +834,7 @@ public class MarketplaceRepositoriesPanel {
if (StringUtils.isBlank(path)) {
return;
}
- MarketplaceRepository imported =
MarketplaceRepositoryDefinition.load(Path.of(path.trim()));
- MarketplaceRepositoryDefinition.applyToConfig(config, imported, false);
- markDirty();
- refreshTable();
- selectRepoId(imported.getId());
- MessageBox box = new MessageBox(shell, SWT.OK | SWT.ICON_INFORMATION);
- box.setText(BaseMessages.getString(PKG,
"ManageRepositoriesDialog.Import.Done.Header"));
- box.setMessage(
- BaseMessages.getString(
- PKG, "ManageRepositoriesDialog.Import.Done.Message",
imported.getId()));
- box.open();
+
applyImported(MarketplaceRepositoryDefinition.load(Path.of(path.trim())));
} catch (Exception e) {
new ErrorDialog(
shell,
@@ -850,6 +844,122 @@ public class MarketplaceRepositoriesPanel {
}
}
+ /**
+ * Import a definition published at a URL. The download is anonymous and any
credentials in the
+ * file are dropped, so what arrives describes only where a repository is.
Because the definition
+ * decides which hosts plugin code is later downloaded from, it is shown for
confirmation before
+ * anything is added.
+ */
+ private void importDefinitionFromUrl() {
+ try {
+ String url =
+ new EnterStringDialog(
+ shell,
+ "",
+ BaseMessages.getString(PKG,
"ManageRepositoriesDialog.ImportUrl.Header"),
+ BaseMessages.getString(PKG,
"ManageRepositoriesDialog.ImportUrl.Message"))
+ .open();
+ if (StringUtils.isBlank(url)) {
+ return;
+ }
+ MarketplaceRepository imported =
MarketplaceRepositoryDefinition.loadFromPublicUrl(url);
+ switch (confirmImport(imported, url.trim())) {
+ case CANCEL:
+ return;
+ case AS_FALLBACK:
+ // Take the repository, leave the install order alone.
+ imported.setPrimary(false);
+ break;
+ case IMPORT:
+ break;
+ }
+ applyImported(imported);
+ } catch (Exception e) {
+ new ErrorDialog(
+ shell,
+ BaseMessages.getString(PKG, "ManageRepositoriesDialog.Error.Header"),
+ BaseMessages.getString(PKG,
"ManageRepositoriesDialog.ImportUrl.Error"),
+ e);
+ }
+ }
+
+ private enum ImportChoice {
+ IMPORT,
+ AS_FALLBACK,
+ CANCEL
+ }
+
+ /**
+ * Show what a downloaded definition would add, and let the user back out.
+ *
+ * <p>A definition that claims {@code primary: true} does more than add a
repository: it decides
+ * which repository every install tries first. That gets its own warning and
a third option, so
+ * the repository can be taken without handing over the install order.
+ */
+ private ImportChoice confirmImport(MarketplaceRepository imported, String
url) {
+ String details =
+ BaseMessages.getString(
+ PKG,
+ "ManageRepositoriesDialog.ImportUrl.Confirm.Message",
+ url,
+ Const.NVL(imported.getId(), "-"),
+ Const.NVL(imported.getName(), "-"),
+ Const.NVL(imported.getUrl(), "-"),
+ Const.NVL(imported.getUrlTemplate(), "-"),
+ Const.NVL(imported.getCatalogUrl(), "-"),
+ Integer.toString(imported.getPlugins() == null ? 0 :
imported.getPlugins().size()));
+
+ MarketplaceRepositoryDefinition.ImportRisk risk =
+ MarketplaceRepositoryDefinition.assess(config, imported);
+ if (risk.noPublicFallback()) {
+ details +=
+ BaseMessages.getString(PKG,
"ManageRepositoriesDialog.ImportUrl.NoFallback.Message");
+ }
+
+ if (!risk.takesOverPrimary()) {
+ MessageBox box = new MessageBox(shell, SWT.OK | SWT.CANCEL |
SWT.ICON_QUESTION);
+ box.setText(BaseMessages.getString(PKG,
"ManageRepositoriesDialog.ImportUrl.Confirm.Header"));
+ box.setMessage(details);
+ return box.open() == SWT.OK ? ImportChoice.IMPORT : ImportChoice.CANCEL;
+ }
+
+ MessageBox box = new MessageBox(shell, SWT.YES | SWT.NO | SWT.CANCEL |
SWT.ICON_WARNING);
+ box.setText(BaseMessages.getString(PKG,
"ManageRepositoriesDialog.ImportUrl.Primary.Header"));
+ box.setMessage(
+ details
+ + BaseMessages.getString(
+ PKG,
+ "ManageRepositoriesDialog.ImportUrl.Primary.Message",
+ imported.displayName(),
+ Const.NVL(risk.currentPrimaryName(), "-")));
+ int answer = box.open();
+ if (answer == SWT.YES) {
+ return ImportChoice.IMPORT;
+ }
+ return answer == SWT.NO ? ImportChoice.AS_FALLBACK : ImportChoice.CANCEL;
+ }
+
+ /**
+ * Add the imported repository and write it out. Importing is a deliberate
one-off action rather
+ * than an edit in progress, and the CLI equivalent saves too, so leaving it
pending only invites
+ * the question of where Save is. A failed write keeps the panel dirty and
reports itself.
+ */
+ private void applyImported(MarketplaceRepository imported) throws
HopException {
+ MarketplaceRepositoryDefinition.applyToConfig(config, imported, false);
+ markDirty();
+ refreshTable();
+ selectRepoId(imported.getId());
+ if (!saveChanges(false)) {
+ return;
+ }
+ MessageBox box = new MessageBox(shell, SWT.OK | SWT.ICON_INFORMATION);
+ box.setText(BaseMessages.getString(PKG,
"ManageRepositoriesDialog.Import.Done.Header"));
+ box.setMessage(
+ BaseMessages.getString(
+ PKG, "ManageRepositoriesDialog.Import.Done.Message",
imported.getId()));
+ box.open();
+ }
+
private void exportDefinition() {
MarketplaceRepository repo = selected();
if (repo == null) {
diff --git
a/plugins/misc/marketplace/src/main/resources/org/apache/hop/marketplace/gui/messages/messages_en_US.properties
b/plugins/misc/marketplace/src/main/resources/org/apache/hop/marketplace/gui/messages/messages_en_US.properties
index c5c2dca2ff..45a47e2ec0 100644
---
a/plugins/misc/marketplace/src/main/resources/org/apache/hop/marketplace/gui/messages/messages_en_US.properties
+++
b/plugins/misc/marketplace/src/main/resources/org/apache/hop/marketplace/gui/messages/messages_en_US.properties
@@ -147,6 +147,7 @@ ManageRepositoriesDialog.Column.Url=URL
ManageRepositoriesDialog.Column.Auth=Auth
ManageRepositoriesDialog.Button.Add=Add
ManageRepositoriesDialog.Button.Import=Import…
+ManageRepositoriesDialog.Button.ImportUrl=Import from URL…
ManageRepositoriesDialog.Button.Export=Export…
ManageRepositoriesDialog.Button.Edit=Edit
ManageRepositoriesDialog.Button.Remove=Remove
@@ -188,8 +189,16 @@ ManageRepositoriesDialog.Filter.RepoDef=Repository
definitions (*.yaml, *.yml, *
ManageRepositoriesDialog.Filter.Yaml=YAML (*.yaml, *.yml)
ManageRepositoriesDialog.Filter.Json=JSON (*.json)
ManageRepositoriesDialog.Import.Done.Header=Repository imported
-ManageRepositoriesDialog.Import.Done.Message=Imported/updated repository
''{0}''. Click Save to persist hop-config.
+ManageRepositoriesDialog.Import.Done.Message=Imported repository ''{0}'' and
saved hop-config.json.
ManageRepositoriesDialog.Import.Error=Could not import repository definition
+ManageRepositoriesDialog.ImportUrl.Header=Import repository definition from URL
+ManageRepositoriesDialog.ImportUrl.Message=https URL of the repository
definition (YAML or JSON). The file is downloaded anonymously and any username
or password in it is ignored.
+ManageRepositoriesDialog.ImportUrl.Error=Could not import repository
definition from URL
+ManageRepositoriesDialog.ImportUrl.Confirm.Header=Add this repository?
+ManageRepositoriesDialog.ImportUrl.Confirm.Message=Downloaded from {0}\n\nID:
{1}\nName: {2}\nURL: {3}\nDownload template: {4}\nCatalog URL: {5}\nPlugins
listed: {6}\n\nPlugins will be downloaded from these addresses, so add this
repository only if you trust the source. Any username or password in the file
was ignored: set credentials on the repository after importing.
+ManageRepositoriesDialog.ImportUrl.NoFallback.Message=\n\nThe Apache and Maven
Central repositories are not configured and enabled, so there is no public
repository left to fall back on if this one cannot be reached.
+ManageRepositoriesDialog.ImportUrl.Primary.Header=This definition wants to
become the primary repository
+ManageRepositoriesDialog.ImportUrl.Primary.Message=\n\nWARNING: this
definition sets itself as the primary repository. {0} would replace {1}, and
every plugin install would try {0} first. If it is private and its credentials
are not set yet, or it cannot be reached, installs that work today will start
failing.\n\nYes: import and make {0} primary.\nNo: import it as a fallback and
keep {1} primary.\nCancel: do not import.
ManageRepositoriesDialog.Export.Select=Select a repository row to export.
ManageRepositoriesDialog.Export.Done.Header=Repository exported
ManageRepositoriesDialog.Export.Done.Message=Wrote:\n{0}
diff --git
a/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/config/PublicUrlImportTest.java
b/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/config/PublicUrlImportTest.java
new file mode 100644
index 0000000000..5ee0a2b835
--- /dev/null
+++
b/plugins/misc/marketplace/src/test/java/org/apache/hop/marketplace/config/PublicUrlImportTest.java
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.marketplace.config;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.hop.core.exception.HopException;
+import org.junit.jupiter.api.Test;
+
+/**
+ * A definition fetched from a URL says where a repository is, never who
connects to it, and must
+ * arrive over a channel nobody can rewrite. Importing the same file from disk
is deliberate and
+ * keeps whatever it contains.
+ */
+class PublicUrlImportTest {
+
+ private static Map<String, Object> definitionWithCredentials() {
+ return Map.of(
+ "id", "acme",
+ "url", "https://nexus.example.org/repository/hop/";,
+ "urlTemplate",
"https://downloads.example.org/${artifactId}-${version}.zip";,
+ "username", "planted-user",
+ "password", "planted-password");
+ }
+
+ @Test
+ void httpsIsRequired() {
+ HopException e =
+ assertThrows(
+ HopException.class,
+ () ->
MarketplaceRepositoryDefinition.requireHttps("http://example.org/repo.yaml";));
+ assertTrue(e.getMessage().contains("https"));
+
+ // Not a transport we can vouch for either.
+ assertThrows(
+ HopException.class,
+ () ->
MarketplaceRepositoryDefinition.requireHttps("ftp://example.org/repo.yaml";));
+ assertThrows(
+ HopException.class, () ->
MarketplaceRepositoryDefinition.requireHttps("/tmp/repo.yaml"));
+ assertThrows(HopException.class, () ->
MarketplaceRepositoryDefinition.requireHttps(" "));
+ assertThrows(HopException.class, () ->
MarketplaceRepositoryDefinition.requireHttps(null));
+ }
+
+ @Test
+ void httpsUrlsAreAcceptedAndTrimmed() throws Exception {
+ assertEquals(
+ "https://example.org/repo.yaml";,
+ MarketplaceRepositoryDefinition.requireHttps("
https://example.org/repo.yaml "));
+ // Scheme comparison is case-insensitive; a valid URL must not be rejected
on casing.
+ assertEquals(
+ "HTTPS://example.org/repo.yaml",
+
MarketplaceRepositoryDefinition.requireHttps("HTTPS://example.org/repo.yaml"));
+ }
+
+ @Test
+ void credentialsAreStrippedFromDownloadedDefinitions() throws Exception {
+ MarketplaceRepository downloaded =
+ MarketplaceRepositoryDefinition.withoutCredentials(
+
MarketplaceRepositoryDefinition.fromMap(definitionWithCredentials()));
+
+ assertNull(downloaded.getUsername());
+ assertNull(downloaded.getPassword());
+ // Everything that describes the repository itself is untouched.
+ assertEquals("acme", downloaded.getId());
+ assertEquals("https://nexus.example.org/repository/hop/";,
downloaded.getUrl());
+ assertEquals(
+ "https://downloads.example.org/${artifactId}-${version}.zip";,
downloaded.getUrlTemplate());
+ }
+
+ @Test
+ void importingTheSameFileFromDiskKeepsItsCredentials() throws Exception {
+ // The asymmetry is the point: a file on disk was put there deliberately,
so an admin can
+ // provision one with credentials. A URL is only ever a pointer.
+ MarketplaceRepository fromDisk =
+ MarketplaceRepositoryDefinition.fromMap(definitionWithCredentials());
+
+ assertEquals("planted-user", fromDisk.getUsername());
+ assertEquals("planted-password", fromDisk.getPassword());
+ }
+
+ private static MarketplaceRepository claiming(boolean primary) {
+ MarketplaceRepository repo = new MarketplaceRepository("putki",
"https://forge.example.org/";);
+ repo.setPrimary(primary);
+ return repo;
+ }
+
+ @Test
+ void aDefinitionThatClaimsPrimaryIsFlaggedAsATakeover() {
+ MarketplaceConfig config = new MarketplaceConfig(); // asf (primary) +
central
+ MarketplaceRepositoryDefinition.ImportRisk risk =
+ MarketplaceRepositoryDefinition.assess(config, claiming(true));
+
+ assertTrue(risk.takesOverPrimary());
+ assertEquals(MarketplaceConfig.DEFAULT_ASF_NAME,
risk.currentPrimaryName());
+ assertFalse(risk.isSafe());
+ }
+
+ @Test
+ void anOrdinaryDefinitionIsSafe() {
+ MarketplaceRepositoryDefinition.ImportRisk risk =
+ MarketplaceRepositoryDefinition.assess(new MarketplaceConfig(),
claiming(false));
+
+ assertFalse(risk.takesOverPrimary());
+ assertNull(risk.currentPrimaryName());
+ assertTrue(risk.isSafe());
+ }
+
+ @Test
+ void reimportingTheRepositoryThatIsAlreadyPrimaryIsNotATakeover() throws
Exception {
+ MarketplaceConfig config = new MarketplaceConfig();
+ config.addRepository(claiming(true));
+ config.setPrimary("putki");
+
+ // Same id, still primary: nothing is being taken over, so do not cry wolf.
+ assertFalse(MarketplaceRepositoryDefinition.assess(config,
claiming(true)).takesOverPrimary());
+ }
+
+ @Test
+ void missingPublicRepositoriesLeaveNoFallback() throws Exception {
+ MarketplaceConfig bare = new MarketplaceConfig();
+ bare.setRepositories(new java.util.ArrayList<>());
+ bare.addRepository(claiming(true));
+
+ assertTrue(MarketplaceRepositoryDefinition.assess(bare,
claiming(false)).noPublicFallback());
+ // The shipped defaults are the fallback, whatever the imported definition
looks like.
+ assertFalse(
+ MarketplaceRepositoryDefinition.assess(new MarketplaceConfig(),
claiming(false))
+ .noPublicFallback());
+ }
+
+ @Test
+ void aDisabledApacheRepositoryIsNotAFallback() throws Exception {
+ MarketplaceConfig config = new MarketplaceConfig();
+ config.setEnabled(MarketplaceConfig.DEFAULT_ASF_ID, false);
+ config.setEnabled(MarketplaceConfig.DEFAULT_CENTRAL_ID, false);
+
+ assertTrue(MarketplaceRepositoryDefinition.assess(config,
claiming(false)).noPublicFallback());
+ }
+
+ @Test
+ void strippingLeavesTheDiscoveryPayloadIntact() throws Exception {
+ Map<String, Object> withPlugins =
+ Map.of(
+ "id",
+ "acme",
+ "url",
+ "https://nexus.example.org/repository/hop/";,
+ "username",
+ "planted-user",
+ "plugins",
+ List.of(Map.of("artifactId", "hop-tech-parquet", "version",
"2.19.0")));
+
+ MarketplaceRepository downloaded =
+ MarketplaceRepositoryDefinition.withoutCredentials(
+ MarketplaceRepositoryDefinition.fromMap(withPlugins));
+
+ assertNull(downloaded.getUsername());
+ assertEquals(1, downloaded.getPlugins().size());
+ assertEquals("hop-tech-parquet",
downloaded.getPlugins().get(0).getArtifactId());
+ }
+}
