This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 3e300c6337 Document hop export and hop driver, align the Hop Tools
navigation and index, merge the hop doc pages, fixes #9269 (#9327)
3e300c6337 is described below
commit 3e300c6337c67f396da23335ce67bab77634f66f
Author: Adalennis <[email protected]>
AuthorDate: Fri Nov 6 14:43:55 2026 +0100
Document hop export and hop driver, align the Hop Tools navigation and
index, merge the hop doc pages, fixes #9269 (#9327)
---
docs/hop-user-manual/modules/ROOT/nav.adoc | 21 ++-
.../ROOT/pages/getting-started/hop-tools.adoc | 51 ++++--
.../modules/ROOT/pages/hop-doc/index.adoc | 103 ------------
.../modules/ROOT/pages/hop-tools/hop-doc.adoc | 175 ++++++++-------------
.../modules/ROOT/pages/hop-tools/hop-driver.adoc | 127 +++++++++++++++
.../modules/ROOT/pages/hop-tools/hop-export.adoc | 117 ++++++++++++++
.../modules/ROOT/pages/hop-tools/hop.adoc | 85 +++++++---
.../modules/ROOT/pages/hop-tools/index.adoc | 31 ++--
8 files changed, 442 insertions(+), 268 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index e0b24ec70b..73a20e3eaf 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -628,17 +628,22 @@ under the License.
** xref:hop-server/web-service.adoc[Web Service]
** xref:hop-server/async-web-service.adoc[Async Web Service]
* xref:hop-tools/index.adoc[Hop Tools]
+** xref:hop-tools/hop.adoc[hop]
+** xref:hop-tools/hop-arrow-command.adoc[hop arrow]
+** xref:hop-tools/hop-conf/hop-conf.adoc[hop conf]
+** xref:hop-tools/hop-doc.adoc[hop doc]
+** xref:hop-tools/hop-driver.adoc[hop driver]
+** xref:hop-tools/hop-encrypt.adoc[hop encrypt]
+** xref:hop-tools/hop-export.adoc[hop export]
+** xref:hop-tools/hop-gui.adoc[hop gui]
+** xref:hop-tools/hop-import.adoc[hop import]
** xref:hop-tools/hop-lint.adoc[hop lint]
** xref:hop-tools/hop-marketplace.adoc[hop marketplace]
+** xref:hop-tools/hop-python/hop-python.adoc[hop python]
+** xref:hop-run/index.adoc[hop run]
+** xref:hop-tools/hop-search.adoc[hop search]
+** xref:hop-server/index.adoc[hop server]
** xref:hop-tools/hop-setup.adoc[hop setup]
-** xref:hop-tools/hop-conf/hop-conf.adoc[hop-conf]
-** xref:hop-tools/hop-encrypt.adoc[hop-encrypt]
-** xref:hop-tools/hop-gui.adoc[hop-gui]
-** xref:hop-doc/index.adoc[hop doc (Generate documentation)]
-** xref:hop-run/index.adoc[hop-run]
-** xref:hop-tools/hop-search.adoc[hop-search]
-** xref:hop-tools/hop-import.adoc[hop-import]
-** xref:hop-server/index.adoc[hop-server]
* xref:best-practices/index.adoc[Best Practices]
* xref:linting/index.adoc[Linting]
** xref:linting/lint-rules.adoc[Lint rules]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
index d47856ea10..b6bcc1d9d5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
@@ -57,15 +57,23 @@ Linux, macOS::
[source,shell]
----
Commands:
- help Display help information about the specified command.
- conf Configure Hop
- encrypt Encrypt secrets
- gui The Hop GUI
- import Import metadata
- run Run a pipeline or workflow
- search Search in Hop metadata
- server Run a Hop server
- setup Configure Hop environment variables
+ help Display help information about the specified command.
+ conf Configure Hop
+ driver Manage JDBC drivers (list, install)
+ encrypt Encrypt secrets
+ export Export pipeline, workflow, or model diagrams to various formats
+ (SVG, Mermaid, etc.)
+ gui The Hop GUI
+ import Import metadata
+ marketplace Install and manage optional Hop plugins from a Maven repository
+ naming-check Validate names against project Naming Schemes
+ run Run a pipeline or workflow
+ search Search in Hop metadata
+ server Run a Hop server
+ setup Configure Hop launcher environment variables
+ (HOP_CONFIG_FOLDER, HOP_AUDIT_FOLDER, HOP_JAVA_HOME,
+ HOP_OPTIONS, HOP_SHARED_JDBC_FOLDERS). Distinct from hop
+ conf, which edits hop-config.json.
----
Two options are worth knowing before anything else, because nearly every
command takes them:
@@ -124,25 +132,34 @@ NOTE: the older `hop-run.sh`, `hop-conf.sh` and friends
still ship and still wor
|Convert a Kettle/PDI repository into a Hop project.
|xref:hop-vs-kettle/import-kettle-projects.adoc[Import Kettle projects]
-|`hop doc`
-|Generate documentation for your own Hop project, in Markdown or HTML.
-|xref:hop-tools/hop-doc.adoc[Hop Doc]
+|`hop driver`
+|List the JDBC drivers Hop can download, and install them.
+|xref:hop-tools/hop-driver.adoc[Hop Driver]
+
+|`hop export`
+|Export pipeline and workflow diagrams to SVG or Mermaid, for one file or a
whole folder.
+|xref:hop-tools/hop-export.adoc[Hop Export]
+
+|`hop marketplace`
+|Install optional plugins into a Hop client from a Maven repository.
+|xref:hop-tools/hop-marketplace.adoc[Hop Marketplace]
|===
-Some tools are optional plugins rather than part of the standard client:
+Some tools are optional plugins rather than part of the standard client.
+Install them with `hop marketplace`:
[options="header",cols="20%,50%,30%"]
|===
|Tool|What you use it for|Read more
-|Marketplace
-|Install optional plugins into a running Hop client from a Maven repository.
-|xref:hop-tools/hop-marketplace.adoc[Hop Marketplace]
-
|`hop lint`
|Static analysis over a project: check pipelines, workflows and metadata
without running them. Useful as a build step.
|xref:hop-tools/hop-lint.adoc[Hop Lint]
+|`hop doc`
+|Generate documentation for your own Hop project, in Markdown or HTML.
+|xref:hop-tools/hop-doc.adoc[Hop Doc]
+
|PyHop
|Drive Hop from Python.
|xref:hop-tools/hop-python/hop-python.adoc[Hop Python]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-doc/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-doc/index.adoc
deleted file mode 100644
index cceb4688e5..0000000000
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-doc/index.adoc
+++ /dev/null
@@ -1,103 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one
-or more contributor license agreements. See the NOTICE file
-distributed with this work for additional information
-regarding copyright ownership. The ASF licenses this file
-to you under the Apache License, Version 2.0 (the
-"License"); you may not use this file except in compliance
-with the License. You may obtain a copy of the License at
- http://www.apache.org/licenses/LICENSE-2.0
-Unless required by applicable law or agreed to in writing,
-software distributed under the License is distributed on an
-"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-KIND, either express or implied. See the License for the
-specific language governing permissions and limitations
-under the License.
-////
-[[HopRun]]
-:imagesdir: ../assets/images
-:description: Hop doc is a command that you can use to document your projects
-
-= hop doc
-
-The `hop doc` can be used to document a project in
https://www.markdownguide.org/basic-syntax/[MarkDown] format.
-It will create a separate file per workflow and pipeline, add an index, and so
on.
-You can then serve up this documentation by MarkDown tools like
https://www.mkdocs.org/[mkdocs]
-
-== Options
-
-You can run the `hop help doc` command to see the options:
-
-On Windows, run `hop.bat help doc`.
-The command on Mac and Linux is `sh hop help doc`.
-
-You'll see the option listed in a similar output to the one below:
-
-
-
-.Usage
-[%collapsible]
-=====
-
-.Output of help
-[source,bash]
-----
-Usage: hop doc [-hV] [-im] [-in] [-ip] [-e=<environmentOption>]
- [-j=<projectOption>] [-n=<projectName>] [-s=<sourceFolder>]
- [-t=<targetParentFolder>]
-Generate documentation
- -e, --environment=<environmentOption>
- The name of the lifecycle environment to use
- -h, --help Show this help message and exit.
- -im, --include-metadata
- Include an overview of the available metadata
- elements
- -in, --include-notes List the text of any notes in alphabetical order
- -ip, --include-parameters
- Include a list of parameters for each pipeline and
- workflow
- -j, --project=<projectOption>
- The name of the project to use
- -n, --project-name=<projectName>
- The name of the project
- -s, --source-folder=<sourceFolder>
- The source folder to document
- -t, --target-folder=<targetParentFolder>
- Specify the target parent folder where the
- documentation should end up
- -V, --version Print version information and exit.
-----
-=====
-
-As you can see, the `projects` plugin contributes options to select a project
or environment, making it not needed to remember which folder the project was
stored in.
-
-== Example
-
-The example below documents the `demo` project and asks to write the MarkDown
files in the `/tmp/hop/docs` folder:
-
- +
-
-[tabs]
-====
-Windows::
-+
---
-Open a command (CMD) window, change to the directory where you unzipped Apache
Hop and run:
-
-[code,shell]
-----
-hop.bat doc -j demo --target-folder /tmp/hop/docs --include-notes
--include-parameters --include-metadata
-----
-
---
-
-Linux, macOS::
-+
-
-Open a terminal, change to the directory where you unzipped Apache Hop and run:
-
-[code,shell]
-----
-./hop doc -j demo -t /tmp/hop/docs --include-notes --include-parameters
--include-metadata
-----
-====
\ No newline at end of file
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-doc.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-doc.adoc
index 8f79408c55..dd4c5e4e42 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-doc.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-doc.adoc
@@ -6,55 +6,36 @@ regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
-
http://www.apache.org/licenses/LICENSE-2.0
-
Unless required by applicable law or agreed to in writing,
-software distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
-OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing
-permissions and limitations under the License.
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
////
-:description: Hop Doc generates project documentation
-= Hop Doc
-
-Hop Doc generates documentation for Hop projects in Markdown format with
optional HTML output.
+[[HopDoc]]
+:description: hop doc writes documentation for a Hop project in Markdown, and
optionally HTML: a page per pipeline and workflow with its diagram, an index,
and an overview of the metadata.
+:page-marketplace: hop-misc-documentation
+:page-aliases: hop-doc/index.adoc
-== Usage
+= hop doc
-Generate documentation for a Hop project:
+`hop doc` writes documentation for a Hop project in
https://www.markdownguide.org/basic-syntax/[Markdown] format.
+It writes a page per pipeline and workflow with its diagram, and an index that
links them.
+You can publish the Markdown with a tool such as
https://www.mkdocs.org/[MkDocs], or let `hop doc` write HTML as well.
-[source,command]
-----
-./hop.bat doc -s /path/to/project -t /path/to/docs [--generate-html]
[--remove-markdown]
-----
+`hop doc` is part of the *Documentation* plugin, which is not in the standard
Hop client.
+Install it from the xref:hop-tools/hop-marketplace.adoc[marketplace], where it
is listed under Miscellaneous, or from the command line:
-Or using Hop project variables:
-[source,command]
+[source,bash]
----
-./hop.bat run -j my-project doc -t /path/to/docs --generate-html
+hop marketplace install hop-misc-documentation
----
-== Options
-
-[options="header"]
-|===
-|Option |Description |Default
-|-s, --source-folder |Source project folder to document |Required (or use
project)
-|-t, --target-folder |Target documentation folder |Required
-|-ip, --include-parameters |Include pipeline/workflow parameters |true
-|-in, --include-notes |Include notes text |false
-|-im, --include-metadata |Include metadata overview |true
-|-n, --project-name |Project name |"Hop" (or HOP_PROJECT_NAME)
-|-gh, --generate-html |Generate HTML versions of all docs |false
-|-rmmd, --remove-markdown |Remove .md files after HTML generation |false
-|===
-
-== Examples
-
-=== Basic Documentation
+The same plugin has the xref:workflow/actions/documentation.adoc[Build
documentation] action, which does the same from a workflow.
-Generate Markdown documentation only:
+== Usage
[tabs]
====
@@ -63,96 +44,74 @@ Windows::
--
[source,shell]
----
-./hop.bat doc -s "C:\Projects\my-hop-project" -t "C:\yourLocation\docs"
+hop.bat doc -j my-project -t C:\docs\my-project
+hop.bat doc -j my-project -t C:\docs\my-project -ip -in -im --generate-html
----
+--
-Linux/macOS::
+Linux, macOS::
+
--
[source,shell]
----
-./hop doc -s ~/projects/my-hop-project -t ~/docs
+./hop doc -j my-project -t /tmp/docs/my-project
+./hop doc -j my-project -t /tmp/docs/my-project -ip -in -im --generate-html
----
+--
====
-=== HTML Documentation
-
-Generate HTML + remove Markdown source:
-
-[tabs]
-====
-Windows::
-+
---
-[source,shell]
-----
-./hop.bat doc -s "C:\Projects\my-hop-project" -t "C:\yourLocation\docs"
--generate-html --remove-markdown
-----
+== Options
-Generates:
-[source]
-----
-C:\docs\
-├── index.html (project TOC)
-├── pipelines\
-│ └── my-pipeline.html
-├── workflows\
-│ └── my-workflow.html
-└── assets\
- ├── styles.css
- └── images\
-----
+[options="header", cols="1,3"]
+|===
+|Option|Description
+|`-j`, `--project <name>`|The project to document. See
<<_which_folder_is_documented,Which folder is documented>>.
+|`-e`, `--environment <name>`|Enable an environment, and document its project.
+|`-t`, `--target-folder <folder>`|The folder to write the documentation to.
Required. It is created when it doesn't exist.
+|`-s`, `--source-folder <folder>`|The folder to document, when no project is
enabled.
+|`-n`, `--project-name <name>`|The name in the title of the index, when no
project is enabled. The default is `Hop`.
+|`-ip`, `--include-parameters`|List the parameters of each pipeline and
workflow.
+|`-in`, `--include-notes`|List the text of the notes of each pipeline and
workflow.
+|`-im`, `--include-metadata`|Add an overview of the metadata elements of the
project, such as connections and run configurations.
+|`-gh`, `--generate-html`|Also write an HTML file next to each Markdown file.
The HTML index links to the HTML pages.
+|`-rmmd`, `--remove-markdown`|With `--generate-html`, remove the Markdown
files once the HTML is written. Without `--generate-html` it does nothing.
+|===
-Linux/macOS::
-+
---
-[source,shell]
-----
-./hop doc -s ~/projects/my-hop-project -t ~/docs --generate-html
--remove-markdown
-----
-====
+All the `-ip`, `-in`, `-im`, `-gh` and `-rmmd` options are off unless you give
them.
-=== Using Hop Project Context
+Run `hop doc --help` for the full list, which also has `-pl`, `--environments`
and the other options of the xref:hop-tools/hop.adoc[hop] command.
-[source,shell]
-----
-./hop.bat run -j IT-mongo doc -t "C:\yourLocation\docs" --generate-html
-----
+== Which folder is documented
-Uses `HOP_PROJECT_HOME` and `HOP_PROJECT_NAME` automatically.
+When a project is enabled, `hop doc` documents the home folder of that
project, and uses the project's name in the index.
+`-s` and `-n` are then not used.
-== Workflow Action Usage
+A project is enabled with `-j` or `-e`, and otherwise the default project of
your Hop configuration is used.
+The standard configuration has the `default` project as its default project,
so name the project you want with `-j`.
-Add "Documentation" action to any Hop workflow:
+`-s` and `-n` are only used when no project is enabled, for example when the
Hop configuration has no default project.
-1. Drag **Documentation** action from Utility category
-2. Set **Target folder** (`C:\yourLocation\docs`)
-3. Optional: Check **Generate HTML**
-4. Optional: Check **Remove markdown** (only when Generate HTML enabled)
-5. Configure content options:
- - ☑ **Include parameters** (optional)
- - ☑ **Include metadata** (optional)
- - ☐ **Include notes** (optional)
+== What is written
-== Generated Structure
+The target folder gets the same subfolders as the project.
+For each pipeline and workflow, `hop doc` writes a page next to where the file
is in the project: `pipelines/load-orders.hpl` becomes
`pipelines/load-orders-pipeline.md`, and `main.hwf` becomes `main-workflow.md`.
+[source]
+----
docs/
-├── index.html (Table of Contents)
-├── pipelines/ (all .hpl files)
-│ ├── pipeline1.html
-│ └── subfolder/
-│ └── pipeline2.html
-├── workflows/ (all .hwf files)
-│ └── workflow1.html
-├── metadata/ (metadata overview)
+├── index.md table of contents, linking every page
+├── pipelines/
+│ └── load-orders-pipeline.md
+├── main-workflow.md
+├── metadata/ with --include-metadata
└── assets/
-├── styles.css
-└── images/
+ ├── styles.css
+ └── images/ an SVG diagram per pipeline and workflow
+----
+
+Each page has the file name, the date the file was last changed and the
diagram, plus the notes and the parameters when you ask for them.
-== Notes
+With `--generate-html`, each `.md` file gets an `.html` file next to it,
including `index.html`.
+With `--remove-markdown` as well, only the HTML files, the images and the
styles remain.
-* `--remove-markdown` is **automatically disabled** if `--generate-html` is
false
-* HTML output includes custom CSS styling and responsive images
-* All links in `index.html` automatically point to `.html` files
-* Supports nested folders and metadata documentation
-* Uses CommonMark for Markdown → HTML conversion
+Files that are already in the target folder stay, so remove the old
documentation first when files were renamed or deleted in the project.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-driver.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-driver.adoc
new file mode 100644
index 0000000000..35135ad112
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-driver.adoc
@@ -0,0 +1,127 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[HopDriver]]
+:description: hop driver lists the JDBC drivers Hop can download and installs
them from Maven Central or another Maven repository.
+
+= hop driver
+
+`hop driver` lists the JDBC drivers that Hop knows how to download, and
installs them with the libraries they need.
+
+The database plugins declare which driver they use.
+`hop driver` reads that list, so it only knows the database types installed in
your Hop.
+
+See xref:database/databases.adoc#_adding_jdbc_drivers[Adding JDBC drivers] for
the other ways to get a driver: the *Download driver* button in the connection
editor, and the `HOP_DRIVERS_DOWNLOAD` variable of the Docker image.
+
+[source,bash]
+----
+hop driver list
+hop driver install <driver-id> [options]
+----
+
+== hop driver list
+
+Lists every downloadable driver, with its license category and whether it is
installed.
+
+[source,bash]
+----
+$ ./hop driver list
+JDBC drivers declared by the database plugins
+Scanned folders: [/opt/hop/lib/jdbc]
+
+ID DATABASE LIC INSTALLED ARTIFACT:VERSION
+---- -------- --- --------- ----------------
+derby Apache Derby A yes
org.apache.derby:derbyclient:<version>
+duckdb DuckDB A no
org.duckdb:duckdb_jdbc:<version>
+mysql MySQL X no
com.mysql:mysql-connector-j:<version>
+oracle Oracle X no
com.oracle.database.jdbc:ojdbc11:<version>
+postgresql PostgreSQL A yes
org.postgresql:postgresql:<version>
+...
+----
+
+[options="header", cols="1,3"]
+|===
+|Column|Description
+|`ID`|The id to pass to `hop driver install`.
+|`DATABASE`|The database type.
+|`LIC`|The ASF license category of the driver. `A` and `B` drivers may ship
with Hop. `X` drivers are restricted, for example proprietary, GPL or LGPL
licensed. Hop never ships them, and only downloads them when you accept the
vendor's license.
+|`INSTALLED`|`yes` when a jar of the driver, in any version, is in one of the
scanned folders.
+|`ARTIFACT:VERSION`|The Maven artifact and the version that `hop driver
install` downloads by default.
+|===
+
+The scanned folders are the entries of `HOP_SHARED_JDBC_FOLDERS`, or
`lib/jdbc` when that variable is not set.
+
+[options="header", cols="1,3"]
+|===
+|Option|Description
+|`--target <folder>`|Only check this folder for installed drivers.
+|===
+
+== hop driver install
+
+Downloads a driver, with the libraries it depends on, and copies the jars into
a JDBC folder.
+Restart Hop, Hop Server or the worker afterwards so the driver is loaded.
+
+[source,bash]
+----
+hop driver install postgresql
+hop driver install oracle --accept-license
+hop driver install mariadb --accept-license --driver-version <version>
+hop driver install duckdb --target /opt/hop-drivers
+----
+
+[options="header", cols="1,3"]
+|===
+|Option|Description
+|`<driver-id>`|The `ID` of the driver in `hop driver list`.
+|`--accept-license`|Accept the vendor's license. Required for restricted
(`LIC` `X`) drivers. Without it, Hop prints the license and the vendor's links
and installs nothing.
+|`--driver-version <version>`|The version to install. By default, the version
shown by `hop driver list`.
+|`--force`|Download and install the driver again, also when the same version
is already installed.
+|`--repo <url>`|The Maven repository to download from instead of Maven
Central, for example an internal mirror. A driver that is published in another
repository, such as MonetDB, is also looked up there.
+|`--target <folder>`|The folder to install into. By default, the first entry
of `HOP_SHARED_JDBC_FOLDERS` that is not `lib/jdbc`. Without such an entry,
`lib/jdbc`.
+|===
+
+The default target keeps downloaded drivers out of Hop's own `lib/jdbc` folder
as soon as `HOP_SHARED_JDBC_FOLDERS` names another folder.
+See xref:database/databases.adoc#_adding_a_driver_manually[Adding a driver
manually] for `HOP_SHARED_JDBC_FOLDERS`.
+
+Some drivers need extra jars at runtime.
+`hop driver install` downloads them too, and installs a missing one when the
driver itself is already there.
+For example, the Oracle driver gets `oraclepki`, which connections with an
Oracle Wallet need.
+
+=== Drivers that are already installed
+
+When the same version is already in the target folder, `hop driver install`
only says so.
+Use `--force` to download it again.
+
+Hop only looks for installed drivers in the scanned folders: the entries of
`HOP_SHARED_JDBC_FOLDERS`, or `lib/jdbc`.
+A `--target` folder outside them is downloaded into again on every install.
+
+When the driver is also in another scanned folder, for example the bundled one
in `lib/jdbc`, Hop prints a note and leaves that copy in place.
+Remove the copy you don't want, so that the driver is not loaded twice.
+
+Downloaded files are cached in `.hop/driver-cache` in your home folder.
+
+=== Exit codes
+
+[options="header", cols="1,3"]
+|===
+|Code|Meaning
+|`0`|The driver is installed, or was already installed.
+|`1`|The download or the install failed. The error is printed.
+|`2`|The driver is restricted and `--accept-license` was not given.
+|`3`|No database type in this Hop uses that id.
+|`4`|The database type exists, but declares no driver that can be downloaded.
Download the driver from the vendor and copy it into the JDBC folder.
+|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-export.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-export.adoc
new file mode 100644
index 0000000000..1a386dc649
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop-export.adoc
@@ -0,0 +1,117 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[HopExport]]
+:description: hop export writes pipeline and workflow diagrams to SVG or
Mermaid files, for one file or a whole folder.
+
+= hop export
+
+`hop export` writes the diagram of a pipeline or a workflow to a file, in SVG
or Mermaid format.
+It exports one file, or every pipeline and workflow in a folder.
+
+It uses the same exporters as *File > Export diagram* in Hop Gui, so you can
create the diagrams from a script or a build.
+The installation also has `hop-export.sh` and `hop-export.bat`, which run the
same command.
+
+[source,bash]
+----
+# One file
+hop export -f pipelines/load-orders.hpl -o diagrams/load-orders.svg
+
+# One file as Mermaid
+hop export -f workflows/main.hwf --format mermaid -o diagrams/main.mmd
+
+# Every pipeline and workflow in a folder and its subfolders
+hop export -s /path/to/project -t /path/to/diagrams --recursive
+
+# The formats this installation can write
+hop export --list-exporters
+----
+
+== Options
+
+[options="header", cols="1,3"]
+|===
+|Option|Description
+|`-f`, `--file <file>`|The pipeline (`.hpl`) or workflow (`.hwf`) to export.
+|`-o`, `--output-file <file>`|The file to write, for `-f`. By default, the
name of the exported file with the extension of the format (`.svg` or `.mmd`),
in the same folder.
+|`--format <format>`|`svg` or `mermaid`, in upper or lower case. Without it,
an `-o` file that ends in `.mmd` is written as Mermaid, and one that ends in
`.svg` as SVG. Without `--format` and `-o`, the format is SVG.
+|`-s`, `--source-folder <folder>`|The folder to export every pipeline and
workflow from.
+|`-t`, `--target-folder <folder>`|The folder to write the diagrams to, for
`-s`. Required with `-s`. See <<_exporting_a_folder,Exporting a folder>>.
+|`-r`, `--recursive`|With `-s`, also export the subfolders.
+|`-m`, `--magnification <factor>`|Scale an SVG diagram, for example `2` for
twice the size. The default is `1.0`. Mermaid diagrams are not scaled.
+|`-in`, `--include-notes`|Notes are included by default. Use `-in=false` to
leave them out of a Mermaid diagram. Giving `-in` without a value also leaves
them out. SVG diagrams always show the notes.
+|`--list-exporters`|List the exporters in this installation, with their format
and file extension.
+|`-j`, `--project <name>`|Enable a project first, so that variables such as
`+${PROJECT_HOME}+` can be used in the file names.
+|`-e`, `--environment <name>`|Enable an environment, and its project, first.
+|===
+
+The project options are the same as for xref:hop-run/index.adoc[hop run]:
+
+[source,bash]
+----
+hop export -j my-project -f '${PROJECT_HOME}/pipelines/load-orders.hpl' -o
/tmp/load-orders.svg
+----
+
+Run `hop export --help` for the full list, which also has `-pl`,
`--environments` and the other options of the xref:hop-tools/hop.adoc[hop]
command.
+
+== Formats
+
+Hop has an SVG and a Mermaid exporter for both pipelines and workflows.
+Plugins can add more; `--list-exporters` shows what your installation has.
+
+[source,bash]
+----
+$ ./hop export --list-exporters
+Available Diagram Exporters (4):
+ ID: pipeline-mermaid
+ Name: Pipeline Mermaid Exporter
+ Format: MERMAID
+ Extension: .mmd
+ Description: Exports a pipeline diagram to Mermaid flowchart (.mmd)
+
+ ID: pipeline-svg
+ Name: Pipeline SVG Exporter
+...
+----
+
+* *SVG* draws the diagram as it looks in Hop Gui, with the transform and
action icons and the notes.
+* *Mermaid* writes a text flowchart (`.mmd`) with a node per transform or
action and an arrow per hop. You can keep it next to the project in version
control, or paste it into a page that renders Mermaid.
+
+== Exporting a folder
+
+With `-s`, `hop export` exports every `.hpl` and `.hwf` file in the folder,
and with `-r` in its subfolders too.
+Each diagram is written under `-t` with the same relative path, for example
`pipelines/load-orders.hpl` becomes `pipelines/load-orders.svg`.
+
+* `-t` must be an existing folder. With `-r`, its subfolders must exist as
well.
+* The export stops at the first file it can't export, for example a diagram
whose folder is missing under `-t`, or a file that is not valid XML. The
diagrams written before it stay.
+* A pipeline and a workflow with the same name in the same folder are written
to the same file, so only the last one is kept.
+
+At the end, the command logs how many diagrams it exported.
+
+== Exit codes
+
+`hop export` ends with `0` when the export is done, and with `1` when it
fails, for example when the source folder doesn't exist or a diagram can't be
written.
+
+The `hop-export.sh` and `hop-export.bat` scripts end with other codes:
+
+[options="header", cols="1,3"]
+|===
+|Code|Meaning
+|`0`|The export is done.
+|`1`|The help was shown, with `--help`.
+|`2`|The export failed, for example when the source folder doesn't exist or a
file is not a valid pipeline or workflow.
+|`9`|An option is not valid.
+|===
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop.adoc
index d0db2c90fd..28d4e9f770 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/hop.adoc
@@ -28,7 +28,7 @@ The main goal is to have a single wrapper script to make
maintenance easier for
== Usage
To see the usage of the `hop` tool, you can simply use the `help` command to
see which commands are available.
-The command list depends on the plugins in your installation; the sample below
shows the common options and a typical set of commands.
+The command list depends on the plugins in your installation; the sample below
is the standard Hop client, without optional plugins.
[source,bash]
----
@@ -40,38 +40,85 @@ Usage: hop [-hV] [--dev-debug] [--dev-debug-wait] [-ime]
<environments>...]]... [-pl=<projectLocations>[,
<projectLocations>...]]... [-s=<systemProperties>[,
<systemProperties>...]]... [COMMAND]
+ --dev-debug Allow a Hop developer to debug remotely
+ --dev-debug-wait Allow a Hop developer to debug remotely. The script
+ will wait until a debugging session is set up.
-e, --environment=<environmentOption>
- The name of the lifecycle environment to use
- -h, --help Show this help message and exit.
- -ime, --in-memory Keep configuration in memory without persisting to
- hop-config.json
+ The name of the lifecycle environment to use
+ --environment-conf-files,
+ --environment-config-files=<environmentConfigFiles>[,
+ <environmentConfigFiles>...]
+ Comma-separated list of configuration files to apply
+ to the project or environment
+ --environments=<environments>[,<environments>...]
+ Comma-separated list of environment definitions
(name=
+ [project:]configFile1;configFile2)
+ -h, --help Show this help message and exit.
+ -ime, --in-memory Keep configuration in memory without persisting to
+ hop-config.json
-j, --project=<projectOption>
- The name of the project to use
+ The name of the project to use
-pl, --project-locations=<projectLocations>[,<projectLocations>...]
- Comma-separated list of project locations
- (name=location or archive files)
+ Comma-separated list of project locations
+ (name=location or archive files)
-s, --system-properties=<systemProperties>[,<systemProperties>...]
- A comma separated list of KEY=VALUE pairs
- -V, --version Print version information and exit.
+ A comma separated list of KEY=VALUE pairs
+ -V, --version Print version information and exit.
Commands:
- help Display help information about the specified command.
- conf Configure Hop
- encrypt Encrypt secrets
- gui The Hop GUI
- import Import metadata
- run Run a pipeline or workflow
- search Search in Hop metadata
- server Run a Hop server
- setup Configure Hop environment variables
+ help Display help information about the specified command.
+ conf Configure Hop
+ driver Manage JDBC drivers (list, install)
+ encrypt Encrypt secrets
+ export Export pipeline, workflow, or model diagrams to various formats
+ (SVG, Mermaid, etc.)
+ gui The Hop GUI
+ import Import metadata
+ marketplace Install and manage optional Hop plugins from a Maven repository
+ naming-check Validate names against project Naming Schemes
+ run Run a pipeline or workflow
+ search Search in Hop metadata
+ server Run a Hop server
+ setup Configure Hop launcher environment variables
+ (HOP_CONFIG_FOLDER, HOP_AUDIT_FOLDER, HOP_JAVA_HOME,
+ HOP_OPTIONS, HOP_SHARED_JDBC_FOLDERS). Distinct from hop
+ conf, which edits hop-config.json.
----
=====
+== Commands
+
+[options="header", cols="1,1,3"]
+|===
+|Command|Plugin|Page
+|`arrow`|Apache Arrow (optional)|xref:hop-tools/hop-arrow-command.adoc[hop
arrow]
+|`conf`||xref:hop-tools/hop-conf/hop-conf.adoc[hop conf]
+|`doc`|Documentation (optional)|xref:hop-tools/hop-doc.adoc[hop doc]
+|`driver`||xref:hop-tools/hop-driver.adoc[hop driver]
+|`encrypt`||xref:hop-tools/hop-encrypt.adoc[hop encrypt]
+|`export`||xref:hop-tools/hop-export.adoc[hop export]
+|`gui`||xref:hop-tools/hop-gui.adoc[hop gui]
+|`import`||xref:hop-tools/hop-import.adoc[hop import]
+|`lint`|Lint (optional)|xref:hop-tools/hop-lint.adoc[hop lint]
+|`marketplace`||xref:hop-tools/hop-marketplace.adoc[hop marketplace]
+|`naming-check`||xref:metadata-types/naming-scheme.adoc#_checking_names_in_ci[Naming
scheme: checking names in CI]
+|`python`|PyHop (optional)|xref:hop-tools/hop-python/hop-python.adoc[hop
python]
+|`run`||xref:hop-run/index.adoc[hop run]
+|`search`||xref:hop-tools/hop-search.adoc[hop search]
+|`server`||xref:hop-server/index.adoc[hop server]
+|`setup`||xref:hop-tools/hop-setup.adoc[hop setup]
+|===
+
+The commands marked optional only exist once their plugin is installed from
the xref:hop-tools/hop-marketplace.adoc[marketplace].
+`hop help` then lists them as well.
+
== Options
[options="header"]
|===
|Short|Extended|Description
+||`--dev-debug`|For Hop developers: let a Java debugger connect on port 5010.
+||`--dev-debug-wait`|As `--dev-debug`, but Hop waits until the debugger is
connected before it starts.
|`-e`|`--environment`|The name of the lifecycle environment to enable before
the subcommand runs.
|`-h`|`--help`|Displays this help message and quits.
|`-ime`|`--in-memory`|Keep Hop configuration in memory. Changes are not
written to `hop-config.json`. Implied by `--project-locations` and
`--environments`.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
index 6dff4ed1f6..ec747b5769 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-tools/index.adoc
@@ -21,17 +21,22 @@ Even though Hop developers will spend the majority of their
time in Hop Gui, Hop
This page provides an overview of the tools that are available in the platform.
-* xref:hop-tools/hop.adoc[Hop]
-* xref:hop-tools/hop-setup.adoc[Hop Setup]
-* xref:hop-tools/hop-marketplace.adoc[Hop Marketplace]
-* xref:hop-tools/hop-conf/hop-conf.adoc[Hop Conf]
-* xref:hop-tools/hop-encrypt.adoc[Hop Encrypt]
-* xref:hop-tools/hop-gui.adoc[Hop Gui]
-* xref:hop-run/index.adoc[Hop Run]
-* xref:hop-tools/hop-search.adoc[Hop Search]
-* xref:hop-tools/hop-import.adoc[Hop Import]
-* xref:hop-tools/hop-doc.adoc[Hop Documentation]
-* xref:hop-tools/hop-python/hop-python.adoc[Hop Python (PyHop)]
-* xref:hop-tools/hop-arrow-command.adoc[Hop Arrow (Flight server)]
+Each tool is a command of the xref:hop-tools/hop.adoc[hop] launcher, for
example `hop run`.
+`hop arrow`, `hop doc`, `hop lint` and `hop python` come with optional
plugins: they are only there once the plugin is installed from the
xref:hop-tools/hop-marketplace.adoc[marketplace].
-* xref:hop-server/index.adoc[Hop Server]
+* xref:hop-tools/hop.adoc[hop]: the launcher, its options and the list of
commands
+* xref:hop-tools/hop-arrow-command.adoc[hop arrow]: start an Apache Arrow
Flight or socket server
+* xref:hop-tools/hop-conf/hop-conf.adoc[hop conf]: manage projects,
environments and other configuration
+* xref:hop-tools/hop-doc.adoc[hop doc]: write documentation for a project
+* xref:hop-tools/hop-driver.adoc[hop driver]: list and install JDBC drivers
+* xref:hop-tools/hop-encrypt.adoc[hop encrypt]: encrypt or obfuscate a password
+* xref:hop-tools/hop-export.adoc[hop export]: export pipeline and workflow
diagrams to SVG or Mermaid
+* xref:hop-tools/hop-gui.adoc[hop gui]: start Hop Gui
+* xref:hop-tools/hop-import.adoc[hop import]: import and convert metadata from
other tools, such as Kettle/PDI
+* xref:hop-tools/hop-lint.adoc[hop lint]: check pipelines, workflows and
metadata against the lint rules
+* xref:hop-tools/hop-marketplace.adoc[hop marketplace]: install and manage
optional plugins
+* xref:hop-tools/hop-python/hop-python.adoc[hop python]: start the gateway for
Hop Python (PyHop)
+* xref:hop-run/index.adoc[hop run]: run a pipeline or a workflow
+* xref:hop-tools/hop-search.adoc[hop search]: search the metadata of a project
+* xref:hop-server/index.adoc[hop server]: start a Hop Server
+* xref:hop-tools/hop-setup.adoc[hop setup]: set the environment variables of
the launcher, such as `HOP_CONFIG_FOLDER`