This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new ebe1e70ab5 structure and content updates for Getting Started guide.
fixes #8447 (#8452)
ebe1e70ab5 is described below
commit ebe1e70ab5c23999ec2dfe42f8caf67de5a43e0f
Author: Bart Maertens <[email protected]>
AuthorDate: Sat Sep 19 17:36:30 2026 +0200
structure and content updates for Getting Started guide. fixes #8447 (#8452)
---
.../gs-hop-gui-create-pipeline-2.png | Bin 42763 -> 0 bytes
.../gs-hop-gui-create-pipeline-3.png | Bin 73652 -> 0 bytes
.../gs-hop-gui-create-pipeline-5.png | Bin 7579 -> 0 bytes
.../gs-hop-gui-create-pipeline-7.png | Bin 115333 -> 0 bytes
.../gs-hop-gui-create-pipeline-8.png | Bin 115236 -> 0 bytes
.../gs-hop-gui-create-pipeline-9.png | Bin 118494 -> 0 bytes
.../gs-hop-gui-create-workflow-2.png | Bin 42763 -> 0 bytes
.../gs-hop-gui-create-workflow-3.png | Bin 66749 -> 0 bytes
.../gs-hop-gui-create-workflow-5.png | Bin 7579 -> 0 bytes
.../gs-hop-gui-create-workflow-7.png | Bin 112901 -> 0 bytes
.../gs-hop-gui-create-workflow-save.png | Bin 40818 -> 0 bytes
.../getting-started/gs-hop-gui-run-pipeline-2.png | Bin 45420 -> 0 bytes
.../getting-started/gs-hop-gui-run-pipeline-4.png | Bin 40551 -> 0 bytes
docs/hop-user-manual/modules/ROOT/nav.adoc | 9 +-
.../ROOT/pages/getting-started/hop-concepts.adoc | 7 +-
.../getting-started/hop-download-install.adoc | 127 +++++-
.../pages/getting-started/hop-gui-pipelines.adoc | 224 +++++++----
.../pages/getting-started/hop-gui-workflows.adoc | 159 +++++---
.../ROOT/pages/getting-started/hop-gui.adoc | 61 +--
.../ROOT/pages/getting-started/hop-next-steps.adoc | 66 +++-
.../ROOT/pages/getting-started/hop-projects.adoc | 107 +++++
.../ROOT/pages/getting-started/hop-tools.adoc | 434 +++++----------------
.../pages/getting-started/hop-what-is-hop.adoc | 103 +++--
.../modules/ROOT/pages/getting-started/index.adoc | 25 +-
.../modules/ROOT/pages/hop-gui/perspectives.adoc | 20 +-
.../hop-gui/perspectives-list.adoc} | 12 -
26 files changed, 763 insertions(+), 591 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-2.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-2.png
deleted file mode 100644
index 19778f0dc9..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-2.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-3.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-3.png
deleted file mode 100644
index 7e05ea8cb6..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-3.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-5.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-5.png
deleted file mode 100644
index c569f11caa..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-5.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-7.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-7.png
deleted file mode 100644
index 3b9fe912e0..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-7.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-8.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-8.png
deleted file mode 100644
index 01e2af4b51..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-8.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-9.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-9.png
deleted file mode 100644
index d2b20cd17b..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-pipeline-9.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-2.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-2.png
deleted file mode 100644
index 19778f0dc9..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-2.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-3.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-3.png
deleted file mode 100644
index d23edd619a..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-3.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-5.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-5.png
deleted file mode 100644
index c569f11caa..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-5.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-7.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-7.png
deleted file mode 100644
index 9f44725687..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-7.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-save.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-save.png
deleted file mode 100644
index 8153b21afa..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-create-workflow-save.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-run-pipeline-2.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-run-pipeline-2.png
deleted file mode 100644
index 2dbdf57596..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-run-pipeline-2.png
and /dev/null differ
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-run-pipeline-4.png
b/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-run-pipeline-4.png
deleted file mode 100644
index c53c1edb04..0000000000
Binary files
a/docs/hop-user-manual/modules/ROOT/assets/images/getting-started/gs-hop-gui-run-pipeline-4.png
and /dev/null differ
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index 371b587e12..eba4f81a41 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -16,12 +16,13 @@ under the License.
////
* xref:getting-started/index.adoc[Getting Started]
** xref:getting-started/hop-what-is-hop.adoc[What is Hop]
-** xref:getting-started/hop-concepts.adoc[Hop Concepts]
** xref:getting-started/hop-download-install.adoc[Download & Install]
-** xref:getting-started/hop-tools.adoc[Tools]
+** xref:getting-started/hop-concepts.adoc[Hop Concepts]
** xref:getting-started/hop-gui.adoc[Hop Gui Overview]
-** xref:getting-started/hop-gui-pipelines.adoc[Pipelines]
-** xref:getting-started/hop-gui-workflows.adoc[Workflows]
+** xref:getting-started/hop-gui-pipelines.adoc[Your first pipeline]
+** xref:getting-started/hop-gui-workflows.adoc[Your first workflow]
+** xref:getting-started/hop-projects.adoc[Projects and environments]
+** xref:getting-started/hop-tools.adoc[The Hop toolbox]
** xref:getting-started/hop-next-steps.adoc[Next Steps]
* xref:hop-vs-kettle/index.adoc[Kettle/PDI and Hop]
** xref:hop-vs-kettle/hop-vs-kettle.adoc[Hop vs Kettle]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
index ba0acffb3e..7628429e17 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-concepts.adoc
@@ -17,7 +17,7 @@ under the License.
[[HopConcepts]]
:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (2/8): before going into the details of
workflows and pipelines, we'll walk through the core terminology used in Hop.
+:description: Getting Started (3/9): before going into the details of
workflows and pipelines, we'll walk through the core terminology used in Hop.
= Hop Concepts
@@ -37,6 +37,11 @@ All transforms in a pipeline run in parallel.
* **Workflows** are collections of **actions**, connected by **hops**.
All actions in a workflow run sequentially by default.
+* **Metadata items** are the definitions you create once and reuse everywhere:
relational database connections,
xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[run
configurations], xref:metadata-types/hop-server.adoc[Hop Servers],
xref:metadata-types/data-set.adoc[data sets],
xref:metadata-types/pipeline-unit-test.adoc[unit tests], and many more.
+A pipeline refers to a database connection by name; the connection itself is
defined once, in the project.
+Change it in one place and everything that uses it follows.
+See the xref:metadata-types/index.adoc[full list of metadata types].
+
* **Projects** are logical collections of hop code and configuration.
**Environments** contain the environment-specific (e.g. dev, uat, prd)
metadata.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-download-install.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-download-install.adoc
index fd5c5256f8..a06ebcf90b 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-download-install.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-download-install.adoc
@@ -17,30 +17,131 @@ under the License.
[[DownloadInstall]]
:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (3/8): Installing Hop is as easy and accessible
as possible: download, unzip, start.
+:description: Getting Started (2/9): installing Hop is download, unzip, start.
This page also covers the Java requirement, the Docker images, and the one
configuration step worth doing before you begin.
-= Download and Install Hop
+= Download and install Hop
Hop is designed to be as easy and accessible to use as possible.
+There is no installer and no server to set up: download, unzip, start.
-Getting and starting Hop couldn't be easier:
+== Check your Java
-== Download
+Hop is self-contained. The only thing it needs is a Java runtime.
-Download Hop from the http://hop.apache.org/download/[downloads page].
+The official Java version for Hop is **21**.
-== Prerequisites
+[source,shell]
+----
+java -version
+----
-Hop is self contained, all you need is a Java runtime.
-Hop official java version is 21. Check the
https://java.com/en/download/help/download_options.html[Java docs] to download
and install a Java runtime for your operating system.
+You should see something like:
-== Install
+[source,shell]
+----
+openjdk version "21.0.10" 2026-01-20
+OpenJDK Runtime Environment (build 21.0.10)
+OpenJDK 64-Bit Server VM (build 21.0.10, mixed mode, sharing)
+----
-Unzip Hop to a folder of your choice.
-From that folder, you'll be able to run the different Hop
xref:hop-tools/index.adoc[tools] through their corresponding scripts.
+If that command is not found, or reports an older version, install a Java 21
runtime first.
+See xref:supported-jvms.adoc[Supported Java Versions] for which runtimes are
known to work, and which one to avoid.
-Recommended next step: point Hop at a user-level config folder so upgrades
keep your projects and settings. In Hop Gui use *Tools → Configure Hop
environment...*, or from the install directory run `./hop setup apply
--defaults` (then restart). See xref:hop-tools/hop-setup.adoc[Hop setup].
+== Download and unzip
-Now that you have Hop installed, let's have a look at all of the Hop tools you
have at your disposal.
+Download the client from the https://hop.apache.org/download/[Apache Hop
downloads page] and unzip it to a folder of your choice.
+That folder is your Hop installation.
+Everything you need is in it:
+[options="header"]
+|===
+|What|Where
+|The launcher scripts|`hop`, `hop.bat` and the individual `hop-*.sh` /
`hop-*.bat` scripts
+|Configuration|`config/`
+|Plugins|`plugins/`
+|Projects created for you|`config/projects/default` and
`config/projects/samples`
+|===
+
+== One step worth doing now
+
+By default Hop keeps its configuration inside the installation folder.
+That means an upgrade - a new unzip - leaves your projects and settings behind
in the old folder.
+
+Point Hop at a user-level configuration folder instead, once, before you start:
+
+[tabs]
+====
+Windows::
++
+--
+[source,shell]
+----
+hop.bat setup apply --defaults
+----
+--
+
+Linux, macOS::
++
+--
+[source,shell]
+----
+./hop setup apply --defaults
+----
+--
+====
+
+Restart Hop afterwards.
+You can do the same thing from the GUI later with *Tools -> Configure Hop
environment...*.
+
+See xref:hop-tools/hop-setup.adoc[Hop Setup] for what it changes and how to
undo it.
+
+== Start Hop Gui
+
+[tabs]
+====
+Windows::
++
+--
+[source,shell]
+----
+hop-gui.bat
+----
+--
+
+Linux, macOS::
++
+--
+[source,shell]
+----
+./hop-gui.sh
+----
+--
+====
+
+The first start takes a little longer than the ones after it, because Hop
scans and registers its plugins.
+
+== Other ways to run Hop
+
+Unzipping the client is the right way to start, but it is not the only option:
+
+[options="header",cols="25%,45%,30%"]
+|===
+|Option|When you'd use it|Read more
+
+|Docker - `apache/hop`
+|Running pipelines and workflows in a container, and Hop Server. This is how
most production deployments run.
+|xref:docker-container.adoc[Hop in Docker]
+
+|Docker - `apache/hop-web`
+|Hop Gui in a browser, for shared or hosted development environments.
+|xref:hop-web-docker.adoc[Hop Web in Docker]
+
+|Hop Web
+|The same Hop Gui, running in a browser instead of on your desktop.
+|xref:hop-gui/hop-web.adoc[Hop Web]
+|===
+
+TIP: xref:installation-configuration.adoc[Installation and Configuration]
covers system requirements, memory settings and the environment variables Hop
reads at startup.
+
+Now that Hop is running, let's look at the words Hop uses for things.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-pipelines.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-pipelines.adoc
index 54d2c62a28..47b07c60d5 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-pipelines.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-pipelines.adoc
@@ -17,119 +17,203 @@ under the License.
[[HopGuiPipelines]]
:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (6/8): Pipelines are one of the two main file
types in Hop. Learn how to add transforms and connect them through hops before
your run, preview and debug your first pipeline.
+:description: Getting Started (5/9): build, preview and run your first
pipeline. You'll create rows, add a sequence number, look at the data before
you run anything, and write the result to a file.
-= Pipelines
+= Your first pipeline
-In xref:getting-started/hop-concepts.adoc[Concepts], we walked through
pipelines, transforms and hops.
-Let’s remember:
+In xref:getting-started/hop-concepts.adoc[Concepts] we said a pipeline is a
set of transforms connected by hops, and that all transforms run in parallel.
-* A pipeline consists of a chain of transforms that read, process or write
data.
-Transforms are connected by hops.
-Hops have a direction but can't create loops, which effectively makes
Pipelines DAGs (Directed Acyclic Graphs).
-* A transform is each basic operation in your pipeline.
-Each transform performs an operation on the data: reads from a source,
operates on data in the pipeline or writes to to a target.
-* A hop in a pipeline passes data between transforms.
+This page turns that into a working pipeline.
+It is deliberately small, but it is a complete one: it creates data, changes
it, lets you look at it, and writes it out.
+That read - change - write shape is what almost every pipeline you build later
will look like.
-== Create a Pipeline
+You will need nothing but a Hop installation.
+No database, no sample files, no network.
-There are two ways to create a pipeline.
+== Create a pipeline
-* click on the **New** option on the Horizontal toolbar and select the
Pipeline option.
+Click *New* in the main toolbar and pick *Pipeline*, or use *File -> New ->
Pipeline*.
-image:getting-started/gs-hop-gui-create-pipeline-1.png[]
+image:getting-started/gs-hop-gui-create-pipeline-1.png[The New option in the
main toolbar]
-* File -> New -> Pipeline 
-image:getting-started/gs-hop-gui-create-pipeline-2.png[] +
-image:getting-started/gs-hop-gui-create-pipeline-3.png[]
+You now have an empty pipeline canvas, with the pipeline toolbar above it.
-Your new pipeline is created.
-You’ll see the dialog below.
+image:getting-started/gs-hop-gui-create-pipeline-4.png[An empty pipeline
canvas]
-image:getting-started/gs-hop-gui-create-pipeline-4.png[width=65%]
+Save it right away with `CTRL-S` and call it `first-pipeline.hpl`.
+Hop can run a pipeline straight from the editor, but you will need the file on
disk in the xref:getting-started/hop-gui-workflows.adoc[next chapter].
-== Add and connect Transforms
+== Add the first transform
-=== Add Transforms
+Click anywhere on the empty canvas.
+The context dialog opens: this is how you add transforms.
-Now you are ready to add the first transform.
-Click anywhere in the pipeline canvas, the area where you’ll see the image
below.
+image:getting-started/gs-hop-gui-create-pipeline-6.png[The context dialog]
-image:getting-started/gs-hop-gui-create-pipeline-5.png[]
+Type in the search box to filter, then click or use the arrow keys and *Enter*
to add.
-After clicking you will see the context dialog.
-This is the dialog you'll use every time you need to select transforms to add
to your pipeline.
+Every pipeline has to start with a
xref:pipeline/pipeline-sources.adoc[pipeline source]: a transform that produces
rows instead of reading them from an incoming hop.
+Search for `Generate rows` and add it.
-image:getting-started/gs-hop-gui-create-pipeline-6.png[]
+TIP: search for `pipeline source` in the context dialog to see all transforms
that can start a pipeline.
-Use the search box in this dialog to find the transforms you need.
-Click or use the arrow keys and hit enter to add a transform to your pipeline.
+=== Configure Generate rows
-Start with a xref:pipeline/pipeline-sources.adoc[pipeline source] (a transform
that does not need incoming hops).
-Search for `pipeline source` in this dialog, or pick Generate Rows.
-For now, add a xref:pipeline/transforms/rowgenerator.adoc[Generate Rows] and a
xref:pipeline/transforms/addsequence.adoc[Add Sequence] transform to your
pipeline.
+Double-click the new transform to open its dialog, and set:
-image:getting-started/gs-hop-gui-create-pipeline-add-two-transforms.png[]
+[options="header"]
+|===
+|Field|Value
+|Transform name|`generate orders`
+|Limit|`100`
+|===
-TIP: check the xref:pipeline/transforms.adoc[full list of transforms].
-There are 186 transforms available in Hop 0.99, but you'll soon get to know
the most commonly used ones.
+In the *Fields* table at the bottom, add one row:
-=== Create a Hop
+[options="header"]
+|===
+|Name|Type|Value
+|`product`|String|`Widget`
+|===
-There are a number of ways to create a hop:
+Click *OK*.
-* **shift-drag**: while holding down the shift key on your keyboard.
-Click on a transform, while holding down your primary mouse button, drag to
the second transform.
-Release the primary mouse button and the shift key.
-* **scroll-drag**: scroll-click on a transform , while holding down your
mouse’s scroll button, drag to the second transform.
-Release the scroll button.
-* click on a transform in your pipeline to open the context dialog (the dialog
you opened in the '**click anywhere**' step).
-Click the 'Create hop' image:getting-started/icons/HOP.svg[Create
hop,25px,align="bottom"] button and select the transform you want to create the
hop to.
+This transform now produces 100 identical rows, each with a single `product`
field.
+Identical rows are not very interesting, so let's make them distinct.
-image:getting-started/gs-hop-gui-create-pipeline-create-hop.png[]
+== Add a second transform and connect it
-== Run your pipeline
+Click the canvas again, search for `Add sequence` and add it.
-Running a pipeline to see how it performs can be done in one of the following
tasks:
+image:getting-started/gs-hop-gui-create-pipeline-add-two-transforms.png[Two
transforms on the canvas]
-* Using the Run icon.
+Open it and set:
-image:getting-started/gs-hop-gui-run-pipeline-1.png[]
+[options="header"]
+|===
+|Field|Value
+|Transform name|`number the orders`
+|Name of value|`order_id`
+|Start at value|`1000`
+|Increment by|`1`
+|===
-* Select Run and click on Start Execution from the toolbar.
+Leave *Use counter to calculate sequence* checked.
+That is the default, and it means Hop counts internally instead of asking a
database for the next value, so this works without a database connection.
-image:getting-started/gs-hop-gui-run-pipeline-2.png[]
+Now connect the two transforms.
+There are three ways to create a hop:
-* Press F8
+* **shift-drag**: hold *Shift*, press your primary mouse button on `generate
orders`, drag to `number the orders` and release.
+* **scroll-drag**: press the scroll button on `generate orders`, drag to
`number the orders` and release.
+* click `generate orders` to open its context dialog and use the *Create hop*
image:getting-started/icons/HOP.svg[Create hop,25px,align="bottom"] button.
-You will see the Run Options dialog.
+image:getting-started/gs-hop-gui-create-pipeline-create-hop.png[Creating a hop
between two transforms]
-image:getting-started/gs-hop-gui-run-pipeline-3.png[] +
+The arrow on the hop shows which way the rows travel.
+Hops in a pipeline cannot form a loop.
-image:getting-started/gs-hop-gui-run-pipeline-4.png[]
+== Look at the data before you run anything
-TIP: a 'local' runtime configuration is created when you first start Hop Gui.
-Check the available
xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[runtime
configurations] for other engines to run your pipelines on.
+This is the step that makes Hop quick to work with, and it is worth learning
before anything else.
-Make sure your configuration is selected and hit Launch.
+Right-click `number the orders` and choose *Preview & debug output*, then
click *Quick Launch*.
-You'll see green check marks in the upper right corner of the transforms when
a pipeline ran successfully.
+Hop runs just enough of the pipeline to fill a preview window, and shows you
the actual rows:
-image:getting-started/gs-hop-gui-run-pipeline-5.png[]
+[options="header"]
+|===
+|product|order_id
+|Widget|1000
+|Widget|1001
+|Widget|1002
+|===
-After each run, the execution results are displayed in the panel on the bottom
of your window.
-The Execution Results contains two tabs:
+Two things are worth noticing.
-* transform metrics
-* logging
+`order_id` exists even though no transform "wrote" it to disk: fields are
created as rows travel down the pipeline.
+And you got this without configuring an output, a connection or a run
configuration.
+Preview early and often - it is much faster than running a whole pipeline to
find out that a field is wrong.
-The Transform Metrics tab displays metrics per transform.
+TIP: xref:pipeline/run-preview-debug-pipeline.adoc[Run, Preview and Debug a
Pipeline] covers previewing a subset of rows, pausing on a condition, and
debugging.
-image:getting-started/gs-hop-gui-run-pipeline-6.png[]
+== Write the result out
-The logging tab displays the log of the pipeline according to the log level
chosen on execution.
+A preview is not a result. Let's write the rows to a file.
-image:getting-started/gs-hop-gui-run-pipeline-7.png[]
+Click the canvas, search for `Text file output` and add it.
+Create a hop from `number the orders` to the new transform.
-TIP: for more detailed information, check out the
xref:pipeline/run-preview-debug-pipeline.adoc[Run, Preview and Debug a
Pipeline] page.
+Open it and, on the *File* tab, set:
+[options="header"]
+|===
+|Field|Value
+|Transform name|`write orders`
+|Filename|`+${java.io.tmpdir}/hop-orders+`
+|Extension|`csv`
+|===
+
+`+${java.io.tmpdir}+` is a xref:fields-parameters-variables.adoc[variable].
+Hop exposes every Java system property as one, so this resolves to your
system's temporary folder on any operating system.
+Anywhere Hop accepts a path, it accepts a variable.
+
+Go to the *Fields* tab and click *Get Fields*.
+Hop reads the fields coming in over the hop and fills the table with `product`
and `order_id`.
+Click *OK*.
+
+NOTE: *Get Fields* only works when the incoming hop exists and the transforms
before it are configured, because Hop asks them what they produce. If the table
stays empty, check the hop.
+
+== Run the pipeline
+
+Press *F8*, or click *Run* in the pipeline toolbar.
+
+image:getting-started/gs-hop-gui-run-pipeline-1.png[The Run button in the
pipeline toolbar]
+
+The run options dialog opens.
+
+image:getting-started/gs-hop-gui-run-pipeline-3.png[The pipeline run options
dialog]
+
+A `local`
xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[pipeline
run configuration] is created the first time you start Hop Gui.
+Select it and click *Launch*.
+
+Green check marks in the corner of each transform mean the pipeline finished
without errors.
+
+image:getting-started/gs-hop-gui-run-pipeline-5.png[A pipeline that ran
successfully]
+
+== Read the results
+
+The *Execution Results* panel at the bottom of the window is where you find
out what happened.
+
+The *Metrics* tab shows one line per transform, with rows read, written and
rejected, and the time each one took.
+`write orders` should show 100 rows written.
+
+image:getting-started/gs-hop-gui-run-pipeline-6.png[The Metrics tab]
+
+The *Logging* tab shows the log for the run, at the log level you picked in
the run options dialog.
+When something goes wrong, this is where the reason is.
+
+image:getting-started/gs-hop-gui-run-pipeline-7.png[The logging tab]
+
+Now open `hop-orders.csv` in your temporary folder.
+The log line for `write orders` tells you the full path if you are not sure
where that is.
+
+[source,csv]
+----
+product;order_id
+Widget;1000
+Widget;1001
+Widget;1002
+----
+
+That is a complete pipeline: it created data, changed it, and wrote it
somewhere you can use it.
+
+== What you learned
+
+* A pipeline starts with a xref:pipeline/pipeline-sources.adoc[source]
transform and moves rows along hops.
+* Fields are created and changed as rows travel; you don't declare them up
front.
+* *Preview* shows you real rows without running or configuring the whole
pipeline.
+* Variables work anywhere Hop accepts a path.
+* *Execution Results* tells you how many rows moved and what went wrong.
+
+Next, we'll build a workflow that runs this pipeline and reacts to whether it
worked.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-workflows.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-workflows.adoc
index b13b8c5aaa..157f861c20 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-workflows.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui-workflows.adoc
@@ -17,81 +17,148 @@ under the License.
[[HopGuiWorkflows]]
:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (7/8): Workflows in Hop are where you'll do most
of the orchestration work: set the stage by fetching files, checking resource
availability etc before your start to do actually run workflows and pipelines.
We'll walk through what it takes to build and run a basic first workflow.
+:description: Getting Started (6/9): build a workflow that runs the pipeline
you just created, and takes a different path depending on whether it worked.
-= Workflows
+= Your first workflow
-In xref:getting-started/hop-concepts.adoc[Concepts], we walked through
workflows, actions and hops.
-Let’s remember:
+A pipeline moves rows.
+A workflow decides *what runs, in what order, and what happens when something
fails*.
-* A _workflow_ is by default a sequential process that has a starting point
and one or more endpoints.
-* An _action_ is an item of workflow functionality that executes pipelines
already implemented or any other element that allows orchestrating the
execution of the data integration processes.
-* A _hop_ in a _workflow_ can conditionally connect _actions_ and determine
which _action_ the _workflow_ needs to execute next.
+That difference is the point of this chapter.
+We'll build a workflow that runs the pipeline from the
xref:getting-started/hop-gui-pipelines.adoc[previous chapter] and reports a
different message depending on the outcome.
-== Create a Workflow
+Remember from xref:getting-started/hop-concepts.adoc[Concepts]:
-There are two ways to create a workflow.
+* A **workflow** is a sequence of **actions** connected by **hops**, with a
start point and one or more end points.
+* Unlike transforms in a pipeline, actions run **one after another**, not in
parallel.
+* A **hop** in a workflow can be conditional: it decides which action runs
next based on whether the previous one succeeded.
-* Click on the New option on the Horizontal toolbar and select the Workflow
option.
+== Create a workflow
-image:getting-started/gs-hop-gui-create-workflow-1.png[]
+Click *New* in the main toolbar and pick *Workflow*, or use *File -> New ->
Workflow*.
-* File -> New -> Workflow
+image:getting-started/gs-hop-gui-create-workflow-1.png[The New option in the
main toolbar]
-image:getting-started/gs-hop-gui-create-workflow-2.png[]
+Your new workflow already contains one action.
-image:getting-started/gs-hop-gui-create-workflow-3.png[]
+image:getting-started/gs-hop-gui-create-workflow-4.png[A new workflow with the
Start action]
-Your new workflow is created, and you’ll see the dialog below.
+NOTE: Hop adds a xref:workflow/actions/start.adoc[Start] action to every new
workflow. It is where execution begins, and it is also where you configure a
schedule if you want the workflow to repeat.
-image:getting-started/gs-hop-gui-create-workflow-4.png[]
+Save the workflow as `first-workflow.hwf`, in the same folder as
`first-pipeline.hpl`.
-NOTE: Note that when you create a workflow, Apache Hop automatically adds
http://localhost:1313/manual/latest/workflow/actions/start.html[Start^] action
by default.
+== Run the pipeline from the workflow
-== Add and connect actions
+Click anywhere on the canvas to open the context dialog, search for `Pipeline`
and add the xref:workflow/actions/pipeline.adoc[Pipeline] action.
-=== Add Actions
+image:getting-started/gs-hop-gui-create-workflow-6.png[The workflow context
dialog]
-Now you are ready to add the first action.
-Click anywhere in the workflow canvas, the area where you’ll see the image
below.
+Open the new action and set:
-image:getting-started/gs-hop-gui-create-workflow-5.png[]
+[options="header"]
+|===
+|Field|Value
+|Action name|`run first pipeline`
+|Pipeline|`+${PROJECT_HOME}/first-pipeline.hpl+`
+|===
-After clicking you will see a dialog:
+This is the step the workflow exists for, so it is worth being precise about
it: the *Pipeline* field is what ties the workflow to the pipeline you built.
+Use the *Browse* button if you prefer to pick the file, but note what Hop
fills in.
-image:getting-started/gs-hop-gui-create-workflow-6.png[]
+`+${PROJECT_HOME}+` is a variable that points at the folder of the current
project.
+Using it instead of an absolute path is what lets the same workflow run on
your laptop, on a colleague's machine and on a server.
+You'll see more of this in xref:getting-started/hop-projects.adoc[Projects and
environments].
-Just like you did with pipelines, use the search box in this dialog to find
the actions you need.
-Click or use the arrow keys and hit enter to add an action to your workflow.
+Now connect *Start* to `run first pipeline`.
+Creating a hop works exactly as it does in a pipeline:
-For now, add a xref:workflow/actions/pipeline.adoc[Pipeline] action to your
workflow.
+* **shift-drag** from one action to the other, or
+* **scroll-drag**, or
+* click the action and use the *Create hop*
image:getting-started/icons/HOP.svg[Create hop,25px,align="bottom"] button in
its context dialog.
-TIP: check the xref:workflow/actions.adoc[full list of actions].
-There are over 50 actions available in Hop 0.70, but you'll soon get to know
the most commonly used ones.
+image:getting-started/gs-hop-gui-create-workflow-add-pipeline-action.png[The
Pipeline action connected to Start]
-=== Create a hop
+== React to the outcome
-Creating a hop is identical to creating a hop in a pipeline:
+A workflow that only runs one thing is not doing much.
+Let's make it respond to what happened.
-* **shift-drag**: while holding down the shift key on your keyboard.
-Click on an action, while holding down your primary mouse button, drag to the
second action.
-Release the primary mouse button and the shift key.
-* **scroll-drag**: scroll-click on an action , while holding down your mouse’s
scroll button, drag to the second action.
-Release the scroll button.
-* click on an action in your pipeline to open the '**click anywhere**' dialog.
-Click the 'Create hop' image:getting-started/icons/HOP.svg[Create
hop,25px,align="bottom"] button and select the action you want to create the
hop to.
+Add two xref:workflow/actions/writetolog.adoc[Write to log] actions:
-image:getting-started/gs-hop-gui-create-workflow-add-pipeline-action.png[]
+[options="header"]
+|===
+|Action name|Log level|Log message
+|`report success`|Basic|`Orders file written.`
+|`report failure`|Error|`The orders pipeline failed.`
+|===
-Save your workflow:
+Create a hop from `run first pipeline` to `report success`, and a second hop
from `run first pipeline` to `report failure`.
-image:getting-started/gs-hop-gui-create-workflow-save.png[]
+By default both hops are green: they fire when the action succeeds.
+Right-click the hop to `report failure` and choose *Failure hop*.
+It turns red.
-At first, we have the Workflow run configuration session.
-A Workflow Run Configuration is a metadata object in Apache Hop and defines
where the workflow is executed.
+You now have the basic shape of every production workflow:
-Select the 'local' workflow runtime configuration that is available by default
in your Hop installation and click 'Launch'.
+....
+Start ──▶ run first pipeline ──✔──▶ report success
+ │
+ └───────────✘──▶ report failure
+....
-image:getting-started/gs-hop-gui-create-workflow-run.png[]
+Right-clicking a hop gives you three choices:
-You’ll now have an execution results pane very similar to the pipeline
execution results.
+[options="header"]
+|===
+|Hop type|Colour|When the next action runs
+|*Unconditional hop*|black|always, whatever the previous action did
+|*Success hop*|green|only when the previous action succeeded
+|*Failure hop*|red|only when the previous action failed
+|===
+
+This is the main thing workflows give you that pipelines cannot: pipelines
have no notion of "and if that fails, do this instead".
+
+== Run the workflow
+
+Save with `CTRL-S`, then press *F8* or click *Run*.
+
+image:getting-started/gs-hop-gui-create-workflow-run.png[The workflow run
options dialog]
+
+A workflow run configuration decides *where* the workflow runs.
+A `local` one is created for you the first time you start Hop Gui.
+Select it and click *Launch*.
+
+The *Execution Results* panel works like the pipeline one.
+Its *Metrics* tab lists one line per action, in the order they ran, with the
comment, result and reason for each.
+
+You should see `Start`, `run first pipeline` and `report success` execute, and
`report failure` skipped.
+In the *Logging* tab you'll find your own message:
+
+[source,shell]
+----
+report success - Orders file written.
+----
+
+== Prove the failure path works
+
+An error path you have never seen run is a guess, not a safety net.
+
+Open `run first pipeline` and change the *Pipeline* field to a file that does
not exist, for example `+${PROJECT_HOME}/nope.hpl+`.
+Run the workflow again.
+
+This time `run first pipeline` fails, `report success` is skipped and your
error message appears in the log.
+The workflow itself is marked as failed, which is what a scheduler or a
xref:hop-server/index.adoc[Hop Server] would pick up.
+
+Set the *Pipeline* field back when you're done.
+
+== What you learned
+
+* Workflows run actions in sequence; pipelines run transforms in parallel.
+* A Pipeline action is how a workflow runs a pipeline, and `+${PROJECT_HOME}+`
is how it stays portable.
+* Hops in a workflow are conditional, and that is where error handling lives.
+* Every new workflow starts from a xref:workflow/actions/start.adoc[Start]
action, which is also where scheduling is configured.
+
+TIP: check the xref:workflow/actions.adoc[full list of actions] for everything
you can orchestrate: files, mail, conditions, scripts, other workflows, and
more.
+
+You have now built and run both Hop file types.
+Next, we'll put them in a project so the paths keep working outside your own
machine.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui.adoc
index f067699de5..1626f44b32 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-gui.adoc
@@ -15,17 +15,17 @@ specific language governing permissions and limitations
under the License.
////
[[HopGui]]
-:imagesdir: ../../assets/images/
+:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (5/8): Hop Gui is Hop's visual development
environment where you design, run and debug your workflows and pipelines. In
this step, we'll take a quick tour of the user interface and describe some of
the functionalities in Hop GUI.
+:description: Getting Started (4/9): Hop Gui is Hop's visual development
environment. A quick tour of the window, the toolbars and the perspectives
before you build your first pipeline.
-= Hop Gui
+= Hop Gui overview
-The Apache Hop GUI is your local development environment to build, run,
preview and debug workflows and pipelines.
+Hop Gui is where you build, run, preview and debug workflows and pipelines.
-== Start the Apache Hop GUI
+Before we start building, here's a short tour so you know what you're looking
at.
-As we've seen in xref:getting-started/hop-tools.adoc[Hop Tools], starting
Hop-Gui is easy:
+== Start Hop Gui
[tabs]
====
@@ -33,51 +33,60 @@ Windows::
+
--
[source,shell]
+----
hop-gui.bat
+----
--
Linux, macOS::
+
--
[source,shell]
-./hop-gui.sh
+----
+./hop-gui.sh
+----
--
====
-== Walking through the Hop GUI
+== The main window
-After starting the Apache Hop GUI, you’ll be presented with the window below.
-
-image:getting-started/gs-hop-gui-overview-1.png[Hop Gui Overview]
+image:getting-started/gs-hop-gui-overview-1.png[The Hop Gui window]
{nbsp} +
-Let's divide the view into two parts:
+There are two toolbars, and everything else is the work area.
-image:getting-started/gs-hop-gui-overview-2.png[Hop Gui Overview]
-
+image:getting-started/gs-hop-gui-overview-2.png[The Hop Gui toolbars]
-* The **main toolbar** offers a range of functions, including creating new
files (such as pipelines and workflows) and adding metadata. It also provides
options for managing projects and environments, as well as tools specifically
for handling pipelines, workflows, and project settings.
-* The **perspectives toolbar** includes switcher icons between the various
perspectives.
+The **main toolbar**, along the top, creates new files (pipelines, workflows,
metadata), opens and saves them, and holds the run and preview controls for
whatever you have open.
+The **perspectives toolbar**, down the left-hand side, switches between the
different things you can do in Hop Gui.
-** image:icons/data_orch.svg[width="24px"]
xref:hop-gui/perspective-data-orchestration.adoc[Data Orchestration]
+At the **bottom left** of the status bar you'll find the active project and
environment.
+You'll use those in xref:getting-started/hop-projects.adoc[Projects and
environments].
-** image:icons/execution.svg[width="24px"]
xref:hop-gui/perspective-execution-information.adoc[Execution Information]
+== Perspectives
-** image:icons/gear.svg[width="24px"]
xref:hop-gui/perspective-configuration.adoc[Configuration]
+A perspective is a mode of the whole window.
+You'll spend nearly all of your time in the first one, Data Orchestration,
which is where pipelines and workflows are edited.
-** image:icons/metadata.svg[width="24px"]
xref:hop-gui/perspective-metadata.adoc[Metadata]
+include::../snippets/hop-gui/perspectives-list.adoc[]
-** image:icons/folder.svg[width="24px"]
xref:hop-gui/perspective-file-explorer.adoc[File explorer]
+Switch perspective by clicking its icon, or cycle through them with
`CTRL-SHIFT-arrow-up` and `CTRL-SHIFT-arrow-down`.
-** image:icons/database.svg[width="24px"]
xref:hop-gui/perspective-database.adoc[Database]
+== Worth knowing early
-** image:icons/plugin.svg[width="24px"]
xref:hop-gui/perspective-plugin.adoc[Plugins explorer]
+Click anywhere::
+Most of Hop Gui is driven by clicking on empty canvas or on an item, which
opens a context dialog with everything you can do from there.
+You'll use it constantly in the next chapter.
-** image:icons/neo4j_logo.svg[width="24px"]
xref:hop-gui/perspective-neo4j.adoc[Neo4j]
+`CTRL-F` searches everything::
+Not just the open file: it searches metadata, pipelines and workflows across
the project.
+See xref:hop-gui/perspective-search.adoc[Search].
-Let's see how we can use Hop Gui to do what Hop is all about: create pipelines
and workflows!
+There are shortcuts for most things::
+See xref:hop-gui/shortcuts.adoc[Keyboard Shortcuts].
-INFO: check the xref:../hop-gui/index.adoc[Hop Gui] section of the Hop
documentation for a more detailed tour of the Hop Gui.
+TIP: the xref:hop-gui/index.adoc[Hop Gui] section of this manual has the
detailed tour, including how to xref:hop-gui/disable-ui-elements.adoc[hide
parts of the UI] for new users.
+Let's build a pipeline.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
index 1867d6699a..6bb033009f 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
@@ -14,26 +14,60 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-[[NetSteps]]
+[[NextSteps]]
:imagesdir: ../../assets/images
:page-pagination:
:page-pagination-no-next:
-:description: Getting Started (8/8): You now know the basic Hop terminology
and tools, and you've created and have run your first pipeline and workflow.
Here's an overview of topics you could explore to continue building your Hop
knowledge.
+:description: Getting Started (9/9): You've built and run a pipeline, a
workflow and a project. Here's where to go next, depending on what you want to
do.
-= Next Steps
+= Next steps
-You now know what Hop is and how to create your first workflows and pipelines.
+You've built a pipeline, orchestrated it from a workflow, handled a failure
path, and put both in a project.
+That is the core loop of working with Hop. Everything else is more of it.
-There's a lot more to discover in Hop.
-Here are a couple of topics you may want to look into:
+Where you go next depends on what you're trying to do.
-* xref:pipeline/pipelines.adoc[Pipelines] takes closer look at the various
aspects of creating and running pipelines, and contains the entire list of
transforms that are at your disposal
-* xref:workflow/workflows.adoc[Workflows] takes a closer look at the various
aspects of create and running workflows, and contains the entire list of
actions that are at your disposal
-* xref:fields-parameters-variables.adoc[Fields, parameters, and variables]
compares the three ways Hop moves data and configuration
-* xref:how-to-guides/change-data-capture.adoc[Change Data Capture] covers the
three ways Hop detects inserts, updates and deletes: log sniffing, filtered
selection, and snapshot comparison.
-* xref:best-practices/index.adoc[Best Practices] covers a number of things you
might want to think about while using Apache Hop.
-* xref:projects/index.adoc[Projects] explains how to work with projects and
environments
-* xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] shows how to start a run from cron, systemd, Jenkins,
a Kubernetes CronJob or Apache Airflow
-* xref:hop-server/deploying.adoc[Deploying Hop Server] covers the three common
ways to get a project onto a running server
-* xref:vfs.adoc[VFS] explains how you can access resources in the 3 main cloud
platforms: AWS, Azure and GCP.
-* xref:logging/logging-basics.adoc[Logging] explains how to configure Hop for
your desired log level and target platform
+== Build more pipelines
+
+* xref:pipeline/pipelines.adoc[Pipelines] - the full picture, including the
complete xref:pipeline/transforms.adoc[list of transforms]
+* xref:pipeline/run-preview-debug-pipeline.adoc[Run, Preview and Debug a
Pipeline] - preview a subset, pause on a condition, step through rows
+* xref:pipeline/errorhandling.adoc[Error Handling] - send failing rows down a
separate hop instead of stopping
+* xref:fields-parameters-variables.adoc[Fields, parameters and variables] -
three things that look alike in the UI and behave differently
+* xref:pipeline/pipeline-unit-testing.adoc[Pipeline Unit Tests] - golden data
sets and assertions for your pipelines
+
+== Orchestrate more
+
+* xref:workflow/workflows.adoc[Workflows] - the full picture, including the
complete xref:workflow/actions.adoc[list of actions]
+* xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] - cron, systemd, Jenkins, Kubernetes CronJob and
Apache Airflow
+
+== Connect to your data
+
+* xref:database/databases.adoc[Databases] - relational connections and the
drivers Hop ships
+* xref:vfs.adoc[VFS] - read and write on AWS, Azure and GCP as if they were
local files
+* xref:how-to-guides/change-data-capture.adoc[Change Data Capture] - log
sniffing, filtered selection and snapshot comparison
+* xref:metadata-types/index.adoc[Metadata types] - everything you can define
once and reuse
+
+== Run it somewhere else
+
+* xref:projects/index.adoc[Projects] - the full reference for projects and
environments
+* xref:hop-server/deploying.adoc[Deploying Hop Server] - three ways to get a
project onto a running server
+* xref:docker-container.adoc[Hop in Docker] - the `apache/hop` image
+*
xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[Pipeline
run configurations] - run the same pipeline on Apache Spark, Apache Flink or
Google Dataflow without changing it
+* xref:logging/logging-basics.adoc[Logging] - log levels and where to send the
output
+
+== Work like a team
+
+* xref:hop-gui/hop-gui-git.adoc[Working with git] - commit, diff and browse
history from inside Hop Gui
+* xref:best-practices/index.adoc[Best Practices] - what experienced Hop
developers wish they had known
+* xref:linting/index.adoc[Linting] - static checks on pipelines, workflows and
metadata, locally or in CI
+
+== Coming from Kettle or PDI?
+
+* xref:hop-vs-kettle/hop-vs-kettle.adoc[Hop vs Kettle] - what changed and why
+* xref:hop-vs-kettle/if-you-know-kettle.adoc[If you know Kettle] - the
vocabulary mapping
+* xref:hop-vs-kettle/import-kettle-projects.adoc[Upgrade Kettle to Hop] -
convert an existing repository
+
+== Get help
+
+Hop is an Apache project, and the community is where questions get answered.
+See https://hop.apache.org/community/[the community pages] for the mailing
lists, chat and issue tracker.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-projects.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-projects.adoc
new file mode 100644
index 0000000000..07b0fb73aa
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-projects.adoc
@@ -0,0 +1,107 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[GettingStartedProjects]]
+:imagesdir: ../../assets/images
+:page-pagination:
+:description: Getting Started (7/9): put your pipeline and workflow in a
project so paths, metadata and configuration keep working on someone else's
machine.
+
+= Projects and environments
+
+So far you have been working in the `default` project, which is what Hop
enables when you haven't chosen anything else.
+That is fine for a first pipeline.
+It stops being fine the moment you want to share your work, run it on a
server, or have a `dev` and a `prd` version of the same connection.
+
+This chapter is the short version.
+xref:projects/projects-environments.adoc[Projects and Environments] has the
full reference.
+
+== What a project gives you
+
+A **project** is a folder plus a `project-config.json`.
+Grouping your work into one buys you three things:
+
+Portable paths::
+`+${PROJECT_HOME}+` always points at the project folder, so
`+${PROJECT_HOME}/first-pipeline.hpl+` resolves on your laptop, on a
colleague's machine and in a container, without anyone editing a path.
+You already used this in the
xref:getting-started/hop-gui-workflows.adoc[Pipeline action].
+
+A home for metadata::
+Database connections, run configurations and every other
xref:projects/metadata.adoc[metadata] object live in the project's metadata
folder.
+They travel with the project instead of sitting in your personal configuration.
+
+Something you can commit::
+A project folder is an ordinary folder of files. It goes into Git as-is, which
is what makes review, history and CI possible.
+
+An **environment** is the other half: it holds the values that differ between
`dev`, `test` and `prd` - hostnames, credentials, file locations - as variables
in one or more property files outside the project.
+Same project, different environment, different target database, and no
branching or editing of pipelines.
+
+== Create a project
+
+The active project is shown at the bottom left of the Hop Gui status bar, next
to a briefcase icon.
+
+image:hop-gui/environment/the-active-project-at-the-bottom-left.png[The active
project at the bottom left]
+
+Click it and choose *Add project...*, then *In a folder*.
+
+image:hop-gui/environment/add-project-menu.png[The Add project menu]
+
+In the project properties dialog, fill in the *Basic* tab:
+
+[options="header"]
+|===
+|Field|Value
+|Name|`getting-started`
+|Home folder|a new, empty folder of your choice
+|===
+
+Everything else can keep its default.
+The *Folders* tab already points metadata, datasets and unit tests at
`+${PROJECT_HOME}+` subfolders, and the *Parent project* tab inherits from
`default`, which is where the `local` run configurations you have been using
come from.
+
+Click *OK*.
+Hop creates the folder structure and switches to the new project.
+
+TIP: *From version control* clones a project straight from Git instead, which
is usually how you pick up a project someone else created.
+
+== Move your work into it
+
+Copy `first-pipeline.hpl` and `first-workflow.hwf` into the new project
folder, then open them from there.
+
+Because you wrote the Pipeline action's path as
`+${PROJECT_HOME}/first-pipeline.hpl+` rather than an absolute path, the
workflow still finds its pipeline.
+Nothing to edit.
+
+That is the whole point, and it is worth doing once deliberately: run the
workflow again from the new project and watch it work unchanged.
+
+NOTE: if you had typed an absolute path such as
`/home/you/first-pipeline.hpl`, this is exactly where it would have broken.
Prefer `+${PROJECT_HOME}+` from the start.
+
+== Add an environment
+
+The active environment sits next to the project in the status bar, with a cube
icon.
+Click it and choose *Add environment...*.
+
+image:hop-gui/environment/environment-add.png[The Add environment menu]
+
+Give it a name and a *Purpose* (`Development`, `Testing`, `Production` and
others are predefined), and point it at one or more property files holding your
environment-specific variables.
+
+Those variables then resolve anywhere Hop accepts one: in a transform field, a
database connection's hostname, a file path.
+Switching environment switches the values, and nothing inside the project
changes.
+
+== What you learned
+
+* A project is a folder of files that can be shared, committed and deployed as
a unit.
+* `+${PROJECT_HOME}+` is what keeps a project working somewhere other than the
machine it was built on.
+* Metadata belongs to the project, not to you.
+* Environments hold the per-stage values, so one project can run against dev,
test and production.
+
+TIP: the xref:hop-gui/hop-gui-git.adoc[git integration] in Hop Gui works on
the project folder, so you can commit, diff and browse history without leaving
Hop.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
index 698ff25ce1..d47856ea10 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-tools.adoc
@@ -17,413 +17,153 @@ under the License.
[[HopTools]]
:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (4/8): Hop comes with a number of GUI and
command line tools to not only design and run workflows and pipelines, but also
to manage your projects and environments, to search import, translate and more.
+:description: Getting Started (8/9): Hop ships a set of command line tools
next to Hop Gui, for running, configuring, searching, importing and serving
your projects.
-= Hop Tools
+= The Hop toolbox
-Hop contains a number of tools that are optimized for the various tasks Hop
developers and administrators need to perform.
+You'll spend most of your time in Hop Gui, but a project that only runs from a
desktop GUI is not much use in production.
-== Hop GUI
+The split is roughly this: **designing** pipelines and workflows and
**creating** metadata is what Hop Gui is for.
+**Configuring** and **running** them is where the command line takes over, and
that is the half a scheduler, a container or a CI/CD pipeline needs.
-include::../snippets/hop-tools/hop-gui.adoc[]
+== One command to start them all
-TIP: Hop Gui is designed to be platform independent. +
-xref:../hop-gui/hop-web.adoc[Hop Web] is a version of Hop Gui that works on
browsers and mobile devices.
-
-The command below starts Hop GUI for your operating system.
-
- +
+Since Hop 2.x there is a single `hop` launcher in the installation folder.
+It replaces the individual `hop-run`, `hop-conf`, `hop-gui`, ... scripts, and
plugins can add their own commands to it.
[tabs]
====
Windows::
+
--
-[source,bash]
-----
-hop-gui.bat
-----
---
-
-Linux, macOS::
-+
---
-[source,bash]
-----
-./hop-gui.sh
-----
---
-====
-
-Expected output: Hop Gui starts.
-
-== Hop Conf
-
-include::../snippets/hop-tools/hop-conf.adoc[]
-
-The example `hop-conf` command below lists your available projects.
-
- +
-
-[tabs]
-====
-
-Windows::
-+
---
-[source,bash]
-----
-hop-conf.bat -pl
-----
-
-Expected output:
[source,shell]
----
-C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\* -Djava.library.path=lib\core;lib\beam
-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Conf -DHOP_AUTO_CREATE_CONFIG=Y
org.apache.hop.config.HopConfig -pl
-===[Starting
HopConfig]=========================================================
-Projects:
- default : config/projects/default
- Configuration file:
file:\C:\<YOUR_PATH>\hop\config\projects\default\project-config.json
- samples : config/projects/samples
- Parent project: default
- Configuration file:
file:\C:\<YOUR_PATH>\hop\config\projects\samples\project-config.json
+hop.bat help
----
-
--
Linux, macOS::
+
--
-[source,bash]
-----
-./hop-conf.sh -pl
-----
-
-Expected output:
-
-[source,bash]
-----
-Projects:
- default : config/projects/default
- Configuration file:
file:<YOUR_PATH>/hop/config/projects/default/project-config.json
- samples : config/projects/samples
- Parent project: default
- Configuration file:
file:<YOUR_PATH>/hop/config/projects/samples/project-config.json
-
-----
---
-
-
-====
-
-See the xref:../hop-tools/hop-conf/hop-conf.adoc[Hop Tools -> Hop Conf]
documentation for more information.
-
-== Hop Encrypt
-
-include::../snippets/hop-tools/hop-encrypt.adoc[]
-
- +
-
-The example below prints the encrypted Hop password `mypassword`.
-
- +
-
-[tabs]
-====
-
-Windows::
-+
---
-[source,bash]
-----
-hop-encrypt.bat -hop mypassword
-----
-
-Expected output:
-
[source,shell]
----
-===[Environment Settings - hop-encrypt.bat]====================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS=-Xmx64m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=GUI
-Command to start Hop will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\* -Djava.library.path=lib\core;lib\beam
-Xmx64m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=GUI org.apache.hop.core.encryption.Encr -hop mypassword
-===[Starting
HopEncrypt]=========================================================
-Encrypted 2be98afc86aa79f9dbb18bd63c99dbdde
+./hop help
----
--
-
-Linux, macOS::
-+
---
-[source,bash]
-----
-./hop-encrypt.sh -hop mypassword
-----
-
-Expected output:
-[source, shell]
-----
-Encrypted 2be98afc86aa79f9dbb18bd63c99dbdde
-----
-
---
-
-====
-
-See the xref:../hop-tools/hop-encrypt.adoc[Hop Tools -> Hop Encrypt]
documentation for more information.
-
-== Hop Run
-
-include::../snippets/hop-tools/hop-run.adoc[]
-
-The example below runs the `switch-case-basic.hpl` pipeline in the samples
project (`-j`) with the
xref:pipeline/pipeline-run-configurations/native-local-pipeline-engine.adoc[local
native pipeline run configuration] (`-r`).
-
- +
-
-[tabs]
====
-Windows::
-+
---
-[source,shell]
-----
-hop-run.bat -r local -j samples -f
${PROJECT_HOME}/transforms/switch-case-basic.hpl
-----
-
-Expected output:
+`help` lists the commands available in *your* installation, which depends on
the plugins you have installed:
[source,shell]
----
-C:\<YOUR_PATH>\hop>echo off
-===[Environment Settings - hop-run.bat]===================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS="-Xmx2048m" -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Run -DHOP_AUTO_CREATE_CONFIG=Y
-Consolidated parameters to pass to HopRun are
--r local -j samples -f ${PROJECT_HOME}/transforms/switch-case-basic.hpl
-Command to start HopRun will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\*
--Djava.library.path=lib\core;lib\beam "-Xmx2048m" -DHOP_AUDIT_FOLDER=.\audit
--DHOP_PLATFORM_OS=Windows -DHOP_PLATFORM_RUNTIME=Run
-DHOP_AUTO_CREATE_CONFIG=Y org.apache.hop.run.HopRun
--r local -j samples -f ${PROJECT_HOME}/transforms/switch-case-basic.hpl
-===[Starting HopRun]=========================================================
-2022/12/16 07:07:47 - HopRun - Enabling project 'samples'
-2022/12/16 07:07:47 - HopRun - Starting pipeline:
config/projects/samples/transforms/switch-case-basic.hpl
-2022/12/16 07:07:47 - switch-case-basic - Executing this pipeline using the
Local Pipeline Engine with run configuration 'local'
-2022/12/16 07:07:47 - switch-case-basic - Execution started for pipeline
[switch-case-basic]
-2022/12/16 07:07:47 - Test Data.0 - Finished processing (I=0, O=0, R=0, W=5,
U=0, E=0)
-2022/12/16 07:07:47 - Switch id.0 - Finished processing (I=0, O=0, R=5, W=5,
U=0, E=0)
-2022/12/16 07:07:47 - Output 2.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 07:07:47 - Output 1.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 07:07:47 - Output 4.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 07:07:47 - Output default.0 - Finished processing (I=0, O=0, R=1,
W=1, U=0, E=0)
-2022/12/16 07:07:47 - Output 3.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 07:07:47 - switch-case-basic - Pipeline duration : 0.435 seconds [
0.435" ]
+Commands:
+ help Display help information about the specified command.
+ conf Configure Hop
+ encrypt Encrypt secrets
+ gui The Hop GUI
+ import Import metadata
+ run Run a pipeline or workflow
+ search Search in Hop metadata
+ server Run a Hop server
+ setup Configure Hop environment variables
----
---
+Two options are worth knowing before anything else, because nearly every
command takes them:
-Linux, macOS::
-+
---
-[source,shell]
-----
-./hop-run.sh -r local -j samples -f
${PROJECT_HOME}/transforms/switch-case-basic.hpl
-----
+[options="header"]
+|===
+|Option|What it does
+|`-j`, `--project`|enable a project before the command runs
+|`-e`, `--environment`|enable an environment before the command runs
+|===
-Expected output:
+So running your workflow from the
xref:getting-started/hop-projects.adoc[previous chapter] looks like this:
[source,shell]
----
-2022/12/16 06:59:03 - HopRun - Enabling project 'samples'
-2022/12/16 06:59:03 - HopRun - Relative path filename specified:
config/projects/samples//transforms/switch-case-basic.hpl
-2022/12/16 06:59:03 - HopRun - Starting pipeline:
config/projects/samples//transforms/switch-case-basic.hpl
-2022/12/16 06:59:03 - switch-case-basic - Executing this pipeline using the
Local Pipeline Engine with run configuration 'local'
-2022/12/16 06:59:03 - switch-case-basic - Execution started for pipeline
[switch-case-basic]
-2022/12/16 06:59:04 - Test Data.0 - Finished processing (I=0, O=0, R=0, W=5,
U=0, E=0)
-2022/12/16 06:59:04 - Switch id.0 - Finished processing (I=0, O=0, R=5, W=5,
U=0, E=0)
-2022/12/16 06:59:04 - Output default.0 - Finished processing (I=0, O=0, R=1,
W=1, U=0, E=0)
-2022/12/16 06:59:04 - Output 1.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 06:59:04 - Output 4.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 06:59:04 - Output 3.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 06:59:04 - Output 2.0 - Finished processing (I=0, O=0, R=1, W=1,
U=0, E=0)
-2022/12/16 06:59:04 - switch-case-basic - Pipeline duration : 0.668 seconds [
0.668" ]
+./hop run -j getting-started -r local -f '${PROJECT_HOME}/first-workflow.hwf'
----
---
-====
+NOTE: the older `hop-run.sh`, `hop-conf.sh` and friends still ship and still
work. New examples in this manual use `hop <command>`.
-xref::../hop-run/index.adoc[Hop Tools -> Hop Run]
+== The tools
-== Hop Search
+[options="header",cols="20%,50%,30%"]
+|===
+|Tool|What you use it for|Read more
-include::../snippets/hop-tools/hop-search.adoc[]
+|`hop gui`
+|The visual development environment: build, preview, run and debug pipelines
and workflows.
+|xref:hop-tools/hop-gui.adoc[Hop Gui]
-The example below searches in the samples project `-j` for `switch-case`.
+|`hop run`
+|Run a pipeline or workflow. This is what a scheduler, a CI job or a container
calls.
+|xref:hop-run/index.adoc[Hop Run]
- +
+|`hop conf`
+|Manage projects, environments and cloud configuration from a script.
+|xref:hop-tools/hop-conf/hop-conf.adoc[Hop Conf]
-[tabs]
-====
+|`hop setup`
+|Point Hop at a user-level config folder so your projects and settings survive
an upgrade.
+|xref:hop-tools/hop-setup.adoc[Hop Setup]
-Windows::
-+
---
-[source, shell]
-----
-hop-search.bat -j samples -x switch-case
-----
+|`hop server`
+|Run a Hop Server to execute pipelines and workflows remotely, over REST.
+|xref:hop-server/index.adoc[Hop Server]
-Expected output:
-[source,shell]
-----
-C:\<YOUR_PATH>\hop>echo off
-===[Environment Settings - hop-search.bat]===================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=Search
--DHOP_AUTO_CREATE_CONFIG=Y
-Command to start Hop will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\*
--Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit
-DHOP_PLATFORM_OS=Windows
--DHOP_PLATFORM_RUNTIME=Search -DHOP_AUTO_CREATE_CONFIG=Y
org.apache.hop.config.HopConfig -j samples -x switch-case
-===[Starting
HopConfig]=========================================================
-Enabling project 'samples'
-Searching in project : samples
-Searching for [switch-case] Case sensitive? true Regular expression? true
-Searching in location : Project samples
------------------------------------------------------------------------------------
-file:///C:/<YOUR_PATH>/hop/config/projects/samples/beam/pipelines/switch-case.hpl
: null(switch-case) : matching property value: switch-case
file:///C:/<YOUR_PATH>/hop/config/projects/samples/beam/pipelines/switch-case.hpl
: switch-case(switch-case) : matching property value: switch-case
-file:///C:/<YOUR_PATH>/hop/config/projects/samples/beam/pipelines/switch-case.hpl
: switch-case(switch-case) : pipeline transform property : filePrefix
-----
---
-
-Linux, macOS::
-+
---
-[source,shell]
-----
-./hop-search.sh -j samples -x switch-case
-----
-
-Expected output:
-[source,shell]
-----
-Enabling project 'samples'
-Searching in project : samples
-Searching for [switch-case] Case sensitive? true Regular expression? true
-Searching in location : Project samples
------------------------------------------------------------------------------------
-file://<YOUR_PATH>/hop/config/projects/samples/beam/pipelines/switch-case.hpl
: null(switch-case) : matching property value: switch-case
-file://<YOUR_PATH>/hop/config/projects/samples/beam/pipelines/switch-case.hpl
: switch-case(switch-case) : matching property value: switch-case
-file://<YOUR_PATH>/hop/config/projects/samples/beam/pipelines/switch-case.hpl
: switch-case(switch-case) : pipeline transform property : filePrefix
-----
---
-
-====
+|`hop search`
+|Search across all the metadata in a project: which pipeline uses this
connection, where is this field set.
+|xref:hop-tools/hop-search.adoc[Hop Search]
-See the xref:../hop-tools/hop-search.adoc[Hop Tools -> Hop Search]
documentation for more information.
+|`hop encrypt`
+|Obfuscate a password so it isn't stored in clear text in a configuration file.
+|xref:hop-tools/hop-encrypt.adoc[Hop Encrypt]
-== Hop Server
+|`hop import`
+|Convert a Kettle/PDI repository into a Hop project.
+|xref:hop-vs-kettle/import-kettle-projects.adoc[Import Kettle projects]
-include::../snippets/hop-tools/hop-server.adoc[]
+|`hop doc`
+|Generate documentation for your own Hop project, in Markdown or HTML.
+|xref:hop-tools/hop-doc.adoc[Hop Doc]
+|===
-The example command below starts the most basic server on your machine
`localhost` on port `8081`.
+Some tools are optional plugins rather than part of the standard client:
- +
+[options="header",cols="20%,50%,30%"]
+|===
+|Tool|What you use it for|Read more
-[tabs]
-====
-Windows::
-+
---
-[source,bash]
-----
-hop-server.bat localhost 8081
-----
+|Marketplace
+|Install optional plugins into a running Hop client from a Maven repository.
+|xref:hop-tools/hop-marketplace.adoc[Hop Marketplace]
-Expected output:
-[source, shell]
-----
-C:\<YOUR_PATH\hop>echo off
-===[Environment Settings - hop-server.bat]====================================
-Java identified as "C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java"
-HOP_OPTIONS=-Xmx2048m -DHOP_AUDIT_FOLDER=.\audit -DHOP_PLATFORM_OS=Windows
-DHOP_PLATFORM_RUNTIME=GUI -DHOP_AUTO_CREATE_CONFIG=Y
-Command to start Hop will be:
-"C:\Program Files\Microsoft\jdk-11.0.17.8-hotspot\\bin\java" -classpath
lib\core\*;lib\beam\*;lib\swt\win64\*
--Djava.library.path=lib\core;lib\beam -Xmx2048m -DHOP_AUDIT_FOLDER=.\audit
-DHOP_PLATFORM_OS=Windows
--DHOP_PLATFORM_RUNTIME=GUI -DHOP_AUTO_CREATE_CONFIG=Y
org.apache.hop.www.HopServer localhost 8081
-===[Starting
HopServer]=========================================================
-2022/12/16 07:33:44 - HopServer - Enabling project 'default'
-2022/12/16 07:33:44 - HopServer - Installing timer to purge stale objects
after 1440 minutes.
-2022/12/16 07:33:44 - HopServer - Created listener for webserver @ address :
localhost:8081
-----
+|`hop lint`
+|Static analysis over a project: check pipelines, workflows and metadata
without running them. Useful as a build step.
+|xref:hop-tools/hop-lint.adoc[Hop Lint]
-Your Hop Server is available at \http://localhost:8081, log on with the
default username `cluster`, password `cluster`.
+|PyHop
+|Drive Hop from Python.
+|xref:hop-tools/hop-python/hop-python.adoc[Hop Python]
-Hit `CTRL-C` to stop the server.
+|Hop Arrow
+|Expose pipeline output over an Apache Arrow Flight server.
+|xref:hop-tools/hop-arrow-command.adoc[Hop Arrow]
+|===
---
+TIP: Hop Gui also runs in a browser. See xref:hop-gui/hop-web.adoc[Hop Web]
and xref:hop-web-docker.adoc[Hop Web in Docker].
-Linux, macOS::
-+
---
-[source,bash]
-----
-./hop-server.sh localhost 8081
-----
+== Try one
-Expected output:
+The quickest way to see the command line in action is to run something you
already built:
[source,shell]
----
-2022/12/16 07:20:19 - HopServer - Enabling project 'default'
-2022/12/16 07:20:19 - HopServer - Installing timer to purge stale objects
after 1440 minutes.
-2022/12/16 07:20:19 - HopServer - Created listener for webserver @ address :
localhost:8081
+./hop run -j getting-started -r local -f '${PROJECT_HOME}/first-pipeline.hpl'
----
-Your Hop Server is available at \http://localhost:8081, log on with the
default username `cluster`, password `cluster`.
-
-Hit `CTRL-C` to stop the server.
-
---
-
-====
-
-Read more about xref::../hop-server/index.adoc[Hop Server]
-
-== Hop Import
-
-include::../snippets/hop-tools/hop-import.adoc[]
-
- +
-
-The example command below prints the `hop-import` help. Check the
xref:hop-vs-kettle/import-kettle-projects.adoc[] docs for more information.
-
- +
-
-[tabs]
-====
-
-Windows::
-+
---
-[code, shell]
-----
-hop-import.bat
-----
---
-
-Linux, macOS::
-+
---
-[code,shell]
-----
-./hop-import.sh
-----
---
-
-====
-
-Read more at xref:hop-vs-kettle/import-kettle-projects.adoc[Import Kettle
(PDI) projects]
+You get the same log you saw in the *Logging* tab of Hop Gui, on stdout.
+That is the whole deployment story for a lot of Hop projects: the same file,
the same run configuration, started by cron, a Kubernetes CronJob or Airflow
instead of by you.
+See xref:how-to-guides/scheduling-workflows-and-pipelines.adoc[Scheduling
workflows and pipelines] for the details.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-what-is-hop.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-what-is-hop.adoc
index a7a052ca0b..2661521449 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-what-is-hop.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-what-is-hop.adoc
@@ -17,43 +17,94 @@ under the License.
[[WhatIsHop]]
:imagesdir: ../../assets/images
:page-pagination:
-:description: Getting Started (1/8): before we're diving into Hop, we'll have
a quick look at the visual design, metadata, kernel + plugin architecture and
runtimes
+:description: Getting Started (1/9): Apache Hop is a metadata-driven data
integration and orchestration platform. Pipelines and workflows are designed
visually and then managed like any other software project.
= What is Apache Hop?
-== Visual Design and Metadata
+Apache Hop is a **metadata-driven data integration and data orchestration
platform**.
-Apache Hop, short for **H**op **O**rchestration **P**latform, is a data
orchestration and data engineering platform that aims to facilitate all aspects
of data and metadata orchestration.
-Hop lets you focus on the problem you're trying to solve without technology
getting in the way.
-Simple tasks should be easy, complex tasks need to be possible.
+== Metadata driven
-Hop allows data professionals to work **visually**, using **metadata** to
describe how data should be processed.
-Visual design enables data developers to focus on *what* they want to do
instead of *how* that task needs to be done.
-This focus on the task at hand lets Hop developers be more productive than
they would be when writing code.
+Everything in Hop is metadata.
-== Architecture
+The pipelines that read, combine and transform your data are metadata.
+So are the workflows that orchestrate them, the connections to the databases,
files and services you work with, and the run configurations that decide where
and how any of it executes.
-Hop was designed to be as flexible as possible: at the core is the small but
powerful Hop engine.
-All functionality is added through plugins: the default Hop installation comes
with about 400 plugins.
-You can remove or add third-party plugins according to your needs to tailor
Hop to be exactly what you need.
-Hop is designed to work in any scenario, from IoT to huge volumes of data,
on-prem, in the cloud, on a bare OS or in containers and kubernetes.
+Nothing in a Hop pipeline is bound to a specific database, file format or
processing engine.
+A pipeline describes *what* should happen to your data; the metadata around it
decides *where* that happens and *against what*.
+That separation is what lets the same pipeline run against a developer's local
database and a production warehouse, or on your laptop and on a Spark cluster,
without being rewritten.
-== Flexible Runtimes
+Hop takes this a step further with
xref:pipeline/metadata-injection.adoc[metadata injection]: a template pipeline
that receives its metadata at runtime, so one pipeline can process a hundred
different files or tables instead of you building a hundred near-identical
pipelines.
-Hop developers create **workflows** and **pipelines** in a visual development
environment called Hop Gui.
-These workflows and pipelines can be executed on a variety of engines:
workflows and pipelines can run on the native Hop engine, both locally and
remotely.
-Pipelines can also run on Apache Spark, Apache Flink and Google Dataflow
through the Apache Beam runtime configurations. +
+== Built to fit your architecture
-In workflows and pipelines, hundreds of operations can be applied on the data:
read from and write to a variety of source and target platforms, but also
combine, enrich, clean and in many other ways manipulate data.
-Depending on the engine and selected functionality, your data can be processed
in batch, streaming or in a batch/streaming hybrid.
+Hop is designed to blend into the data architecture you already run:
relational databases, files, object stores on AWS, Azure and Google Cloud,
NoSQL and graph databases, message queues, REST and SOAP services, SaaS
platforms.
+Hop connects them and moves data between them.
-== Use Cases
+That is also why Hop is deliberately lightweight.
+At the core is a small, fast engine; everything else is a plugin, and you add
or remove plugins to make your installation exactly what you need and nothing
more.
+The plugin architecture is open, so you can write your own and load them the
same way.
-A number of common use cases for Hop are:
+The result runs anywhere from an edge device in an IoT scenario to a cluster
processing very large volumes, on bare metal, in a container, or on Kubernetes.
-* Loading large data sets into databases taking advantage of the cloud,
clustered, and massively parallel processing environments.
-* Data warehouse population with built-in support for Slowly Changing
Dimensions (SCD), xref:how-to-guides/change-data-capture.adoc[Change Data
Capture (CDC)], and surrogate key creation.
-* Integrate between diverse data architectures, combining relational
databases, files, NoSQL databases like Neo4j, MongoDB, Cassandra etc
-* Data migration between different databases and applications.
-* Data profiling and data cleansing.
+== Designed visually, managed like software
+Hop developers build pipelines and workflows **visually**, in a full
development environment called xref:hop-gui/index.adoc[Hop Gui], available on
Windows, macOS and Linux and in the browser as xref:hop-gui/hop-web.adoc[Hop
Web].
+
+Visual design means you describe the problem instead of the plumbing, and it
means the next person to open your pipeline can see what it does at a glance
instead of reading through hundreds of lines of code.
+
+What Hop does *not* do is trap that work in a proprietary tool.
+A pipeline is a plain file on disk, and a project is an ordinary folder of
them.
+From that point on, a Hop project is managed like any other software project:
+
+Version control::
+Projects and environments live in Git. Branch, review, diff and roll back like
any other repository. Hop Gui has xref:hop-gui/hop-gui-git.adoc[git
integration] built in, so you can do most of it without leaving the tool.
+
+Unit tests::
+xref:pipeline/pipeline-unit-testing.adoc[Pipeline unit tests] validate a
pipeline's output against a golden data set. Logging tells you a pipeline ran;
a unit test tells you it produced the right data. Regression tests keep a fixed
bug fixed.
+
+Static analysis::
+xref:linting/index.adoc[Linting] checks pipelines, workflows and metadata
without running them or connecting to anything, so it is safe to run on every
commit.
+
+CI/CD::
+Everything Hop needs to run is on the command line, so building, testing and
deploying a Hop project is an ordinary build pipeline. Promote the same project
from development through test to production by pointing it at a different
environment.
+
+== Flexible runtimes
+
+You design a pipeline once and choose at runtime where it executes.
+A
xref:pipeline/pipeline-run-configurations/pipeline-run-configurations.adoc[run
configuration] is metadata like everything else, so switching engines does not
mean redesigning anything.
+
+Workflows run on Hop's own engine,
xref:pipeline/pipeline-run-configurations/native-local-pipeline-engine.adoc[locally]
or on a remote xref:hop-server/index.adoc[Hop Server].
+
+Pipelines have more options:
+
+[options="header",cols="30%,70%"]
+|===
+|Engine|What it's for
+
+|xref:pipeline/pipeline-run-configurations/native-local-pipeline-engine.adoc[Native
local]
+|The default. Runs on the machine you're on, and handles a very large share of
real workloads on its own.
+
+|xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Native
remote]
+|The same engine on a xref:hop-server/index.adoc[Hop Server], with
xref:pipeline/pipeline-run-configurations/native-load-balancing-pipeline-engine.adoc[load
balancing] across several servers when you need it.
+
+|xref:pipeline/pipeline-run-configurations/native-spark-pipeline-engine.adoc[Native
Spark]
+|Executes the pipeline directly as a Spark job, without translating it through
another framework. Runs locally, on a cluster through `spark-submit`, and on
xref:pipeline/spark/databricks.adoc[Databricks]. This is also the path to
xref:pipeline/spark/lakehouse.adoc[Delta Lake and Apache Iceberg] tables.
+
+|xref:pipeline/beam/getting-started-with-beam.adoc[Apache Beam]
+|Mainly used to run pipelines on
xref:pipeline/pipeline-run-configurations/beam-dataflow-pipeline-engine.adoc[Google
Cloud Dataflow]. Beam run configurations for
xref:pipeline/pipeline-run-configurations/beam-spark-pipeline-engine.adoc[Spark]
and
xref:pipeline/pipeline-run-configurations/beam-flink-pipeline-engine.adoc[Flink]
are also available.
+|===
+
+Depending on the engine and the transforms you use, data is processed in
batch, streaming, or a mix of the two.
+
+== Use cases
+
+Hop is used for, among many other things:
+
+* Populating and maintaining data warehouses, with built-in support for Slowly
Changing Dimensions (SCD), xref:how-to-guides/change-data-capture.adoc[Change
Data Capture (CDC)] and surrogate keys.
+* Loading large data sets into databases, data warehouses and cloud object
storage.
+* Building and maintaining xref:pipeline/spark/lakehouse.adoc[lakehouse
tables] on Delta Lake and Apache Iceberg.
+* Integrating across diverse architectures: relational databases, files,
object stores, NoSQL and graph databases such as MongoDB, Cassandra and Neo4j,
REST and SOAP services, and SaaS platforms.
+* Processing streaming data, for example from Apache Kafka, on its own or
alongside batch.
+* Migrating data between databases and applications.
+* Data profiling, validation and cleansing.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/getting-started/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/index.adoc
index d0c41d40c7..48589a7a7c 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/getting-started/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/getting-started/index.adoc
@@ -24,17 +24,20 @@ under the License.
This guide walks you through the Hop basics.
-We'll focus on the core knowledge you need to move around in Hop, without
going in detail.
-At the end of this guide, there will be links to dive deeper into various Hop
topics.
+It is built around one small example that you'll carry from chapter to
chapter: a pipeline that creates and writes data, a workflow that runs it and
handles failure, and a project that holds both.
+By the end you'll have built and run something real, and you'll know where to
look for everything else.
-This is what we'll cover:
+Set aside about an hour.
+All you need is a machine with Java on it.
-* xref:getting-started/hop-what-is-hop.adoc[What is Hop?]
-* xref:getting-started/hop-concepts.adoc[Concepts]
-* xref:getting-started/hop-download-install.adoc[Download & Install]
-* xref:getting-started/hop-tools.adoc[Tools]
-* xref:getting-started/hop-gui.adoc[Hop Gui overview]
-* xref:getting-started/hop-gui-pipelines.adoc[Pipelines]
-* xref:getting-started/hop-gui-workflows.adoc[Workflows]
-* xref:getting-started/hop-next-steps.adoc[Next Steps]
+This is what we'll cover:
+. xref:getting-started/hop-what-is-hop.adoc[What is Hop?] - what the platform
is for
+. xref:getting-started/hop-download-install.adoc[Download & Install] - get it
running
+. xref:getting-started/hop-concepts.adoc[Concepts] - the vocabulary
+. xref:getting-started/hop-gui.adoc[Hop Gui overview] - find your way around
+. xref:getting-started/hop-gui-pipelines.adoc[Your first pipeline] - build,
preview and run it
+. xref:getting-started/hop-gui-workflows.adoc[Your first workflow] -
orchestrate it
+. xref:getting-started/hop-projects.adoc[Projects and environments] - make it
portable
+. xref:getting-started/hop-tools.adoc[The Hop toolbox] - run it without the GUI
+. xref:getting-started/hop-next-steps.adoc[Next Steps] - where to go from here
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
index 1c031ed6cc..87eeed35e4 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
@@ -26,22 +26,4 @@ Use the keyboard shortcuts `CTRL-SHIFT-arrow-up` and
`CTRL-SHIFT-arrow-down` to
Here are the available perspectives:
-* image:icons/data_orch.svg[width="24px"]
xref:hop-gui/perspective-data-orchestration.adoc[Data Orchestration]
-
-* image:icons/execution.svg[width="24px"]
xref:hop-gui/perspective-execution-information.adoc[Execution Information]
-
-* image:icons/gear.svg[width="24px"]
xref:hop-gui/perspective-configuration.adoc[Configuration]
-
-* image:icons/metadata.svg[width="24px"]
xref:hop-gui/perspective-metadata.adoc[Metadata]
-
-* image:icons/folder.svg[width="24px"]
xref:hop-gui/perspective-file-explorer.adoc[File explorer]
-
-* image:icons/database.svg[width="24px"]
xref:hop-gui/perspective-database.adoc[Database] (also as a floating window or
bottom-dock tab)
-
-* image:icons/ai-provider.svg[width="24px"]
xref:hop-gui/perspective-ai-advisor.adoc[AI Assistant] (also as a floating
window or bottom-dock tab; off by default)
-
-* image:icons/plugin.svg[width="24px"]
xref:hop-gui/perspective-plugin.adoc[Plugins explorer]
-
-* image:icons/neo4j_logo.svg[width="24px"]
xref:hop-gui/perspective-neo4j.adoc[Neo4j]
-
-TIP: Project-wide search is not a left-toolbar perspective. Open it with
`CTRL-F` (*Edit → Find*). See xref:hop-gui/perspective-search.adoc[Search].
\ No newline at end of file
+include::../snippets/hop-gui/perspectives-list.adoc[]
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-gui/perspectives-list.adoc
similarity index 73%
copy from docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
copy to
docs/hop-user-manual/modules/ROOT/pages/snippets/hop-gui/perspectives-list.adoc
index 1c031ed6cc..f35d87c5b3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-gui/perspectives.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/hop-gui/perspectives-list.adoc
@@ -14,18 +14,6 @@ KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
////
-
-= Hop Gui Perspectives
-:imagesdir: ../assets/images
-:description: Hop Gui provides a number of perspectives for a variety of
tasks. You'll spend most of your time in the data orchestration perspective,
building workflows and pipelines, but other perspectives for metadata
management, file management and more offer tons of functionality.
-
-Hop comes with a number of perspectives built in.
-
-Change the current perspectives by clicking on the icon in the left-hand
toolbar, or through the corresponding keyboard shortcut.
-Use the keyboard shortcuts `CTRL-SHIFT-arrow-up` and `CTRL-SHIFT-arrow-down`
to cycle through perspectives.
-
-Here are the available perspectives:
-
* image:icons/data_orch.svg[width="24px"]
xref:hop-gui/perspective-data-orchestration.adoc[Data Orchestration]
* image:icons/execution.svg[width="24px"]
xref:hop-gui/perspective-execution-information.adoc[Execution Information]
