This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 282ba3e937 issue #3062: document Hop Server deployment patterns (#7920)
282ba3e937 is described below
commit 282ba3e937a5e7042956b8c50c835af5945c2f10
Author: Matt Casters <[email protected]>
AuthorDate: Thu Aug 13 09:10:07 2026 +0200
issue #3062: document Hop Server deployment patterns (#7920)
Add a deployment manual for getting a git-hosted project onto Hop
Server (image, export resources, git checkout), document the
short-lived container alternative, and add the missing OpenSearch
execution information location page.
Co-authored-by: Hans Van Akelyen <[email protected]>
---
.../modules/ROOT/pages/docker-container.adoc | 6 +-
.../ROOT/assets/files/hop-server/Dockerfile | 28 ++
.../assets/files/hop-server/ci-build-image.yml | 54 ++++
.../assets/files/hop-server/docker-compose.yaml | 40 +++
.../assets/files/hop-server/hop-server.service | 39 +++
docs/hop-user-manual/modules/ROOT/nav.adoc | 7 +
.../ROOT/pages/getting-started/hop-next-steps.adoc | 1 +
.../modules/ROOT/pages/hop-run/index.adoc | 4 +-
.../pages/hop-server/deploy-export-resources.adoc | 180 +++++++++++++
.../ROOT/pages/hop-server/deploy-git-checkout.adoc | 138 ++++++++++
.../pages/hop-server/deploy-project-image.adoc | 299 +++++++++++++++++++++
.../modules/ROOT/pages/hop-server/deploying.adoc | 173 ++++++++++++
.../modules/ROOT/pages/hop-server/index.adoc | 4 +
.../apache-hop-web-services-docker.adoc | 2 +
.../modules/ROOT/pages/how-to-guides/index.adoc | 1 +
.../how-to-guides/run-hop-in-apache-airflow.adoc | 6 +
.../metadata-types/elastic-location-type.adoc | 11 +-
.../execution-information-location.adoc | 11 +-
.../modules/ROOT/pages/metadata-types/index.adoc | 1 +
.../metadata-types/opensearch-location-type.adoc | 96 +++++++
.../native-remote-pipeline-engine.adoc | 3 +
.../modules/ROOT/pages/projects/index.adoc | 1 +
.../pages/snippets/best-practices/governance.adoc | 1 +
.../native-remote-workflow-engine.adoc | 3 +
helm/README.md | 2 +
25 files changed, 1107 insertions(+), 4 deletions(-)
diff --git a/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
b/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
index 156f591e0a..337c40cd72 100644
--- a/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
+++ b/docs/hop-tech-manual/modules/ROOT/pages/docker-container.adoc
@@ -270,6 +270,8 @@ That image will contain the last built Development snapshot
of Apache Hop. rxq77
=== Running the short-lived container
The most common use case will be that you run a **short-lived container** to
just complete one Hop workflow or pipeline.
+If CI already baked the project into the image, this is often all you need in
production: a scheduler starts the container, hop-run finishes, the container
exits -- no long-lived Hop Server.
+See
xref:manual::hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers].
The first example below runs the sample `switch-case-basic.hpl` **pipeline**
from the samples project.
@@ -340,7 +342,9 @@ The value of `HOP_PARENT_PROJECT_NAME` should match
`parentProjectName` in the c
If you need a **long-lived container**, this option is also available.
-For more information on the long-lived container please also see the
xref:manual::hop-server/index.adoc[Hop Server] documentation as it describes
what can be done using the Hop Server
+For more information on the long-lived container please also see the
xref:manual::hop-server/index.adoc[Hop Server] documentation as it describes
what can be done using the Hop Server.
+
+To ship a git-hosted project *with* the image (CI builds `FROM apache/hop`,
copies the project to `/your-project`, then rolls the container) see
xref:manual::hop-server/deploy-project-image.adoc[Project in a Docker image].
Run this command to start a Hop Server in a docker container:
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/Dockerfile
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/Dockerfile
new file mode 100644
index 0000000000..d0d6395c76
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/Dockerfile
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample image: copy a Hop project into the official long-lived Hop Server
image.
+# Pin a released Hop version. Do not use "latest" in production.
+# Build from the root of your project repository:
+# docker build -t registry.example.com/your-project-hop-server:${GIT_SHA} .
+
+FROM apache/hop:2.20.0
+
+COPY --chown=hop:hop . /your-project
+
+# Optional extra JDBC drivers the official image does not ship:
+# COPY --chown=hop:hop lib/jdbc/*.jar /opt/hop/lib/jdbc/
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/ci-build-image.yml
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/ci-build-image.yml
new file mode 100644
index 0000000000..99da759be7
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/ci-build-image.yml
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample GitHub Actions workflow: on a push to main, build the Hop Server
+# image that contains this project and push it to a registry.
+# Copy to .github/workflows/ in the project repository and adjust the
+# registry login to match your registry.
+
+name: Build Hop Server image
+
+on:
+ push:
+ branches:
+ - main
+
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ permissions:
+ contents: read
+ packages: write
+ steps:
+ - uses: actions/checkout@v4
+
+ - uses: docker/setup-buildx-action@v3
+
+ - uses: docker/login-action@v3
+ with:
+ registry: ghcr.io
+ username: ${{ github.actor }}
+ password: ${{ secrets.GITHUB_TOKEN }}
+
+ - uses: docker/build-push-action@v6
+ with:
+ context: .
+ file: Dockerfile
+ push: true
+ tags: |
+ ghcr.io/${{ github.repository }}/hop-server:${{ github.sha }}
+ ghcr.io/${{ github.repository }}/hop-server:main
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/docker-compose.yaml
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/docker-compose.yaml
new file mode 100644
index 0000000000..a3817f4ef4
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/docker-compose.yaml
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample Compose file for a Hop Server image that already contains the project
+# at /your-project. Environment JSON is mounted from the host so secrets stay
+# out of the image.
+
+services:
+ hop-server:
+ image: registry.example.com/your-project-hop-server:${GIT_SHA:-latest}
+ ports:
+ - "8080:8080"
+ environment:
+ HOP_PROJECT_FOLDER: /your-project
+ HOP_PROJECT_NAME: your-project
+ HOP_ENVIRONMENT_NAME: prod
+ HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS: /config/prod.json
+ HOP_SERVER_HOSTNAME: "0.0.0.0"
+ HOP_SERVER_PORT: "8080"
+ HOP_SERVER_USER: hop-admin
+ HOP_SERVER_PASS: change-me
+ HOP_SERVER_SHUTDOWN_TIMEOUT: "120"
+ volumes:
+ - /etc/hop/prod.json:/config/prod.json:ro
+ restart: unless-stopped
+ stop_grace_period: 120s
diff --git
a/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/hop-server.service
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/hop-server.service
new file mode 100644
index 0000000000..764a01b2bf
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/assets/files/hop-server/hop-server.service
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample systemd unit for a Hop Server that uses a git checkout of the
+# project. The environment already references the project, so -e is enough.
+# Install to /etc/systemd/system/hop-server.service and adjust the paths.
+
+[Unit]
+Description=Apache Hop Server
+After=network.target
+
+[Service]
+Type=simple
+User=hop
+Group=hop
+Environment=HOP_CONFIG_FOLDER=/etc/hop/config
+Environment=HOP_SERVER_SHUTDOWN_TIMEOUT=120
+WorkingDirectory=/opt/hop
+ExecStart=/opt/hop/hop-server.sh -e prod /etc/hop/hop-server.xml
+TimeoutStopSec=130
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
diff --git a/docs/hop-user-manual/modules/ROOT/nav.adoc
b/docs/hop-user-manual/modules/ROOT/nav.adoc
index ab900fd349..3370f79813 100644
--- a/docs/hop-user-manual/modules/ROOT/nav.adoc
+++ b/docs/hop-user-manual/modules/ROOT/nav.adoc
@@ -467,6 +467,9 @@ under the License.
** xref:metadata-types/data-set.adoc[Data Set]
** xref:metadata-types/execution-data-profile.adoc[]
** xref:metadata-types/execution-information-location.adoc[]
+*** xref:metadata-types/elastic-location-type.adoc[Elastic location]
+*** xref:metadata-types/opensearch-location-type.adoc[OpenSearch location]
+*** xref:metadata-types/neo4j/neo4j-location-type.adoc[Neo4j location]
** xref:metadata-types/ftp-connection.adoc[FTP Connection]
** xref:metadata-types/google-storage-authentication.adoc[Google Storage
Connection]
** xref:metadata-types/hop-server.adoc[Hop Server]
@@ -516,6 +519,10 @@ under the License.
** xref:technology/redis/index.adoc[Redis]
** xref:technology/google/index.adoc[Google]
* xref:hop-server/index.adoc[Hop Server]
+** xref:hop-server/deploying.adoc[Deploying Hop Server]
+*** xref:hop-server/deploy-project-image.adoc[Project in a Docker image]
+*** xref:hop-server/deploy-export-resources.adoc[Remote run with export
resources]
+*** xref:hop-server/deploy-git-checkout.adoc[Git checkout on the server]
** xref:hop-server/rest-api.adoc[REST api]
** xref:hop-server/web-service.adoc[Web Service]
** xref:hop-server/async-web-service.adoc[Async Web Service]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
index af0d1ae9c0..f8ed66a5ed 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/getting-started/hop-next-steps.adoc
@@ -31,5 +31,6 @@ Here are a couple of topics you may want to look into:
* xref:workflow/workflows.adoc[Workflows] takes a closer look at the various
aspects of create and running workflows, and contains the entire list of
actions that are at your disposal
* xref:best-practices/index.adoc[Best Practices] covers a number of things you
might want to think about while using Apache Hop.
* xref:projects/index.adoc[Projects] explains how to work with projects and
environments
+* xref:hop-server/deploying.adoc[Deploying Hop Server] covers the three common
ways to get a project onto a running server
* xref:vfs.adoc[VFS] explains how you can access resources in the 3 main cloud
platforms: AWS, Azure and GCP.
* xref:logging/logging-basics.adoc[Logging] explains how to configure Hop for
your desired log level and target platform
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
index b2ac81216d..b313f6eaa3 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-run/index.adoc
@@ -86,6 +86,7 @@ The available options are listed in more detail in the table
below:
|```-e```
|```--environment```
|The name of the environment to use.
+The project name is taken from that environment, so you do not need `-j` as
well.
Check the xref:projects/projects-environments.adoc[documentation on
environments] for more details.
@@ -109,7 +110,8 @@ Strict by default — when omitted, Hop Run refuses to start
such a run and prin
|```-j```
|```--project```
-|The name of the project to use when running a pipeline or workflow
+|The name of the project to use when running a pipeline or workflow.
+Not needed when `-e` is set.
|```-l```
|```--level```
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-export-resources.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-export-resources.adoc
new file mode 100644
index 0000000000..2fbcfcc8e8
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-export-resources.adoc
@@ -0,0 +1,180 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[DeployExportResources]]
+:imagesdir: ../../assets/images
+:openvar: ${
+:closevar: }
+:description: Run a bare Hop Server and send it work from hop-run, Hop Gui or
Airflow with a remote run configuration and export resources enabled.
+
+= Remote run with export resources
+
+Hop Server runs on a machine -- or in a stock `apache/hop` long-lived
container -- *without* a project of its own.
+A client (Hop Gui, xref:hop-run/index.adoc[hop-run], an Airflow worker that
has a Hop client) executes a workflow or pipeline with a
xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Remote]
run configuration and *Export linked resources to server* enabled.
+Everything Hop can pack is sent to the server in a ZIP.
+
+See xref:hop-server/deploying.adoc[Deploying Hop Server] for how this compares
to the other patterns.
+
+Environment configuration happens on the *client*.
+The server never reads your environment JSON files.
+
+== How the ZIP is built
+
+When export resources is enabled, Hop walks the parent pipeline or workflow
and every referenced pipeline or workflow (mappings, workflow actions,
metadata-referenced files such as a Pipeline Log).
+It then:
+
+* serializes the client's project metadata to `metadata.json` inside the ZIP
+* rewrites sibling Hop files to
`{openvar}Internal.Entry.Current.Folder{closevar}/...`
+* rewrites named data folders to `{openvar}DATA_PATH_n{closevar}/...`
+* adds the execution configuration XML, including the variables that were
resolved on the client
+* sends the ZIP to the server, which runs from the archive *without* unzipping
it into a project home
+
+The remote run configuration has two folder fields that matter as soon as you
touch anything that is not a `.hpl` / `.hwf` file:
+
+[cols="1,3",options="header"]
+|===
+|Option |What to put there
+
+|Named resources reference source folder
+|Typically `{openvar}PROJECT_HOME{closevar}`
+
+|Named resources reference target folder
+|A directory that *exists on the server* (or is mounted there)
+|===
+
+Result: `{openvar}DATA_PATH_1{closevar}` is set to the target mapping of that
folder, for example `/data/incoming`.
+
+If you leave the target empty, `{openvar}DATA_PATH_n{closevar}` keeps the
*client's* absolute path, which is almost never valid on the server.
+
+The same options are documented on the
xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Remote
pipeline] and
xref:workflow/workflow-run-configurations/native-remote-workflow-engine.adoc[Remote
workflow] pages.
+
+== What is not in the ZIP
+
+WARNING: Export resources is easy to get wrong.
+File locations lose their meaning once they have been rewritten into a ZIP,
and extra dependencies are not sent.
+
+* Environment JSON files are *not* sent.
+Enable the environment on the client (`hop-run.sh -e prod ...`) so the
variables are already in the execution configuration.
+* Extra configuration files (`.properties`, JSON or Avro schemas, templates,
snippets of `hop-config.json`) are *not* sent unless a transform or action
implements `exportResources()` for that filename.
+Many do not.
+* JDBC drivers, plugins and native libraries are never sent.
+Install them on the server (or in the server image) yourself.
+* Large data files are *renamed*, not packed.
+Stage them on the server or on a shared VFS (S3, NFS, ...) and map the folder.
+* Only transforms and actions that implement `exportResources` participate.
+A "Write to file" transform with a hardcoded laptop path will fail on the
server.
+
+Practical rules:
+
+* Express every path as a variable (`{openvar}PROJECT_HOME{closevar}`,
`{openvar}DATA_IN{closevar}`, ...), never as a developer laptop path.
+* Prefer environment *variables* over sidecar files if you must use this
pattern.
+* Pre-stage shared files on the server and set the named-resource target
folder to that location.
+* If the project has many file dependencies, use
xref:hop-server/deploy-project-image.adoc[a project image] or a
xref:hop-server/deploy-git-checkout.adoc[git checkout] instead.
+
+== The client
+
+The environment already references its project, so `-e` is enough.
+You do not pass `-j` as well.
+
+[source,bash]
+----
+hop-run.sh -e prod \
+ -r remote-on-server \
+ -f '{openvar}PROJECT_HOME{closevar}/main.hwf'
+----
+
+`remote-on-server` is a Remote run configuration in the project metadata:
+
+* Hop Server host, port and credentials
+* *Export linked resources to server?* enabled
+* Named resources source folder: `{openvar}PROJECT_HOME{closevar}`
+* Named resources target folder: a path that exists on the server, for example
`/data/project`
+
+Hop Gui uses the same run configuration.
+That is the usual way to try a shared test server from a laptop.
+
+=== Apache Airflow
+
+This is *not* the
xref:how-to-guides/run-hop-in-apache-airflow.adoc[short-lived `apache/hop`
container] how-to.
+That page runs `hop-run` inside Docker with a *local* engine and never talks
to Hop Server.
+
+To drive a Hop Server from Airflow you have two honest options:
+
+. Give the Airflow worker a Hop client (installed on the worker, or a thin
image that contains Hop) and call `hop-run` with the remote run configuration
above.
+. Stay on the short-lived-container how-to and skip Hop Server entirely.
+
+A DAG task that shells out to hop-run looks like this:
+
+[source,python]
+----
+from airflow.operators.bash import BashOperator
+
+run_on_hop_server = BashOperator(
+ task_id="run_on_hop_server",
+ bash_command=(
+ "hop-run.sh -e prod "
+ "-r remote-on-server "
+ "-f '{openvar}PROJECT_HOME{closevar}/main.hwf'"
+ ),
+)
+----
+
+The worker must have `HOP_CONFIG_FOLDER` (or a Hop install whose
`config/hop-config.json` already lists the project and the `prod` environment)
and network access to the server.
+
+== The server
+
+The server does not need the project.
+
+[tabs]
+====
+Linux, macOS::
++
+--
+[source,bash]
+----
+./hop-server.sh 0.0.0.0 8080
+----
+--
+
+Docker::
++
+--
+[source,bash]
+----
+docker run -d --name hop-server \
+ -p 8080:8080 \
+ -e HOP_SERVER_USER=hop-admin \
+ -e HOP_SERVER_PASS=change-me \
+ apache/hop:<version>
+----
+
+Do *not* set `HOP_PROJECT_*` on this container.
+The project arrives in the ZIP.
+--
+====
+
+The server *does* need:
+
+* a Hop version compatible with the client (same release is the safe choice)
+* the same JDBC drivers and extra plugins the project uses
+* the named-resource *target* directories (they can be empty if you only send
Hop files)
+* network reachability from the client
+* credentials and, in production, TLS -- see
xref:hop-server/index.adoc#_ssl_configuration[SSL configuration]
+
+xref:hop-server/web-service.adoc[Web services] look up metadata *on the
server*.
+They will not see the client's project unless you also set `<metadata_folder>`
/ `HOP_SERVER_METADATA_FOLDER`.
+Do not mix the two patterns by accident.
+If you need web services, put the project on the server with
xref:hop-server/deploy-project-image.adoc[an image] or a
xref:hop-server/deploy-git-checkout.adoc[git checkout].
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-git-checkout.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-git-checkout.adoc
new file mode 100644
index 0000000000..e8640145e4
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-git-checkout.adoc
@@ -0,0 +1,138 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[DeployGitCheckout]]
+:imagesdir: ../../assets/images
+:openvar: ${
+:closevar: }
+:description: Clone a Hop project onto the server host, start Hop Server
against that folder, and update with git pull.
+
+= Git checkout on the server
+
+The host can use git.
+You clone the project, register it, and start Hop Server with that folder as
the project home.
+Updating is `git pull`.
+
+See xref:hop-server/deploying.adoc[Deploying Hop Server] for how this compares
to the other patterns.
+Environment configuration files still live *on the server*, usually outside
the clone.
+
+== One-time setup
+
+. Install a Hop client package (the same archive you use for hop-run), or run
the official `apache/hop` image with the clone bind-mounted (see
<<docker-mount>> below).
+. Clone the repository:
++
+[source,bash]
+----
+git clone [email protected]:data/your-project.git
/opt/hop-projects/your-project
+----
+. Point `HOP_CONFIG_FOLDER` at a directory *outside* the Hop install so an
upgrade does not lose the project registration, for example `/etc/hop/config`.
+. Register the project and a production environment.
+The environment references the project; after this, `-e` is enough to start
the server.
+
+[source,bash]
+----
+export HOP_CONFIG_FOLDER=/etc/hop/config
+
+hop-conf.sh --project-create \
+ --project=your-project \
+ --project-home=/opt/hop-projects/your-project \
+ --project-keep-config-file
+
+hop-conf.sh --environment-create \
+ --environment=prod \
+ --environment-project=your-project \
+ --environment-purpose=Production \
+ --environment-config-files=/etc/hop/environments/prod.json
+----
+
+. Start the server with the environment.
+The project name is taken from that environment; you do not pass `-j`.
+
+[source,bash]
+----
+hop-server.sh -e prod /etc/hop/hop-server.xml
+----
+
+If `hop-server.xml` lives in the project, the projects plugin resolves it
relative to `{openvar}PROJECT_HOME{closevar}` (see
xref:hop-server/index.adoc#_start_hop_server_with_a_configuration_file[Start
Hop Server with a configuration file]).
+Prefer keeping the server XML and the environment JSON *outside* the clone.
+
+A systemd unit is in the Hop repository at
`docs/hop-user-manual/modules/ROOT/assets/files/hop-server/hop-server.service`.
+It sets `HOP_CONFIG_FOLDER`, starts with `-e prod`, and uses a stop timeout
that matches `HOP_SERVER_SHUTDOWN_TIMEOUT`.
+
+[#docker-mount]
+== Docker with a bind-mounted clone
+
+You do not need a custom image.
+Mount the clone and set the same variables as
xref:hop-server/deploy-project-image.adoc[the image pattern]:
+
+[source,bash]
+----
+docker run -d --name hop-server \
+ -p 8080:8080 \
+ -e HOP_PROJECT_FOLDER=/your-project \
+ -e HOP_PROJECT_NAME=your-project \
+ -e HOP_ENVIRONMENT_NAME=prod \
+ -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/config/prod.json \
+ -e HOP_SERVER_USER=hop-admin \
+ -e HOP_SERVER_PASS=change-me \
+ -v /opt/hop-projects/your-project:/your-project \
+ -v /etc/hop/environments/prod.json:/config/prod.json:ro \
+ apache/hop:<version>
+----
+
+`git pull` on the host updates the files the container sees.
+Restart the container after you change environment JSON or `hop-server.xml`.
+
+The
xref:tech-manual::docker-container.adoc#_custom_entrypoint_extension_shell_script[custom
entrypoint] example on the Docker page shows how to `git clone` (or `git
pull`) inside the container at start if the host itself should not hold the
clone.
+
+== Updating
+
+[source,bash]
+----
+cd /opt/hop-projects/your-project
+git pull --ff-only
+----
+
+[cols="2,3",options="header"]
+|===
+|Change |Restart Hop Server?
+
+|`.hpl` / `.hwf` in the clone
+|No -- files are read when the next execution starts
+
+|Project metadata JSON (`metadata/**`)
+|Usually no -- metadata is read from disk on use
+
+|Environment JSON (`prod.json`)
+|*Yes* -- variables are applied when the environment is enabled at startup
+
+|`hop-server.xml`, JVM options, plugins, JDBC
+|*Yes*
+
+|`project-config.json` / parent project
+|*Yes* -- the project is enabled at startup
+|===
+
+Use `--ff-only` and, in production, check out a known tag or commit rather
than a floating branch.
+
+`git pull` while an execution is reading a file is a rare race.
+Busy servers are better served by xref:hop-server/deploy-project-image.adoc[an
immutable image].
+
+== Environment files
+
+`/etc/hop/environments/prod.json` is not in the clone.
+Production hosts get that file from configuration management, a second
repository, or a secret store.
+See xref:hop-server/deploying.adoc#_environment_configuration[Environment
configuration].
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
new file mode 100644
index 0000000000..751f5a5f3f
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploy-project-image.adoc
@@ -0,0 +1,299 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[DeployProjectImage]]
+:imagesdir: ../../assets/images
+:openvar: ${
+:closevar: }
+:description: Build a Docker image that contains your Hop project. Run it as a
short-lived job from a scheduler, or as a long-lived Hop Server, with docker
run, Compose, Kubernetes or OpenShift.
+
+= Project in a Docker image
+
+This is the classic enterprise pattern.
+
+The Hop project lives in git.
+When a change is merged, a Jenkins pipeline, GitHub Action or GitLab CI job
builds a Docker image that copies the project into a folder such as
`/your-project`.
+
+That image can run in two ways:
+
+* *Short-lived* -- a scheduler starts the container, hop-run executes one
pipeline or workflow, the container exits.
+You often do not need a Hop Server at all.
+* *Long-lived* -- the same image starts Hop Server; you roll it with `docker
run`, Compose, Kubernetes or OpenShift.
+
+See xref:hop-server/deploying.adoc[Deploying Hop Server] for how this compares
to the other patterns.
+
+== The image
+
+Extend the official `apache/hop` image rather than assembling Hop yourself.
+A copy of this Dockerfile is in the Hop repository at
`docs/hop-user-manual/modules/ROOT/assets/files/hop-server/Dockerfile`.
+
+[source,dockerfile]
+----
+FROM apache/hop:<version>
+
+COPY --chown=hop:hop . /your-project
+
+# Optional extra JDBC drivers the official image does not ship:
+# COPY --chown=hop:hop lib/jdbc/*.jar /opt/hop/lib/jdbc/
+----
+
+Pin a Hop version.
+Do not use `latest` in production.
+The image user is `hop`; keep `/your-project` as the project home in CI,
Compose and Helm so the rest of this page stays consistent.
+
+The official entrypoint already knows how to turn a folder into either a
short-lived hop-run or a long-lived Hop Server.
+When you set the variables below it:
+
+. registers the project with `hop-conf --project-create
--project-keep-config-file`
+. registers the environment (the environment references that project)
+. if `HOP_FILE_PATH` and `HOP_RUN_CONFIG` are set, runs that file with hop-run
and exits
+. otherwise starts Hop Server with the environment enabled
+
+You do not need a custom `CMD`.
+The xref:tech-manual::docker-container.adoc[Docker container] page lists every
variable.
+
+[cols="2,3",options="header"]
+|===
+|Variable |Role
+
+|`HOP_PROJECT_FOLDER`
+|`/your-project`
+
+|`HOP_PROJECT_NAME`
+|Name used when the entrypoint *creates* the project in the container
+
+|`HOP_ENVIRONMENT_NAME`
+|Name used when the entrypoint *creates* the environment
+
+|`HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS`
+|Comma-separated paths to environment JSON files *inside* the container
+
+|`HOP_FILE_PATH`
+|Pipeline or workflow to run.
+When set (with `HOP_RUN_CONFIG`), the container is short-lived.
+
+|`HOP_RUN_CONFIG`
+|Run configuration name for a short-lived run (usually a *local* engine).
+
+|`HOP_SERVER_USER` / `HOP_SERVER_PASS`
+|Hop Server credentials (long-lived only).
+Change the default `cluster` / `cluster`.
+
+|`HOP_SERVER_PORT` / `HOP_SERVER_HOSTNAME`
+|Defaults `8080` and `0.0.0.0`
+
+|`HOP_SERVER_SHUTDOWN_TIMEOUT`
+|Seconds to wait for running work on `docker stop` / `SIGTERM`.
+`0` exits immediately.
+
+|`HOP_PARENT_PROJECT_NAME` / `HOP_PARENT_PROJECT_FOLDER`
+|Optional one-level parent project.
+See the Docker page.
+|===
+
+`HOP_PROJECT_NAME` is required in the image because the entrypoint has to
*create* the project before it can create the environment that points at it.
+After that, enabling the environment is enough: the project name is taken from
the environment.
+
+== Do not bake secrets
+
+Copy the project into the image.
+Inject environment JSON at run time.
+
+[cols="1,2",options="header"]
+|===
+|Mechanism |Typical use
+
+|Bind mount
+|`docker run -v /etc/hop/prod.json:/config/prod.json:ro`
+
+|Compose `configs` / `secrets`
+|Same idea, declared in Compose
+
+|Kubernetes Secret or ConfigMap
+|Mount at `/config/prod.json`
+
+|`HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS`
+|`/config/prod.json` (add more files, comma-separated)
+|===
+
+If the environment file contains only non-secret values or
xref:metadata-types/variable-resolver/index.adoc[resolver] expressions, it
*may* live at `/your-project/config/prod.json` and be copied with the project.
+Treat that as the exception.
+
+`<metadata_folder>` / `HOP_SERVER_METADATA_FOLDER` is the older hook for
xref:hop-server/web-service.adoc[web services].
+Prefer `HOP_PROJECT_*` so `{openvar}PROJECT_HOME{closevar}` and metadata
inheritance work.
+
+JDBC drivers the image is not allowed to ship can be installed at start with
`HOP_DRIVERS_DOWNLOAD` and `HOP_DRIVERS_ACCEPT_LICENSE`, or `COPY`'d into
`/opt/hop/lib/jdbc/`.
+Extra plugins go into the image the same way.
+
+== Continuous integration
+
+The pipeline is the same whether you use Jenkins, GitHub Actions or GitLab CI:
+
+. Check out the project.
+. `docker build -t registry.example.com/your-project-hop-server:<git-sha>`
(and a moving branch tag if you want).
+. `docker push`.
+. Either start short-lived jobs from the scheduler (Airflow, a Kubernetes Job,
...), or deploy / restart a long-lived Hop Server: `docker compose pull &&
docker compose up -d`, `helm upgrade`, `kubectl set image` / `oc rollout`.
+
+A minimal GitHub Actions workflow is in the Hop repository at
`docs/hop-user-manual/modules/ROOT/assets/files/hop-server/ci-build-image.yml`.
+Log in to the registry the way that registry documents (OIDC, a deploy token,
...).
+
+[cols="1,3",options="header"]
+|===
+|System |Same four steps
+
+|GitHub Actions
+|`actions/checkout` -> `docker/build-push-action` -> deploy job or
`workflow_dispatch`
+
+|GitLab CI
+|`docker build` / `docker push` in `.gitlab-ci.yml`; deploy with `kubectl` or
a runner on the host
+
+|Jenkins
+|Pipeline stages for checkout, `docker.build(...).push()`, then a deploy stage
+|===
+
+[#skip_hop_server_short_lived_containers]
+== Skip Hop Server: short-lived containers
+
+If a scheduler already decides *when* work runs, do not also run a Hop Server.
+Use the same baked image as a job: start it, run one file, exit.
+
+[source,bash]
+----
+docker run --rm \
+ -e HOP_PROJECT_FOLDER=/your-project \
+ -e HOP_PROJECT_NAME=your-project \
+ -e HOP_ENVIRONMENT_NAME=prod \
+ -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/config/prod.json \
+ -e HOP_FILE_PATH='{openvar}PROJECT_HOME{closevar}/main.hwf' \
+ -e HOP_RUN_CONFIG=local \
+ -v /etc/hop/prod.json:/config/prod.json:ro \
+ registry.example.com/your-project-hop-server:<git-sha>
+----
+
+Airflow can do this with the DockerOperator (or KubernetesPodOperator on a
cluster).
+That is exactly the xref:how-to-guides/run-hop-in-apache-airflow.adoc[Airflow
how-to], except the image already contains `/your-project` so you do not mount
the project from the worker.
+On Kubernetes or OpenShift the same idea is a Job or CronJob whose container
spec is this image plus those environment variables.
+
+The container is gone when hop-run finishes, so anything you still want to
look at later must be written *outside* it.
+
+Attach an xref:metadata-types/execution-information-location.adoc[execution
information location] to the `local` run configuration (it is metadata, so it
is already in the image):
+
+[cols="1,3",options="header"]
+|===
+|Location type |Typical target for a short-lived job
+
+|File / caching file
+|A volume or PVC mounted at a fixed path, or a VFS URI (S3, ...).
+The folder must still be there after the pod is deleted.
+
+|Caching database
+|A relational database the container can reach.
+Survives every job.
+
+|xref:metadata-types/opensearch-location-type.adoc[OpenSearch] or
xref:metadata-types/elastic-location-type.adoc[Elastic]
+|A search cluster.
+Useful when many jobs write in parallel.
+
+|Neo4j
+|A graph database, if you already use Neo4j for this.
+|===
+
+Inspect those runs later from the
xref:hop-gui/perspective-execution-information.adoc[Execution Information
perspective] in Hop Gui.
+Do *not* use a *Remote* execution information location that points at a Hop
Server you are not running.
+
+Keep a long-lived Hop Server when you need
xref:hop-server/web-service.adoc[web services] or
xref:hop-server/async-web-service.adoc[asynchronous web services], or when many
clients submit work to one always-on engine.
+
+== Run a long-lived Hop Server
+
+=== docker run
+
+[source,bash]
+----
+docker run -d --name hop-server \
+ -p 8080:8080 \
+ -e HOP_PROJECT_FOLDER=/your-project \
+ -e HOP_PROJECT_NAME=your-project \
+ -e HOP_ENVIRONMENT_NAME=prod \
+ -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/config/prod.json \
+ -e HOP_SERVER_USER=hop-admin \
+ -e HOP_SERVER_PASS=change-me \
+ -e HOP_SERVER_SHUTDOWN_TIMEOUT=120 \
+ -v /etc/hop/prod.json:/config/prod.json:ro \
+ registry.example.com/your-project-hop-server:<git-sha>
+----
+
+=== Docker Compose
+
+A full file is in the Hop repository at
`docs/hop-user-manual/modules/ROOT/assets/files/hop-server/docker-compose.yaml`.
+The service is the same image and the same variables; a bind mount supplies
`prod.json`.
+
+[source,bash]
+----
+docker compose pull
+docker compose up -d
+----
+
+After a new image is pushed:
+
+[source,bash]
+----
+docker compose pull
+docker compose up -d --force-recreate
+----
+
+=== Kubernetes and OpenShift
+
+Use the baked image in a Deployment (or the
https://github.com/apache/hop/tree/main/helm[Helm chart] under `helm/`).
+
+* Create the Hop Server credential as a Secret.
+The chart expects `<release_name>-server` with key `pass`.
+* Mount environment JSON from a Secret or ConfigMap at `/config/prod.json`.
+* Set the same `HOP_PROJECT_*` and `HOP_ENVIRONMENT_*` variables.
+They are already listed, commented, in `helm/hop/values.yaml`.
+
+[source,yaml]
+----
+server:
+ image:
+ name: registry.example.com/your-project-hop-server
+ tag: "abc1234"
+ env:
+ HOP_PROJECT_FOLDER: /your-project
+ HOP_PROJECT_NAME: your-project
+ HOP_ENVIRONMENT_NAME: prod
+ HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS: /config/prod.json
+----
+
+The image already runs as non-root user `hop`.
+On OpenShift that matches a typical restricted SecurityContextConstraint; you
rarely need extra SCC work.
+
+Roll a new image with `helm upgrade` or `kubectl rollout restart`.
+Set `HOP_SERVER_SHUTDOWN_TIMEOUT` (and a matching
`terminationGracePeriodSeconds`) so in-flight work can finish before the new
replica takes over.
+See xref:hop-server/index.adoc#_graceful_shutdown[graceful shutdown].
+
+== After deploy
+
+Point a browser at `http://host:8080`.
+Change the default `cluster` / `cluster` credentials.
+
+xref:hop-server/web-service.adoc[Web services] work because the project's
`metadata/` folder is on disk.
+
+Clients (Hop Gui, hop-run, an Airflow worker with a Hop client) use a *Remote*
run configuration that points at this server.
+Export resources is optional here: the server already has the project.
+Prefer executing a filename under `{openvar}PROJECT_HOME{closevar}` that
exists in the image.
+
+For the short-lived alternative, see
<<skip_hop_server_short_lived_containers>>.
+The xref:tech-manual::docker-container.adoc[Docker container] page lists every
hop-run variable. The
xref:how-to-guides/apache-hop-web-services-docker.adoc[web services in Docker]
how-to is the long-lived case.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
new file mode 100644
index 0000000000..d068167014
--- /dev/null
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-server/deploying.adoc
@@ -0,0 +1,173 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[DeployingHopServer]]
+:imagesdir: ../../assets/images
+:openvar: ${
+:closevar: }
+:description: How to get a Hop project onto a Hop Server: bake it into a
Docker image, send it from a client with export resources, or check it out with
git on the host.
+
+= Deploying Hop Server
+
+xref:hop-server/index.adoc[Hop Server] is a long-lived process that executes
pipelines and workflows.
+It is not a scheduler: something else (Hop Gui,
xref:hop-run/index.adoc[hop-run], Apache Airflow, cron, Jenkins, a Kubernetes
CronJob, ...) has to submit the work.
+
+This page is the deployment story: after you develop a project, how does a
running Hop Server get the files and the configuration it needs?
+
+There are three patterns that cover almost every production setup:
+
+* xref:hop-server/deploy-project-image.adoc[Project in a Docker image] -- CI
builds an image that contains the project.
+You can roll that image as a long-lived Hop Server, but you often do not need
to: a scheduler can start a *short-lived* copy, run one pipeline or workflow,
and exit.
+* xref:hop-server/deploy-export-resources.adoc[Remote run with export
resources] -- the client owns the project and sends a ZIP to a "dumb" server.
+* xref:hop-server/deploy-git-checkout.adoc[Git checkout on the server] -- the
host has a clone; updating is `git pull`.
+
+== What you deploy
+
+Two things make a Hop project runnable.
+
+The *project* lives in git: `project-config.json`, the `metadata/` folder,
pipelines, workflows, and any other files the project itself owns.
+See xref:projects/index.adoc[Projects and environments].
+
+The *lifecycle environment* is per runtime: one or more JSON files with
hostnames, credentials, inbound paths, and a purpose (Development, Test,
Production).
+Those values change between laptops, test and production, and they usually
must *not* live in the project repository.
+
+Hop Server then needs the project in one of two ways:
+
+* *On disk*, registered and enabled.
+The official Docker image does this from `HOP_PROJECT_*` / `HOP_ENVIRONMENT_*`
variables.
+On a host you register the project and environment once with
xref:hop-tools/hop-conf/hop-conf.adoc[hop-conf], then start the server with
`-e`.
+The environment already references its project, so you do not pass `-j` as
well.
+This is how xref:hop-server/deploy-project-image.adoc[the image] and
xref:hop-server/deploy-git-checkout.adoc[the git checkout] work.
+Environment configuration files must be present *on the server*.
+
+* *As a payload from the client*.
+A
xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Remote
pipeline] or
xref:workflow/workflow-run-configurations/native-remote-workflow-engine.adoc[Remote
workflow] run configuration with *Export linked resources to server* sends a
ZIP (the files Hop can pack, plus serialized metadata, plus the execution
configuration).
+This is xref:hop-server/deploy-export-resources.adoc[export resources].
+Environment configuration is applied *on the client*; variables travel with
the execution.
+
+`<metadata_folder>` in `hop-server.xml` is the older, lower-level hook.
+Prefer enabling a project so `{openvar}PROJECT_HOME{closevar}`, metadata
inheritance and environment variables all work.
+When both a project and `<metadata_folder>` are set, both providers sit on the
metadata stack.
+
+== Choose a pattern
+
+[cols="1,2,2,2",options="header"]
+|===
+| |Project in a Docker image |Export resources from a client |Git checkout on
the server
+
+|Who owns the project files
+|The image (immutable per build)
+|The client (Airflow, hop-run, Hop Gui)
+|The server host (working tree)
+
+|How you update
+|Rebuild and roll the container
+|Change the client project; the next run sends a new ZIP
+|`git pull` (restart only for environment or server config)
+
+|Environment configuration
+|Mounted or injected on the *server*
+|Applied on the *client*
+|Files on the *server*, usually outside the clone
+
+|Extra files (JDBC, plugins, `.properties`, JSON schemas)
+|`COPY` into the image or a volume
+|*Not sent.* Must already exist on the server, or be rewritten with the
named-resource folder mapping
+|Present in the clone or installed on the host
+
+|File paths
+|`{openvar}PROJECT_HOME{closevar}` is real on the server
+|Paths are rewritten; see the
xref:hop-server/deploy-export-resources.adoc[export resources caveats]
+|`{openvar}PROJECT_HOME{closevar}` is the clone directory
+
+|Best for
+|Enterprise CI/CD, Kubernetes, OpenShift; the same image as a short-lived job
*or* a long-lived server
+|Orchestrators that already have the project; a server that should not own
files
+|Smaller or ops-friendly hosts with git access
+
+|Poor fit
+|You cannot rebuild images
+|Heavy file I/O, extra config files, large data files
+|You cannot (or must not) run `git pull` on production hosts
+|===
+
+== You often do not need a Hop Server
+
+If the project is already in an image, a scheduler can start that image as a
*short-lived* container (locally, or on Docker, Kubernetes, OpenShift, ...),
run one pipeline or workflow, and exit.
+There is no long-lived Hop Server to keep alive, upgrade or authenticate.
+
+The official `apache/hop` image does both jobs.
+Set `HOP_FILE_PATH` and `HOP_RUN_CONFIG` and the entrypoint runs
xref:hop-run/index.adoc[hop-run] and exits.
+Omit them and it starts Hop Server.
+
+Because the container disappears when the run finishes, send
xref:metadata-types/execution-information-location.adoc[execution information]
somewhere that outlives it: a mounted volume (file or caching-file location), a
relational database, OpenSearch, Neo4j or Elastic.
+You can inspect those runs later from the
xref:hop-gui/perspective-execution-information.adoc[Execution Information
perspective] in Hop Gui.
+
+This is the usual production pattern when a scheduler (Airflow, Jenkins, cron,
a Kubernetes Job or CronJob) already owns the calendar.
+Keep a long-lived Hop Server when you need
xref:hop-server/web-service.adoc[web services], or when many clients submit
work to a shared always-on engine.
+
+See
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers] and the
xref:how-to-guides/run-hop-in-apache-airflow.adoc[Airflow] how-to.
+
+== Environment configuration
+
+Use project variables for values that are stable and safe to commit (folder
names relative to `{openvar}PROJECT_HOME{closevar}`).
+Use environment variables for infrastructure:
`{openvar}DB_HOSTNAME{closevar}`, credentials, inbound paths.
+
+Keep environment JSON *outside* the project folder when it contains secrets.
+Typical places:
+
+* a second, private repository
+* files dropped onto the host by the deployment pipeline
+* a Kubernetes Secret or ConfigMap mounted as a file
+* a xref:metadata-types/variable-resolver/index.adoc[variable resolver]
(HashiCorp Vault, Azure Key Vault, Google Secret Manager) so the committed file
only holds resolver expressions
+
+An environment can list several configuration files.
+In Docker that is the comma-separated `HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS`
variable.
+
+A production file looks like this:
+
+[source,json]
+----
+{
+ "variables" : [ {
+ "name" : "DB_HOSTNAME",
+ "value" : "db.prod.example.com",
+ "description" : "Warehouse host"
+ }, {
+ "name" : "DB_PASSWORD",
+ "value" : "#{vault:hop/data/some-db:password}",
+ "description" : "Resolved at runtime"
+ } ]
+}
+----
+
+Default metadata password protection is obfuscation, not encryption.
+For production use the AES2 encoder or a secrets resolver, change the default
Hop Server credential (`cluster` / `cluster`), and enable TLS.
+See xref:password/passwords.adoc[Passwords] and the project's `SECURITY.md`.
+
+For xref:hop-server/deploy-export-resources.adoc[export resources] the *same*
environment files are used on the *client* (`hop-run.sh -e prod ...`).
+The server does not read them.
+
+== What Hop Server does not do
+
+* It does not schedule work.
+Use Airflow, cron, Jenkins, a Kubernetes CronJob, or hop-run from a wrapper.
+* It does not keep a durable work queue.
+A restart drops in-flight executions unless you set `--shutdown-timeout` /
`HOP_SERVER_SHUTDOWN_TIMEOUT` and the orchestrator waits.
+* A short-lived `apache/hop` container that runs `hop-run` and exits is *not*
Hop Server.
+For a project baked into an image that is often the better production pattern
-- see <<_you_often_do_not_need_a_hop_server>>.
+
+How you start, stop and query a server is still on the
xref:hop-server/index.adoc[Hop Server] reference page.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-server/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-server/index.adoc
index 9ea54d5c1d..e36f777530 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-server/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-server/index.adoc
@@ -22,6 +22,8 @@ under the License.
Hop Server is a lightweight server to run workflows and pipelines with the
xref:pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc[Remote
pipeline] or
xref:workflow/workflow-run-configurations/native-remote-workflow-engine.adoc[Remote
workflow] run configurations.
+How you get a project onto a Hop Server is covered in
xref:hop-server/deploying.adoc[Deploying Hop Server].
+
== Starting and Stopping Hop Server
=== General Use
@@ -111,6 +113,7 @@ Specify a comma separated list of KEY=VALUE pairs.
|-e
|--environment
|The name of the project lifecycle environment to enable before startup.
+The project name is taken from that environment, so you do not need `-j` as
well.
This is provided for by the 'projects' plugin.
|-n
@@ -120,6 +123,7 @@ This is provided for by the 'projects' plugin.
|-j
|--project
|The name of the project to enable before startup.
+Not needed when `-e` is set.
This is provided for by the 'projects' plugin.
|-gs
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
index 93396f1a66..989ece5595 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/apache-hop-web-services-docker.adoc
@@ -172,6 +172,8 @@ As mentioned at the beginning of the tutorial, the API
layer above can now proce
== Conclusion
+To bake the whole project into an image in CI and roll Hop Server on Docker,
Compose, Kubernetes or OpenShift, see
xref:hop-server/deploy-project-image.adoc[Project in a Docker image].
+
Apache Hop Pipelines as web services are easy to set up via meta-data
configurations and can be easily deployed via Docker. They are suitable
everywhere to synchronously return results quickly that do not require a long
processing time. As of version 1.1, Apache Hop also supports the use of
workflows as web services.
Here, the call is asynchronous, i.e. when the service is called, a unique ID
of the workflow to be executed is returned immediately and the execution starts
in the background on the HOP server (for details check
https://hop.apache.org/manual/latest/hop-server/async-web-service.html[the
docs]).
diff --git a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
index 45d884bab3..ecaeeaa86e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/index.adoc
@@ -26,3 +26,4 @@ This page contains a collection of how-to guides to perform a
variety of tasks,
* xref:how-to-guides/loops-in-apache-hop.adoc[Loops in Apache Hop]
* xref:how-to-guides/run-hop-in-apache-airflow.adoc[Run Pipelines and
Workflows from Apache Airflow]
* xref:how-to-guides/apache-hop-web-services-docker.adoc[Using Apache Hop web
services in Docker]
+* xref:hop-server/deploying.adoc[Deploying Hop Server]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
index 6e37b58070..769649fa64 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/how-to-guides/run-hop-in-apache-airflow.adoc
@@ -80,6 +80,12 @@
image:how-to-guides/run-hop-in-apache-airflow/apache-airflow-empty-server.png[Ap
== Your first Apache Airflow and Apache Hop DAG
+NOTE: This page runs a *short-lived* `apache/hop` container with a local
engine.
+That is often what you want: if the project is already in the image, you do
not need a Hop Server at all.
+Send xref:metadata-types/execution-information-location.adoc[execution
information] to a volume, database, OpenSearch, Neo4j or Elastic so you can
inspect the run after the container exits.
+See
xref:hop-server/deploy-project-image.adoc#skip_hop_server_short_lived_containers[Skip
Hop Server: short-lived containers].
+To have Airflow submit work to a running Hop Server instead (remote run
configuration, export resources), see
xref:hop-server/deploy-export-resources.adoc[Remote run with export resources].
+
We'll use the Apache Airflow
https://airflow.apache.org/docs/apache-airflow-providers-docker/stable/_api/airflow/providers/docker/operators/docker/index.html[DockerOperator^]
to run Apache Hop workflows and pipelines from an embedded container in Apache
Airflow.
Again, you don't need to be an Apache Airflow, Docker, or Python expert to
create DAGs, we'll treat DAGs as just another text file.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/elastic-location-type.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/elastic-location-type.adoc
index a1457ad464..1e05d76390 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/elastic-location-type.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/elastic-location-type.adoc
@@ -22,7 +22,11 @@ under the License.
== Description
-With this location type you can store execution information in an Elastic
database.
+With this location type you can store execution information in an
Elasticsearch index.
+
+It is a *caching* location: Hop keeps the execution in memory and writes one
JSON document per pipeline or workflow to the index.
+
+If the cluster is OpenSearch rather than Elasticsearch, use the
xref:metadata-types/opensearch-location-type.adoc[OpenSearch location type]
instead.
== Options
@@ -31,3 +35,8 @@ With this location type you can store execution information
in an Elastic databa
* *The index name to use*: The name of the index to store the execution
information in.
* *Button 'Create and configure Elastic index'*: This executes a number of
commands to create and configure the specified index in such a way that Hop can
store and retrieve information from Elastic.
+== See also
+
+* xref:metadata-types/execution-information-location.adoc[Execution
Information Location]
+* xref:metadata-types/opensearch-location-type.adoc[OpenSearch location type]
+
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
index e34ce38d6d..13e1367214 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/execution-information-location.adoc
@@ -89,7 +89,16 @@ With this location type you can store execution information
in a xref:technology
=== Elastic location
-The Elastic location type stores information in an Elastic index. See the
xref:metadata-types/elastic-location-type.adoc[Elastic location type]
+The Elastic location type is a caching location that stores one JSON document
per execution in an Elasticsearch index.
+Authenticate with an API key.
+See the xref:metadata-types/elastic-location-type.adoc[Elastic location type].
+
+=== OpenSearch location
+
+The OpenSearch location type is the same idea as Elastic -- a caching
location, one JSON document per execution -- for an
https://opensearch.org/[OpenSearch] cluster.
+Authenticate with a username and password.
+Listing uses OpenSearch SQL (`/_plugins/_sql`) rather than Elasticsearch
`_search`.
+See the xref:metadata-types/opensearch-location-type.adoc[OpenSearch location
type] for the options and a comparison with Elastic.
=== Caching Database location
diff --git a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
index 81c2f45404..1225ff7ca8 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/index.adoc
@@ -43,6 +43,7 @@ By default, Hop contains the following metadata types:
xref:metadata-types/data-stream/data-stream.adoc[Data Stream]: A Data Stream
can be used for Inter Process Communication (IPC) or fast data serialization.
* xref:metadata-types/databricks-connection.adoc[Databricks Connection]:
Workspace connection for the Databricks Jobs / REST API (PAT); not the JDBC SQL
warehouse connection
* xref:metadata-types/execution-data-profile.adoc[Execution Data Profile]:
Collects and profiles data as it flows through a pipeline using configurable
samplers for insight into value ranges, nulls, and row samples.
+* xref:metadata-types/execution-information-location.adoc[Execution
Information Location]: Defines where and how Apache Hop stores execution
metadata, supporting local files, remote servers, a database, Neo4j, Elastic or
OpenSearch for later inspection and analysis.
* xref:metadata-types/execution-information-location.adoc[Execution
Information Location]: Defines where and how Apache Hop stores execution
metadata, supporting local files, remote servers, Neo4j, or Elastic for later
inspection and analysis.
* xref:metadata-types/ftp-connection.adoc[FTP Connection]: Named FTP or FTPS
server used as a VFS scheme in file paths and by the FTP actions
* xref:metadata-types/google-storage-authentication.adoc[Google Storage
Connection]: A Google Cloud Storage connection type.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/opensearch-location-type.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/opensearch-location-type.adoc
new file mode 100644
index 0000000000..8cf7a42452
--- /dev/null
+++
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/opensearch-location-type.adoc
@@ -0,0 +1,96 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+ http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+////
+[[OpenSearchLocationType]]
+:imagesdir: ../../assets/images
+:description: Store Hop execution information in an OpenSearch index.
+
+= OpenSearch Location Type
+
+== Description
+
+With this location type you can store execution information in an
https://opensearch.org/[OpenSearch] index.
+
+It is a *caching* location: Hop keeps the execution in memory and writes one
JSON document per pipeline or workflow to the index, the same document model as
the Caching File, Caching Database and
xref:metadata-types/elastic-location-type.adoc[Elastic] locations.
+Inspect the runs later from the
xref:hop-gui/perspective-execution-information.adoc[Execution Information
perspective].
+
+OpenSearch is the right choice when the cluster is OpenSearch (Amazon
OpenSearch Service, a self-hosted cluster, ...).
+Use the Elastic location type against Elasticsearch.
+
+== Options
+
+* *OpenSearch server base URL*: The base URL of the OpenSearch REST API,
including the port.
+For example: `https://localhost:9200`
+* *The OpenSearch username to use*: HTTP basic authentication user.
+* *The OpenSearch password to use*: HTTP basic authentication password.
+* *The index name to use*: The index that stores the execution documents.
+* *Ignore SSL certificate check*: Skip TLS certificate verification.
+Useful against a local or self-signed cluster; leave this off in production.
+* *Persistence delay*: Maximum time to wait before a dirty cache entry is
written to OpenSearch, in milliseconds (default `5000`).
+* *Maximum cache age*: Maximum time to keep an entry in memory, in
milliseconds (default `86400000`, one day).
+* *Button 'Create and configure OpenSearch index'*: Creates the specified
index with the mappings and settings Hop needs, and shows the HTTP result.
+
+Click *Create and configure OpenSearch index* once, before the first run.
+The request is a `PUT` of the index name with a mapping that:
+
+* treats `execution`, `executionState` and `summary` as nested objects
+* stores dates with `epoch_millis`, `yyyy-MM-dd HH:mm:ss.SSS` and
`strict_date_optional_time`
+* disables indexing of `childExecutions`, `childExecutionStates` and
`childExecutionData` (they are stored but not searched)
+* raises `index.mapping.total_fields.limit` to `500`
+
+The dialog that opens afterwards shows the status code and response body so
you can see whether the index was created or already existed.
+
+== Compared to the Elastic location
+
+Both plugins cache in memory and persist one JSON document per execution to
`/<index>/_doc`.
+They are not interchangeable:
+
+[cols="1,2,2",options="header"]
+|===
+| |OpenSearch |Elastic
+
+|Authentication
+|Username and password (`Authorization: Basic`)
+|API key (`Authorization: ApiKey`)
+
+|TLS
+|*Ignore SSL certificate check* option
+|Uses the JVM default trust store; no ignore-SSL option
+
+|HTTP calls
+|Hop REST transform (anonymous pipeline)
+|Java `HttpClient`
+
+|Create index
+|Nested mappings (`execution`, `executionState`, `summary`) and explicit date
formats
+|Flatter dotted field names (`execution.executionStartDate`, ...) and default
date types
+
+|Listing executions
+|OpenSearch SQL (`POST /_plugins/_sql`), ordered by `creationDate`, limited to
50 rows; a "last period" date filter is converted to UTC
+|`_search` with `match_all`, ordered by `execution.executionStartDate`,
respects the requested limit
+
+|Extra GUI filters
+|Parent / pipeline / workflow / finished filters are not sent: OpenSearch SQL
rejects those nested conditions
+|Not applied on the Elastic side either; listing is `match_all` plus sort
+|===
+
+Pick the plugin that matches the product you run.
+An OpenSearch cluster will not accept Elastic API keys, and Elasticsearch will
not serve `/_plugins/_sql`.
+
+== See also
+
+* xref:metadata-types/execution-information-location.adoc[Execution
Information Location]
+* xref:metadata-types/elastic-location-type.adoc[Elastic location type]
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc
index e8347f687a..21ad7c3fb8 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/pipeline-run-configurations/native-remote-pipeline-engine.adoc
@@ -87,3 +87,6 @@ In that example variable `DATA_PATH_1` will get value
`/server/files/`.
This in turn allows you to transfer required files in advance or map a folder
into a docker container and so on.
It gives you flexibility when executing remotely while having ease of
development on your client.
+Extra configuration files, JDBC drivers and plugins are *not* sent in the ZIP.
+See xref:hop-server/deploy-export-resources.adoc[Remote run with export
resources] for the full list of limitations and when to put the project on the
server instead.
+
diff --git a/docs/hop-user-manual/modules/ROOT/pages/projects/index.adoc
b/docs/hop-user-manual/modules/ROOT/pages/projects/index.adoc
index f1e5371cd7..66e6b01e0d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/projects/index.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/projects/index.adoc
@@ -72,6 +72,7 @@ Just like projects, environments are defined in one or a
number of configuration
TIP: store your environment configuration files outside of the project folder.
You may even want to check them in to a separate version control repository.
+How those files get onto a running Hop Server is covered in
xref:hop-server/deploying.adoc[Deploying Hop Server].
==== Import variables
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
index d27d2c57f9..9c88799bf6 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/snippets/best-practices/governance.adoc
@@ -25,4 +25,5 @@ The items below will make your Apache Hop project easier to
manage, to monitor a
* Make sure to have a backup and restore strategy, and test it.
* Run continuous integration
* Set up lifecycle environments (development, test, acceptance, production)
+* Deploy Hop Server using one of the xref:hop-server/deploying.adoc[documented
patterns]
* Test your pipelines with xref:pipeline/pipeline-unit-testing.adoc[unit
tests]. Run all your unit tests regularly, validate the results & take action
if needed
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
index 657bcabd82..3e2a4de3c6 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/workflow/workflow-run-configurations/native-remote-workflow-engine.adoc
@@ -91,3 +91,6 @@ In that example variable `DATA_PATH_1` will get value
`/server/files/`.
This in turn allows you to transfer required files in advance or map a folder
into a docker container and so on.
It gives you flexibility when executing remotely while having ease of
development on your client.
+Extra configuration files, JDBC drivers and plugins are *not* sent in the ZIP.
+See xref:hop-server/deploy-export-resources.adoc[Remote run with export
resources] for the full list of limitations and when to put the project on the
server instead.
+
diff --git a/helm/README.md b/helm/README.md
index ee1c45d383..4346424889 100644
--- a/helm/README.md
+++ b/helm/README.md
@@ -19,6 +19,8 @@ under the License.
A **Hop Helm chart** supporting **long-lived** setups.
+To run a Hop Server whose image already contains your project, set
`HOP_PROJECT_FOLDER`, `HOP_PROJECT_NAME`, `HOP_ENVIRONMENT_NAME` and
`HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS` (commented examples are in
`hop/values.yaml`). The full pattern — bake the project in CI, inject
environment JSON at run time — is documented in the user manual: [Project in a
Docker
image](https://hop.apache.org/manual/latest/hop-server/deploy-project-image.html).
+
You can provide values for the following variables in the values.yaml file:
## Variables hop server
