This is an automated email from the ASF dual-hosted git repository.
mattcasters pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 592c5b288e Fixes #2547 : Add a sample env file and document cloud
storage creden… (#8746)
592c5b288e is described below
commit 592c5b288e72d2303bd35cb0653febb6372c669c
Author: Bart Maertens <[email protected]>
AuthorDate: Fri Oct 9 15:21:31 2026 +0200
Fixes #2547 : Add a sample env file and document cloud storage creden…
(#8746)
* Fixes #2547 : Add a sample env file and document cloud storage
credentials for the Docker image
* Issue #2547 : Mount the Google Drive tokens folder read-write, mention
docker logs as a secret exposure, limit the sample to docker run --env-file
---
docker/hop.env.sample | 143 +++++++++++++++++++++
docs/hop-user-manual/modules/ROOT/pages/cloud.adoc | 1 +
.../modules/ROOT/pages/docker-container.adoc | 111 ++++++++++++++++
.../modules/ROOT/pages/hop-web-docker.adoc | 5 +
.../modules/ROOT/pages/vfs/aws-s3-vfs.adoc | 2 +
.../ROOT/pages/vfs/azure-blob-storage-vfs.adoc | 2 +
.../modules/ROOT/pages/vfs/dropbox-vfs.adoc | 2 +
.../ROOT/pages/vfs/google-cloud-storage-vfs.adoc | 2 +
.../modules/ROOT/pages/vfs/google-drive-vfs.adoc | 2 +
9 files changed, 270 insertions(+)
diff --git a/docker/hop.env.sample b/docker/hop.env.sample
new file mode 100644
index 0000000000..e9fe39e25c
--- /dev/null
+++ b/docker/hop.env.sample
@@ -0,0 +1,143 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample environment file for the apache/hop image (hop-run and Hop Server).
+#
+# Copy this file, uncomment what you need and pass it to the container:
+#
+# docker run --rm --env-file hop.env -v /path/to/project:/files
apache/hop:<tag>
+#
+# This file is for 'docker run --env-file' only. Do not use it as a Docker
Compose
+# 'env_file': Compose strips quotes and expands ${...}, which breaks the quoted
+# Azure key and HOP_FILE_PATH below.
+#
+# Notes on the file format (docker run --env-file):
+# - One VARIABLE=value per line. Everything after the first '=' is the value,
+# quotes included. Do not wrap values in quotes.
+# - Lines starting with '#' are comments. Comments after a value are not
supported.
+# - A line with only a variable name (no '=') copies that variable from the
+# shell that runs 'docker run'. Use this for secrets so they stay out of
the file.
+#
+# Values passed as environment variables are visible to anyone who can run
+# 'docker inspect'. The values in HOP_CONFIG_OPTIONS are also printed to the
+# container log ('docker logs'). Prefer mounted key files or a variable
resolver
+# (AWS Secrets Manager, Azure Key Vault, Google Secret Manager, HashiCorp
Vault)
+# for credentials.
+#
+# Documentation:
+# https://hop.apache.org/manual/latest/docker-container.html
+# https://hop.apache.org/manual/latest/docker-container.html#CloudCredentials
+
+#
+# Project and execution
+#
+
+HOP_LOG_LEVEL=Basic
+#HOP_PROJECT_NAME=my-project
+#HOP_PROJECT_FOLDER=/files
+#HOP_ENVIRONMENT_NAME=my-project-prod
+#HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS=/files/config/my-project-prod.json
+
+# Run a pipeline or workflow (short-lived container). Leave HOP_FILE_PATH empty
+# to start a long-lived Hop Server instead.
+#HOP_FILE_PATH=${PROJECT_HOME}/main.hwf
+#HOP_RUN_CONFIG=local
+#HOP_RUN_PARAMETERS=PARAM_1=value1,PARAM_2=value2
+
+# Restricted JDBC drivers to download on start-up (Oracle, MySQL, ...).
+#HOP_DRIVERS_DOWNLOAD=oracle,mariadb
+#HOP_DRIVERS_ACCEPT_LICENSE=true
+
+#
+# Hop configuration (hop-conf)
+#
+# HOP_CONFIG_OPTIONS is passed to hop-conf.sh before Hop starts.
+# Separate options with spaces. Quote a value that contains spaces,
+# for example: --azure-account=myaccount --describe-variable=FOO="A
description"
+# The Azure, Google Cloud, Google Drive and Dropbox defaults below all go into
+# this one variable: combine the options you need on a single line.
+#
+#HOP_CONFIG_OPTIONS=
+
+#
+# Amazon S3 (s3://)
+#
+# Hop uses the AWS SDK default credential chain, so the standard AWS variables
+# work without any Hop configuration. Leave the secrets without a value to copy
+# them from your shell.
+#
+#AWS_ACCESS_KEY_ID
+#AWS_SECRET_ACCESS_KEY
+#AWS_SESSION_TOKEN
+#AWS_REGION=eu-west-1
+#
+# Or mount a credentials file and pick a profile:
+# -v ~/.aws:/home/hop/.aws:ro
+#AWS_PROFILE=default
+
+#
+# Google Cloud Storage (gs://)
+#
+# Mount a service account key file, for example with -v /path/to/keys:/keys:ro,
+# and point the standard Google variable at it:
+#
+#GOOGLE_APPLICATION_CREDENTIALS=/keys/gcp-service-account.json
+#
+# Or configure it in Hop:
+#HOP_CONFIG_OPTIONS=--google-cloud-service-account-key-file=/keys/gcp-service-account.json
+
+#
+# Azure Storage (azure://)
+#
+# The default azure:// file system uses a storage account name and key.
+# The key can be encrypted with 'hop-encrypt.sh -hop <key>'. The result
contains
+# a space, so quote it.
+#
+#HOP_CONFIG_OPTIONS=--azure-account=mystorageaccount --azure-key="Encrypted
<hex string printed by hop-encrypt.sh>"
+#
+# Managed identity or a service principal (AZURE_CLIENT_ID, AZURE_TENANT_ID,
+# AZURE_CLIENT_SECRET) only works with an Azure Storage connection in the
+# project metadata that uses the 'Managed Identity' authentication type.
+
+#
+# Google Drive (googledrive://)
+#
+# Requires a credentials file and a tokens folder that already holds a token.
+# The token is created by an interactive browser login, so generate it with
+# Hop GUI on a desktop first and mount both into the container.
+# The credentials file can stay on the read-only /keys mount. Hop writes to the
+# tokens folder, so mount it read-write and make it owned by uid 501 (the 'hop'
+# user in the image):
+# chown -R 501 /path/to/google-drive-tokens
+# -v /path/to/google-drive-tokens:/home/hop/google-drive-tokens
+#
+#HOP_CONFIG_OPTIONS=--google-drive-credentials-file=/keys/google-drive-credentials.json
--google-drive-tokens-folder=/home/hop/google-drive-tokens
+
+#
+# Dropbox (dropbox://)
+#
+#HOP_CONFIG_OPTIONS=--dropbox-access-token=your-access-token
+
+#
+# Named connections (S3, MinIO, Azure, Google Cloud Storage, ...)
+#
+# Connections defined as metadata in your project are read from the project,
+# not from this file. Use variables such as ${S3_SECRET_KEY} in the connection
+# and provide them through the environment config file
+# (HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS) or as system properties:
+#
+#HOP_SYSTEM_PROPERTIES=S3_ACCESS_KEY=my-key,S3_SECRET_KEY=my-secret
diff --git a/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
b/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
index a1a9d00db5..09e7c1d40f 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/cloud.adoc
@@ -245,6 +245,7 @@ The provider-specific part is narrow, and it is already
documented per provider:
* *Storage* — xref:vfs.adoc[Virtual File System], with pages for
xref:vfs/aws-s3-vfs.adoc[Amazon S3], xref:vfs/azure-blob-storage-vfs.adoc[Azure
Storage], xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage],
xref:vfs/minio-vfs.adoc[MinIO] and xref:vfs/databricks-vfs.adoc[Databricks].
Any S3-compatible service, including those of providers without a dedicated
page, is reachable through the S3 or MinIO file system.
+In a container, pass the credentials as described in
xref:docker-container.adoc#CloudCredentials[Cloud storage credentials].
* *Secrets* — xref:metadata-types/variable-resolver/index.adoc[Variable
Resolver], with resolvers for
xref:metadata-types/variable-resolver/aws-secrets-manager-variable-resolver.adoc[AWS
Secrets Manager],
xref:metadata-types/variable-resolver/azure-key-vault-variable-resolver.adoc[Azure
Key Vault],
xref:metadata-types/variable-resolver/google-secret-manager-variable-resolver.adoc[Google
Secret Manager],
xref:metadata-types/variable-resolver/hashicorp-vault-variable-resolver.adoc[HashiCor
[...]
* *Databases* — xref:database/databases.adoc[Relational Database Connections].
Managed database services are ordinary JDBC connections; the driver still has
to be in the image.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
b/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
index 4e45eb12ed..c7d6d457bc 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc
@@ -171,6 +171,7 @@ You can specify the properties as a comma separated list,
e.g. `PROP1=xxx,PROP2=
|
|If you specify a value for this variable, they will be used to execute
`hop-conf.sh` right before execution of the short- or long-lived container.
You can use it for example to configure values for keys, plugins, and so on.
+See <<CloudCredentials>> for the cloud storage options.
Specify multiple options separated by spaces. Enclose values containing spaces
in quotes, for example `--set-variable=FOO=bar --describe-variable=FOO="An
example variable"`.
|```HOP_COMMAND```
@@ -518,6 +519,116 @@ docker run -it --rm \
NOTE: `HOP_PLUGINS_DOWNLOAD` is available from Hop version 2.20.
+[[CloudCredentials]]
+== Cloud storage credentials
+
+The cloud file systems in Hop (`s3://`, `gs://`, `azure://`, `googledrive://`,
`dropbox://`) get their default credentials from one of two places:
+
+* the standard environment variables of the provider's SDK (AWS and Google
Cloud), or
+* the Hop configuration (`hop-config.json`), which you set in the container
with `hop-conf` options in `HOP_CONFIG_OPTIONS`.
+
+The sample file
https://github.com/apache/hop/blob/main/docker/hop.env.sample[`docker/hop.env.sample`]
lists the options below with comments.
+Copy it, uncomment what you need and pass it with `--env-file`:
+
+[source,bash]
+----
+docker run -it --rm \
+ --env-file hop.env \
+ -v /path/to/project:/files \
+ -v /path/to/keys:/keys:ro \
+ apache/hop:<tag>
+----
+
+`docker run --env-file` takes everything after the first `=` literally, quotes
included, so do not wrap values in quotes.
+A line with only a variable name, for example `AWS_SECRET_ACCESS_KEY`, copies
that variable from the shell that runs `docker run`.
+
+The sample is for `docker run --env-file` only.
+Do not use it as a Docker Compose `env_file`: Compose strips quotes and
expands `${...}`, which breaks the quoted Azure key and `HOP_FILE_PATH`.
+
+[options="header"]
+|===
+|File system |Environment variables |`HOP_CONFIG_OPTIONS`
+
+|xref:vfs/aws-s3-vfs.adoc[Amazon S3] (`s3://`)
+|`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`,
`AWS_REGION`.
+Or mount a credentials file with `-v ~/.aws:/home/hop/.aws:ro` and set
`AWS_PROFILE`.
+|Not needed.
+
+|xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage] (`gs://`)
+|`GOOGLE_APPLICATION_CREDENTIALS`, pointing to a mounted service account key
file.
+|`--google-cloud-service-account-key-file=/keys/key.json`
+
+|xref:vfs/azure-blob-storage-vfs.adoc[Azure Storage] (`azure://`)
+|None.
+|`--azure-account=<account> --azure-key=<key>`, optionally
`--azure-emulator=<url>`
+
+|xref:vfs/google-drive-vfs.adoc[Google Drive] (`googledrive://`)
+|None.
+|`--google-drive-credentials-file=<file> --google-drive-tokens-folder=<folder>`
+
+|xref:vfs/dropbox-vfs.adoc[Dropbox] (`dropbox://`)
+|None.
+|`--dropbox-access-token=<token>`
+|===
+
+`HOP_CONFIG_OPTIONS` is a single variable.
+To configure more than one file system, put all options on one line, separated
by spaces:
+
+[source,properties]
+----
+HOP_CONFIG_OPTIONS=--azure-account=mystorageaccount --azure-key="Encrypted
2be98..." --dropbox-access-token=my-token
+----
+
+The Azure key may be encrypted with `hop-encrypt.sh -hop <key>`.
+The result contains a space, so quote it as shown above.
+
+Google Drive needs a token that is created by an interactive browser login.
+Log in once with Hop GUI on a desktop, then mount the credentials file and the
tokens folder into the container.
+The credentials file can stay on the read-only `/keys` mount.
+Hop writes to the tokens folder, so mount it read-write and make it owned by
uid 501, the `hop` user in the image:
+
+[source,bash]
+----
+chown -R 501 /path/to/google-drive-tokens
+
+docker run -it --rm \
+ --env-file hop.env \
+ -v /path/to/keys:/keys:ro \
+ -v /path/to/google-drive-tokens:/home/hop/google-drive-tokens \
+ apache/hop:<tag>
+----
+
+with `--google-drive-credentials-file=/keys/google-drive-credentials.json
--google-drive-tokens-folder=/home/hop/google-drive-tokens` in
`HOP_CONFIG_OPTIONS`.
+
+=== Named connections
+
+Connections that are defined as metadata in your project, such as
xref:vfs/aws-s3-vfs.adoc[S3], xref:vfs/minio-vfs.adoc[MinIO],
xref:vfs/azure-blob-storage-vfs.adoc[Azure Storage] or
xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage] connections, are
read from the project and not from the environment file.
+
+* An S3 connection with the `Default (environment / instance profile)`
authentication type uses the `AWS_*` variables from the table above.
+* An Azure connection with the `Managed Identity` authentication type also
accepts a service principal through `AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and
`AZURE_CLIENT_SECRET`.
+
+Other connection fields can use Hop variables such as `${S3_SECRET_KEY}`.
+Container environment variables are not Hop variables.
+Set them in the environment configuration file
(`HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS`) or pass them as system properties
with `HOP_SYSTEM_PROPERTIES`.
+
+=== Keeping secrets safe
+
+Anyone who can run `docker inspect` on the container can read its environment
variables, including the values in `HOP_CONFIG_OPTIONS`.
+`hop-conf` also writes those values to `hop-config.json` inside the container.
+The values in `HOP_CONFIG_OPTIONS` also end up in the container log (`docker
logs`): the start-up script prints the variable, and the Azure key is logged
when it is set.
+
+* Prefer mounted, read-only key files over keys in environment variables.
+* Leave secrets out of the environment file and copy them from your shell, as
described above.
+* Use a xref:metadata-types/variable-resolver/index.adoc[Variable Resolver] to
read secrets from AWS Secrets Manager, Azure Key Vault, Google Secret Manager
or HashiCorp Vault at runtime.
+
+=== Hop Web image
+
+The `apache/hop-web` image does not read `HOP_CONFIG_OPTIONS`.
+The provider SDK variables (`AWS_*`, `GOOGLE_APPLICATION_CREDENTIALS`) work as
described above.
+For Azure, Google Drive and Dropbox, set the defaults in the *Plugins* section
of the xref:hop-gui/perspective-configuration.adoc[configuration perspective]
in Hop Web.
+They are stored in `hop-config.json`, so keep that file on a persistent volume.
+See xref:hop-web-docker.adoc[Hop Web in Docker].
+
== Custom Entrypoint Extension Shell Script
To make the Hop Docker image even more flexible, we added a
```HOP_CUSTOM_ENTRYPOINT_EXTENSION_SHELL_FILE_PATH``` variable that accepts a
path to a custom shell script (that you provide).This shell script will run
when you start the container before your Hop project is registered with the
container's Hop config and before your Hop workflow or pipeline gets kicked off.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/hop-web-docker.adoc
b/docs/hop-user-manual/modules/ROOT/pages/hop-web-docker.adoc
index d9a1aac6af..d8e7639e03 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/hop-web-docker.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/hop-web-docker.adoc
@@ -174,6 +174,11 @@ Keep `/usr/local/tomcat/jdbc-drivers` and add `/hop/jdbc`
when you mount extra j
| Download restricted drivers on start.
See xref:docker-container.adoc#DownloadingJDBCDrivers[Downloading JDBC
drivers].
+| `AWS_*` / `GOOGLE_APPLICATION_CREDENTIALS`
+| Default credentials for `s3://` and `gs://`.
+This image does not read `HOP_CONFIG_OPTIONS`.
+See xref:docker-container.adoc#CloudCredentials[Cloud storage credentials].
+
| `HOP_WEB_SECURITY_MODE` and related `HOP_WEB_*` variables
| Authentication bootstrap.
See xref:hop-gui/hop-web.adoc#hop-web-authentication[Hop Web authentication].
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
index be93109ce4..132c2ea440 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
@@ -56,6 +56,8 @@ You can set the following system environment variables:
* `AWS_ACCESS_KEY_ID` : set it to your AWS access key
* `AWS_SECRET_ACCESS_KEY` : set it to your secret access key
+TIP: Running Hop in Docker? See
xref:docker-container.adoc#CloudCredentials[Cloud storage credentials] to pass
these settings to the container.
+
== S3-compatible services
Storage services that implement the S3 API can be used through this plugin as
well.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
index 773155d44e..5cda6fcae2 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
@@ -68,6 +68,8 @@ Once done you will see an `azure` entry in the central
`hop-config.json` file:
}
----
+TIP: Running Hop in Docker? See
xref:docker-container.adoc#CloudCredentials[Cloud storage credentials] to pass
these settings to the container.
+
== Options
The options below are the ones in the *Plugins* section of the
xref:hop-gui/perspective-configuration.adoc[configuration perspective]. Each is
also settable with `hop-conf`.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
index 3e62a4254b..86889d496e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/dropbox-vfs.adoc
@@ -56,6 +56,8 @@ Dropbox displays the App Settings panel for the app that you
created
Once done you will see a `dropbox` entry in the central `hop-config.json` file:
+TIP: Running Hop in Docker? See
xref:docker-container.adoc#CloudCredentials[Cloud storage credentials] to pass
these settings to the container.
+
== Options
The options below are the ones in the *Plugins* section of the
xref:hop-gui/perspective-configuration.adoc[configuration perspective]. Each is
also settable with `hop-conf`.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
index ea6d59b32c..6b6b0bed54 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
@@ -51,6 +51,8 @@ Once done you will see a `googleCloud` entry in the central
`hop-config.json` fi
}
----
+TIP: Running Hop in Docker? See
xref:docker-container.adoc#CloudCredentials[Cloud storage credentials] to pass
these settings to the container.
+
== Options
The options below are the ones in the *Plugins* section of the
xref:hop-gui/perspective-configuration.adoc[configuration perspective]. Each is
also settable with `hop-conf`.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
index bdaab12d7d..73760b88f6 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-drive-vfs.adoc
@@ -68,6 +68,8 @@ Please open the following address in your browser:
Open that URL in a browser and authenticate the given client ID.
You'll then get a token in the configured `tokens folder` with which you can
work.
+TIP: Running Hop in Docker? See
xref:docker-container.adoc#CloudCredentials[Cloud storage credentials] to pass
these settings to the container.
+
== Options
The options below are the ones in the *Plugins* section of the
xref:hop-gui/perspective-configuration.adoc[configuration perspective]. Each is
also settable with `hop-conf`.
