mattcasters commented on code in PR #8746: URL: https://github.com/apache/hop/pull/8746#discussion_r4187699324
########## docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc: ########## @@ -448,6 +449,97 @@ The download is supported by the client/server image (running pipelines, workflo NOTE: `HOP_DRIVERS_DOWNLOAD` is available from Hop version 2.19. +[[CloudCredentials]] +== Cloud storage credentials + +The cloud file systems in Hop (`s3://`, `gs://`, `azure://`, `googledrive://`, `dropbox://`) get their default credentials from one of two places: + +* the standard environment variables of the provider's SDK (AWS and Google Cloud), or +* the Hop configuration (`hop-config.json`), which you set in the container with `hop-conf` options in `HOP_CONFIG_OPTIONS`. + +The sample file https://github.com/apache/hop/blob/main/docker/hop.env.sample[`docker/hop.env.sample`] lists the options below with comments. +Copy it, uncomment what you need and pass it with `--env-file`: + +[source,bash] +---- +docker run -it --rm \ + --env-file hop.env \ + -v /path/to/project:/files \ + -v /path/to/keys:/keys:ro \ + apache/hop:<tag> +---- + +`docker run --env-file` takes everything after the first `=` literally, quotes included, so do not wrap values in quotes. +A line with only a variable name, for example `AWS_SECRET_ACCESS_KEY`, copies that variable from the shell that runs `docker run`. + +[options="header"] +|=== +|File system |Environment variables |`HOP_CONFIG_OPTIONS` + +|xref:vfs/aws-s3-vfs.adoc[Amazon S3] (`s3://`) +|`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION`. +Or mount a credentials file with `-v ~/.aws:/home/hop/.aws:ro` and set `AWS_PROFILE`. +|Not needed. + +|xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage] (`gs://`) +|`GOOGLE_APPLICATION_CREDENTIALS`, pointing to a mounted service account key file. +|`--google-cloud-service-account-key-file=/keys/key.json` + +|xref:vfs/azure-blob-storage-vfs.adoc[Azure Storage] (`azure://`) +|None. +|`--azure-account=<account> --azure-key=<key>`, optionally `--azure-emulator=<url>` + +|xref:vfs/google-drive-vfs.adoc[Google Drive] (`googledrive://`) +|None. +|`--google-drive-credentials-file=<file> --google-drive-tokens-folder=<folder>` + +|xref:vfs/dropbox-vfs.adoc[Dropbox] (`dropbox://`) +|None. +|`--dropbox-access-token=<token>` +|=== + +`HOP_CONFIG_OPTIONS` is a single variable. +To configure more than one file system, put all options on one line, separated by spaces: + +[source,properties] +---- +HOP_CONFIG_OPTIONS=--azure-account=mystorageaccount --azure-key="Encrypted 2be98..." --dropbox-access-token=my-token +---- + +The Azure key may be encrypted with `hop-encrypt.sh -hop <key>`. +The result contains a space, so quote it as shown above. + +Google Drive needs a token that is created by an interactive browser login. +Log in once with Hop GUI on a desktop, then mount the credentials file and the tokens folder into the container. Review Comment: **[bug]** Mounting the desktop Google Drive tokens folder the way this section and `docker/hop.env.sample` describe does not work in the image. `GoogleDriveFileObject` builds a `FileDataStoreFactory` on that folder. The constructor calls `Files.setPosixFilePermissions` (owner read/write/execute only), and a later refresh rewrites the stored credential. With google-http-client 2.1.0, a read-only mount throws `Read-only file system`, and a read-write mount owned by the host user (uid 1000) opened as the image user `hop` (uid 501) throws `Operation not permitted`. The sample puts the tokens at `/keys/google-drive-tokens`, and the `docker run` example above mounts `/keys:ro`, so following both never reaches the token. **Suggestion:** Keep the credentials JSON on the read-only keys mount. Mount the tokens folder read-write, owned by uid 501, and say so here and on the `/keys/google-drive-tokens` line in `docker/hop.env.sample`. ########## docs/hop-user-manual/modules/ROOT/pages/docker-container.adoc: ########## @@ -448,6 +449,97 @@ The download is supported by the client/server image (running pipelines, workflo NOTE: `HOP_DRIVERS_DOWNLOAD` is available from Hop version 2.19. +[[CloudCredentials]] +== Cloud storage credentials + +The cloud file systems in Hop (`s3://`, `gs://`, `azure://`, `googledrive://`, `dropbox://`) get their default credentials from one of two places: + +* the standard environment variables of the provider's SDK (AWS and Google Cloud), or +* the Hop configuration (`hop-config.json`), which you set in the container with `hop-conf` options in `HOP_CONFIG_OPTIONS`. + +The sample file https://github.com/apache/hop/blob/main/docker/hop.env.sample[`docker/hop.env.sample`] lists the options below with comments. +Copy it, uncomment what you need and pass it with `--env-file`: + +[source,bash] +---- +docker run -it --rm \ + --env-file hop.env \ + -v /path/to/project:/files \ + -v /path/to/keys:/keys:ro \ + apache/hop:<tag> +---- + +`docker run --env-file` takes everything after the first `=` literally, quotes included, so do not wrap values in quotes. +A line with only a variable name, for example `AWS_SECRET_ACCESS_KEY`, copies that variable from the shell that runs `docker run`. + +[options="header"] +|=== +|File system |Environment variables |`HOP_CONFIG_OPTIONS` + +|xref:vfs/aws-s3-vfs.adoc[Amazon S3] (`s3://`) +|`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION`. +Or mount a credentials file with `-v ~/.aws:/home/hop/.aws:ro` and set `AWS_PROFILE`. +|Not needed. + +|xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage] (`gs://`) +|`GOOGLE_APPLICATION_CREDENTIALS`, pointing to a mounted service account key file. +|`--google-cloud-service-account-key-file=/keys/key.json` + +|xref:vfs/azure-blob-storage-vfs.adoc[Azure Storage] (`azure://`) +|None. +|`--azure-account=<account> --azure-key=<key>`, optionally `--azure-emulator=<url>` + +|xref:vfs/google-drive-vfs.adoc[Google Drive] (`googledrive://`) +|None. +|`--google-drive-credentials-file=<file> --google-drive-tokens-folder=<folder>` + +|xref:vfs/dropbox-vfs.adoc[Dropbox] (`dropbox://`) +|None. +|`--dropbox-access-token=<token>` +|=== + +`HOP_CONFIG_OPTIONS` is a single variable. +To configure more than one file system, put all options on one line, separated by spaces: + +[source,properties] +---- +HOP_CONFIG_OPTIONS=--azure-account=mystorageaccount --azure-key="Encrypted 2be98..." --dropbox-access-token=my-token +---- + +The Azure key may be encrypted with `hop-encrypt.sh -hop <key>`. +The result contains a space, so quote it as shown above. + +Google Drive needs a token that is created by an interactive browser login. +Log in once with Hop GUI on a desktop, then mount the credentials file and the tokens folder into the container. + +=== Named connections + +Connections that are defined as metadata in your project, such as xref:vfs/aws-s3-vfs.adoc[S3], xref:vfs/minio-vfs.adoc[MinIO], xref:vfs/azure-blob-storage-vfs.adoc[Azure Storage] or xref:vfs/google-cloud-storage-vfs.adoc[Google Cloud Storage] connections, are read from the project and not from the environment file. + +* An S3 connection with the `Default (environment / instance profile)` authentication type uses the `AWS_*` variables from the table above. +* An Azure connection with the `Managed Identity` authentication type also accepts a service principal through `AZURE_CLIENT_ID`, `AZURE_TENANT_ID` and `AZURE_CLIENT_SECRET`. + +Other connection fields can use Hop variables such as `${S3_SECRET_KEY}`. +Container environment variables are not Hop variables. +Set them in the environment configuration file (`HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS`) or pass them as system properties with `HOP_SYSTEM_PROPERTIES`. + +=== Keeping secrets safe + +Anyone who can run `docker inspect` on the container can read its environment variables, including the values in `HOP_CONFIG_OPTIONS`. Review Comment: **[suggestion]** `docker inspect` and `hop-config.json` are not the only places `HOP_CONFIG_OPTIONS` shows up. `docker/resources/load-and-execute.sh` prints the raw value (`echo "Configuring Hop with : ${HOP_CONFIG_OPTIONS}"`), and `AzureConfigPlugin` logs the key at basic level (`The Azure key is set to '...'`). A Dropbox token or Azure key copied in from the shell still lands in `docker logs`. **Suggestion:** Call out the container log as a third exposure, next to inspect and `hop-config.json`. ########## docker/hop.env.sample: ########## @@ -0,0 +1,133 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Sample environment file for the apache/hop image (hop-run and Hop Server). +# +# Copy this file, uncomment what you need and pass it to the container: +# +# docker run --rm --env-file hop.env -v /path/to/project:/files apache/hop:<tag> +# +# Notes on the file format (docker run --env-file): +# - One VARIABLE=value per line. Everything after the first '=' is the value, +# quotes included. Do not wrap values in quotes. Review Comment: **[suggestion]** These quote and `${...}` rules are right for `docker run --env-file` (quotes stay, nothing is interpolated), which is what lets `--azure-key="Encrypted <hex>"` survive until Alpine `xargs` splits it. A Compose `env_file` does the opposite: it strips one matching pair of quotes and interpolates `${PROJECT_HOME}`. Using this sample from Compose would split the encrypted Azure key on the space and expand `HOP_FILE_PATH` on the host. **Suggestion:** State that the file is for `docker run --env-file` only, not a Compose `env_file`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
