dsanderbi commented on PR #8780:
URL: https://github.com/apache/hop/pull/8780#issuecomment-6061535969
Hi @mattcasters ,
thanks for this PR. I tested it locally and there are a few things I'd like
to point out / discuss.
**My setup**
- Project `test-project` with one embedded environment `project-environment`
(variables `DB_NAME`, `DB_PASS`, `VAR1`, `VAR2`).
- The configuration file of the lifecycle environment is stored in the
project under `.local/environments/`. `.local` is in `.gitignore`, so it is not
committed.
- The project has one configuration file per stage,
`config/dev/project-environment-config.json` and
`config/prod/test-project-prod.json`. Both are committed, just like
`project-config.json`.
```
test-project
├── .gitignore # contains ".local"
├── .local # not committed
│ └── environments
│ └── project-environment.json # local lifecycle environment
config (created by Hop)
├── config # could be a git-submodule
│ ├── dev
│ │ └── project-environment-config.json # committed, stage values for
dev
│ └── prod
│ └── test-project-prod.json # committed, stage values for
prod
├── main.hwf
└── project-config.json # committed, contains
embeddedEnvironments
```
### 1. Mandatory/secret defaults end up in the repository *and* in the local
file
When I set the variables with their defaults under *Project → Environments*,
the values are stored in `project-config.json` **and** in the local environment
file. For a secret this means the value is always committed:
```json
{ "name": "DB_PASS", "defaultValue": "password", "mandatory": true,
"secret": true }
```
This doesn't match the message Hop shows when the environments are created:
<img width="1019" height="137" alt="image"
src="https://github.com/user-attachments/assets/dd2caf10-eb7c-4b50-aee2-5b25661f5b8f";
/>
I read *"Mandatory variables and secrets are written to a configuration file
… Other variables stay in the project"* as: mandatory variables and secrets
move to the local file, and only the other variables stay in the project.
Right now the `secret` flag doesn't change how the value is handled. It
behaves like `mandatory`: the value is committed, stored in clear text locally,
and not masked in the dialog.
**Suggestion:** for secret variables, keep only the name and description in
`project-config.json`. Ask for the value when the environment is created and
store it only in the local file, ideally with the existing `Encrypted …`
mechanism.
### 2. hop-run doesn't use the defaults from project-config.json
Changes to the defaults under *Project → Environments* are not copied to an
existing local file. I assume that's intended.
What I didn't expect: when I run the project from the command line, the
defaults from `project-config.json` are not used at all. After a deployment
there is no `.local` file. I expected Hop to take the defaults from
`project-config.json` and only replace the values that come from the stage
configuration file, here `config/prod/test-project-prod.json` with `DB_NAME`
and `VAR3`.
Repro with a fresh client (fresh `hop-config.json`):
```bash
./hop-run.sh \
--project-locations test-project=/path/to/test-project \
--environments
'prod=test-project:${PROJECT_HOME}/config/prod/test-project-prod.json' \
-e prod \
-f '${PROJECT_HOME}/main.hwf' \
-r local
```
| Variable | Default in `project-config.json` | `config/prod/…` | Result |
Expected |
|---|---|---|---|---|
| `DB_NAME` | `localhost` | `prod-db-server` | `prod-db-server` ✅ |
`prod-db-server` |
| `VAR1` | `var-1-value` | – | `${VAR1}` ❌ | `var-1-value` |
| `VAR3` | – | `prod-var-3` | `prod-var-3` ✅ | `prod-var-3` |
For comparison, in Hop GUI:
<img width="757" height="371" alt="image"
src="https://github.com/user-attachments/assets/adcfacb4-6fe7-466b-9ea4-e47f44c94b80";
/>
With hop-run:
<img width="589" height="252" alt="image"
src="https://github.com/user-attachments/assets/93f25ae6-ba7f-480f-a1b8-6c835ab51fe7";
/>
The values from the configuration file are applied, but none of the defaults
from `project-config.json` are.
As far as I can see, `Project.applyEmbeddedEnvironmentDefaults()` only
applies the defaults when the lifecycle environment has
`embeddedEnvironmentName` set. That field is only set from the GUI. `hop-conf
--environment-create` and `hop-run --environments` have no option for it, and
an environment with the same name as the embedded definition is not linked
automatically either.
**Question:** Is this intentional? If so, what's the idea behind it? Maybe
the intention is that in production every value is set explicitly, through the
environment's configuration files or through environment variables, and the
embedded definition is only a template for creating environments in the GUI? In
that case it would help to mention this in the documentation, because I
expected the defaults to be applied everywhere.
If it's not intentional, I'd prefer an option on `hop-run` to link an
environment to an embedded definition. Then a project can be run without any
registration in `hop-config.json`, which is what I want for a deployment (e.g.
Docker or CI). Another option would be to link automatically when the
environment name matches an embedded definition.
### 3. Minor: the name "Mandatory"
The dialog explains what happens to mandatory variables (they are written to
the local configuration file). Still, when I first looked at the flag, the name
was misleading for me. I read *Mandatory* as "the project needs this variable",
not as "has to be set on each computer". A name like *Local* or *Per computer*
would describe the behavior more directly. Alternatively, `secret` alone could
decide where the value is stored, and `mandatory` would only check that the
value is not empty.
### 4. Question: development and deployment workflow
More generally, I'd like to understand how you see the workflow from
development to deployment. Do you expect the environments to be registered in
`hop-config.json` on the target system (e.g. with `hop-conf`) as part of every
deployment, or should a deployment work with `hop-run` alone? With the first
approach, every deployment needs an extra, often manual, step to create or
update the environment. With `hop-run` alone, a single command is enough to run
the project with the right stage configuration.
---
I attached the
[test-project.zip](https://github.com/user-attachments/files/33207141/test-project.zip).
Like after a deployment, it doesn't contain the `.local` folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
