CarlosJuncher03 opened a new issue, #8045:
URL: https://github.com/apache/hop/issues/8045
### Apache Hop version?
2.20
### Java version?
21
### Operating system
Linux
### What happened?
I am testing the new **Load Balancing** run configuration in the Apache Hop
2.20.0 pre-release and have identified two possible issues, as well as a
question regarding execution from a custom Docker image.
### 1. Pipeline Variable Resolver does not work with exported resources
The main pipeline contains an **ETL Metadata Injection** transform that uses
another pipeline as its template. The template contains the following Variable
Resolver expression:
`#{pipeline_token:x:authorization}`
The pipeline works correctly when executed locally. However, when I use the
Load Balancing run configuration with **Export linked resources to server**
enabled, the following error occurs:
```text
Variavel.0 - ERROR: Unexpected error
Variavel.0 - org.apache.hop.core.exception.HopRuntimeException:
Error resolving variable '#{pipeline_token:x:authorization}'
with variable resolver metadata
```
The template pipeline appears to be located and started successfully because
its internal transforms begin processing. The failure occurs specifically while
resolving the expression.
Expected behavior:
* The `pipeline_token` Variable Resolver should be included in the
`metadata.json` sent to the Hop Server.
* The pipeline used by the resolver should also be included in the exported
package.
* The run configuration used by the Pipeline Variable Resolver, which must
be of type **Native Local**, should be available on the selected server.
* The expression should be resolved in the same way as during local
execution.
Observed behavior:
* The template pipeline starts, but the Variable Resolver cannot complete
the resolution.
* The error message does not clarify whether the problem is related to the
resolver metadata, the pipeline used by the resolver, or the Native Local run
configuration.
### 2. Parquet File Output does not correctly use the MinIO connection
inside a template pipeline
In another test, the template pipeline contains a **Parquet File Output**
transform configured to write a file to local S3-compatible storage through a
MinIO connection defined in the project metadata.
In this case, the MinIO connection metadata is received by the Hop Server,
and the variables used in its configuration are resolved correctly.
However, when the pipeline is submitted through Load Balancing with **Export
linked resources to server** enabled, the Parquet File Output transform does
not recognize the destination as a MinIO connection.
Expected behavior:
* The transform should use the MinIO connection received through the
exported metadata.
* The Parquet file should be written to the configured bucket and path in
the local S3-compatible storage.
Observed behavior:
* The MinIO metadata is available.
* The configuration variables are resolved.
* Nevertheless, the transform interprets the destination as a local
filesystem path.
* As a result, the Parquet file is created in a local directory on the Hop
Server instead of being written to MinIO.
The MinIO plugin is installed on the Hop Server. This behavior occurs
specifically when **Parquet File Output** is inside a template pipeline
executed by **ETL Metadata Injection**. When the same structure is executed
locally, the MinIO destination is recognized correctly.
### 3. Question about `${PROJECT_HOME}` when the project is already included
in the Docker image
I also tested a custom Hop Server Docker image. During the image build, I
copied the project into the container and configured the project and
environment to be enabled when the server starts.
In this scenario, I attempted to execute a pipeline without enabling
**Export linked resources to server**, because the project is already included
in the Docker image and configured on the Hop Server.
However, the `${PROJECT_HOME}` value used during execution appears to be the
path resolved on the client. Since this path does not exist inside the
container, the execution fails.
The execution works when resource export is enabled. However, in that mode,
the client sends the project again even though it is already available in the
server image.
My questions are:
* When the project and environment are already registered and enabled on the
Hop Server, should it not use the `${PROJECT_HOME}` value configured on the
server?
* For this deployment model, should the pipeline be executed without
enabling **Export linked resources to server**?
* Is there an additional setting that prevents the client’s
`${PROJECT_HOME}` value from being sent to or overriding the project path on
the server?
* Are **Named resources reference source folder** and **Named resources
reference target folder** applicable only when resource export is enabled, or
should they also support path mapping in this scenario?
Expected configuration for this deployment model:
* The project is added during the Docker image build.
* The project and environment are registered and enabled when the Hop Server
starts.
* `${PROJECT_HOME}` points to the project directory inside the container.
* Pipelines are executed using the files and metadata already available on
the server.
* Resources already included in the image do not need to be exported again.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
