This is an automated email from the ASF dual-hosted git repository.

bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git


The following commit(s) were added to refs/heads/main by this push:
     new 75cc127535 Pipeline executor converts every field of every row to a 
String and r… (#9447)
75cc127535 is described below

commit 75cc127535de60a7ef1eda67f38d6ebe1c790642
Author: Hans Van Akelyen <[email protected]>
AuthorDate: Sat Nov 14 13:05:33 2026 +0100

    Pipeline executor converts every field of every row to a String and r… 
(#9447)
    
    * Pipeline executor converts every field of every row to a String and 
reloads metadata for every child run, fixes #9445
    
    * fix remarks
---
 .../ROOT/pages/metadata-types/pipeline-log.adoc    |   6 +
 .../ROOT/pages/metadata-types/pipeline-probe.adoc  |   6 +
 .../ROOT/pages/metadata-types/workflow-log.adoc    |   6 +
 .../pipeline/transforms/pipeline-executor.adoc     |   1 +
 .../hop/pipeline/engine/PipelineEngineFactory.java |  36 ++++
 .../logging/003-pipeline-executor-child.hpl        |  98 +++++++++
 .../logging/003-pipeline-executor-parent.hpl       | 112 ++++++++++
 .../logging/003-pipeline-log-verify.hpl            | 121 +++++++++++
 integration-tests/logging/003-pipeline-log.hpl     | 142 +++++++++++++
 .../golden-pipeline-executor-children-log.csv      |   5 +
 .../main-003-pipeline-executor-child-logging.hwf   | 133 ++++++++++++
 .../golden-pipeline-executor-children-log.json     |  24 +++
 .../pipeline-log/pipeline-executor-children.json   |  20 ++
 .../unit-test/003-pipeline-log-verify UNIT.json    |  33 +++
 .../org/apache/hop/reflection/RunMetadata.java     | 127 +++++++++++
 .../pipeline/xp/PipelineStartLoggingXp.java        |   9 +-
 .../reflection/probe/xp/PipelineDataProbeXp.java   |   9 +-
 .../workflow/xp/WorkflowStartLoggingXp.java        |   9 +-
 .../org/apache/hop/reflection/RunMetadataTest.java | 176 ++++++++++++++++
 .../pipelineexecutor/PipelineExecutor.java         |  93 +++++++--
 .../pipelineexecutor/PipelineExecutorData.java     |  10 +
 .../pipelineexecutor/PipelineExecutorTest.java     | 232 +++++++++++++++++++++
 22 files changed, 1370 insertions(+), 38 deletions(-)

diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-log.adoc 
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-log.adoc
index 5c42be2e65..a30984ca3b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-log.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-log.adoc
@@ -31,6 +31,12 @@ A pipeline log streams logging information from a running 
pipeline to another pi
 The only requirement for the receiving pipeline is that it starts with a 
xref:pipeline/transforms/pipeline-logging.adoc[Pipeline logging] transform. 
Other than that, the logging pipeline is "just another pipeline".
 In this logging pipeline, you can process the logging information and e.g. 
write to a relational or NoSQL database, a Kafka topic etc
 
+== When changes take effect
+
+A run reads the pipeline logs once, when it first needs them, and every 
pipeline and workflow it starts uses what it read: the child pipelines of a 
Pipeline executor, the pipelines of a workflow, a mapping and so on. A run is a 
top-level pipeline or workflow, the one that Hop Run, Hop GUI, Hop Server or 
the API starts.
+
+A pipeline log that you add or change while a run is going on takes effect 
with the next run. This matters for runs that go on for a long time, such as a 
workflow that loops for hours or a streaming pipeline.
+
 == Samples
 
 The samples project comes with an example pipeline log.
diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-probe.adoc 
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-probe.adoc
index 0944a7a741..ea20a0285e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-probe.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/pipeline-probe.adoc
@@ -32,6 +32,12 @@ The Pipeline Probe metadata type works by specifying a 
receiving pipeline (`Pipe
 
 The receiving pipeline can then continue to process the probe data with all 
the functionality Apache Hop pipelines have to offer.
 
+== When changes take effect
+
+A run reads the pipeline probes once, when it first needs them, and every 
pipeline and workflow it starts uses what it read: the child pipelines of a 
Pipeline executor, the pipelines of a workflow, a mapping and so on. A run is a 
top-level pipeline or workflow, the one that Hop Run, Hop GUI, Hop Server or 
the API starts.
+
+A pipeline probe that you add or change while a run is going on takes effect 
with the next run. This matters for runs that go on for a long time, such as a 
workflow that loops for hours or a streaming pipeline.
+
 == Related Plugins
 
 * Pipeline Probe
diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/workflow-log.adoc 
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/workflow-log.adoc
index 9889caab98..e78bd519f5 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/workflow-log.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/workflow-log.adoc
@@ -31,6 +31,12 @@ A workflow log streams logging information from a running 
workflow to a pipeline
 The only requirement for the receiving pipeline is that it starts with a 
xref:pipeline/transforms/workflow-logging.adoc[Workflow logging] transform. 
Other than that, the logging pipeline is "just another pipeline".
 In this logging pipeline, you can process the logging information and e.g. 
write to a relational or NoSQL database, a Kafka topic etc
 
+== When changes take effect
+
+A run reads the workflow logs once, when it first needs them, and every 
pipeline and workflow it starts uses what it read: the child pipelines of a 
Pipeline executor, the pipelines of a workflow, a mapping and so on. A run is a 
top-level pipeline or workflow, the one that Hop Run, Hop GUI, Hop Server or 
the API starts.
+
+A workflow log that you add or change while a run is going on takes effect 
with the next run. This matters for runs that go on for a long time, such as a 
workflow that loops for hours or a streaming pipeline.
+
 == Samples
 
 The samples project comes with an example workflow log.
diff --git 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
index 44a4de16c6..3d6fb4d531 100644
--- 
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
+++ 
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/pipeline-executor.adoc
@@ -89,6 +89,7 @@ Click *Open* to open it in Hop Gui. When the dialog has 
unsaved changes, Hop fir
 
 *NOTE:* It must be considered that, by specifying the pipeline file name using 
this option, we can experience a little performance penalty because the 
pipeline that will be executed will be initialized at runtime and not during 
the initialization phase as usual. On the other side this option gives greater 
flexibility in specifying the pipeline filename dynamically and this could be 
useful for many use-cases.
 |Run configuration|Specify the Pipeline Run Configuration to be used for 
execution.
+The transform reads the run configuration when it starts its first child 
pipeline and uses it for every child pipeline that follows: a change to the run 
configuration while the pipeline runs takes effect with the next run.
 |Stop after (milliseconds)|Maximum time to wait for each child pipeline 
execution attempt. Empty or 0 means wait indefinitely. When the time is reached 
the child pipeline is stopped and the execution-results stream reports a 
failure (`ExecutionResult=N`, `ExecutionNrErrors>=1`) for that attempt. The 
value may contain a variable.
 |Retry attempts on failure|Number of retries after a failed child pipeline 
execution attempt. `0` means no retries (default), so the transform executes 
the child once. The value can use integer shorthand and variables (for example 
`1,000`, `100k`, `1.5m`, `1e3`, `+'${PIPELINE_RETRY_ATTEMPTS}'+`).
 |Retry delay (milliseconds)|Delay between retry attempts. Empty or `0` means 
retry immediately. The delay happens only between attempts and is not part of 
the per-attempt stop-after timeout. The value can use integer shorthand and 
variables (for example `1,000`, `100k`, `1.5m`, `1e3`, 
`+'${PIPELINE_RETRY_DELAY_MS}'+`).
diff --git 
a/engine/src/main/java/org/apache/hop/pipeline/engine/PipelineEngineFactory.java
 
b/engine/src/main/java/org/apache/hop/pipeline/engine/PipelineEngineFactory.java
index 8ad8b0d04f..ffbe2905c5 100644
--- 
a/engine/src/main/java/org/apache/hop/pipeline/engine/PipelineEngineFactory.java
+++ 
b/engine/src/main/java/org/apache/hop/pipeline/engine/PipelineEngineFactory.java
@@ -51,6 +51,22 @@ public class PipelineEngineFactory {
       T pipelineMeta)
       throws HopException {
 
+    return createPipelineEngine(
+        parentVariables,
+        loadRunConfiguration(runConfigurationName, metadataProvider),
+        metadataProvider,
+        pipelineMeta);
+  }
+
+  /**
+   * Loads a pipeline run configuration from the metadata.
+   *
+   * @param runConfigurationName the name of the run configuration
+   * @return the run configuration, never null
+   * @throws HopException when the name is empty or the run configuration 
can't be loaded or found
+   */
+  public static PipelineRunConfiguration loadRunConfiguration(
+      String runConfigurationName, IHopMetadataProvider metadataProvider) 
throws HopException {
     if (StringUtils.isEmpty(runConfigurationName)) {
       throw new HopException("Please specify a run configuration to execute 
the pipeline with");
     }
@@ -69,6 +85,26 @@ public class PipelineEngineFactory {
               + "' in metadata provider: "
               + metadataProvider.getDescription());
     }
+    return pipelineRunConfiguration;
+  }
+
+  /**
+   * Creates a pipeline engine for a run configuration that is loaded already, 
see {@link
+   * #loadRunConfiguration}. A caller that starts the same pipeline many times 
can load the run
+   * configuration once and pass a copy of it for every execution.
+   *
+   * @param parentVariables The parent variables to inherit from
+   * @param pipelineRunConfiguration The run configuration to use. The engine 
keeps it and the
+   *     parent variables are applied to it: do not share one object between 
engines.
+   * @param metadataProvider the metadata of the new engine
+   * @param pipelineMeta the pipeline to run
+   */
+  public static final <T extends PipelineMeta> IPipelineEngine<T> 
createPipelineEngine(
+      IVariables parentVariables,
+      PipelineRunConfiguration pipelineRunConfiguration,
+      IHopMetadataProvider metadataProvider,
+      T pipelineMeta)
+      throws HopException {
 
     // Apply the variables from the run configuration
     //
diff --git a/integration-tests/logging/003-pipeline-executor-child.hpl 
b/integration-tests/logging/003-pipeline-executor-child.hpl
new file mode 100644
index 0000000000..9f711e2a88
--- /dev/null
+++ b/integration-tests/logging/003-pipeline-executor-child.hpl
@@ -0,0 +1,98 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+  <info>
+    <name>003-pipeline-executor-child</name>
+    <name_sync_with_filename>Y</name_sync_with_filename>
+    <description>Started by 003-pipeline-executor-parent for every row it 
gets.</description>
+    <extended_description/>
+    <pipeline_version/>
+    <pipeline_type>Normal</pipeline_type>
+    <parameters>
+    </parameters>
+    <capture_transform_performance>N</capture_transform_performance>
+    
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+    
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+    <created_user>-</created_user>
+    <created_date>2026/11/13 12:00:00.000</created_date>
+    <modified_user>-</modified_user>
+    <modified_date>2026/11/13 12:00:00.000</modified_date>
+  </info>
+  <notepads>
+  </notepads>
+  <order>
+    <hop>
+      <from>one row</from>
+      <to>done</to>
+      <enabled>Y</enabled>
+    </hop>
+  </order>
+  <transform>
+    <name>one row</name>
+    <type>DataGrid</type>
+    <description/>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <data>
+      <line>
+        <item>done</item>
+      </line>
+    </data>
+    <fields>
+      <field>
+        <length>-1</length>
+        <precision>-1</precision>
+        <set_empty_string>N</set_empty_string>
+        <name>child</name>
+        <type>String</type>
+      </field>
+    </fields>
+    <attributes/>
+    <GUI>
+      <xloc>112</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform>
+    <name>done</name>
+    <type>Dummy</type>
+    <description/>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <attributes/>
+    <GUI>
+      <xloc>304</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform_error_handling>
+  </transform_error_handling>
+  <attributes/>
+</pipeline>
diff --git a/integration-tests/logging/003-pipeline-executor-parent.hpl 
b/integration-tests/logging/003-pipeline-executor-parent.hpl
new file mode 100644
index 0000000000..c039c428a9
--- /dev/null
+++ b/integration-tests/logging/003-pipeline-executor-parent.hpl
@@ -0,0 +1,112 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+  <info>
+    <name>003-pipeline-executor-parent</name>
+    <name_sync_with_filename>Y</name_sync_with_filename>
+    <description>Runs 003-pipeline-executor-child three times. The pipeline 
log logs this pipeline and each of its children.</description>
+    <extended_description/>
+    <pipeline_version/>
+    <pipeline_type>Normal</pipeline_type>
+    <parameters>
+    </parameters>
+    <capture_transform_performance>N</capture_transform_performance>
+    
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+    
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+    <created_user>-</created_user>
+    <created_date>2026/11/13 12:00:00.000</created_date>
+    <modified_user>-</modified_user>
+    <modified_date>2026/11/13 12:00:00.000</modified_date>
+  </info>
+  <notepads>
+  </notepads>
+  <order>
+    <hop>
+      <from>three rows</from>
+      <to>one child per row</to>
+      <enabled>Y</enabled>
+    </hop>
+  </order>
+  <transform>
+    <name>three rows</name>
+    <type>DataGrid</type>
+    <description/>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <data>
+      <line>
+        <item>1</item>
+      </line>
+      <line>
+        <item>2</item>
+      </line>
+      <line>
+        <item>3</item>
+      </line>
+    </data>
+    <fields>
+      <field>
+        <length>-1</length>
+        <precision>-1</precision>
+        <set_empty_string>N</set_empty_string>
+        <name>id</name>
+        <type>String</type>
+      </field>
+    </fields>
+    <attributes/>
+    <GUI>
+      <xloc>112</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform>
+    <name>one child per row</name>
+    <type>PipelineExecutor</type>
+    <description>One child pipeline for every row.</description>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <run_configuration>local</run_configuration>
+    <filename>${PROJECT_HOME}/003-pipeline-executor-child.hpl</filename>
+    <group_size>1</group_size>
+    <group_field/>
+    <group_time/>
+    <parameters>
+      <inherit_all_vars>Y</inherit_all_vars>
+    </parameters>
+    <attributes/>
+    <GUI>
+      <xloc>304</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform_error_handling>
+  </transform_error_handling>
+  <attributes/>
+</pipeline>
diff --git a/integration-tests/logging/003-pipeline-log-verify.hpl 
b/integration-tests/logging/003-pipeline-log-verify.hpl
new file mode 100644
index 0000000000..0d226f525b
--- /dev/null
+++ b/integration-tests/logging/003-pipeline-log-verify.hpl
@@ -0,0 +1,121 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+  <info>
+    <name>003-pipeline-log-verify</name>
+    <name_sync_with_filename>Y</name_sync_with_filename>
+    <description/>
+    <extended_description/>
+    <pipeline_version/>
+    <pipeline_type>Normal</pipeline_type>
+    <parameters>
+    </parameters>
+    <capture_transform_performance>N</capture_transform_performance>
+    
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+    
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+    <created_user>-</created_user>
+    <created_date>2026/11/13 12:00:00.000</created_date>
+    <modified_user>-</modified_user>
+    <modified_date>2026/11/13 12:00:00.000</modified_date>
+  </info>
+  <notepads>
+  </notepads>
+  <order>
+    <hop>
+      <from>log file</from>
+      <to>validate</to>
+      <enabled>Y</enabled>
+    </hop>
+  </order>
+  <transform>
+    <name>log file</name>
+    <type>CSVInput</type>
+    <description/>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <filename>${java.io.tmpdir}/hop-it-logging-003-pipeline-log.csv</filename>
+    <filename_field/>
+    <rownum_field/>
+    <include_filename>N</include_filename>
+    <separator>,</separator>
+    <enclosure>"</enclosure>
+    <header>N</header>
+    <buffer_size>50000</buffer_size>
+    <lazy_conversion>N</lazy_conversion>
+    <add_filename_result>N</add_filename_result>
+    <parallel>N</parallel>
+    <newline_possible>N</newline_possible>
+    <encoding>UTF-8</encoding>
+    <fields>
+      <field>
+        <name>pipelineName</name>
+        <type>String</type>
+        <format/>
+        <currency/>
+        <decimal/>
+        <group/>
+        <length>-1</length>
+        <precision>-1</precision>
+        <trim_type>none</trim_type>
+      </field>
+      <field>
+        <name>loggingPhase</name>
+        <type>String</type>
+        <format/>
+        <currency/>
+        <decimal/>
+        <group/>
+        <length>-1</length>
+        <precision>-1</precision>
+        <trim_type>none</trim_type>
+      </field>
+    </fields>
+    <attributes/>
+    <GUI>
+      <xloc>112</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform>
+    <name>validate</name>
+    <type>Dummy</type>
+    <description/>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <attributes/>
+    <GUI>
+      <xloc>304</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform_error_handling>
+  </transform_error_handling>
+  <attributes/>
+</pipeline>
diff --git a/integration-tests/logging/003-pipeline-log.hpl 
b/integration-tests/logging/003-pipeline-log.hpl
new file mode 100644
index 0000000000..45f445af48
--- /dev/null
+++ b/integration-tests/logging/003-pipeline-log.hpl
@@ -0,0 +1,142 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+  <info>
+    <name>003-pipeline-log</name>
+    <name_sync_with_filename>Y</name_sync_with_filename>
+    <description>The pipeline that the pipeline log pipeline-executor-children 
runs at the end of every pipeline it logs.</description>
+    <extended_description/>
+    <pipeline_version/>
+    <pipeline_type>Normal</pipeline_type>
+    <parameters>
+    </parameters>
+    <capture_transform_performance>N</capture_transform_performance>
+    
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
+    
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
+    <created_user>-</created_user>
+    <created_date>2026/11/13 12:00:00.000</created_date>
+    <modified_user>-</modified_user>
+    <modified_date>2026/11/13 12:00:00.000</modified_date>
+  </info>
+  <notepads>
+  </notepads>
+  <order>
+    <hop>
+      <from>pipeline run</from>
+      <to>append to the log file</to>
+      <enabled>Y</enabled>
+    </hop>
+  </order>
+  <transform>
+    <name>pipeline run</name>
+    <type>PipelineLogging</type>
+    <description/>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <log_transforms>N</log_transforms>
+    <attributes/>
+    <GUI>
+      <xloc>112</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform>
+    <name>append to the log file</name>
+    <type>TextFileOutput</type>
+    <description>One row for every pipeline run that is logged.</description>
+    <distribute>Y</distribute>
+    <custom_distribution/>
+    <copies>1</copies>
+    <partitioning>
+      <method>none</method>
+      <schema_name/>
+    </partitioning>
+    <separator>,</separator>
+    <enclosure>"</enclosure>
+    <enclosure_forced>N</enclosure_forced>
+    <enclosure_fix_disabled>N</enclosure_fix_disabled>
+    <header>N</header>
+    <footer>N</footer>
+    <format>UNIX</format>
+    <compression>None</compression>
+    <encoding>UTF-8</encoding>
+    <endedLine/>
+    <fileNameInField>N</fileNameInField>
+    <fileNameField/>
+    <create_parent_folder>Y</create_parent_folder>
+    <file>
+      <name>${java.io.tmpdir}/hop-it-logging-003-pipeline-log</name>
+      <do_not_open_new_file_init>Y</do_not_open_new_file_init>
+      <extension>csv</extension>
+      <append>Y</append>
+      <split>N</split>
+      <haspartno>N</haspartno>
+      <add_date>N</add_date>
+      <add_time>N</add_time>
+      <SpecifyFormat>N</SpecifyFormat>
+      <date_time_format/>
+      <add_to_result_filenames>N</add_to_result_filenames>
+      <pad>N</pad>
+      <fast_dump>N</fast_dump>
+      <splitevery/>
+    </file>
+    <fields>
+      <field>
+        <name>pipelineName</name>
+        <type>String</type>
+        <format/>
+        <currency/>
+        <decimal/>
+        <group/>
+        <nullif/>
+        <trim_type>none</trim_type>
+        <roundingType>half_even</roundingType>
+        <length>-1</length>
+        <precision>-1</precision>
+      </field>
+      <field>
+        <name>loggingPhase</name>
+        <type>String</type>
+        <format/>
+        <currency/>
+        <decimal/>
+        <group/>
+        <nullif/>
+        <trim_type>none</trim_type>
+        <roundingType>half_even</roundingType>
+        <length>-1</length>
+        <precision>-1</precision>
+      </field>
+    </fields>
+    <attributes/>
+    <GUI>
+      <xloc>304</xloc>
+      <yloc>64</yloc>
+    </GUI>
+  </transform>
+  <transform_error_handling>
+  </transform_error_handling>
+  <attributes/>
+</pipeline>
diff --git 
a/integration-tests/logging/datasets/golden-pipeline-executor-children-log.csv 
b/integration-tests/logging/datasets/golden-pipeline-executor-children-log.csv
new file mode 100644
index 0000000000..817aef90ca
--- /dev/null
+++ 
b/integration-tests/logging/datasets/golden-pipeline-executor-children-log.csv
@@ -0,0 +1,5 @@
+pipelineName,loggingPhase
+003-pipeline-executor-child,end
+003-pipeline-executor-child,end
+003-pipeline-executor-child,end
+003-pipeline-executor-parent,end
diff --git 
a/integration-tests/logging/main-003-pipeline-executor-child-logging.hwf 
b/integration-tests/logging/main-003-pipeline-executor-child-logging.hwf
new file mode 100644
index 0000000000..1e7e7a8afa
--- /dev/null
+++ b/integration-tests/logging/main-003-pipeline-executor-child-logging.hwf
@@ -0,0 +1,133 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+  <name>main-003-pipeline-executor-child-logging</name>
+  <name_sync_with_filename>Y</name_sync_with_filename>
+  <description>A pipeline log that does not log parents only logs every child 
pipeline of a Pipeline executor, as well as the parent: four runs, four 
lines.</description>
+  <extended_description/>
+  <workflow_version/>
+  <created_user>-</created_user>
+  <created_date>2026/11/13 12:00:00.000</created_date>
+  <modified_user>-</modified_user>
+  <modified_date>2026/11/13 12:00:00.000</modified_date>
+  <parameters>
+    </parameters>
+  <actions>
+    <action>
+      <name>Start</name>
+      <description/>
+      <type>SPECIAL</type>
+      <attributes/>
+      <DayOfMonth>1</DayOfMonth>
+      <hour>12</hour>
+      <intervalMinutes>60</intervalMinutes>
+      <intervalSeconds>0</intervalSeconds>
+      <minutes>0</minutes>
+      <repeat>N</repeat>
+      <schedulerType>0</schedulerType>
+      <weekDay>1</weekDay>
+      <parallel>N</parallel>
+      <xloc>112</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>delete the log file</name>
+      <description/>
+      <type>DELETE_FILE</type>
+      <attributes/>
+      <fail_if_file_not_exists>N</fail_if_file_not_exists>
+      
<filename>${java.io.tmpdir}/hop-it-logging-003-pipeline-log.csv</filename>
+      <parallel>N</parallel>
+      <xloc>304</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>003-pipeline-executor-parent</name>
+      <description/>
+      <type>PIPELINE</type>
+      <attributes/>
+      <filename>${PROJECT_HOME}/003-pipeline-executor-parent.hpl</filename>
+      <params_from_previous>N</params_from_previous>
+      <exec_per_row>N</exec_per_row>
+      <clear_rows>N</clear_rows>
+      <clear_files>N</clear_files>
+      <set_logfile>N</set_logfile>
+      <logfile/>
+      <logext/>
+      <add_date>N</add_date>
+      <add_time>N</add_time>
+      <loglevel>Basic</loglevel>
+      <set_append_logfile>N</set_append_logfile>
+      <wait_until_finished>Y</wait_until_finished>
+      <create_parent_folder>N</create_parent_folder>
+      <run_configuration>local</run_configuration>
+      <parameters>
+        <pass_all_parameters>Y</pass_all_parameters>
+      </parameters>
+      <parallel>N</parallel>
+      <xloc>496</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+    <action>
+      <name>003-pipeline-log-verify UNIT</name>
+      <description/>
+      <type>RunPipelineTests</type>
+      <attributes/>
+      <test_names>
+        <test_name>
+          <name>003-pipeline-log-verify UNIT</name>
+        </test_name>
+      </test_names>
+      <parallel>N</parallel>
+      <xloc>688</xloc>
+      <yloc>64</yloc>
+      <attributes_hac/>
+    </action>
+  </actions>
+  <hops>
+    <hop>
+      <from>Start</from>
+      <to>delete the log file</to>
+      <enabled>Y</enabled>
+      <evaluation>Y</evaluation>
+      <unconditional>Y</unconditional>
+    </hop>
+    <hop>
+      <from>delete the log file</from>
+      <to>003-pipeline-executor-parent</to>
+      <enabled>Y</enabled>
+      <evaluation>Y</evaluation>
+      <unconditional>N</unconditional>
+    </hop>
+    <hop>
+      <from>003-pipeline-executor-parent</from>
+      <to>003-pipeline-log-verify UNIT</to>
+      <enabled>Y</enabled>
+      <evaluation>Y</evaluation>
+      <unconditional>N</unconditional>
+    </hop>
+  </hops>
+  <notepads>
+  </notepads>
+  <attributes/>
+</workflow>
diff --git 
a/integration-tests/logging/metadata/dataset/golden-pipeline-executor-children-log.json
 
b/integration-tests/logging/metadata/dataset/golden-pipeline-executor-children-log.json
new file mode 100644
index 0000000000..1beeca7919
--- /dev/null
+++ 
b/integration-tests/logging/metadata/dataset/golden-pipeline-executor-children-log.json
@@ -0,0 +1,24 @@
+{
+  "base_filename": "golden-pipeline-executor-children-log.csv",
+  "name": "golden-pipeline-executor-children-log",
+  "description": "",
+  "dataset_fields": [
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 2,
+      "field_precision": -1,
+      "field_name": "pipelineName",
+      "field_format": ""
+    },
+    {
+      "field_comment": "",
+      "field_length": -1,
+      "field_type": 2,
+      "field_precision": -1,
+      "field_name": "loggingPhase",
+      "field_format": ""
+    }
+  ],
+  "folder_name": ""
+}
diff --git 
a/integration-tests/logging/metadata/pipeline-log/pipeline-executor-children.json
 
b/integration-tests/logging/metadata/pipeline-log/pipeline-executor-children.json
new file mode 100644
index 0000000000..9054f4713b
--- /dev/null
+++ 
b/integration-tests/logging/metadata/pipeline-log/pipeline-executor-children.json
@@ -0,0 +1,20 @@
+{
+  "name": "pipeline-executor-children",
+  "enabled": true,
+  "loggingParentsOnly": false,
+  "pipelineFilename": "${PROJECT_HOME}/003-pipeline-log.hpl",
+  "executingAtStart": false,
+  "executingPeriodically": false,
+  "intervalInSeconds": "30",
+  "executingAtEnd": true,
+  "logLevel": "BASIC",
+  "pipelinesToLog": [
+    {
+      "pipelineToLogFilename": 
"${PROJECT_HOME}/003-pipeline-executor-parent.hpl"
+    },
+    {
+      "pipelineToLogFilename": 
"${PROJECT_HOME}/003-pipeline-executor-child.hpl"
+    }
+  ],
+  "failParentOnLoggingFailure": true
+}
diff --git 
a/integration-tests/logging/metadata/unit-test/003-pipeline-log-verify 
UNIT.json 
b/integration-tests/logging/metadata/unit-test/003-pipeline-log-verify UNIT.json
new file mode 100644
index 0000000000..bef5e5b995
--- /dev/null
+++ b/integration-tests/logging/metadata/unit-test/003-pipeline-log-verify 
UNIT.json    
@@ -0,0 +1,33 @@
+{
+  "database_replacements": [],
+  "autoOpening": true,
+  "description": "",
+  "persist_filename": "",
+  "test_type": "UNIT_TEST",
+  "variableValues": [],
+  "basePath": "${HOP_UNIT_TESTS_FOLDER}",
+  "golden_data_sets": [
+    {
+      "field_mappings": [
+        {
+          "transform_field": "pipelineName",
+          "data_set_field": "pipelineName"
+        },
+        {
+          "transform_field": "loggingPhase",
+          "data_set_field": "loggingPhase"
+        }
+      ],
+      "field_order": [
+        "pipelineName",
+        "loggingPhase"
+      ],
+      "data_set_name": "golden-pipeline-executor-children-log",
+      "transform_name": "validate"
+    }
+  ],
+  "input_data_sets": [],
+  "name": "003-pipeline-log-verify UNIT",
+  "trans_test_tweaks": [],
+  "pipeline_filename": "./003-pipeline-log-verify.hpl"
+}
diff --git 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/RunMetadata.java
 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/RunMetadata.java
new file mode 100644
index 0000000000..93c0ca3982
--- /dev/null
+++ 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/RunMetadata.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.reflection;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.WeakHashMap;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.metadata.api.IHopMetadata;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+import org.apache.hop.pipeline.engine.IPipelineEngine;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * The metadata objects of a type as they were when a run first needed them.
+ *
+ * <p>Pipeline logs, workflow logs and pipeline probes are looked at every 
time a pipeline or
+ * workflow starts. A run that starts thousands of child pipelines (a Pipeline 
executor, a mapping,
+ * a loop in a workflow) would read the same metadata from disk thousands of 
times. Here the objects
+ * are read once per run and type: the run is the top-level pipeline or 
workflow, and everything it
+ * starts shares what was read. A change to this metadata while the run is 
going on takes effect
+ * with the next run.
+ *
+ * <p>What a run read is dropped when its top-level pipeline or workflow 
finishes: an engine can run
+ * more than once, a pipeline that Hop Server starts again for one, and every 
run reads again.
+ */
+public final class RunMetadata {
+
+  /**
+   * Per top-level pipeline or workflow, until it finishes. The weak keys are 
a backstop for an
+   * engine that never does.
+   */
+  private static final Map<Object, Map<Class<?>, List<?>>> RUNS =
+      Collections.synchronizedMap(new WeakHashMap<>());
+
+  private RunMetadata() {}
+
+  /**
+   * @param engine the pipeline or workflow that starts
+   * @param metadataProvider where to read the objects when the run does not 
have them yet
+   * @param type the metadata type
+   * @return all objects of the type, read once per run; do not change the 
list or its objects
+   */
+  @SuppressWarnings("unchecked")
+  public static <T extends IHopMetadata> List<T> loadAll(
+      Object engine, IHopMetadataProvider metadataProvider, Class<T> type) 
throws HopException {
+    Map<Class<?>, List<?>> run = run(root(engine));
+    List<?> objects = run.get(type);
+    if (objects == null) {
+      // Two children starting at the same moment may both read: harmless, the 
first one stays
+      List<T> loaded = 
List.copyOf(metadataProvider.getSerializer(type).loadAll());
+      List<?> earlier = run.putIfAbsent(type, loaded);
+      objects = earlier == null ? loaded : earlier;
+    }
+    return (List<T>) objects;
+  }
+
+  /** What the run of this top-level pipeline or workflow read so far. */
+  private static Map<Class<?>, List<?>> run(Object root) {
+    Map<Class<?>, List<?>> run;
+    synchronized (RUNS) {
+      run = RUNS.get(root);
+      if (run != null) {
+        return run;
+      }
+      run = new ConcurrentHashMap<>();
+      RUNS.put(root, run);
+    }
+    forgetWhenFinished(root, run);
+    return run;
+  }
+
+  /**
+   * Drops what the run read when it finishes. Only this run's entry goes: the 
listener stays on an
+   * engine that runs again, and must not drop what the next run read.
+   */
+  private static void forgetWhenFinished(Object root, Map<Class<?>, List<?>> 
run) {
+    if (root instanceof IPipelineEngine<?> pipeline) {
+      pipeline.addExecutionFinishedListener(finished -> RUNS.remove(root, 
run));
+    } else if (root instanceof IWorkflowEngine<?> workflow) {
+      workflow.addExecutionFinishedListener(finished -> RUNS.remove(root, 
run));
+    }
+  }
+
+  /** The top-level pipeline or workflow that the engine runs in. */
+  static Object root(Object engine) {
+    Object current = engine;
+    while (true) {
+      Object parent = parent(current);
+      if (parent == null) {
+        return current;
+      }
+      current = parent;
+    }
+  }
+
+  private static Object parent(Object engine) {
+    if (engine instanceof IPipelineEngine<?> pipeline) {
+      return pipeline.getParentPipeline() != null
+          ? pipeline.getParentPipeline()
+          : pipeline.getParentWorkflow();
+    }
+    if (engine instanceof IWorkflowEngine<?> workflow) {
+      return workflow.getParentWorkflow() != null
+          ? workflow.getParentWorkflow()
+          : workflow.getParentPipeline();
+    }
+    return null;
+  }
+}
diff --git 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/pipeline/xp/PipelineStartLoggingXp.java
 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/pipeline/xp/PipelineStartLoggingXp.java
index 66ea83a8fa..82acf10ac8 100644
--- 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/pipeline/xp/PipelineStartLoggingXp.java
+++ 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/pipeline/xp/PipelineStartLoggingXp.java
@@ -31,13 +31,12 @@ import org.apache.hop.core.logging.ILogChannel;
 import org.apache.hop.core.util.ExecutorUtil;
 import org.apache.hop.core.variables.IVariables;
 import org.apache.hop.core.vfs.HopVfs;
-import org.apache.hop.metadata.api.IHopMetadataProvider;
-import org.apache.hop.metadata.api.IHopMetadataSerializer;
 import org.apache.hop.pipeline.Pipeline;
 import org.apache.hop.pipeline.PipelineMeta;
 import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.engines.local.LocalPipelineEngine;
 import org.apache.hop.pipeline.transform.TransformMetaDataCombi;
+import org.apache.hop.reflection.RunMetadata;
 import org.apache.hop.reflection.pipeline.meta.PipelineLog;
 import org.apache.hop.reflection.pipeline.meta.PipelineToLogLocation;
 import org.apache.hop.reflection.pipeline.transform.PipelineLogging;
@@ -66,10 +65,8 @@ public class PipelineStartLoggingXp implements 
IExtensionPoint<Pipeline> {
       return;
     }
 
-    IHopMetadataProvider metadataProvider = pipeline.getMetadataProvider();
-    IHopMetadataSerializer<PipelineLog> serializer =
-        metadataProvider.getSerializer(PipelineLog.class);
-    List<PipelineLog> pipelineLogs = serializer.loadAll();
+    List<PipelineLog> pipelineLogs =
+        RunMetadata.loadAll(pipeline, pipeline.getMetadataProvider(), 
PipelineLog.class);
 
     for (PipelineLog pipelineLog : pipelineLogs) {
       handlePipelineLog(log, pipelineLog, pipeline, variables);
diff --git 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/probe/xp/PipelineDataProbeXp.java
 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/probe/xp/PipelineDataProbeXp.java
index 4b488550cd..f474f3d9d0 100644
--- 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/probe/xp/PipelineDataProbeXp.java
+++ 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/probe/xp/PipelineDataProbeXp.java
@@ -31,8 +31,6 @@ import org.apache.hop.core.row.IRowMeta;
 import org.apache.hop.core.util.Utils;
 import org.apache.hop.core.variables.IVariables;
 import org.apache.hop.core.vfs.HopVfs;
-import org.apache.hop.metadata.api.IHopMetadataProvider;
-import org.apache.hop.metadata.api.IHopMetadataSerializer;
 import org.apache.hop.pipeline.Pipeline;
 import org.apache.hop.pipeline.PipelineMeta;
 import org.apache.hop.pipeline.RowProducer;
@@ -41,6 +39,7 @@ import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.engines.local.LocalPipelineEngine;
 import org.apache.hop.pipeline.transform.RowAdapter;
 import org.apache.hop.pipeline.transform.TransformMetaDataCombi;
+import org.apache.hop.reflection.RunMetadata;
 import org.apache.hop.reflection.pipeline.xp.PipelineStartLoggingXp;
 import org.apache.hop.reflection.probe.meta.DataProbeLocation;
 import org.apache.hop.reflection.probe.meta.PipelineProbe;
@@ -64,10 +63,8 @@ public class PipelineDataProbeXp implements 
IExtensionPoint<Pipeline> {
       return;
     }
 
-    IHopMetadataProvider metadataProvider = pipeline.getMetadataProvider();
-    IHopMetadataSerializer<PipelineProbe> serializer =
-        metadataProvider.getSerializer(PipelineProbe.class);
-    List<PipelineProbe> pipelineDataProbes = serializer.loadAll();
+    List<PipelineProbe> pipelineDataProbes =
+        RunMetadata.loadAll(pipeline, pipeline.getMetadataProvider(), 
PipelineProbe.class);
 
     for (PipelineProbe pipelineProbe : pipelineDataProbes) {
       handlePipelineProbe(log, pipelineProbe, pipeline, variables);
diff --git 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/workflow/xp/WorkflowStartLoggingXp.java
 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/workflow/xp/WorkflowStartLoggingXp.java
index cc1c90a914..46b1b39f94 100644
--- 
a/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/workflow/xp/WorkflowStartLoggingXp.java
+++ 
b/plugins/misc/reflection/src/main/java/org/apache/hop/reflection/workflow/xp/WorkflowStartLoggingXp.java
@@ -30,11 +30,10 @@ import org.apache.hop.core.logging.ILogChannel;
 import org.apache.hop.core.util.ExecutorUtil;
 import org.apache.hop.core.variables.IVariables;
 import org.apache.hop.core.vfs.HopVfs;
-import org.apache.hop.metadata.api.IHopMetadataProvider;
-import org.apache.hop.metadata.api.IHopMetadataSerializer;
 import org.apache.hop.pipeline.PipelineMeta;
 import org.apache.hop.pipeline.engines.local.LocalPipelineEngine;
 import org.apache.hop.pipeline.transform.TransformMetaDataCombi;
+import org.apache.hop.reflection.RunMetadata;
 import org.apache.hop.reflection.pipeline.xp.PipelineStartLoggingXp;
 import org.apache.hop.reflection.workflow.meta.WorkflowLog;
 import org.apache.hop.reflection.workflow.transform.WorkflowLogging;
@@ -52,10 +51,8 @@ public class WorkflowStartLoggingXp implements 
IExtensionPoint<IWorkflowEngine<W
       ILogChannel log, IVariables variables, IWorkflowEngine<WorkflowMeta> 
workflow)
       throws HopException {
 
-    IHopMetadataProvider metadataProvider = workflow.getMetadataProvider();
-    IHopMetadataSerializer<WorkflowLog> serializer =
-        metadataProvider.getSerializer(WorkflowLog.class);
-    List<WorkflowLog> workflowLogs = serializer.loadAll();
+    List<WorkflowLog> workflowLogs =
+        RunMetadata.loadAll(workflow, workflow.getMetadataProvider(), 
WorkflowLog.class);
 
     for (WorkflowLog workflowLog : workflowLogs) {
       handleWorkflowLog(log, workflowLog, workflow, variables);
diff --git 
a/plugins/misc/reflection/src/test/java/org/apache/hop/reflection/RunMetadataTest.java
 
b/plugins/misc/reflection/src/test/java/org/apache/hop/reflection/RunMetadataTest.java
new file mode 100644
index 0000000000..eb30e809e2
--- /dev/null
+++ 
b/plugins/misc/reflection/src/test/java/org/apache/hop/reflection/RunMetadataTest.java
@@ -0,0 +1,176 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.reflection;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.metadata.api.IHopMetadata;
+import org.apache.hop.metadata.api.IHopMetadataSerializer;
+import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.apache.hop.pipeline.IExecutionFinishedListener;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.engine.IPipelineEngine;
+import org.apache.hop.reflection.pipeline.meta.PipelineLog;
+import org.apache.hop.reflection.workflow.meta.WorkflowLog;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+
+class RunMetadataTest {
+
+  /** Counts how often metadata is asked for. */
+  private static final class CountingMetadataProvider extends 
MemoryMetadataProvider {
+    final AtomicInteger reads = new AtomicInteger();
+
+    @Override
+    public <T extends IHopMetadata> IHopMetadataSerializer<T> 
getSerializer(Class<T> managedClass)
+        throws HopException {
+      reads.incrementAndGet();
+      return super.getSerializer(managedClass);
+    }
+  }
+
+  @SuppressWarnings("unchecked")
+  private static IPipelineEngine<PipelineMeta> pipeline(
+      IPipelineEngine<PipelineMeta> parentPipeline, 
IWorkflowEngine<WorkflowMeta> parentWorkflow) {
+    IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+    when(pipeline.getParentPipeline()).thenReturn(parentPipeline);
+    when(pipeline.getParentWorkflow()).thenReturn(parentWorkflow);
+    return pipeline;
+  }
+
+  @SuppressWarnings("unchecked")
+  private static IWorkflowEngine<WorkflowMeta> workflow(
+      IWorkflowEngine<WorkflowMeta> parentWorkflow, 
IPipelineEngine<PipelineMeta> parentPipeline) {
+    IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+    when(workflow.getParentWorkflow()).thenReturn(parentWorkflow);
+    when(workflow.getParentPipeline()).thenReturn(parentPipeline);
+    return workflow;
+  }
+
+  @Test
+  void theRunIsTheTopLevelPipelineOrWorkflow() {
+    IWorkflowEngine<WorkflowMeta> top = workflow(null, null);
+    IPipelineEngine<PipelineMeta> inWorkflow = pipeline(null, top);
+    IPipelineEngine<PipelineMeta> child = pipeline(inWorkflow, null);
+    IWorkflowEngine<WorkflowMeta> workflowInPipeline = workflow(null, child);
+    IPipelineEngine<PipelineMeta> deepest = pipeline(null, workflowInPipeline);
+
+    assertSame(top, RunMetadata.root(top));
+    assertSame(top, RunMetadata.root(child));
+    assertSame(top, RunMetadata.root(deepest));
+
+    IPipelineEngine<PipelineMeta> alone = pipeline(null, null);
+    assertSame(alone, RunMetadata.root(alone));
+  }
+
+  @Test
+  void metadataIsReadOncePerRunAndType() throws Exception {
+    CountingMetadataProvider provider = new CountingMetadataProvider();
+    provider.getSerializer(PipelineLog.class).save(new PipelineLog("first"));
+    provider.reads.set(0);
+
+    IPipelineEngine<PipelineMeta> top = pipeline(null, null);
+    IPipelineEngine<PipelineMeta> childA = pipeline(top, null);
+    IPipelineEngine<PipelineMeta> childB = pipeline(top, null);
+
+    List<PipelineLog> forTop = RunMetadata.loadAll(top, provider, 
PipelineLog.class);
+    List<PipelineLog> forA = RunMetadata.loadAll(childA, provider, 
PipelineLog.class);
+    List<PipelineLog> forB = RunMetadata.loadAll(childB, provider, 
PipelineLog.class);
+
+    assertEquals(1, provider.reads.get());
+    assertEquals(1, forTop.size());
+    assertEquals("first", forTop.get(0).getName());
+    assertSame(forTop, forA);
+    assertSame(forTop, forB);
+
+    // another type is read on its own, once
+    assertEquals(0, RunMetadata.loadAll(childA, provider, 
WorkflowLog.class).size());
+    RunMetadata.loadAll(childB, provider, WorkflowLog.class);
+    assertEquals(2, provider.reads.get());
+  }
+
+  @Test
+  void aChangeIsSeenByTheNextRunNotByTheOneGoingOn() throws Exception {
+    CountingMetadataProvider provider = new CountingMetadataProvider();
+    provider.getSerializer(PipelineLog.class).save(new PipelineLog("first"));
+
+    IPipelineEngine<PipelineMeta> run = pipeline(null, null);
+    assertEquals(1, RunMetadata.loadAll(run, provider, 
PipelineLog.class).size());
+
+    provider.getSerializer(PipelineLog.class).save(new PipelineLog("second"));
+
+    // the run that is going on keeps what it read when it started
+    assertEquals(1, RunMetadata.loadAll(pipeline(run, null), provider, 
PipelineLog.class).size());
+    // a new run reads again
+    IPipelineEngine<PipelineMeta> nextRun = pipeline(null, null);
+    assertEquals(2, RunMetadata.loadAll(nextRun, provider, 
PipelineLog.class).size());
+  }
+
+  /**
+   * An engine can run more than once, a pipeline that Hop Server starts again 
for one. What a run
+   * read goes when its top-level pipeline or workflow finishes, so the next 
run reads again.
+   */
+  @Test
+  @SuppressWarnings("unchecked")
+  void anEngineThatRunsAgainReadsAgain() throws Exception {
+    CountingMetadataProvider provider = new CountingMetadataProvider();
+    provider.getSerializer(PipelineLog.class).save(new PipelineLog("first"));
+    provider.getSerializer(WorkflowLog.class).save(new WorkflowLog("first"));
+
+    IPipelineEngine<PipelineMeta> pipeline = pipeline(null, null);
+    assertEquals(1, RunMetadata.loadAll(pipeline, provider, 
PipelineLog.class).size());
+    ArgumentCaptor<IExecutionFinishedListener<IPipelineEngine<PipelineMeta>>> 
pipelineFinished =
+        ArgumentCaptor.forClass(IExecutionFinishedListener.class);
+    verify(pipeline).addExecutionFinishedListener(pipelineFinished.capture());
+
+    IWorkflowEngine<WorkflowMeta> workflow = workflow(null, null);
+    assertEquals(1, RunMetadata.loadAll(workflow, provider, 
WorkflowLog.class).size());
+    ArgumentCaptor<IExecutionFinishedListener<IWorkflowEngine<WorkflowMeta>>> 
workflowFinished =
+        ArgumentCaptor.forClass(IExecutionFinishedListener.class);
+    verify(workflow).addExecutionFinishedListener(workflowFinished.capture());
+
+    provider.getSerializer(PipelineLog.class).save(new PipelineLog("second"));
+    provider.getSerializer(WorkflowLog.class).save(new WorkflowLog("second"));
+
+    // the children of the runs that are going on share what was read, and add 
no listeners
+    assertEquals(
+        1, RunMetadata.loadAll(pipeline(pipeline, null), provider, 
PipelineLog.class).size());
+    assertEquals(
+        1, RunMetadata.loadAll(workflow(workflow, null), provider, 
WorkflowLog.class).size());
+    verify(pipeline, times(1)).addExecutionFinishedListener(any());
+    verify(workflow, times(1)).addExecutionFinishedListener(any());
+
+    pipelineFinished.getValue().finished(pipeline);
+    workflowFinished.getValue().finished(workflow);
+
+    // started again: the same engines read again
+    assertEquals(2, RunMetadata.loadAll(pipeline, provider, 
PipelineLog.class).size());
+    assertEquals(2, RunMetadata.loadAll(workflow, provider, 
WorkflowLog.class).size());
+  }
+}
diff --git 
a/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutor.java
 
b/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutor.java
index 7f68413bf7..893abda6ef 100644
--- 
a/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutor.java
+++ 
b/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutor.java
@@ -20,6 +20,7 @@ package org.apache.hop.pipeline.transforms.pipelineexecutor;
 import com.google.common.annotations.VisibleForTesting;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.HashSet;
 import java.util.LinkedHashMap;
 import java.util.LinkedHashSet;
 import java.util.List;
@@ -45,6 +46,7 @@ import org.apache.hop.i18n.BaseMessages;
 import org.apache.hop.pipeline.Pipeline;
 import org.apache.hop.pipeline.PipelineMeta;
 import org.apache.hop.pipeline.TransformWithMappingMeta;
+import org.apache.hop.pipeline.config.PipelineRunConfiguration;
 import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.engine.PipelineEngineFactory;
 import org.apache.hop.pipeline.transform.BaseTransform;
@@ -84,14 +86,6 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
         return false;
       }
 
-      List<String> incomingFieldValues = new ArrayList<>();
-      if (getInputRowMeta() != null) {
-        for (int i = 0; i < getInputRowMeta().size(); i++) {
-          String fieldvalue = getInputRowMeta().getString(row, i);
-          incomingFieldValues.add(fieldvalue);
-        }
-      }
-
       if (first) {
 
         first = false;
@@ -112,7 +106,7 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
           if (pipelineExecutorData.prevFilename != null
               && pipelineExecutorData.groupSize > 0
               && !pipelineExecutorData.groupBuffer.isEmpty()) {
-            executePipeline(incomingFieldValues);
+            executePipeline(fieldValues(getInputRowMeta(), row));
           }
           if (isDetailed()) {
             logDetailed("Identified a new pipeline to execute: '" + filename + 
"'");
@@ -148,7 +142,7 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
         } else if (pipelineExecutorData.groupTime > 0) { // grouping by 
execution time
           long now = System.currentTimeMillis();
           if (now - pipelineExecutorData.groupTimeStart >= 
pipelineExecutorData.groupTime) {
-            executePipeline(incomingFieldValues);
+            executePipeline(fieldValues(getInputRowMeta(), row));
           }
         }
       }
@@ -162,7 +156,7 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
       // If group buffer size exceeds specified limit, then execute pipeline 
and flush group buffer.
       if (pipelineExecutorData.groupSize > 0
           && pipelineExecutorData.groupBuffer.size() >= 
pipelineExecutorData.groupSize) {
-        executePipeline(incomingFieldValues);
+        executePipeline(fieldValues(getInputRowMeta(), row));
       }
 
       return true;
@@ -171,6 +165,46 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
     }
   }
 
+  /**
+   * The values of a row as text, for the parameters that take their value 
from a field, see {@link
+   * #passParametersToPipeline(List)}. Only asked for when a child pipeline is 
about to run, and
+   * only the fields a parameter takes its value from are converted: with a 
group size of one a
+   * child pipeline runs for every row.
+   *
+   * @return one value per field of the row, null for a field that no 
parameter uses
+   */
+  private List<String> fieldValues(IRowMeta rowMeta, Object[] row) throws 
HopValueException {
+    List<String> values = new ArrayList<>();
+    if (rowMeta != null) {
+      Set<String> parameterFields = parameterFields();
+      for (int i = 0; i < rowMeta.size(); i++) {
+        values.add(
+            parameterFields.contains(rowMeta.getValueMeta(i).getName())
+                ? rowMeta.getString(row, i)
+                : null);
+      }
+    }
+    return values;
+  }
+
+  /** The names of the fields that parameters take their value from, as they 
are matched. */
+  private Set<String> parameterFields() {
+    PipelineExecutorData pipelineExecutorData = getData();
+    if (pipelineExecutorData.parameterFields == null) {
+      Set<String> fields = new HashSet<>();
+      List<PipelineExecutorParameters> parameters = meta.getParameters();
+      if (parameters != null) {
+        for (PipelineExecutorParameters parameter : parameters) {
+          if (parameter != null && parameter.getField() != null) {
+            fields.add(parameter.getField());
+          }
+        }
+      }
+      pipelineExecutorData.parameterFields = fields;
+    }
+    return pipelineExecutorData.parameterFields;
+  }
+
   private void initOnFirstProcessingIteration() throws HopException {
     PipelineExecutorData pipelineExecutorData = getData();
     // internal pipeline's first transform has exactly the same input
@@ -431,7 +465,10 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
     // HOP_DATA the same way when running under Native Spark mapPartitions.
     IPipelineEngine<PipelineMeta> executorPipeline =
         PipelineEngineFactory.createPipelineEngine(
-            this, runConfigurationName, metadataProvider, 
getData().getExecutorPipelineMeta());
+            this,
+            runConfiguration(runConfigurationName),
+            metadataProvider,
+            getData().getExecutorPipelineMeta());
     executorPipeline.setParentPipeline(getPipeline());
     executorPipeline.setParent(this);
     executorPipeline.setLogLevel(getLogLevel());
@@ -441,6 +478,22 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
     return executorPipeline;
   }
 
+  /**
+   * The run configuration of the child pipeline. It is read from the metadata 
the first time this
+   * transform copy needs it and kept for the rest of the run: a change to the 
run configuration
+   * while the pipeline runs does not reach the executions that follow. Every 
execution gets its own
+   * copy, because the engine keeps it and applies variables to it.
+   */
+  private PipelineRunConfiguration runConfiguration(String name) throws 
HopException {
+    Map<String, PipelineRunConfiguration> loaded = getData().runConfigurations;
+    PipelineRunConfiguration runConfiguration = loaded.get(name);
+    if (runConfiguration == null) {
+      runConfiguration = PipelineEngineFactory.loadRunConfiguration(name, 
metadataProvider);
+      loaded.put(name, runConfiguration);
+    }
+    return new PipelineRunConfiguration(runConfiguration);
+  }
+
   @VisibleForTesting
   void passParametersToPipeline(List<String> incomingFieldValues) {
     // The values of the incoming fields from the previous transform.
@@ -742,6 +795,10 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
     return getData().getExecutorPipeline();
   }
 
+  /**
+   * @return the values of the last row in the group, as text, for the 
parameters that take their
+   *     value from a field: null for a field that no parameter uses. Null 
when there is no row.
+   */
   protected List<String> getLastIncomingFieldValues() throws HopException {
     PipelineExecutorData pipelineExecutorData = getData();
     if (pipelineExecutorData == null
@@ -758,14 +815,10 @@ public class PipelineExecutor extends 
BaseTransform<PipelineExecutorMeta, Pipeli
       return null;
     }
 
-    List<String> lastIncomingFieldValues = new ArrayList<>(rowMeta.size());
-    for (int i = 0; i < rowMeta.size(); i++) {
-      try {
-        lastIncomingFieldValues.add(rowMeta.getString(rowData, i));
-      } catch (HopValueException e) {
-        throw new HopException("Error converting buffered row values", e);
-      }
+    try {
+      return fieldValues(rowMeta, rowData);
+    } catch (HopValueException e) {
+      throw new HopException("Error converting buffered row values", e);
     }
-    return lastIncomingFieldValues;
   }
 }
diff --git 
a/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorData.java
 
b/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorData.java
index 40a921ff20..131e495630 100644
--- 
a/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorData.java
+++ 
b/plugins/transforms/pipelineexecutor/src/main/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorData.java
@@ -17,7 +17,10 @@
 
 package org.apache.hop.pipeline.transforms.pipelineexecutor;
 
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
+import java.util.Set;
 import lombok.Getter;
 import lombok.Setter;
 import org.apache.hop.core.IRowSet;
@@ -25,6 +28,7 @@ import org.apache.hop.core.RowMetaAndData;
 import org.apache.hop.core.row.IRowMeta;
 import org.apache.hop.core.row.IValueMeta;
 import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.config.PipelineRunConfiguration;
 import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.transform.BaseTransformData;
 import org.apache.hop.pipeline.transform.ITransformData;
@@ -61,6 +65,12 @@ public class PipelineExecutorData extends BaseTransformData 
implements ITransfor
 
   public Object prevGroupFieldData;
 
+  /** The run configurations this copy loaded, by name: read once and kept for 
the run. */
+  public final Map<String, PipelineRunConfiguration> runConfigurations = new 
HashMap<>();
+
+  /** The names of the fields that parameters take their value from, worked 
out once. */
+  public Set<String> parameterFields;
+
   private IRowSet executorTransformOutputRowSet;
   private IRowSet resultRowsRowSet;
   private IRowSet resultFilesRowSet;
diff --git 
a/plugins/transforms/pipelineexecutor/src/test/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorTest.java
 
b/plugins/transforms/pipelineexecutor/src/test/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorTest.java
index e0f88c1c14..57731f2bf6 100644
--- 
a/plugins/transforms/pipelineexecutor/src/test/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorTest.java
+++ 
b/plugins/transforms/pipelineexecutor/src/test/java/org/apache/hop/pipeline/transforms/pipelineexecutor/PipelineExecutorTest.java
@@ -20,7 +20,10 @@ package org.apache.hop.pipeline.transforms.pipelineexecutor;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
 import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 import static org.mockito.ArgumentMatchers.any;
 import static org.mockito.ArgumentMatchers.anyLong;
@@ -41,6 +44,7 @@ import static org.mockito.Mockito.when;
 
 import java.net.URISyntaxException;
 import java.nio.file.Paths;
+import java.time.Duration;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
@@ -48,21 +52,27 @@ import java.util.Date;
 import java.util.List;
 import java.util.Objects;
 import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hop.core.BlockingRowSet;
 import org.apache.hop.core.HopEnvironment;
 import org.apache.hop.core.Result;
 import org.apache.hop.core.RowMetaAndData;
 import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.exception.HopValueException;
 import org.apache.hop.core.row.RowMeta;
 import org.apache.hop.core.row.value.ValueMetaBoolean;
 import org.apache.hop.core.row.value.ValueMetaDate;
 import org.apache.hop.core.row.value.ValueMetaString;
 import org.apache.hop.core.variables.Variables;
 import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.metadata.api.IHopMetadata;
+import org.apache.hop.metadata.api.IHopMetadataSerializer;
 import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
 import org.apache.hop.pipeline.PipelineMeta;
 import org.apache.hop.pipeline.TransformWithMappingMeta;
+import org.apache.hop.pipeline.config.PipelineRunConfiguration;
 import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.engines.local.LocalPipelineEngine;
+import org.apache.hop.pipeline.engines.local.LocalPipelineRunConfiguration;
 import org.apache.hop.pipeline.transform.TransformMeta;
 import org.apache.hop.workflow.engines.local.LocalWorkflowEngine;
 import org.junit.jupiter.api.BeforeEach;
@@ -95,6 +105,19 @@ class PipelineExecutorTest {
     return executor;
   }
 
+  /** Parameters P_field that take their value from these fields. */
+  private static List<PipelineExecutorParameters> parametersFrom(String... 
fields) {
+    List<PipelineExecutorParameters> parameters = new ArrayList<>();
+    for (String field : fields) {
+      PipelineExecutorParameters parameter = new PipelineExecutorParameters();
+      parameter.setVariable("P_" + field);
+      parameter.setField(field);
+      parameter.setInput("");
+      parameters.add(parameter);
+    }
+    return parameters;
+  }
+
   @Test
   void getLastIncomingFieldValuesReturnsNullWhenBufferEmpty() throws 
HopException {
     PipelineExecutorMeta meta = new PipelineExecutorMeta();
@@ -110,6 +133,7 @@ class PipelineExecutorTest {
   void getLastIncomingFieldValuesPreservesNullsFromLastBufferedRow() throws 
HopException {
     PipelineExecutorMeta meta = new PipelineExecutorMeta();
     meta.setDefault();
+    meta.setParameters(parametersFrom("a", "b"));
     PipelineExecutorData data = new PipelineExecutorData();
     data.groupBuffer = new ArrayList<>();
     RowMeta rowMeta = new RowMeta();
@@ -129,6 +153,7 @@ class PipelineExecutorTest {
   void getLastIncomingFieldValuesUsesRowMetaStringConversion() throws 
HopException {
     PipelineExecutorMeta meta = new PipelineExecutorMeta();
     meta.setDefault();
+    meta.setParameters(parametersFrom("s1", "d1", "b1"));
     PipelineExecutorData data = new PipelineExecutorData();
     data.groupBuffer = new ArrayList<>();
     RowMeta rowMeta = new RowMeta();
@@ -897,4 +922,211 @@ class PipelineExecutorTest {
     assertFalse(result.isResult());
     verify(executor, times(1)).executePipelineAttempt(any(), anyLong());
   }
+
+  // --- field values for parameters are taken when a child pipeline runs, not 
for every row ---
+
+  /** Runs rows through processRow() and returns the field values each child 
execution got. */
+  private List<List<String>> parameterValuesPerExecution(
+      PipelineExecutorData data, RowMeta rowMeta, Object[]... rows) throws 
HopException {
+    PipelineExecutorMeta meta = new PipelineExecutorMeta();
+    meta.setDefault();
+    meta.setParameters(parametersFrom("grp", "name"));
+    return parameterValuesPerExecution(meta, data, rowMeta, rows);
+  }
+
+  /** Runs rows through processRow() and returns the field values each child 
execution got. */
+  private List<List<String>> parameterValuesPerExecution(
+      PipelineExecutorMeta meta, PipelineExecutorData data, RowMeta rowMeta, 
Object[]... rows)
+      throws HopException {
+    data.groupBuffer = new ArrayList<>();
+    PipelineExecutor executor = spy(newExecutor(meta, data));
+    // getRow() waits for the parent pipeline to be running
+    IPipelineEngine<PipelineMeta> parentPipeline = executor.getPipeline();
+    doReturn(true).when(parentPipeline).isRunning();
+
+    BlockingRowSet input = new BlockingRowSet(rows.length + 1);
+    input.setThreadNameFromToCopy("previous", 0, "pipeline_executor", 0);
+    input.setRowMeta(rowMeta);
+    for (Object[] row : rows) {
+      input.putRow(rowMeta, row);
+    }
+    input.setDone();
+    executor.addRowSetToInputRowSets(input);
+
+    List<List<String>> executions = new ArrayList<>();
+    doAnswer(
+            invocation -> {
+              List<String> values = invocation.getArgument(0);
+              executions.add(values == null ? null : new ArrayList<>(values));
+              Result result = new Result();
+              result.setResult(true);
+              return result;
+            })
+        .when(executor)
+        .executeWithRetries(any());
+
+    assertTimeoutPreemptively(
+        Duration.ofSeconds(30),
+        () -> {
+          for (int i = 0; i <= rows.length; i++) {
+            executor.processRow();
+          }
+        });
+    return executions;
+  }
+
+  private static RowMeta groupAndNameRowMeta() {
+    RowMeta rowMeta = new RowMeta();
+    rowMeta.addValueMeta(new ValueMetaString("grp"));
+    rowMeta.addValueMeta(new ValueMetaString("name"));
+    return rowMeta;
+  }
+
+  @Test
+  void groupingBySizeUsesTheRowThatCompletesTheGroup() throws HopException {
+    PipelineExecutorData data = new PipelineExecutorData();
+    data.groupSize = 2;
+
+    List<List<String>> executions =
+        parameterValuesPerExecution(
+            data,
+            groupAndNameRowMeta(),
+            new Object[] {"1", "a"},
+            new Object[] {"1", "b"},
+            new Object[] {"2", "c"},
+            new Object[] {"2", "d"},
+            new Object[] {"3", "e"});
+
+    // two full groups and the rest at the end of the input
+    assertEquals(List.of(List.of("1", "b"), List.of("2", "d"), List.of("3", 
"e")), executions);
+  }
+
+  @Test
+  void groupingByFieldUsesTheLastRowOfTheGroup() throws HopException {
+    PipelineExecutorData data = new PipelineExecutorData();
+    data.groupSize = -1;
+    data.groupField = "grp";
+
+    List<List<String>> executions =
+        parameterValuesPerExecution(
+            data,
+            groupAndNameRowMeta(),
+            new Object[] {"1", "a"},
+            new Object[] {"1", "b"},
+            new Object[] {"2", "c"},
+            new Object[] {"3", "d"},
+            new Object[] {"3", "e"});
+
+    assertEquals(List.of(List.of("1", "b"), List.of("2", "c"), List.of("3", 
"e")), executions);
+  }
+
+  @Test
+  void aGroupOfOneRowRunsTheChildForEveryRow() throws HopException {
+    PipelineExecutorData data = new PipelineExecutorData();
+    data.groupSize = 1;
+
+    List<List<String>> executions =
+        parameterValuesPerExecution(
+            data, groupAndNameRowMeta(), new Object[] {"1", "a"}, new Object[] 
{"2", null});
+
+    assertEquals(List.of(List.of("1", "a"), Arrays.asList("2", null)), 
executions);
+  }
+
+  /** A field that no parameter takes its value from is never converted to 
text. */
+  @Test
+  void onlyTheFieldsOfParametersAreConverted() throws HopException {
+    AtomicInteger conversions = new AtomicInteger();
+    RowMeta rowMeta = new RowMeta();
+    rowMeta.addValueMeta(new ValueMetaString("grp"));
+    rowMeta.addValueMeta(
+        new ValueMetaString("payload") {
+          @Override
+          public String getString(Object object) throws HopValueException {
+            conversions.incrementAndGet();
+            return super.getString(object);
+          }
+        });
+
+    // a child pipeline for every row
+    PipelineExecutorData data = new PipelineExecutorData();
+    data.groupSize = 1;
+    PipelineExecutorMeta meta = new PipelineExecutorMeta();
+    meta.setDefault();
+    meta.setParameters(parametersFrom("grp"));
+    List<List<String>> executions =
+        parameterValuesPerExecution(
+            meta, data, rowMeta, new Object[] {"1", "a"}, new Object[] {"2", 
"b"});
+
+    assertEquals(List.of(Arrays.asList("1", null), Arrays.asList("2", null)), 
executions);
+    assertEquals(0, conversions.get());
+
+    // the last row of a group, as grouping by field and the end of the input 
use it
+    data.groupBuffer.add(new RowMetaAndData(rowMeta, new Object[] {"3", "c"}));
+    PipelineExecutor executor = newExecutor(meta, data);
+    assertEquals(Arrays.asList("3", null), 
executor.getLastIncomingFieldValues());
+    assertEquals(0, conversions.get());
+  }
+
+  // --- the run configuration is read once per transform copy ---
+
+  /** Counts how often the run configurations are asked for. */
+  private static final class CountingMetadataProvider extends 
MemoryMetadataProvider {
+    final AtomicInteger runConfigurationReads = new AtomicInteger();
+
+    @Override
+    public <T extends IHopMetadata> IHopMetadataSerializer<T> 
getSerializer(Class<T> managedClass)
+        throws HopException {
+      if (PipelineRunConfiguration.class.equals(managedClass)) {
+        runConfigurationReads.incrementAndGet();
+      }
+      return super.getSerializer(managedClass);
+    }
+  }
+
+  @Test
+  void theRunConfigurationIsReadOnceAndEveryExecutionGetsItsOwnCopy() throws 
HopException {
+    CountingMetadataProvider provider = new CountingMetadataProvider();
+    LocalPipelineRunConfiguration localEngine = new 
LocalPipelineRunConfiguration();
+    localEngine.setEnginePluginId("Local");
+    provider
+        .getSerializer(PipelineRunConfiguration.class)
+        .save(
+            new PipelineRunConfiguration(
+                "local", "", null, new ArrayList<>(), localEngine, null, 
false));
+    provider.runConfigurationReads.set(0);
+
+    PipelineExecutorMeta meta = new PipelineExecutorMeta();
+    meta.setDefault();
+    meta.setRunConfigurationName("local");
+    PipelineExecutorData data = new PipelineExecutorData();
+    data.setExecutorPipelineMeta(new PipelineMeta());
+    PipelineExecutor executor = newExecutor(meta, data);
+    executor.setMetadataProvider(provider);
+
+    IPipelineEngine<PipelineMeta> first = executor.createInternalPipeline();
+    IPipelineEngine<PipelineMeta> second = executor.createInternalPipeline();
+    IPipelineEngine<PipelineMeta> third = executor.createInternalPipeline();
+
+    assertEquals(1, provider.runConfigurationReads.get());
+    assertEquals("local", third.getPipelineRunConfiguration().getName());
+    // the engine keeps its run configuration and applies variables to it: 
never a shared object
+    assertNotSame(first.getPipelineRunConfiguration(), 
second.getPipelineRunConfiguration());
+    assertNotSame(second.getPipelineRunConfiguration(), 
third.getPipelineRunConfiguration());
+    assertNotSame(
+        first.getPipelineRunConfiguration().getEngineRunConfiguration(),
+        second.getPipelineRunConfiguration().getEngineRunConfiguration());
+  }
+
+  @Test
+  void aMissingRunConfigurationStillFailsWithItsName() {
+    PipelineExecutorMeta meta = new PipelineExecutorMeta();
+    meta.setDefault();
+    meta.setRunConfigurationName("nowhere");
+    PipelineExecutorData data = new PipelineExecutorData();
+    data.setExecutorPipelineMeta(new PipelineMeta());
+    PipelineExecutor executor = newExecutor(meta, data);
+
+    HopException e = assertThrows(HopException.class, 
executor::createInternalPipeline);
+    assertTrue(e.getMessage().contains("nowhere"));
+  }
 }

Reply via email to