This is an automated email from the ASF dual-hosted git repository.

hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git


The following commit(s) were added to refs/heads/main by this push:
     new ab71f635f5 Lineage: action start events, nested-run flushing and run 
context for… (#9339)
ab71f635f5 is described below

commit ab71f635f5a39458c907a6ff472e0211d896ca9a
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Nov 12 09:04:55 2026 +0100

    Lineage: action start events, nested-run flushing and run context for… 
(#9339)
    
    * Lineage: action start events, nested-run flushing and run context for 
sinks, fixes #9322
    
    * Address review comments on lineage action events and nested-run flushing, 
#9322
---
 .../hop-dev-manual/modules/ROOT/pages/lineage.adoc |  20 +-
 .../java/org/apache/hop/core/HopEnvironment.java   |  20 +-
 .../hop/lineage/LineageActionStartListener.java    | 132 ++++++++++
 .../apache/hop/lineage/LineageFileIoEmitter.java   |  33 +++
 .../hop/lineage/LineageRelationalIoEmitter.java    |  18 +-
 .../hop/lineage/LineageRunLifecycleEmitter.java    | 292 +++++++++++++++++++--
 .../org/apache/hop/lineage/hub/LineageHub.java     |  45 ++++
 .../apache/hop/lineage/model/LineageEventKind.java |   8 +-
 .../lineage/model/PipelineGraphLineagePayload.java |   4 +-
 .../lineage/xp/LineageHubPipelineCompletedXp.java  |   8 +-
 .../xp/LineageHubWorkflowActionAfterXp.java        |   2 +
 .../xp/LineageHubWorkflowActionBeforeXp.java       |  41 ---
 .../hop/lineage/xp/LineageHubWorkflowFinishXp.java |  11 +-
 .../hop/lineage/xp/LineageHubWorkflowStartXp.java  |  11 +-
 .../transforms/file/BaseFileInputTransform.java    |  55 +++-
 .../main/java/org/apache/hop/run/HopRunBase.java   |   4 +-
 .../hop/lineage/LineageActionLifecycleTest.java    | 236 +++++++++++++++++
 .../hop/lineage/LineageFileIoEmitterTest.java      |  27 ++
 .../hop/lineage/LineagePipelineGraphTest.java      |  70 ++++-
 .../hop/lineage/LineagePipelineParentTest.java     | 241 +++++++++++++++++
 .../hub/LineageHubConcurrentShutdownTest.java      |  47 +++-
 .../org/apache/hop/lineage/hub/LineageHubTest.java |  51 ++++
 .../file/BaseFileInputTransformLineageTest.java    | 183 +++++++++++++
 .../openlineage/OpenLineageEventMapper.java        |   1 +
 .../transforms/fileinput/text/TextFileInput.java   |  10 +
 25 files changed, 1448 insertions(+), 122 deletions(-)

diff --git a/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc 
b/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
index 4f13aac749..95f0ed1c8d 100644
--- a/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
+++ b/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
@@ -105,7 +105,7 @@ Each observation is a **`LineageEvent`**:
 
 * **Event id** — Unique id (UUID by default).
 * **Timestamp** — Milliseconds since epoch.
-* **`LineageEventKind`** — One of `RUN_LIFECYCLE`, `TRANSFORM_SCHEMA`, 
`FILE_IO`, `RELATIONAL_IO`, `HTTP_IO`, `PIPELINE_GRAPH`. New kinds are only 
ever appended: a sink must skip a kind it does not handle rather than fail the 
batch.
+* **`LineageEventKind`** — One of `RUN_LIFECYCLE`, `TRANSFORM_SCHEMA`, 
`FILE_IO`, `RELATIONAL_IO`, `HTTP_IO`, `PIPELINE_GRAPH`, `WORKFLOW_GRAPH`. New 
kinds are only ever appended: a sink must skip a kind it does not handle rather 
than fail the batch.
 * **`LineageContext`** — Who/what produced the observation (see above).
 * **`LineagePayload`** — Optional typed payload; may be null if the kind and 
context are enough.
 
@@ -115,7 +115,7 @@ Payload types are plain Java types (strings, longs, enums) 
so sinks can serializ
 * **`RelationalIoLineagePayload`** — Relational/JDBC table access: operation 
(`READ`, `WRITE`, `DELETE`, `EXEC`), the resolved **dataset namespace**, the 
input/output **`RelationalTable`** list (catalog/schema/table), the raw SQL 
when the engine cannot resolve the tables itself, an optional column schema, 
the connection's default catalog and schema, optional per-column provenance 
(**`RelationalWriteColumn`**: target column, stream field, origin transform), 
and an optional **`RelationalLi [...]
 * **`HttpLineagePayload`** — `CLIENT` vs `SERVER`, method, URL, status, 
request/response bytes, duration, success, message.
 * **`RunLifecycleLineagePayload`** — `STARTED` / `FINISHED` plus optional 
detail text.
-* **`PipelineGraphLineagePayload`** — The pipeline's hop graph as a list of 
`from` → `to` transform-name transitions: enabled hops with both ends named, in 
pipeline order. Informational hops are included. A sink uses it to attribute 
datasets to the transform that touched them and to propagate them along the 
graph, instead of pairing all of a pipeline's inputs with all of its outputs.
+* **`PipelineGraphLineagePayload`** — The pipeline's hop graph as a list of 
`from` → `to` transform-name transitions: enabled hops with both ends named, in 
pipeline order. Informational hops are included. A sink uses it to attribute 
datasets to the transform that touched them and to propagate them along the 
graph, instead of pairing all of a pipeline's inputs with all of its outputs. A 
pipeline without hops gets an empty list. The same payload carries a workflow's 
hop graph (`WORKFLOW_GR [...]
 * **`TransformSchemaLineagePayload`** — `INPUT` or `OUTPUT` plus a list of 
field names, Hop type names, length, and precision (boundary schema; emitted 
when a transform finishes). Fields usually come from runtime row metadata on 
the transform or row sets. If input was never observed (e.g. _Abort_ when 
`getRow()` immediately returns null), input is derived via 
`PipelineMeta#getPrevTransformFields`. If output was never written (e.g. 
_Detect empty stream_ with a non-empty stream), output is [...]
 
 Convenience factory: `LineageEvent.of(kind, context, payload)` assigns id and 
current time.
@@ -131,9 +131,9 @@ When enabled:
 . The worker builds **batches** up to `HOP_LINEAGE_BATCH_MAX`, waiting up to 
`HOP_LINEAGE_BATCH_LINGER_MS` for additional events before sending a partial 
batch.
 . Each batch is passed to **every** loaded sink via 
`ILineageSink.accept(List<LineageEvent>)`. If one sink throws, others still 
receive the same batch; failures are logged.
 
-**`flush()`** blocks until queued events (and coalesced flush markers) have 
been processed. **`flushQuietly()`** catches and logs errors.
+**`flush()`** blocks until queued events (and coalesced flush markers) have 
been processed. **`flushQuietly()`** catches and logs errors. 
**`flushIfBacklogged()`** flushes only once the queue is at least half full: a 
nested run calls it, so that a run per row does not drain the hub after every 
row, but still waits for the sinks before the queue overflows and events are 
dropped.
 
-**`shutdown()`** stops the worker and calls `shutdown()` on sinks. It is 
invoked from `HopEnvironment.shutdown()` and `HopEnvironment.reset()`.
+**`shutdown()`** stops the worker and calls `shutdown()` on sinks. It is 
invoked from `HopEnvironment.shutdown()` and `HopEnvironment.reset()`, and from 
a JVM exit hook that the hub registers when it starts its worker. That hook 
waits for a hop-run that is still stopping on a termination signal 
(`holdShutdownOnExit`), so the end of the run is delivered too.
 
 After `PluginRegistry.init()`, `HopEnvironment` calls 
`LineageHub.getInstance().environmentReady()` so sink lists can be rebuilt when 
the environment is re-initialized.
 
@@ -144,20 +144,20 @@ The queue is allocated lazily on the first `emit` from 
the resolved configuratio
 
 The `org.apache.hop.lineage.xp` package registers extension points that 
**emit** `RUN_LIFECYCLE` events (via `LineageRunLifecycleEmitter`) and 
**flush** the hub where needed:
 
-* **`PipelineStart`** — `LineageHubPipelineStartXp` — phase `STARTED`, 
followed by one `PIPELINE_GRAPH` event with the same context when the pipeline 
has enabled hops. `PipelineStart` fires after the transform threads have 
started, so the run's transform `STARTED` events, and early I/O such as a Table 
Input read, can arrive before the graph, possibly in an earlier batch. A sink 
that uses the graph collects a run's events by the pipeline's log channel id 
(transform events carry it in the  [...]
+* **`PipelineStart`** — `LineageHubPipelineStartXp` — phase `STARTED`, 
followed by one `PIPELINE_GRAPH` event with the same context, with no 
transitions when the pipeline has no enabled hops. `PipelineStart` fires after 
the transform threads have started, so the run's transform `STARTED` events, 
and early I/O such as a Table Input read, can arrive before the graph, possibly 
in an earlier batch. A sink that uses the graph collects a run's events by the 
pipeline's log channel id (transform [...]
 * **`PipelineFinish`** — `LineageHubPipelineFinishXp` — phase `FINISHING` 
(transforms stopped, cleanup)
-* **`PipelineCompleted`** — `LineageHubPipelineCompletedXp` — phase `FINISHED` 
or `FAILED` (from pipeline error count), then `flushQuietly()`
-* **`WorkflowStart`** — `LineageHubWorkflowStartXp` — phase `STARTED`
-* **`WorkflowFinish`** — `LineageHubWorkflowFinishXp` — phase `FINISHED` or 
`FAILED` (from workflow/result errors), then `flushQuietly()`
+* **`PipelineCompleted`** — `LineageHubPipelineCompletedXp` — phase `FINISHED` 
or `FAILED` (from pipeline error count), then `flushQuietly()` for a top-level 
run and `flushIfBacklogged()` for a nested one
+* **`WorkflowStart`** — `LineageHubWorkflowStartXp` — phase `STARTED`, 
followed by one `WORKFLOW_GRAPH` event with the same context (empty when the 
workflow has no enabled hops)
+* **`WorkflowFinish`** — `LineageHubWorkflowFinishXp` — first action `FAILED` 
for every action run that started but never ended (its `execute()` threw), then 
the workflow's phase `FINISHED` or `FAILED` (from workflow/result errors), then 
`flushQuietly()` for a top-level run and `flushIfBacklogged()` for a nested one
 * **`HopServerShutdown`** — `LineageHubServerShutdownFlushXp` — flush only
 
 * **`TransformBeforeStart`** — `LineageHubTransformStartXp` — transform phase 
`STARTED` (local pipeline engine; transform log channel + pipeline correlation 
attributes)
 
 * **`TransformFinished`** — `LineageHubTransformFinishXp` — transform 
`FINISHED` or `FAILED` from transform error count, then `TRANSFORM_SCHEMA` 
events for observed input/output row shapes when non-empty
 
-* **`WorkflowBeforeActionExecution`** — `LineageHubWorkflowActionBeforeXp` — 
action `STARTED` (workflow log channel + action name; template action log 
channel id as attribute when present)
+* **Action start** — `LineageActionStartListener`, an `IActionListener` that 
`LineageHubWorkflowStartXp` adds to each workflow — action `STARTED` from 
`beforeExecution`, once the workflow has cloned the action, so the event 
carries the executing action's log channel id. `WorkflowBeforeActionExecution` 
fires before that clone exists. An error while emitting is logged and does not 
fail the action.
 
-* **`WorkflowAfterActionExecution`** — `LineageHubWorkflowActionAfterXp` — 
action `FINISHED` or `FAILED` from the action `Result`; uses the executed 
action’s log channel id when available 
(`WorkflowExecutionExtension.actionLogChannelId` / `actionExecutionResult`)
+* **`WorkflowAfterActionExecution`** — `LineageHubWorkflowActionAfterXp` — 
action `FINISHED` or `FAILED` from the action `Result`, under the executed 
action's log channel id (`WorkflowExecutionExtension.actionLogChannelId`). An 
action the workflow skipped has no run of its own and emits nothing. The 
workflow does not reach this extension point when the action's `execute()` 
throws; `WorkflowFinish` ends that run instead.
 
 Remote (`RemotePipelineEngine`) and Beam (`BeamPipelineEngine`) engines also 
call **`PipelineStart`** and **`PipelineFinish`** so they align with the local 
pipeline engine.
 
diff --git a/engine/src/main/java/org/apache/hop/core/HopEnvironment.java 
b/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
index 02a8e6e58c..081840964e 100644
--- a/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
+++ b/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
@@ -238,24 +238,8 @@ public class HopEnvironment {
     }
   }
 
-  /**
-   * Alert all Lifecycle plugins that the Hop environment is being initialized.
-   *
-   * @throws HopException when a lifecycle listener throws an exception
-   */
-  private static void initLifecycleListeners() {
-    // Register a shutdown hook to invoke the listener's onExit() methods
-    Runtime.getRuntime()
-        .addShutdownHook(
-            new Thread() {
-              @Override
-              public void run() {
-                // waits for a hop-run that is still stopping, so that the end 
of the run is
-                // delivered too
-                LineageHub.getInstance().shutdownOnExit();
-              }
-            });
-  }
+  // The JVM exit hook that shuts the lineage hub down is registered by the 
hub itself, once it has
+  // a worker; see LineageHub#shutdownOnExit.
 
   // Shutdown the Hop environment programmatically
   public static void shutdown() {
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java 
b/engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java
new file mode 100644
index 0000000000..261a5747dc
--- /dev/null
+++ 
b/engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.action.IAction;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * Emits an action's {@code STARTED} lineage event once the workflow has 
cloned the action, so the
+ * event carries the run id of the action that actually executes. One listener 
per workflow, added
+ * by {@link #register(IWorkflowEngine)}.
+ *
+ * <p>The listener remembers the action runs it started. An action whose 
{@code execute()} throws
+ * never reaches the {@code WorkflowAfterActionExecution} extension point that 
emits its {@code
+ * FINISHED} or {@code FAILED}, so {@link 
#failUnfinishedActions(IWorkflowEngine)} closes those runs
+ * as {@code FAILED} when the workflow ends.
+ */
+public final class LineageActionStartListener implements 
IActionListener<WorkflowMeta> {
+
+  /** The action runs that started and have no terminal event yet, by run id. 
*/
+  private final Map<String, IAction> unfinishedActions = new 
ConcurrentHashMap<>();
+
+  private LineageActionStartListener() {}
+
+  /** Adds the listener to a workflow, unless it is already there (a workflow 
can run again). */
+  @SuppressWarnings("unchecked")
+  public static void register(IWorkflowEngine<WorkflowMeta> workflow) {
+    if (workflow == null || workflow.getActionListeners() == null || 
find(workflow) != null) {
+      return;
+    }
+    workflow.addActionListener(new LineageActionStartListener());
+  }
+
+  /** The action run ended with a {@code FINISHED} or {@code FAILED} event. */
+  public static void actionEnded(IWorkflowEngine<WorkflowMeta> workflow, 
String actionRunId) {
+    LineageActionStartListener listener = find(workflow);
+    if (listener != null && !Utils.isEmpty(actionRunId)) {
+      listener.unfinishedActions.remove(actionRunId);
+    }
+  }
+
+  /**
+   * Emits {@code FAILED} for every action run of this workflow that started 
but never ended: its
+   * {@code execute()} threw, so the workflow skipped the extension point that 
ends it. Without this
+   * a sink keeps those runs open forever.
+   */
+  public static void failUnfinishedActions(IWorkflowEngine<WorkflowMeta> 
workflow) {
+    LineageActionStartListener listener = find(workflow);
+    if (listener == null || listener.unfinishedActions.isEmpty()) {
+      return;
+    }
+    List<IAction> unfinished = new 
ArrayList<>(listener.unfinishedActions.values());
+    listener.unfinishedActions.clear();
+    for (IAction action : unfinished) {
+      LineageRunLifecycleEmitter.emitActionFailedWithoutResult(workflow, 
action);
+    }
+  }
+
+  @SuppressWarnings("unchecked")
+  private static LineageActionStartListener find(IWorkflowEngine<WorkflowMeta> 
workflow) {
+    List<IActionListener> listeners = workflow != null ? 
workflow.getActionListeners() : null;
+    if (listeners != null) {
+      for (IActionListener listener : listeners) {
+        if (listener instanceof LineageActionStartListener lineageListener) {
+          return lineageListener;
+        }
+      }
+    }
+    return null;
+  }
+
+  /**
+   * Runs inside the workflow's action execution, before the action's own 
error handling: a lineage
+   * problem is logged and may never fail the action.
+   */
+  @Override
+  public void beforeExecution(
+      IWorkflowEngine<WorkflowMeta> workflow, ActionMeta actionMeta, IAction 
action) {
+    try {
+      LineageRunLifecycleEmitter.emitActionStarted(workflow, action);
+      ILogChannel actionLog = action != null ? action.getLogChannel() : null;
+      String runId = actionLog != null ? actionLog.getLogChannelId() : null;
+      if (!Utils.isEmpty(runId)) {
+        unfinishedActions.put(runId, action);
+      }
+    } catch (RuntimeException e) {
+      ILogChannel log = workflow != null ? workflow.getLogChannel() : null;
+      if (log != null) {
+        log.logError(
+            "Unable to emit the lineage start event of action "
+                + (actionMeta != null ? actionMeta.getName() : null),
+            e);
+      }
+    }
+  }
+
+  @Override
+  public void afterExecution(
+      IWorkflowEngine<WorkflowMeta> workflow,
+      ActionMeta actionMeta,
+      IAction action,
+      Result result) {
+    // FINISHED and FAILED come from the WorkflowAfterActionExecution 
extension point, which the
+    // workflow reaches whenever the action returned a result. An action that 
threw is closed by
+    // failUnfinishedActions() at the end of the workflow.
+  }
+}
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java 
b/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
index d9d45b8588..6656071fb6 100644
--- a/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
+++ b/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
@@ -17,13 +17,18 @@
 
 package org.apache.hop.lineage;
 
+import java.util.ArrayList;
+import java.util.List;
 import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
 import org.apache.hop.core.util.Utils;
 import org.apache.hop.lineage.context.LineageContext;
 import org.apache.hop.lineage.hub.LineageHub;
 import org.apache.hop.lineage.model.FileIoContentSchema;
 import org.apache.hop.lineage.model.FileIoLineagePayload;
 import org.apache.hop.lineage.model.FileIoOperation;
+import org.apache.hop.lineage.model.FileIoTabularColumn;
 import org.apache.hop.lineage.model.LineageEvent;
 import org.apache.hop.lineage.model.LineageEventKind;
 import org.apache.hop.pipeline.transform.ITransform;
@@ -47,6 +52,34 @@ public final class LineageFileIoEmitter {
     return LineageHub.getInstance().isEnabled();
   }
 
+  /**
+   * Converts a Hop {@link IRowMeta} to the neutral {@link 
FileIoContentSchema} a file event
+   * carries, so a sink can describe the file's columns (name and Hop type). 
Returns {@code null}
+   * for a null/empty row shape. Columns have no path locator and no structure 
tree here: a reader
+   * that knows a hierarchy derives its own.
+   *
+   * @param rowMeta the row shape read from or written to the file
+   * @param formatHint short format name ({@code csv}, {@code json}, …), or 
null when unknown
+   */
+  public static FileIoContentSchema toContentSchema(IRowMeta rowMeta, String 
formatHint) {
+    if (rowMeta == null || rowMeta.isEmpty()) {
+      return null;
+    }
+    List<FileIoTabularColumn> columns = new ArrayList<>(rowMeta.size());
+    for (IValueMeta valueMeta : rowMeta.getValueMetaList()) {
+      columns.add(
+          new FileIoTabularColumn(
+              valueMeta.getName(),
+              valueMeta.getTypeDesc(),
+              valueMeta.getLength(),
+              valueMeta.getPrecision(),
+              null,
+              null,
+              false));
+    }
+    return new FileIoContentSchema(formatHint == null ? "" : formatHint, 
columns, List.of());
+  }
+
   /**
    * Records a file/object-store operation performed by a workflow action 
during {@link
    * IAction#execute}.
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java 
b/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
index 8646f1583e..d1baa44224 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
@@ -28,7 +28,6 @@ import org.apache.hop.core.util.Utils;
 import org.apache.hop.lineage.context.LineageContext;
 import org.apache.hop.lineage.hub.LineageHub;
 import org.apache.hop.lineage.model.FileIoContentSchema;
-import org.apache.hop.lineage.model.FileIoTabularColumn;
 import org.apache.hop.lineage.model.LineageEvent;
 import org.apache.hop.lineage.model.LineageEventKind;
 import org.apache.hop.lineage.model.RelationalIoLineagePayload;
@@ -462,22 +461,7 @@ public final class LineageRelationalIoEmitter {
    * locator, so the structure-roots tree is empty.
    */
   static FileIoContentSchema toContentSchema(IRowMeta rowMeta) {
-    if (rowMeta == null || rowMeta.isEmpty()) {
-      return null;
-    }
-    List<FileIoTabularColumn> columns = new ArrayList<>(rowMeta.size());
-    for (IValueMeta valueMeta : rowMeta.getValueMetaList()) {
-      columns.add(
-          new FileIoTabularColumn(
-              valueMeta.getName(),
-              valueMeta.getTypeDesc(),
-              valueMeta.getLength(),
-              valueMeta.getPrecision(),
-              null,
-              null,
-              false));
-    }
-    return new FileIoContentSchema(JDBC_FORMAT_HINT, columns, List.of());
+    return LineageFileIoEmitter.toContentSchema(rowMeta, JDBC_FORMAT_HINT);
   }
 
   /**
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java 
b/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
index 53d1669793..010b557c91 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
@@ -40,6 +40,7 @@ import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.transform.ITransform;
 import org.apache.hop.pipeline.transform.TransformMetaDataCombi;
 import org.apache.hop.workflow.WorkflowExecutionExtension;
+import org.apache.hop.workflow.WorkflowHopMeta;
 import org.apache.hop.workflow.WorkflowMeta;
 import org.apache.hop.workflow.action.ActionMeta;
 import org.apache.hop.workflow.action.IAction;
@@ -55,8 +56,164 @@ public final class LineageRunLifecycleEmitter {
   public static final String FAILED = "failed";
   public static final String SUCCESS = "success";
 
+  /**
+   * The run that started this pipeline or workflow, when the engine knows it: 
the workflow action,
+   * or the transform of a parent pipeline (Pipeline Executor, mapping), that 
ran it. {@code
+   * parentRunId} is that subject's log-channel id and {@code parentJobName} 
is the job name the
+   * lineage mapper gives it (e.g. {@code my-workflow/Run orders} or {@code 
my-pipeline/Run per
+   * row}); both are needed to form a ParentRunFacet. Sinks that write the job 
tree themselves
+   * (DataHub) read the same pair.
+   */
+  public static final String ATTR_PARENT_RUN_ID = "parentRunId";
+
+  public static final String ATTR_PARENT_JOB_NAME = "parentJobName";
+
+  /**
+   * The parts {@link #ATTR_PARENT_JOB_NAME} is made of, for a sink that 
models workflows and
+   * pipelines apart from the jobs inside them (DataHub): the workflow and 
action that ran this run,
+   * or the pipeline and transform (Pipeline Executor, mapping) that did. A 
job name cannot be split
+   * back reliably, since names may contain the separator.
+   */
+  public static final String ATTR_PARENT_WORKFLOW_NAME = "parentWorkflowName";
+
+  public static final String ATTR_PARENT_ACTION_NAME = "parentActionName";
+  public static final String ATTR_PARENT_PIPELINE_NAME = "parentPipelineName";
+  public static final String ATTR_PARENT_TRANSFORM_NAME = 
"parentTransformName";
+
+  /** The description the subject's metadata gives it (pipeline, workflow, 
transform, action). */
+  public static final String ATTR_DESCRIPTION = "description";
+
+  private static void fillDescription(LineageContext.Builder ctx, String 
description) {
+    if (!Utils.isEmpty(description)) {
+      ctx.putAttribute(ATTR_DESCRIPTION, description);
+    }
+  }
+
   private LineageRunLifecycleEmitter() {}
 
+  /**
+   * Records who started this run, when the engine knows: an action that ran a 
pipeline or workflow,
+   * or a parent pipeline that ran a sub-pipeline. Left out entirely for a 
top-level run.
+   */
+  private static void fillParentFields(
+      LineageContext.Builder ctx, String parentJobName, String parentRunId) {
+    if (!Utils.isEmpty(parentJobName)) {
+      ctx.putAttribute(ATTR_PARENT_JOB_NAME, parentJobName);
+    }
+    if (!Utils.isEmpty(parentRunId)) {
+      ctx.putAttribute(ATTR_PARENT_RUN_ID, parentRunId);
+    }
+  }
+
+  /**
+   * The run that started a pipeline: its sub-pipeline parent, or the workflow 
action that ran it
+   * (with the workflow that owns the action). Package-private so the naming 
rule can be checked
+   * without starting an engine.
+   */
+  static void fillPipelineParentFields(
+      LineageContext.Builder ctx, IPipelineEngine<PipelineMeta> pipeline) {
+    IPipelineEngine<PipelineMeta> parentPipeline = 
pipeline.getParentPipeline();
+    if (parentPipeline != null) {
+      fillPipelineParentFields(ctx, parentPipeline, pipeline.getParent());
+      return;
+    }
+    if (pipeline.getParent() instanceof IAction parentAction) {
+      fillActionParentFields(ctx, parentAction, pipeline.getParentWorkflow());
+    }
+  }
+
+  /**
+   * A run started from a pipeline. When the transform that started it 
(Pipeline Executor, mapping,
+   * Workflow Executor) is its logging parent, the parent is that transform's 
run, named {@code
+   * <pipeline>/<transform>}: the same shape as a run started by a workflow 
action. Only when the
+   * transform is unknown does the parent pipeline's own run stand in.
+   */
+  private static void fillPipelineParentFields(
+      LineageContext.Builder ctx, IPipelineEngine<PipelineMeta> 
parentPipeline, Object parent) {
+    PipelineMeta parentMeta = parentPipeline.getPipelineMeta();
+    String parentPipelineName = parentMeta != null ? parentMeta.getName() : 
null;
+    if (Utils.isEmpty(parentPipelineName)) {
+      fillParentFields(ctx, null, parentPipeline.getLogChannelId());
+      return;
+    }
+    ctx.putAttribute(ATTR_PARENT_PIPELINE_NAME, parentPipelineName);
+    if (parent instanceof ITransform transform && 
!Utils.isEmpty(transform.getTransformName())) {
+      ctx.putAttribute(ATTR_PARENT_TRANSFORM_NAME, 
transform.getTransformName());
+      ILogChannel transformLog = transform.getLogChannel();
+      String transformRunId = transformLog != null ? 
transformLog.getLogChannelId() : null;
+      if (!Utils.isEmpty(transformRunId)) {
+        fillParentFields(
+            ctx, parentPipelineName + "/" + transform.getTransformName(), 
transformRunId);
+        return;
+      }
+    }
+    fillParentFields(ctx, parentPipelineName, 
parentPipeline.getLogChannelId());
+  }
+
+  /**
+   * The run that started a workflow: the transform that ran it (Workflow 
Executor), the workflow
+   * action that ran it (with the workflow that owns the action), or failing 
that the parent
+   * workflow itself.
+   */
+  static void fillWorkflowParentFields(
+      LineageContext.Builder ctx, IWorkflowEngine<WorkflowMeta> workflow) {
+    IPipelineEngine<PipelineMeta> parentPipeline = 
workflow.getParentPipeline();
+    if (parentPipeline != null) {
+      fillPipelineParentFields(ctx, parentPipeline, workflow.getParent());
+      return;
+    }
+    IWorkflowEngine<WorkflowMeta> parentWorkflow = 
workflow.getParentWorkflow();
+    if (workflow.getParent() instanceof IAction parentAction) {
+      fillActionParentFields(ctx, parentAction, parentWorkflow);
+      return;
+    }
+    WorkflowMeta parentMeta = parentWorkflow != null ? 
parentWorkflow.getWorkflowMeta() : null;
+    if (parentMeta != null) {
+      fillParentFields(ctx, parentMeta.getName(), 
parentWorkflow.getLogChannelId());
+    }
+  }
+
+  /**
+   * Whether a pipeline is a top-level run: not a sub-pipeline, not started by 
a workflow action.
+   * Only a top-level run always waits for the lineage hub to drain when it 
completes. A nested run
+   * (a Pipeline Executor row, a mapping, a Pipeline action) waits only once 
the queue is filling up
+   * ({@link LineageHub#flushIfBacklogged()}): a full drain after every nested 
run would sit on the
+   * path of every caller waiting for it, and no drain at all would let the 
queue overflow.
+   */
+  public static boolean isTopLevel(IPipelineEngine<PipelineMeta> pipeline) {
+    return pipeline != null
+        && pipeline.getParentPipeline() == null
+        && pipeline.getParentWorkflow() == null
+        && !(pipeline.getParent() instanceof IAction);
+  }
+
+  /** Whether a workflow is a top-level run; see {@link 
#isTopLevel(IPipelineEngine)}. */
+  public static boolean isTopLevel(IWorkflowEngine<WorkflowMeta> workflow) {
+    return workflow != null
+        && workflow.getParentPipeline() == null
+        && workflow.getParentWorkflow() == null
+        && !(workflow.getParent() instanceof IAction);
+  }
+
+  /** An action is named {@code <workflow>/<action>}, the convention the 
lineage mapper uses. */
+  private static void fillActionParentFields(
+      LineageContext.Builder ctx,
+      IAction parentAction,
+      IWorkflowEngine<WorkflowMeta> parentWorkflow) {
+    WorkflowMeta parentMeta = parentWorkflow != null ? 
parentWorkflow.getWorkflowMeta() : null;
+    String parentWorkflowName = parentMeta != null ? parentMeta.getName() : 
null;
+    String parentJobName =
+        !Utils.isEmpty(parentWorkflowName) && 
!Utils.isEmpty(parentAction.getName())
+            ? parentWorkflowName + "/" + parentAction.getName()
+            : null;
+    ILogChannel parentLog = parentAction.getLogChannel();
+    fillParentFields(ctx, parentJobName, parentLog != null ? 
parentLog.getLogChannelId() : null);
+    if (parentJobName != null) {
+      ctx.putAttribute(ATTR_PARENT_WORKFLOW_NAME, parentWorkflowName);
+      ctx.putAttribute(ATTR_PARENT_ACTION_NAME, parentAction.getName());
+    }
+  }
+
   private static void fillFilenameFields(
       LineageContext.Builder ctx, String filename, IVariables variables) {
     ctx.hopFilename(filename);
@@ -135,6 +292,7 @@ public final class LineageRunLifecycleEmitter {
     if (!Utils.isEmpty(action.getPluginId())) {
       ctx.putAttribute("actionPluginId", action.getPluginId());
     }
+    fillDescription(ctx, action.getDescription());
     return ctx;
   }
 
@@ -147,6 +305,9 @@ public final class LineageRunLifecycleEmitter {
       ctx.transformName(combi.transformName);
     }
     ctx.copyNr(String.valueOf(combi.copy));
+    if (combi.transformMeta != null) {
+      fillDescription(ctx, combi.transformMeta.getDescription());
+    }
     return ctx;
   }
 
@@ -168,6 +329,13 @@ public final class LineageRunLifecycleEmitter {
     if (!Utils.isEmpty(pluginId)) {
       ctx.putAttribute("pipelineEnginePluginId", pluginId);
     }
+    // A pipeline run by a workflow action knows the action (its logging 
parent) and the workflow
+    // that owns it; a sub-pipeline knows its parent pipeline and the 
transform that ran it. Either
+    // way the parent is that action's or transform's run, named <owner>/<job> 
as the mapper does.
+    fillPipelineParentFields(ctx, pipeline);
+    if (meta != null) {
+      fillDescription(ctx, meta.getDescription());
+    }
     ctx.putAttribute("errors", String.valueOf(pipeline.getErrors()));
     if (phase == RunLifecyclePhase.FINISHED || phase == 
RunLifecyclePhase.FAILED) {
       ctx.putAttribute(OUTCOME, phase == RunLifecyclePhase.FAILED ? FAILED : 
SUCCESS);
@@ -182,18 +350,16 @@ public final class LineageRunLifecycleEmitter {
 
     // The hop graph is a fact about the pipeline, not about a phase: emit it 
once, next to the
     // STARTED event. Sinks that attribute I/O to individual transforms need 
it to know which
-    // transform fed which; see PipelineGraphLineagePayload.
+    // transform fed which; see PipelineGraphLineagePayload. A pipeline 
without hops gets an empty
+    // graph: without one a sink cannot tell it from an engine that sends no 
graph at all.
     //
     if (phase == RunLifecyclePhase.STARTED && meta != null) {
-      List<Transition> transitions = pipelineTransitions(meta);
-      if (!transitions.isEmpty()) {
-        LineageHub.getInstance()
-            .emit(
-                LineageEvent.of(
-                    LineageEventKind.PIPELINE_GRAPH,
-                    context,
-                    new PipelineGraphLineagePayload(transitions)));
-      }
+      LineageHub.getInstance()
+          .emit(
+              LineageEvent.of(
+                  LineageEventKind.PIPELINE_GRAPH,
+                  context,
+                  new PipelineGraphLineagePayload(pipelineTransitions(meta))));
     }
   }
 
@@ -237,6 +403,10 @@ public final class LineageRunLifecycleEmitter {
             .logChannelId(workflow.getLogChannelId())
             .workflowName(name);
     fillFilenameFields(ctx, filename, workflow);
+    fillWorkflowParentFields(ctx, workflow);
+    if (meta != null) {
+      fillDescription(ctx, meta.getDescription());
+    }
     Result result = workflow.getResult();
     ctx.putAttribute("errors", String.valueOf(result != null ? 
result.getNrErrors() : 0));
     if (result != null) {
@@ -246,12 +416,50 @@ public final class LineageRunLifecycleEmitter {
     if (phase == RunLifecyclePhase.FINISHED || phase == 
RunLifecyclePhase.FAILED) {
       ctx.putAttribute(OUTCOME, phase == RunLifecyclePhase.FAILED ? FAILED : 
SUCCESS);
     }
+    LineageContext context = ctx.build();
     LineageHub.getInstance()
         .emit(
             LineageEvent.of(
                 LineageEventKind.RUN_LIFECYCLE,
-                ctx.build(),
+                context,
                 new RunLifecycleLineagePayload(phase, detail)));
+
+    // Like a pipeline's, the hop graph is a fact about the workflow: emitted 
once, next to STARTED,
+    // and empty when the workflow has no hops.
+    if (phase == RunLifecyclePhase.STARTED && meta != null) {
+      LineageHub.getInstance()
+          .emit(
+              LineageEvent.of(
+                  LineageEventKind.WORKFLOW_GRAPH,
+                  context,
+                  new PipelineGraphLineagePayload(workflowTransitions(meta))));
+    }
+  }
+
+  /**
+   * The enabled hops of a workflow, in workflow order, as {@code from -> to} 
transitions between
+   * action names. Every hop counts, whatever its condition: each is a way the 
workflow can go.
+   */
+  public static List<Transition> workflowTransitions(WorkflowMeta meta) {
+    if (meta == null) {
+      return List.of();
+    }
+    List<Transition> transitions = new ArrayList<>();
+    for (int i = 0; i < meta.nrWorkflowHops(); i++) {
+      WorkflowHopMeta hop = meta.getWorkflowHop(i);
+      if (hop == null
+          || !hop.isEnabled()
+          || hop.getFromAction() == null
+          || hop.getToAction() == null) {
+        continue;
+      }
+      String from = hop.getFromAction().getName();
+      String to = hop.getToAction().getName();
+      if (!Utils.isEmpty(from) && !Utils.isEmpty(to)) {
+        transitions.add(new Transition(from, to));
+      }
+    }
+    return transitions;
   }
 
   /** Terminal failure for a workflow at {@code WorkflowFinish} time. */
@@ -294,9 +502,55 @@ public final class LineageRunLifecycleEmitter {
     return combi != null && combi.transform != null && 
combi.transform.getErrors() > 0;
   }
 
+  /**
+   * Action {@code STARTED}, emitted by {@link LineageActionStartListener} 
once the workflow has
+   * cloned the action: the clone's log channel is the action run's identity, 
the same one its
+   * {@code FINISHED} event, its I/O events and any pipeline it starts refer 
to. The {@code
+   * WorkflowBeforeActionExecution} extension point fires before that clone 
exists, which is why the
+   * start is not emitted from there.
+   */
+  public static void emitActionStarted(IWorkflowEngine<WorkflowMeta> wf, 
IAction action) {
+    if (wf == null || action == null) {
+      return;
+    }
+    LineageContext.Builder ctx = actionContextBuilder(wf, action);
+    LineageHub.getInstance()
+        .emit(
+            LineageEvent.of(
+                LineageEventKind.RUN_LIFECYCLE,
+                ctx.build(),
+                new RunLifecycleLineagePayload(RunLifecyclePhase.STARTED, 
null)));
+  }
+
+  /**
+   * Action {@code FAILED} for a run that started but whose {@code execute()} 
threw: the workflow
+   * then skips the extension point that ends it. See {@link
+   * LineageActionStartListener#failUnfinishedActions(IWorkflowEngine)}.
+   */
+  static void emitActionFailedWithoutResult(IWorkflowEngine<WorkflowMeta> wf, 
IAction action) {
+    LineageContext.Builder ctx = actionContextBuilder(wf, action);
+    if (ctx == null) {
+      return;
+    }
+    ctx.putAttribute(OUTCOME, FAILED);
+    LineageHub.getInstance()
+        .emit(
+            LineageEvent.of(
+                LineageEventKind.RUN_LIFECYCLE,
+                ctx.build(),
+                new RunLifecycleLineagePayload(
+                    RunLifecyclePhase.FAILED, "The action ended without a 
result")));
+  }
+
+  /**
+   * Action {@code FINISHED} or {@code FAILED}. An action the workflow skipped 
(an extension point
+   * set {@code executeAction} to false) never ran and has no run of its own, 
so it emits nothing.
+   */
   public static void emitWorkflowAction(
       WorkflowExecutionExtension extension, RunLifecyclePhase phase, String 
detail) {
-    if (extension == null || extension.workflow == null) {
+    if (extension == null
+        || extension.workflow == null
+        || Utils.isEmpty(extension.actionLogChannelId)) {
       return;
     }
     IWorkflowEngine<WorkflowMeta> wf = extension.workflow;
@@ -309,26 +563,18 @@ public final class LineageRunLifecycleEmitter {
     LineageContext.Builder ctx =
         LineageContext.builder()
             .subjectType(LineageSubjectType.ACTION)
+            .logChannelId(extension.actionLogChannelId)
             .workflowName(workflowName)
             .actionName(actionName);
     fillFilenameFields(ctx, filename, wf);
 
     ctx.putAttribute("workflowLogChannelId", wf.getLogChannelId());
-    ctx.putAttribute("executeAction", String.valueOf(extension.executeAction));
 
     if (am != null && am.getAction() != null && am.getAction().getPluginId() 
!= null) {
       ctx.putAttribute("actionPluginId", am.getAction().getPluginId());
     }
-
-    if (phase == RunLifecyclePhase.STARTED) {
-      ctx.logChannelId(wf.getLogChannelId());
-      if (am != null && am.getAction() != null) {
-        ctx.putAttribute(
-            "actionTemplateLogChannelId", 
am.getAction().getLogChannel().getLogChannelId());
-      }
-    } else {
-      String actionLogId = extension.actionLogChannelId;
-      ctx.logChannelId(!Utils.isEmpty(actionLogId) ? actionLogId : 
wf.getLogChannelId());
+    if (am != null) {
+      fillDescription(ctx, am.getDescription());
     }
 
     if (extension.actionExecutionResult != null) {
diff --git a/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java 
b/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
index 07133b3b85..8ad95039ec 100644
--- a/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
+++ b/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
@@ -30,6 +30,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicLong;
 import java.util.concurrent.locks.ReadWriteLock;
 import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Consumer;
 import org.apache.hop.core.exception.HopException;
 import org.apache.hop.core.exception.HopPluginException;
 import org.apache.hop.core.logging.ILogChannel;
@@ -62,6 +63,14 @@ public final class LineageHub {
   private volatile BlockingQueue<Object> queue;
 
   private final AtomicLong droppedEvents = new AtomicLong();
+
+  /**
+   * Whether the JVM exit hook is in place. A sink delivers asynchronously, so 
a process that exits
+   * right after its last run (hop-run calls System.exit) would lose whatever 
the sink still holds,
+   * typically the runs' terminal events, unless the hub shuts its sinks down 
on the way out.
+   */
+  private final AtomicBoolean exitHookRegistered = new AtomicBoolean(false);
+
   private final AtomicBoolean running = new AtomicBoolean(false);
 
   /**
@@ -185,6 +194,23 @@ public final class LineageHub {
     }
   }
 
+  /**
+   * {@link #flushQuietly()}, but only once the queue is at least half full. A 
nested run (a
+   * Pipeline Executor running a pipeline per row, say) calls this when it 
ends: it does not wait
+   * for the sinks while they keep up, and waits for them before the queue 
fills up and events get
+   * dropped.
+   */
+  public void flushIfBacklogged() {
+    BlockingQueue<Object> q = queue;
+    if (q == null || !running.get()) {
+      return;
+    }
+    int size = q.size();
+    if (size > 0 && size >= q.remainingCapacity()) {
+      flushQuietly();
+    }
+  }
+
   /**
    * Stop the worker thread and shut down sinks. Calls are serialized: a 
second call waits until the
    * first one has delivered the remaining events, instead of shutting the 
sinks down under it. An
@@ -307,6 +333,25 @@ public final class LineageHub {
       thread.setDaemon(true);
       worker = thread;
       thread.start();
+      if (this == INSTANCE) {
+        registerExitHook(Runtime.getRuntime()::addShutdownHook);
+      }
+    }
+  }
+
+  /**
+   * Register the JVM exit hook, once. It goes through {@link 
#shutdownOnExit()}: the JVM starts
+   * every shutdown hook at once, and hop-run may still be stopping its run on 
a termination signal.
+   */
+  void registerExitHook(Consumer<Thread> addShutdownHook) {
+    if (!exitHookRegistered.compareAndSet(false, true)) {
+      return;
+    }
+    try {
+      addShutdownHook.accept(new Thread(this::shutdownOnExit, 
"Hop-LineageHub-Exit"));
+    } catch (IllegalStateException e) {
+      // The JVM is already exiting. Too late for a hook, but the event must 
still be queued.
+      log.logDebug("The JVM is exiting, no lineage exit hook registered");
     }
   }
 
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java 
b/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
index a81905eade..90a9284cd3 100644
--- a/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
+++ b/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
@@ -37,5 +37,11 @@ public enum LineageEventKind {
    * PipelineGraphLineagePayload}). Sinks need it to attribute a dataset to 
the transform that
    * touched it rather than to the whole pipeline.
    */
-  PIPELINE_GRAPH
+  PIPELINE_GRAPH,
+  /**
+   * The workflow's hop graph, emitted once per run next to its start 
(payload: {@link
+   * PipelineGraphLineagePayload}, its transitions naming actions). Sinks use 
it to say which action
+   * follows which.
+   */
+  WORKFLOW_GRAPH
 }
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
 
b/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
index 13a837fb75..b739d89a7d 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
@@ -31,7 +31,9 @@ import lombok.Getter;
  * (its own reads plus what its upstream transforms produced) and its own 
outputs.
  *
  * <p>Only enabled hops with two known ends are listed, in pipeline order. 
Informational hops are
- * included: data flows through them, even though they do no I/O of their own.
+ * included: data flows through them, even though they do no I/O of their own. 
A pipeline without
+ * such hops still gets the event, with no transitions, so a sink can tell it 
from an engine that
+ * sends no graph.
  *
  * <p>The graph is emitted with the pipeline's {@code STARTED} event, which 
fires after the
  * transform threads have started. A transform's own events, including early 
I/O, can therefore
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
index de1e93201f..c17696df0a 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
@@ -44,6 +44,12 @@ public class LineageHubPipelineCompletedXp
             ? RunLifecyclePhase.FAILED
             : RunLifecyclePhase.FINISHED;
     LineageRunLifecycleEmitter.emitPipeline(pipeline, phase, null);
-    LineageHub.getInstance().flushQuietly();
+    // A top-level run waits for delivery. A nested run only waits once the 
queue fills up: that
+    // keeps a run per row from dropping events, without a full drain after 
every row.
+    if (LineageRunLifecycleEmitter.isTopLevel(pipeline)) {
+      LineageHub.getInstance().flushQuietly();
+    } else {
+      LineageHub.getInstance().flushIfBacklogged();
+    }
   }
 }
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
index 1283e485c3..86af8bd70a 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
@@ -22,6 +22,7 @@ import org.apache.hop.core.extension.ExtensionPoint;
 import org.apache.hop.core.extension.IExtensionPoint;
 import org.apache.hop.core.logging.ILogChannel;
 import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.lineage.LineageActionStartListener;
 import org.apache.hop.lineage.LineageRunLifecycleEmitter;
 import org.apache.hop.lineage.model.RunLifecyclePhase;
 import org.apache.hop.workflow.WorkflowExecutionExtension;
@@ -41,5 +42,6 @@ public class LineageHubWorkflowActionAfterXp
             ? RunLifecyclePhase.FAILED
             : RunLifecyclePhase.FINISHED;
     LineageRunLifecycleEmitter.emitWorkflowAction(extension, phase, null);
+    LineageActionStartListener.actionEnded(extension.workflow, 
extension.actionLogChannelId);
   }
 }
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionBeforeXp.java
 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionBeforeXp.java
deleted file mode 100644
index 833ccf9d02..0000000000
--- 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionBeforeXp.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements.  See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License.  You may obtain a copy of the License at
- *
- *      http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hop.lineage.xp;
-
-import org.apache.hop.core.exception.HopException;
-import org.apache.hop.core.extension.ExtensionPoint;
-import org.apache.hop.core.extension.IExtensionPoint;
-import org.apache.hop.core.logging.ILogChannel;
-import org.apache.hop.core.variables.IVariables;
-import org.apache.hop.lineage.LineageRunLifecycleEmitter;
-import org.apache.hop.lineage.model.RunLifecyclePhase;
-import org.apache.hop.workflow.WorkflowExecutionExtension;
-
-@ExtensionPoint(
-    id = "LineageHubWorkflowActionBeforeXp",
-    extensionPointId = "WorkflowBeforeActionExecution",
-    description = "Emits lineage RUN_LIFECYCLE STARTED before a workflow 
action runs")
-public class LineageHubWorkflowActionBeforeXp
-    implements IExtensionPoint<WorkflowExecutionExtension> {
-  @Override
-  public void callExtensionPoint(
-      ILogChannel log, IVariables variables, WorkflowExecutionExtension 
extension)
-      throws HopException {
-    LineageRunLifecycleEmitter.emitWorkflowAction(extension, 
RunLifecyclePhase.STARTED, null);
-  }
-}
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
index 780036e277..cd6225fa83 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
@@ -22,6 +22,7 @@ import org.apache.hop.core.extension.ExtensionPoint;
 import org.apache.hop.core.extension.IExtensionPoint;
 import org.apache.hop.core.logging.ILogChannel;
 import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.lineage.LineageActionStartListener;
 import org.apache.hop.lineage.LineageRunLifecycleEmitter;
 import org.apache.hop.lineage.hub.LineageHub;
 import org.apache.hop.lineage.model.RunLifecyclePhase;
@@ -38,11 +39,19 @@ public class LineageHubWorkflowFinishXp implements 
IExtensionPoint<IWorkflowEngi
   public void callExtensionPoint(
       ILogChannel log, IVariables variables, IWorkflowEngine<WorkflowMeta> 
workflow)
       throws HopException {
+    // An action that threw never got its FINISHED or FAILED; close it before 
the workflow ends.
+    LineageActionStartListener.failUnfinishedActions(workflow);
     RunLifecyclePhase phase =
         LineageRunLifecycleEmitter.workflowFailed(workflow, 
workflow.getResult())
             ? RunLifecyclePhase.FAILED
             : RunLifecyclePhase.FINISHED;
     LineageRunLifecycleEmitter.emitWorkflow(workflow, phase, null);
-    LineageHub.getInstance().flushQuietly();
+    // A top-level run waits for delivery. A nested run only waits once the 
queue fills up: that
+    // keeps a run per row from dropping events, without a full drain after 
every row.
+    if (LineageRunLifecycleEmitter.isTopLevel(workflow)) {
+      LineageHub.getInstance().flushQuietly();
+    } else {
+      LineageHub.getInstance().flushIfBacklogged();
+    }
   }
 }
diff --git 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
index ed6c2099b9..a4b071f78c 100644
--- 
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
+++ 
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
@@ -22,7 +22,9 @@ import org.apache.hop.core.extension.ExtensionPoint;
 import org.apache.hop.core.extension.IExtensionPoint;
 import org.apache.hop.core.logging.ILogChannel;
 import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.lineage.LineageActionStartListener;
 import org.apache.hop.lineage.LineageRunLifecycleEmitter;
+import org.apache.hop.lineage.hub.LineageHub;
 import org.apache.hop.lineage.model.RunLifecyclePhase;
 import org.apache.hop.workflow.WorkflowMeta;
 import org.apache.hop.workflow.engine.IWorkflowEngine;
@@ -30,12 +32,19 @@ import org.apache.hop.workflow.engine.IWorkflowEngine;
 @ExtensionPoint(
     id = "LineageHubWorkflowStartXp",
     extensionPointId = "WorkflowStart",
-    description = "Emits lineage RUN_LIFECYCLE STARTED for workflow 
executions")
+    description =
+        "Emits lineage RUN_LIFECYCLE STARTED for workflow executions and 
registers the action"
+            + " start listener")
 public class LineageHubWorkflowStartXp implements 
IExtensionPoint<IWorkflowEngine<WorkflowMeta>> {
   @Override
   public void callExtensionPoint(
       ILogChannel log, IVariables variables, IWorkflowEngine<WorkflowMeta> 
workflow)
       throws HopException {
     LineageRunLifecycleEmitter.emitWorkflow(workflow, 
RunLifecyclePhase.STARTED, null);
+    // Action starts are emitted from an action listener: only there does the 
cloned action, and
+    // with it the action run's id, exist.
+    if (LineageHub.getInstance().isEnabled()) {
+      LineageActionStartListener.register(workflow);
+    }
   }
 }
diff --git 
a/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
 
b/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
index 22ad30841c..5d87e30c77 100644
--- 
a/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
+++ 
b/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
@@ -19,6 +19,8 @@ package org.apache.hop.pipeline.transforms.file;
 
 import java.util.Date;
 import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
 import org.apache.commons.vfs2.FileObject;
 import org.apache.commons.vfs2.FileSystemException;
 import org.apache.hop.core.IRowSet;
@@ -26,6 +28,7 @@ import org.apache.hop.core.ResultFile;
 import org.apache.hop.core.exception.HopException;
 import org.apache.hop.core.exception.HopFileException;
 import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
 import org.apache.hop.core.row.RowMeta;
 import org.apache.hop.core.row.value.ValueMetaFactory;
 import org.apache.hop.core.vfs.HopVfs;
@@ -44,6 +47,9 @@ public abstract class BaseFileInputTransform<
 
   private static final Class<?> PKG = BaseFileInputTransform.class;
 
+  /** The file schema for lineage, built on the first file this transform 
opens. */
+  private IRowMeta lineageFileRowMeta;
+
   /** Create reader for specific file. */
   protected abstract IBaseFileInputReader createReader(Meta meta, Data data, 
FileObject file)
       throws Exception;
@@ -164,9 +170,12 @@ public abstract class BaseFileInputTransform<
 
   /** Best-effort {@link FileIoOperation#READ} lineage for the file being 
opened. */
   protected void emitLineageFileRead(FileObject file) {
-    if (file == null) {
+    if (file == null || !LineageFileIoEmitter.isEnabled()) {
       return;
     }
+    if (lineageFileRowMeta == null) {
+      lineageFileRowMeta = fileRowMeta();
+    }
     Long size = null;
     try {
       if (file.getType().hasContent()) {
@@ -175,8 +184,50 @@ public abstract class BaseFileInputTransform<
     } catch (Exception ignored) {
       // optional metadata for lineage only
     }
+    String extension = file.getName().getExtension();
     LineageFileIoEmitter.emitTransformFileIo(
-        this, FileIoOperation.READ, file, null, size, true, null);
+        this,
+        FileIoOperation.READ,
+        file,
+        null,
+        size,
+        true,
+        null,
+        LineageFileIoEmitter.toContentSchema(
+            lineageFileRowMeta, extension == null ? null : 
extension.toLowerCase(Locale.ROOT)));
+  }
+
+  /**
+   * The names of the fields this transform reads from the file itself, in the 
order it reads them,
+   * which make up the file's schema in its lineage; {@code null} when every 
output field comes from
+   * the file. In the output row these fields follow the ones passed through 
from the input rows,
+   * and precede the fields the transform adds (file name, row number, ...). 
Asked once, on the
+   * first file the transform opens with lineage on.
+   */
+  protected List<String> fileFieldNames() {
+    return null;
+  }
+
+  /**
+   * The file's own fields: types as read, from their position in the output 
row, and names as the
+   * file has them. The output row cannot be matched by name: a passed-through 
field can have the
+   * name of a file field (which the row then renames to {@code name_1}), and 
the file name can be
+   * prepended to every file field.
+   */
+  private IRowMeta fileRowMeta() {
+    IRowMeta output = data.outputRowMeta;
+    List<String> names = fileFieldNames();
+    if (output == null || names == null) {
+      return output;
+    }
+    int from = meta.getFileInput().isPassingThruFields() ? 
data.nrPassThruFields : 0;
+    RowMeta fileRow = new RowMeta();
+    for (int i = 0; i < names.size() && from + i < output.size(); i++) {
+      IValueMeta valueMeta = output.getValueMeta(from + i).clone();
+      valueMeta.setName(names.get(i));
+      fileRow.addValueMeta(valueMeta);
+    }
+    return fileRow;
   }
 
   /** Read files from previous transform. */
diff --git a/engine/src/main/java/org/apache/hop/run/HopRunBase.java 
b/engine/src/main/java/org/apache/hop/run/HopRunBase.java
index bf27b7229b..ce769999c5 100644
--- a/engine/src/main/java/org/apache/hop/run/HopRunBase.java
+++ b/engine/src/main/java/org/apache/hop/run/HopRunBase.java
@@ -202,8 +202,8 @@ public abstract class HopRunBase implements Runnable, 
IHasHopMetadataProvider {
               Duration.ofSeconds(Math.max(0L, shutdownTimeout)),
               () -> LineageHub.getInstance().shutdown());
       shutdownHookThread = new Thread(shutdownHook, 
"hop-run-stop-on-shutdown");
-      // The lineage shutdown hook of HopEnvironment waits for this run, so 
that its end is
-      // delivered as well.
+      // The exit hook of the lineage hub waits for this run, so that its end 
is delivered as
+      // well.
       LineageHub.getInstance().holdShutdownOnExit(shutdownHook.getRunOver());
       Runtime.getRuntime().addShutdownHook(shutdownHookThread);
 
diff --git 
a/engine/src/test/java/org/apache/hop/lineage/LineageActionLifecycleTest.java 
b/engine/src/test/java/org/apache/hop/lineage/LineageActionLifecycleTest.java
new file mode 100644
index 0000000000..b39f4a2490
--- /dev/null
+++ 
b/engine/src/test/java/org/apache/hop/lineage/LineageActionLifecycleTest.java
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.lineage.hub.LineageHub;
+import org.apache.hop.lineage.model.LineageEvent;
+import org.apache.hop.lineage.model.RunLifecycleLineagePayload;
+import org.apache.hop.lineage.model.RunLifecyclePhase;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowExecutionExtension;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionBase;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+/**
+ * An action run is identified by the log channel of the action clone the 
workflow executes. Its
+ * STARTED and FINISHED events have to carry that id, not the workflow's: the 
action is a job of its
+ * own ({@code <workflow>/<action>}), and a pipeline it starts names that same 
id as its parent.
+ */
+class LineageActionLifecycleTest {
+
+  @SuppressWarnings("unchecked")
+  private static IWorkflowEngine<WorkflowMeta> workflow() {
+    IWorkflowEngine<WorkflowMeta> wf = mock(IWorkflowEngine.class);
+    WorkflowMeta meta = new WorkflowMeta();
+    meta.setName("nightly-load");
+    when(wf.getWorkflowMeta()).thenReturn(meta);
+    when(wf.getLogChannelId()).thenReturn("workflow-run-id");
+    return wf;
+  }
+
+  private static ActionBase executingAction() {
+    ActionBase action = mock(ActionBase.class);
+    ILogChannel log = mock(ILogChannel.class);
+    when(log.getLogChannelId()).thenReturn("action-run-id");
+    when(action.getName()).thenReturn("Run orders");
+    when(action.getDescription()).thenReturn("Loads the orders");
+    when(action.getLogChannel()).thenReturn(log);
+    return action;
+  }
+
+  @Test
+  void actionStartCarriesTheExecutingActionsRunId() {
+    LineageHub hub = mock(LineageHub.class);
+    IWorkflowEngine<WorkflowMeta> wf = workflow();
+    ActionBase action = executingAction();
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      List<IActionListener> listeners = new ArrayList<>();
+      when(wf.getActionListeners()).thenReturn(listeners);
+      doAnswer(invocation -> listeners.add(invocation.getArgument(0)))
+          .when(wf)
+          .addActionListener(any());
+
+      LineageActionStartListener.register(wf);
+      listeners.get(0).beforeExecution(wf, new ActionMeta(action), action);
+    }
+
+    ArgumentCaptor<LineageEvent> event = 
ArgumentCaptor.forClass(LineageEvent.class);
+    verify(hub).emit(event.capture());
+    assertEquals(
+        RunLifecyclePhase.STARTED,
+        ((RunLifecycleLineagePayload) 
event.getValue().getPayload()).getPhase());
+    assertEquals("action-run-id", 
event.getValue().getContext().getLogChannelId());
+    assertEquals("Run orders", event.getValue().getContext().getActionName());
+    assertEquals("nightly-load", 
event.getValue().getContext().getWorkflowName());
+    assertEquals(
+        "Loads the orders",
+        event
+            .getValue()
+            .getContext()
+            .getAttributes()
+            .get(LineageRunLifecycleEmitter.ATTR_DESCRIPTION));
+  }
+
+  @Test
+  void actionStartListenerIsRegisteredOncePerWorkflow() {
+    IWorkflowEngine<WorkflowMeta> wf = workflow();
+    List<IActionListener> listeners = new ArrayList<>();
+    when(wf.getActionListeners()).thenReturn(listeners);
+    doAnswer(invocation -> listeners.add(invocation.getArgument(0)))
+        .when(wf)
+        .addActionListener(any());
+
+    LineageActionStartListener.register(wf);
+    LineageActionStartListener.register(wf);
+
+    verify(wf, times(1)).addActionListener(any());
+  }
+
+  @Test
+  void actionFinishCarriesTheExecutingActionsRunId() {
+    LineageHub hub = mock(LineageHub.class);
+    WorkflowExecutionExtension extension =
+        new WorkflowExecutionExtension(
+            workflow(), new Result(), new ActionMeta(executingAction()), true);
+    extension.actionLogChannelId = "action-run-id";
+    extension.actionExecutionResult = new Result();
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      LineageRunLifecycleEmitter.emitWorkflowAction(extension, 
RunLifecyclePhase.FINISHED, null);
+    }
+
+    ArgumentCaptor<LineageEvent> event = 
ArgumentCaptor.forClass(LineageEvent.class);
+    verify(hub).emit(event.capture());
+    assertEquals("action-run-id", 
event.getValue().getContext().getLogChannelId());
+  }
+
+  @Test
+  void skippedActionEmitsNothing() {
+    LineageHub hub = mock(LineageHub.class);
+    // An extension point told the workflow not to execute the action: no 
clone, no run id.
+    WorkflowExecutionExtension extension =
+        new WorkflowExecutionExtension(
+            workflow(), new Result(), new ActionMeta(executingAction()), 
false);
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      LineageRunLifecycleEmitter.emitWorkflowAction(extension, 
RunLifecyclePhase.FINISHED, null);
+    }
+
+    verify(hub, never()).emit(any());
+  }
+
+  private static IActionListener 
registeredListener(IWorkflowEngine<WorkflowMeta> wf) {
+    List<IActionListener> listeners = new ArrayList<>();
+    when(wf.getActionListeners()).thenReturn(listeners);
+    doAnswer(invocation -> listeners.add(invocation.getArgument(0)))
+        .when(wf)
+        .addActionListener(any());
+    LineageActionStartListener.register(wf);
+    return listeners.get(0);
+  }
+
+  // The listener runs inside the workflow's action execution: a lineage 
problem may not fail it.
+  @Test
+  void aFailingStartEmitDoesNotFailTheAction() {
+    LineageHub hub = mock(LineageHub.class);
+    doThrow(new IllegalStateException("sink exploded")).when(hub).emit(any());
+    IWorkflowEngine<WorkflowMeta> wf = workflow();
+    ILogChannel workflowLog = mock(ILogChannel.class);
+    when(wf.getLogChannel()).thenReturn(workflowLog);
+    ActionBase action = executingAction();
+    IActionListener listener = registeredListener(wf);
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      assertDoesNotThrow(() -> listener.beforeExecution(wf, new 
ActionMeta(action), action));
+    }
+
+    verify(workflowLog).logError(anyString(), any(Throwable.class));
+  }
+
+  // execute() threw: the workflow skips WorkflowAfterActionExecution, so the 
run is closed as
+  // FAILED when the workflow ends, under the run id its STARTED carried.
+  @Test
+  void anActionThatThrewIsClosedAsFailedWhenTheWorkflowEnds() {
+    LineageHub hub = mock(LineageHub.class);
+    IWorkflowEngine<WorkflowMeta> wf = workflow();
+    ActionBase action = executingAction();
+    IActionListener listener = registeredListener(wf);
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      listener.beforeExecution(wf, new ActionMeta(action), action);
+      LineageActionStartListener.failUnfinishedActions(wf);
+      // Closed once: a second end of the workflow has nothing left to close.
+      LineageActionStartListener.failUnfinishedActions(wf);
+    }
+
+    ArgumentCaptor<LineageEvent> events = 
ArgumentCaptor.forClass(LineageEvent.class);
+    verify(hub, times(2)).emit(events.capture());
+    LineageEvent failed = events.getAllValues().get(1);
+    assertEquals(
+        RunLifecyclePhase.FAILED, ((RunLifecycleLineagePayload) 
failed.getPayload()).getPhase());
+    assertEquals("action-run-id", failed.getContext().getLogChannelId());
+    assertEquals(
+        LineageRunLifecycleEmitter.FAILED,
+        
failed.getContext().getAttributes().get(LineageRunLifecycleEmitter.OUTCOME));
+  }
+
+  @Test
+  void anActionThatEndedIsNotClosedAgain() {
+    LineageHub hub = mock(LineageHub.class);
+    IWorkflowEngine<WorkflowMeta> wf = workflow();
+    ActionBase action = executingAction();
+    IActionListener listener = registeredListener(wf);
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      listener.beforeExecution(wf, new ActionMeta(action), action);
+      LineageActionStartListener.actionEnded(wf, "action-run-id");
+      LineageActionStartListener.failUnfinishedActions(wf);
+    }
+
+    verify(hub, times(1)).emit(any());
+  }
+}
diff --git 
a/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java 
b/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
index 7f7f3117ab..6f63f44a41 100644
--- a/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
+++ b/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
@@ -17,6 +17,10 @@
 
 package org.apache.hop.lineage;
 
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
 import static org.mockito.ArgumentMatchers.argThat;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.verify;
@@ -25,6 +29,9 @@ import static org.mockito.Mockito.when;
 import org.apache.commons.vfs2.FileObject;
 import org.apache.commons.vfs2.VFS;
 import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaInteger;
+import org.apache.hop.core.row.value.ValueMetaString;
 import org.apache.hop.lineage.context.LineageSubjectType;
 import org.apache.hop.lineage.hub.LineageHub;
 import org.apache.hop.lineage.model.FileIoContentSchema;
@@ -249,6 +256,26 @@ class LineageFileIoEmitterTest {
                 }));
   }
 
+  @Test
+  void toContentSchema_describesTheRowShapeWithTheFileFormat() {
+    RowMeta rowMeta = new RowMeta();
+    rowMeta.addValueMeta(new ValueMetaString("customer"));
+    rowMeta.addValueMeta(new ValueMetaInteger("id"));
+
+    FileIoContentSchema schema = LineageFileIoEmitter.toContentSchema(rowMeta, 
"csv");
+
+    assertEquals("csv", schema.getFormatHint());
+    assertEquals(2, schema.getColumns().size());
+    assertEquals("customer", schema.getColumns().get(0).getName());
+    assertEquals("id", schema.getColumns().get(1).getName());
+    // The type name comes from the value-meta plugin registry, which a bare 
unit test does not
+    // load.
+    assertNotNull(schema.getColumns().get(0).getHopTypeName());
+    assertTrue(schema.getStructureRoots().isEmpty());
+    assertNull(LineageFileIoEmitter.toContentSchema(new RowMeta(), "csv"));
+    assertNull(LineageFileIoEmitter.toContentSchema(null, null));
+  }
+
   @Test
   void emitTransformFileIo_postsFileIoEventWithTransformContext() throws 
Exception {
     LineageHub hub = mock(LineageHub.class);
diff --git 
a/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java 
b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
index a197d70d9c..67d87a0215 100644
--- a/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
+++ b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
@@ -36,6 +36,11 @@ import org.apache.hop.pipeline.PipelineHopMeta;
 import org.apache.hop.pipeline.PipelineMeta;
 import org.apache.hop.pipeline.engine.IPipelineEngine;
 import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.workflow.WorkflowHopMeta;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.actions.dummy.ActionDummy;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
 import org.junit.jupiter.api.Test;
 import org.mockito.ArgumentCaptor;
 import org.mockito.MockedStatic;
@@ -160,14 +165,19 @@ class LineagePipelineGraphTest {
   }
 
   @Test
-  void pipelineWithoutHopsEmitsNoGraph() {
+  void pipelineWithoutHopsEmitsAnEmptyGraph() {
+    // An empty graph, not none: a sink that gets no graph assumes an engine 
that sends none and
+    // files the run's datasets under the pipeline instead of under its only 
transform.
     PipelineMeta meta = new PipelineMeta();
     meta.getTransforms().add(transform("Alone"));
 
     List<LineageEvent> events = emitPipeline(meta, RunLifecyclePhase.STARTED);
 
-    assertEquals(1, events.size());
+    assertEquals(2, events.size());
     assertEquals(LineageEventKind.RUN_LIFECYCLE, events.get(0).getKind());
+    assertEquals(LineageEventKind.PIPELINE_GRAPH, events.get(1).getKind());
+    assertTrue(
+        ((PipelineGraphLineagePayload) 
events.get(1).getPayload()).getTransitions().isEmpty());
   }
 
   private static PipelineMeta chain() {
@@ -196,4 +206,60 @@ class LineagePipelineGraphTest {
     verify(hub, atLeastOnce()).emit(captor.capture());
     return captor.getAllValues();
   }
+
+  // Like a pipeline's: a workflow without hops still tells the sinks so, with 
an empty graph.
+  @Test
+  void workflowWithoutHopsEmitsAnEmptyGraph() {
+    WorkflowMeta meta = new WorkflowMeta();
+    meta.addAction(new ActionMeta(new ActionDummy("Alone")));
+
+    List<LineageEvent> events = emitWorkflow(meta, RunLifecyclePhase.STARTED);
+
+    assertEquals(2, events.size());
+    assertEquals(LineageEventKind.RUN_LIFECYCLE, events.get(0).getKind());
+    assertEquals(LineageEventKind.WORKFLOW_GRAPH, events.get(1).getKind());
+    assertTrue(
+        ((PipelineGraphLineagePayload) 
events.get(1).getPayload()).getTransitions().isEmpty());
+  }
+
+  @SuppressWarnings("unchecked")
+  private static List<LineageEvent> emitWorkflow(WorkflowMeta meta, 
RunLifecyclePhase phase) {
+    LineageHub hub = mock(LineageHub.class);
+    IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+    when(workflow.getWorkflowMeta()).thenReturn(meta);
+    when(workflow.getLogChannelId()).thenReturn("workflow-log");
+
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      LineageRunLifecycleEmitter.emitWorkflow(workflow, phase, null);
+    }
+
+    ArgumentCaptor<LineageEvent> captor = 
ArgumentCaptor.forClass(LineageEvent.class);
+    verify(hub, atLeastOnce()).emit(captor.capture());
+    return captor.getAllValues();
+  }
+
+  // A workflow's hops say which action follows which, whatever their 
condition; disabled hops are
+  // not part of what ran.
+  @Test
+  void workflowHopsAreReportedBetweenActions() {
+    WorkflowMeta meta = new WorkflowMeta();
+    ActionMeta load = new ActionMeta(new ActionDummy("Load"));
+    ActionMeta report = new ActionMeta(new ActionDummy("Report"));
+    ActionMeta alert = new ActionMeta(new ActionDummy("Alert"));
+    meta.addAction(load);
+    meta.addAction(report);
+    meta.addAction(alert);
+    meta.addWorkflowHop(new WorkflowHopMeta(load, report));
+    WorkflowHopMeta onFailure = new WorkflowHopMeta(load, alert);
+    onFailure.setEvaluation(false);
+    meta.addWorkflowHop(onFailure);
+    WorkflowHopMeta disabled = new WorkflowHopMeta(report, alert);
+    disabled.setEnabled(false);
+    meta.addWorkflowHop(disabled);
+
+    assertEquals(
+        List.of(new Transition("Load", "Report"), new Transition("Load", 
"Alert")),
+        LineageRunLifecycleEmitter.workflowTransitions(meta));
+  }
 }
diff --git 
a/engine/src/test/java/org/apache/hop/lineage/LineagePipelineParentTest.java 
b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineParentTest.java
new file mode 100644
index 0000000000..0755a32c2d
--- /dev/null
+++ b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineParentTest.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.lineage.context.LineageContext;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.engine.IPipelineEngine;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionBase;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+import org.junit.jupiter.api.Test;
+
+/**
+ * A pipeline's lineage run has to say which run started it. The engine knows 
either the action that
+ * ran it (inside its workflow) or the parent pipeline of a sub-pipeline; the 
job name it reports is
+ * the one the OpenLineage mapper gives that parent, so a collector can draw 
the hierarchy.
+ */
+class LineagePipelineParentTest {
+
+  private static LineageContext.Builder context() {
+    return LineageContext.builder();
+  }
+
+  @Test
+  void pipelineRunByAWorkflowActionNamesTheActionAsItsParent() {
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+    @SuppressWarnings("unchecked")
+    IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+    ActionBase action = mock(ActionBase.class);
+    ILogChannel actionLog = mock(ILogChannel.class);
+    WorkflowMeta workflowMeta = new WorkflowMeta();
+    workflowMeta.setName("nightly-load");
+    when(pipeline.getParent()).thenReturn(action);
+    when(pipeline.getParentWorkflow()).thenReturn(workflow);
+    when(workflow.getWorkflowMeta()).thenReturn(workflowMeta);
+    when(action.getName()).thenReturn("Run orders pipeline");
+    when(action.getLogChannel()).thenReturn(actionLog);
+    when(actionLog.getLogChannelId()).thenReturn("action-run-id");
+
+    LineageContext.Builder ctx = context();
+    LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+    LineageContext built = ctx.build();
+
+    assertEquals(
+        "nightly-load/Run orders pipeline",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+    assertEquals(
+        "nightly-load",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_WORKFLOW_NAME));
+    assertEquals(
+        "Run orders pipeline",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_ACTION_NAME));
+    assertEquals(
+        "action-run-id", 
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+  }
+
+  @Test
+  void subPipelineNamesTheParentPipelineAsItsParent() {
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> parentPipeline = mock(IPipelineEngine.class);
+    PipelineMeta parentMeta = new PipelineMeta();
+    parentMeta.setName("orders-parent");
+    when(pipeline.getParentPipeline()).thenReturn(parentPipeline);
+    when(parentPipeline.getPipelineMeta()).thenReturn(parentMeta);
+    when(parentPipeline.getLogChannelId()).thenReturn("parent-run-id");
+
+    LineageContext.Builder ctx = context();
+    LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+    LineageContext built = ctx.build();
+
+    assertEquals(
+        "orders-parent",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+    assertEquals(
+        "orders-parent",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_PIPELINE_NAME));
+    assertEquals(
+        "parent-run-id", 
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+  }
+
+  @Test
+  void workflowRunByAWorkflowActionNamesTheActionAsItsParent() {
+    @SuppressWarnings("unchecked")
+    IWorkflowEngine<WorkflowMeta> child = mock(IWorkflowEngine.class);
+    @SuppressWarnings("unchecked")
+    IWorkflowEngine<WorkflowMeta> parent = mock(IWorkflowEngine.class);
+    ActionBase action = mock(ActionBase.class);
+    ILogChannel actionLog = mock(ILogChannel.class);
+    WorkflowMeta parentMeta = new WorkflowMeta();
+    parentMeta.setName("nightly-load");
+    when(child.getParent()).thenReturn(action);
+    when(child.getParentWorkflow()).thenReturn(parent);
+    when(parent.getWorkflowMeta()).thenReturn(parentMeta);
+    when(parent.getLogChannelId()).thenReturn("parent-workflow-run-id");
+    when(action.getName()).thenReturn("Run sub-workflow");
+    when(action.getLogChannel()).thenReturn(actionLog);
+    when(actionLog.getLogChannelId()).thenReturn("action-run-id");
+
+    LineageContext.Builder ctx = context();
+    LineageRunLifecycleEmitter.fillWorkflowParentFields(ctx, child);
+    LineageContext built = ctx.build();
+
+    assertEquals(
+        "nightly-load/Run sub-workflow",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+    assertEquals(
+        "action-run-id", 
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+  }
+
+  // A Pipeline Executor or mapping runs its pipeline as its logging parent. 
The parent is that
+  // transform's run, <pipeline>/<transform>: the same shape as a pipeline run 
by a workflow action.
+  @Test
+  void subPipelineRunByATransformNamesThatTransform() {
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> parentPipeline = mock(IPipelineEngine.class);
+    BaseTransform<?, ?> executor = mock(BaseTransform.class);
+    ILogChannel executorLog = mock(ILogChannel.class);
+    PipelineMeta parentMeta = new PipelineMeta();
+    parentMeta.setName("orders-parent");
+    when(pipeline.getParentPipeline()).thenReturn(parentPipeline);
+    when(pipeline.getParent()).thenReturn(executor);
+    when(parentPipeline.getPipelineMeta()).thenReturn(parentMeta);
+    when(parentPipeline.getLogChannelId()).thenReturn("parent-run-id");
+    when(executor.getTransformName()).thenReturn("Run per order");
+    when(executor.getLogChannel()).thenReturn(executorLog);
+    when(executorLog.getLogChannelId()).thenReturn("executor-run-id");
+
+    LineageContext.Builder ctx = context();
+    LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+    LineageContext built = ctx.build();
+
+    assertEquals(
+        "orders-parent/Run per order",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+    assertEquals(
+        "executor-run-id",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+    assertEquals(
+        "orders-parent",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_PIPELINE_NAME));
+    assertEquals(
+        "Run per order",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_TRANSFORM_NAME));
+  }
+
+  // A Workflow Executor is the logging parent of the workflow it runs: same 
shape.
+  @Test
+  void workflowRunByATransformNamesThatTransform() {
+    @SuppressWarnings("unchecked")
+    IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> parentPipeline = mock(IPipelineEngine.class);
+    BaseTransform<?, ?> executor = mock(BaseTransform.class);
+    ILogChannel executorLog = mock(ILogChannel.class);
+    PipelineMeta parentMeta = new PipelineMeta();
+    parentMeta.setName("orders-parent");
+    when(workflow.getParentPipeline()).thenReturn(parentPipeline);
+    when(workflow.getParent()).thenReturn(executor);
+    when(parentPipeline.getPipelineMeta()).thenReturn(parentMeta);
+    when(executor.getTransformName()).thenReturn("Run workflow per order");
+    when(executor.getLogChannel()).thenReturn(executorLog);
+    when(executorLog.getLogChannelId()).thenReturn("executor-run-id");
+
+    LineageContext.Builder ctx = context();
+    LineageRunLifecycleEmitter.fillWorkflowParentFields(ctx, workflow);
+    LineageContext built = ctx.build();
+
+    assertEquals(
+        "orders-parent/Run workflow per order",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+    assertEquals(
+        "executor-run-id",
+        
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+  }
+
+  // Only a top-level run flushes the lineage hub when it completes.
+  @Test
+  void onlyRunsWithoutAParentAreTopLevel() {
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> topLevel = mock(IPipelineEngine.class);
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> subPipeline = mock(IPipelineEngine.class);
+    when(subPipeline.getParentPipeline()).thenReturn(topLevel);
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> runByAction = mock(IPipelineEngine.class);
+    when(runByAction.getParent()).thenReturn(mock(ActionBase.class));
+    @SuppressWarnings("unchecked")
+    IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+    @SuppressWarnings("unchecked")
+    IWorkflowEngine<WorkflowMeta> childWorkflow = mock(IWorkflowEngine.class);
+    when(childWorkflow.getParentWorkflow()).thenReturn(workflow);
+
+    assertTrue(LineageRunLifecycleEmitter.isTopLevel(topLevel));
+    assertFalse(LineageRunLifecycleEmitter.isTopLevel(subPipeline));
+    assertFalse(LineageRunLifecycleEmitter.isTopLevel(runByAction));
+    assertTrue(LineageRunLifecycleEmitter.isTopLevel(workflow));
+    assertFalse(LineageRunLifecycleEmitter.isTopLevel(childWorkflow));
+  }
+
+  @Test
+  void topLevelPipelineHasNoParentFields() {
+    @SuppressWarnings("unchecked")
+    IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+
+    LineageContext.Builder ctx = context();
+    LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+    LineageContext built = ctx.build();
+
+    
assertNull(built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+    
assertNull(built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+  }
+}
diff --git 
a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
 
b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
index 9fcaf7c0a8..0b8ef189dd 100644
--- 
a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
+++ 
b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
@@ -17,10 +17,12 @@
 
 package org.apache.hop.lineage.hub;
 
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
 import static org.junit.jupiter.api.Assertions.assertEquals;
 import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Set;
 import java.util.concurrent.CountDownLatch;
@@ -160,8 +162,8 @@ class LineageHubConcurrentShutdownTest {
   }
 
   /**
-   * The JVM runs the shutdown hook of HopEnvironment at the same time as the 
one of hop-run. While
-   * hop-run holds the hub, that hook may not shut it down: the run still has 
to emit its end.
+   * The JVM runs the exit hook of the hub at the same time as the one of 
hop-run. While hop-run
+   * holds the hub, that hook may not shut it down: the run still has to emit 
its end.
    */
   @Test
   void shutdownOnExitWaitsForTheRunThatHoldsIt() throws Exception {
@@ -206,6 +208,47 @@ class LineageHubConcurrentShutdownTest {
     assertEquals(1, sinkShutdowns.get());
   }
 
+  /** The exit hook the hub registers is the one that waits for hop-run, and 
it is added once. */
+  @Test
+  void theRegisteredExitHookWaitsForTheRunThatHoldsIt() throws Exception {
+    AtomicInteger sinkShutdowns = new AtomicInteger();
+    LineageHub hub =
+        LineageHub.newIsolatedForTesting(
+            configuration(), List.of(sink(new AtomicInteger(), 
sinkShutdowns)));
+    hub.emit(event());
+    CountDownLatch runOver = new CountDownLatch(1);
+    hub.holdShutdownOnExit(runOver);
+    List<Thread> hooks = new ArrayList<>();
+    hub.registerExitHook(hooks::add);
+    hub.registerExitHook(hooks::add);
+    assertEquals(1, hooks.size(), "The exit hook should be registered once");
+
+    Thread exitHook = hooks.get(0);
+    exitHook.start();
+    Thread.sleep(300);
+    assertTrue(exitHook.isAlive(), "The exit hook shut the hub down while a 
run held it");
+    assertEquals(0, sinkShutdowns.get());
+
+    runOver.countDown();
+    exitHook.join(10_000);
+    assertFalse(exitHook.isAlive());
+    assertEquals(1, sinkShutdowns.get());
+  }
+
+  /** The first event can come in while the JVM is already exiting: it may not 
fail the emit. */
+  @Test
+  void aJvmThatIsAlreadyExitingDoesNotFailTheRegistration() {
+    LineageHub hub =
+        LineageHub.newIsolatedForTesting(
+            configuration(), List.of(sink(new AtomicInteger(), new 
AtomicInteger())));
+    assertDoesNotThrow(
+        () ->
+            hub.registerExitHook(
+                hook -> {
+                  throw new IllegalStateException("Shutdown in progress");
+                }));
+  }
+
   private static LineageConfiguration configuration() {
     return LineageConfiguration.forTesting(true, 100, 1, 5000L, Set.of());
   }
diff --git 
a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java 
b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
index b1d9fdd7ca..f049c4b40e 100644
--- a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
+++ b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
@@ -18,6 +18,7 @@
 package org.apache.hop.lineage.hub;
 
 import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
 import static org.junit.jupiter.api.Assertions.assertTrue;
 
 import java.util.ArrayList;
@@ -539,6 +540,56 @@ class LineageHubTest {
     assertEquals(2, delivered.get());
   }
 
+  /**
+   * A nested run (one per row of a Pipeline Executor, say) does not wait for 
the sinks while the
+   * queue has room, and waits for them once it is half full, before events 
get dropped.
+   */
+  @Test
+  void flushIfBackloggedWaitsOnlyOnceTheQueueIsHalfFull() throws Exception {
+    LineageConfiguration cfg = LineageConfiguration.forTesting(true, 10, 1, 
5000L, Set.of());
+    CountDownLatch inSink = new CountDownLatch(1);
+    CountDownLatch release = new CountDownLatch(1);
+    AtomicInteger delivered = new AtomicInteger();
+    ILineageSink sink =
+        batch -> {
+          inSink.countDown();
+          try {
+            release.await(10, TimeUnit.SECONDS);
+          } catch (InterruptedException e) {
+            Thread.currentThread().interrupt();
+          }
+          delivered.addAndGet(batch.size());
+        };
+    LineageHub hub = LineageHub.newIsolatedForTesting(cfg, List.of(sink));
+    hub.emit(sampleEvent());
+    assertTrue(inSink.await(5, TimeUnit.SECONDS), "the sink never got the 
first event");
+
+    // The worker is busy in the sink, so these stay queued: 3 of 10.
+    for (int i = 0; i < 3; i++) {
+      hub.emit(sampleEvent());
+    }
+    Thread belowHalf = new Thread(hub::flushIfBacklogged);
+    belowHalf.start();
+    belowHalf.join(2000);
+    assertFalse(belowHalf.isAlive(), "a nested run waited for the sinks with 
the queue 3/10 full");
+
+    // 5 of 10 queued: now it waits until the sinks have caught up.
+    for (int i = 0; i < 2; i++) {
+      hub.emit(sampleEvent());
+    }
+    Thread halfFull = new Thread(hub::flushIfBacklogged);
+    halfFull.start();
+    halfFull.join(300);
+    assertTrue(halfFull.isAlive(), "a nested run did not wait with the queue 
half full");
+
+    release.countDown();
+    halfFull.join(5000);
+    assertFalse(halfFull.isAlive());
+    assertEquals(6, delivered.get());
+    assertEquals(0, hub.getDroppedEventCount());
+    hub.shutdown();
+  }
+
   private static LineageEvent sampleEvent() {
     return LineageEvent.of(LineageEventKind.RUN_LIFECYCLE, 
LineageContext.empty(), null);
   }
diff --git 
a/engine/src/test/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransformLineageTest.java
 
b/engine/src/test/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransformLineageTest.java
new file mode 100644
index 0000000000..74dda8544a
--- /dev/null
+++ 
b/engine/src/test/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransformLineageTest.java
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.file;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.logging.ILoggingObject;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaInteger;
+import org.apache.hop.core.row.value.ValueMetaNumber;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.vfs.HopVfs;
+import org.apache.hop.lineage.hub.LineageHub;
+import org.apache.hop.lineage.model.FileIoContentSchema;
+import org.apache.hop.lineage.model.FileIoLineagePayload;
+import org.apache.hop.lineage.model.FileIoTabularColumn;
+import org.apache.hop.lineage.model.LineageEvent;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.mock.TransformMockHelper;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+/** The schema a file input transform reports for the file it reads, in its 
FILE_IO lineage. */
+@SuppressWarnings({"rawtypes", "unchecked"})
+class BaseFileInputTransformLineageTest {
+
+  /** A file input that reads the fields "tag" and "amount" from its files. */
+  static final class FileInput
+      extends BaseFileInputTransform<BaseFileInputMeta, 
BaseFileInputTransformData> {
+    final AtomicInteger fileFieldNamesCalls = new AtomicInteger();
+
+    FileInput(
+        TransformMeta transformMeta,
+        BaseFileInputMeta meta,
+        BaseFileInputTransformData data,
+        PipelineMeta pipelineMeta,
+        Pipeline pipeline) {
+      super(transformMeta, meta, data, 0, pipelineMeta, pipeline);
+    }
+
+    @Override
+    protected IBaseFileInputReader createReader(
+        BaseFileInputMeta meta, BaseFileInputTransformData data, FileObject 
file) {
+      return null;
+    }
+
+    @Override
+    public boolean failAfterBadFile(String errorMsg, boolean errorIgnored, 
boolean skipBadFiles) {
+      return true;
+    }
+
+    @Override
+    protected List<String> fileFieldNames() {
+      fileFieldNamesCalls.incrementAndGet();
+      return List.of("tag", "amount");
+    }
+  }
+
+  private TransformMockHelper<BaseFileInputMeta, BaseFileInputTransformData> 
helper;
+  private FileInput transform;
+  private FileObject file;
+
+  @BeforeAll
+  static void initHop() throws Exception {
+    HopEnvironment.init();
+  }
+
+  @BeforeEach
+  void setUp() throws Exception {
+    helper =
+        new TransformMockHelper<>(
+            "File input", BaseFileInputMeta.class, 
BaseFileInputTransformData.class);
+    when(helper.logChannelFactory.create(any(), any(ILoggingObject.class)))
+        .thenReturn(helper.iLogChannel);
+    BaseFileInput fileInput = new BaseFileInput();
+    fileInput.setPassingThruFields(true);
+    when(helper.iTransformMeta.getFileInput()).thenReturn(fileInput);
+    BaseFileInputTransformData data = new BaseFileInputTransformData();
+    transform =
+        new FileInput(
+            helper.transformMeta,
+            helper.iTransformMeta,
+            data,
+            helper.pipelineMeta,
+            helper.pipeline);
+
+    // Passed through from the filename rows: "filename" and a "tag" of their 
own. The file's "tag"
+    // comes after them and is renamed "tag_1" in the row; "rownum" is added 
by the transform.
+    RowMeta passedThrough = new RowMeta();
+    passedThrough.addValueMeta(new ValueMetaString("filename"));
+    passedThrough.addValueMeta(new ValueMetaString("tag"));
+    RowMeta output = new RowMeta();
+    output.mergeRowMeta(passedThrough, "File input");
+    output.addValueMeta(new ValueMetaInteger("tag"));
+    output.addValueMeta(new ValueMetaNumber("amount"));
+    output.addValueMeta(new ValueMetaInteger("rownum"));
+    data.outputRowMeta = output;
+    data.nrPassThruFields = passedThrough.size();
+
+    file = HopVfs.getFileObject("ram:///lineage/orders.csv");
+    try (OutputStream out = file.getContent().getOutputStream()) {
+      out.write("7,12.5\n".getBytes(StandardCharsets.UTF_8));
+    }
+  }
+
+  @AfterEach
+  void tearDown() throws Exception {
+    file.delete();
+    helper.cleanUp();
+  }
+
+  @Test
+  void theFileSchemaIsTheFilesOwnFieldsWithTheirTypes() {
+    LineageHub hub = mock(LineageHub.class);
+    when(hub.isEnabled()).thenReturn(true);
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      transform.emitLineageFileRead(file);
+      transform.emitLineageFileRead(file);
+    }
+
+    ArgumentCaptor<LineageEvent> events = 
ArgumentCaptor.forClass(LineageEvent.class);
+    verify(hub, times(2)).emit(events.capture());
+    FileIoContentSchema schema =
+        ((FileIoLineagePayload) 
events.getValue().getPayload()).getContentSchema();
+    List<FileIoTabularColumn> columns = schema.getColumns();
+    assertEquals(2, columns.size());
+    assertEquals("tag", columns.get(0).getName());
+    assertEquals("Integer", columns.get(0).getHopTypeName());
+    assertEquals("amount", columns.get(1).getName());
+    assertEquals("Number", columns.get(1).getHopTypeName());
+    assertEquals("csv", schema.getFormatHint());
+    // Worked out once, not on every file.
+    assertEquals(1, transform.fileFieldNamesCalls.get());
+  }
+
+  @Test
+  void nothingIsWorkedOutWithLineageOff() {
+    LineageHub hub = mock(LineageHub.class);
+    when(hub.isEnabled()).thenReturn(false);
+    try (MockedStatic<LineageHub> staticHub = 
Mockito.mockStatic(LineageHub.class)) {
+      staticHub.when(LineageHub::getInstance).thenReturn(hub);
+      transform.emitLineageFileRead(file);
+    }
+
+    verify(hub, never()).emit(any());
+    assertEquals(0, transform.fileFieldNamesCalls.get());
+  }
+}
diff --git 
a/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
 
b/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
index e8326cc085..4b41d94896 100644
--- 
a/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
+++ 
b/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
@@ -135,6 +135,7 @@ public final class OpenLineageEventMapper {
       case TRANSFORM_SCHEMA -> Optional.empty();
       case HTTP_IO -> Optional.empty();
       case PIPELINE_GRAPH -> Optional.empty();
+      default -> Optional.empty();
     };
   }
 
diff --git 
a/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
 
b/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
index 7eaaf125d2..643c866b68 100644
--- 
a/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
+++ 
b/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
@@ -623,4 +623,14 @@ public class TextFileInput extends 
BaseFileInputTransform<TextFileInputMeta, Tex
       }
     }
   }
+
+  /** The reader fills one output field per input field, right after the 
passed-through ones. */
+  @Override
+  protected List<String> fileFieldNames() {
+    List<String> names = new ArrayList<>();
+    for (TextFileInputField field : meta.getInputFields()) {
+      names.add(field.getName());
+    }
+    return names;
+  }
 }

Reply via email to