This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new ab71f635f5 Lineage: action start events, nested-run flushing and run
context for… (#9339)
ab71f635f5 is described below
commit ab71f635f5a39458c907a6ff472e0211d896ca9a
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Nov 12 09:04:55 2026 +0100
Lineage: action start events, nested-run flushing and run context for…
(#9339)
* Lineage: action start events, nested-run flushing and run context for
sinks, fixes #9322
* Address review comments on lineage action events and nested-run flushing,
#9322
---
.../hop-dev-manual/modules/ROOT/pages/lineage.adoc | 20 +-
.../java/org/apache/hop/core/HopEnvironment.java | 20 +-
.../hop/lineage/LineageActionStartListener.java | 132 ++++++++++
.../apache/hop/lineage/LineageFileIoEmitter.java | 33 +++
.../hop/lineage/LineageRelationalIoEmitter.java | 18 +-
.../hop/lineage/LineageRunLifecycleEmitter.java | 292 +++++++++++++++++++--
.../org/apache/hop/lineage/hub/LineageHub.java | 45 ++++
.../apache/hop/lineage/model/LineageEventKind.java | 8 +-
.../lineage/model/PipelineGraphLineagePayload.java | 4 +-
.../lineage/xp/LineageHubPipelineCompletedXp.java | 8 +-
.../xp/LineageHubWorkflowActionAfterXp.java | 2 +
.../xp/LineageHubWorkflowActionBeforeXp.java | 41 ---
.../hop/lineage/xp/LineageHubWorkflowFinishXp.java | 11 +-
.../hop/lineage/xp/LineageHubWorkflowStartXp.java | 11 +-
.../transforms/file/BaseFileInputTransform.java | 55 +++-
.../main/java/org/apache/hop/run/HopRunBase.java | 4 +-
.../hop/lineage/LineageActionLifecycleTest.java | 236 +++++++++++++++++
.../hop/lineage/LineageFileIoEmitterTest.java | 27 ++
.../hop/lineage/LineagePipelineGraphTest.java | 70 ++++-
.../hop/lineage/LineagePipelineParentTest.java | 241 +++++++++++++++++
.../hub/LineageHubConcurrentShutdownTest.java | 47 +++-
.../org/apache/hop/lineage/hub/LineageHubTest.java | 51 ++++
.../file/BaseFileInputTransformLineageTest.java | 183 +++++++++++++
.../openlineage/OpenLineageEventMapper.java | 1 +
.../transforms/fileinput/text/TextFileInput.java | 10 +
25 files changed, 1448 insertions(+), 122 deletions(-)
diff --git a/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
b/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
index 4f13aac749..95f0ed1c8d 100644
--- a/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
+++ b/docs/hop-dev-manual/modules/ROOT/pages/lineage.adoc
@@ -105,7 +105,7 @@ Each observation is a **`LineageEvent`**:
* **Event id** — Unique id (UUID by default).
* **Timestamp** — Milliseconds since epoch.
-* **`LineageEventKind`** — One of `RUN_LIFECYCLE`, `TRANSFORM_SCHEMA`,
`FILE_IO`, `RELATIONAL_IO`, `HTTP_IO`, `PIPELINE_GRAPH`. New kinds are only
ever appended: a sink must skip a kind it does not handle rather than fail the
batch.
+* **`LineageEventKind`** — One of `RUN_LIFECYCLE`, `TRANSFORM_SCHEMA`,
`FILE_IO`, `RELATIONAL_IO`, `HTTP_IO`, `PIPELINE_GRAPH`, `WORKFLOW_GRAPH`. New
kinds are only ever appended: a sink must skip a kind it does not handle rather
than fail the batch.
* **`LineageContext`** — Who/what produced the observation (see above).
* **`LineagePayload`** — Optional typed payload; may be null if the kind and
context are enough.
@@ -115,7 +115,7 @@ Payload types are plain Java types (strings, longs, enums)
so sinks can serializ
* **`RelationalIoLineagePayload`** — Relational/JDBC table access: operation
(`READ`, `WRITE`, `DELETE`, `EXEC`), the resolved **dataset namespace**, the
input/output **`RelationalTable`** list (catalog/schema/table), the raw SQL
when the engine cannot resolve the tables itself, an optional column schema,
the connection's default catalog and schema, optional per-column provenance
(**`RelationalWriteColumn`**: target column, stream field, origin transform),
and an optional **`RelationalLi [...]
* **`HttpLineagePayload`** — `CLIENT` vs `SERVER`, method, URL, status,
request/response bytes, duration, success, message.
* **`RunLifecycleLineagePayload`** — `STARTED` / `FINISHED` plus optional
detail text.
-* **`PipelineGraphLineagePayload`** — The pipeline's hop graph as a list of
`from` → `to` transform-name transitions: enabled hops with both ends named, in
pipeline order. Informational hops are included. A sink uses it to attribute
datasets to the transform that touched them and to propagate them along the
graph, instead of pairing all of a pipeline's inputs with all of its outputs.
+* **`PipelineGraphLineagePayload`** — The pipeline's hop graph as a list of
`from` → `to` transform-name transitions: enabled hops with both ends named, in
pipeline order. Informational hops are included. A sink uses it to attribute
datasets to the transform that touched them and to propagate them along the
graph, instead of pairing all of a pipeline's inputs with all of its outputs. A
pipeline without hops gets an empty list. The same payload carries a workflow's
hop graph (`WORKFLOW_GR [...]
* **`TransformSchemaLineagePayload`** — `INPUT` or `OUTPUT` plus a list of
field names, Hop type names, length, and precision (boundary schema; emitted
when a transform finishes). Fields usually come from runtime row metadata on
the transform or row sets. If input was never observed (e.g. _Abort_ when
`getRow()` immediately returns null), input is derived via
`PipelineMeta#getPrevTransformFields`. If output was never written (e.g.
_Detect empty stream_ with a non-empty stream), output is [...]
Convenience factory: `LineageEvent.of(kind, context, payload)` assigns id and
current time.
@@ -131,9 +131,9 @@ When enabled:
. The worker builds **batches** up to `HOP_LINEAGE_BATCH_MAX`, waiting up to
`HOP_LINEAGE_BATCH_LINGER_MS` for additional events before sending a partial
batch.
. Each batch is passed to **every** loaded sink via
`ILineageSink.accept(List<LineageEvent>)`. If one sink throws, others still
receive the same batch; failures are logged.
-**`flush()`** blocks until queued events (and coalesced flush markers) have
been processed. **`flushQuietly()`** catches and logs errors.
+**`flush()`** blocks until queued events (and coalesced flush markers) have
been processed. **`flushQuietly()`** catches and logs errors.
**`flushIfBacklogged()`** flushes only once the queue is at least half full: a
nested run calls it, so that a run per row does not drain the hub after every
row, but still waits for the sinks before the queue overflows and events are
dropped.
-**`shutdown()`** stops the worker and calls `shutdown()` on sinks. It is
invoked from `HopEnvironment.shutdown()` and `HopEnvironment.reset()`.
+**`shutdown()`** stops the worker and calls `shutdown()` on sinks. It is
invoked from `HopEnvironment.shutdown()` and `HopEnvironment.reset()`, and from
a JVM exit hook that the hub registers when it starts its worker. That hook
waits for a hop-run that is still stopping on a termination signal
(`holdShutdownOnExit`), so the end of the run is delivered too.
After `PluginRegistry.init()`, `HopEnvironment` calls
`LineageHub.getInstance().environmentReady()` so sink lists can be rebuilt when
the environment is re-initialized.
@@ -144,20 +144,20 @@ The queue is allocated lazily on the first `emit` from
the resolved configuratio
The `org.apache.hop.lineage.xp` package registers extension points that
**emit** `RUN_LIFECYCLE` events (via `LineageRunLifecycleEmitter`) and
**flush** the hub where needed:
-* **`PipelineStart`** — `LineageHubPipelineStartXp` — phase `STARTED`,
followed by one `PIPELINE_GRAPH` event with the same context when the pipeline
has enabled hops. `PipelineStart` fires after the transform threads have
started, so the run's transform `STARTED` events, and early I/O such as a Table
Input read, can arrive before the graph, possibly in an earlier batch. A sink
that uses the graph collects a run's events by the pipeline's log channel id
(transform events carry it in the [...]
+* **`PipelineStart`** — `LineageHubPipelineStartXp` — phase `STARTED`,
followed by one `PIPELINE_GRAPH` event with the same context, with no
transitions when the pipeline has no enabled hops. `PipelineStart` fires after
the transform threads have started, so the run's transform `STARTED` events,
and early I/O such as a Table Input read, can arrive before the graph, possibly
in an earlier batch. A sink that uses the graph collects a run's events by the
pipeline's log channel id (transform [...]
* **`PipelineFinish`** — `LineageHubPipelineFinishXp` — phase `FINISHING`
(transforms stopped, cleanup)
-* **`PipelineCompleted`** — `LineageHubPipelineCompletedXp` — phase `FINISHED`
or `FAILED` (from pipeline error count), then `flushQuietly()`
-* **`WorkflowStart`** — `LineageHubWorkflowStartXp` — phase `STARTED`
-* **`WorkflowFinish`** — `LineageHubWorkflowFinishXp` — phase `FINISHED` or
`FAILED` (from workflow/result errors), then `flushQuietly()`
+* **`PipelineCompleted`** — `LineageHubPipelineCompletedXp` — phase `FINISHED`
or `FAILED` (from pipeline error count), then `flushQuietly()` for a top-level
run and `flushIfBacklogged()` for a nested one
+* **`WorkflowStart`** — `LineageHubWorkflowStartXp` — phase `STARTED`,
followed by one `WORKFLOW_GRAPH` event with the same context (empty when the
workflow has no enabled hops)
+* **`WorkflowFinish`** — `LineageHubWorkflowFinishXp` — first action `FAILED`
for every action run that started but never ended (its `execute()` threw), then
the workflow's phase `FINISHED` or `FAILED` (from workflow/result errors), then
`flushQuietly()` for a top-level run and `flushIfBacklogged()` for a nested one
* **`HopServerShutdown`** — `LineageHubServerShutdownFlushXp` — flush only
* **`TransformBeforeStart`** — `LineageHubTransformStartXp` — transform phase
`STARTED` (local pipeline engine; transform log channel + pipeline correlation
attributes)
* **`TransformFinished`** — `LineageHubTransformFinishXp` — transform
`FINISHED` or `FAILED` from transform error count, then `TRANSFORM_SCHEMA`
events for observed input/output row shapes when non-empty
-* **`WorkflowBeforeActionExecution`** — `LineageHubWorkflowActionBeforeXp` —
action `STARTED` (workflow log channel + action name; template action log
channel id as attribute when present)
+* **Action start** — `LineageActionStartListener`, an `IActionListener` that
`LineageHubWorkflowStartXp` adds to each workflow — action `STARTED` from
`beforeExecution`, once the workflow has cloned the action, so the event
carries the executing action's log channel id. `WorkflowBeforeActionExecution`
fires before that clone exists. An error while emitting is logged and does not
fail the action.
-* **`WorkflowAfterActionExecution`** — `LineageHubWorkflowActionAfterXp` —
action `FINISHED` or `FAILED` from the action `Result`; uses the executed
action’s log channel id when available
(`WorkflowExecutionExtension.actionLogChannelId` / `actionExecutionResult`)
+* **`WorkflowAfterActionExecution`** — `LineageHubWorkflowActionAfterXp` —
action `FINISHED` or `FAILED` from the action `Result`, under the executed
action's log channel id (`WorkflowExecutionExtension.actionLogChannelId`). An
action the workflow skipped has no run of its own and emits nothing. The
workflow does not reach this extension point when the action's `execute()`
throws; `WorkflowFinish` ends that run instead.
Remote (`RemotePipelineEngine`) and Beam (`BeamPipelineEngine`) engines also
call **`PipelineStart`** and **`PipelineFinish`** so they align with the local
pipeline engine.
diff --git a/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
b/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
index 02a8e6e58c..081840964e 100644
--- a/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
+++ b/engine/src/main/java/org/apache/hop/core/HopEnvironment.java
@@ -238,24 +238,8 @@ public class HopEnvironment {
}
}
- /**
- * Alert all Lifecycle plugins that the Hop environment is being initialized.
- *
- * @throws HopException when a lifecycle listener throws an exception
- */
- private static void initLifecycleListeners() {
- // Register a shutdown hook to invoke the listener's onExit() methods
- Runtime.getRuntime()
- .addShutdownHook(
- new Thread() {
- @Override
- public void run() {
- // waits for a hop-run that is still stopping, so that the end
of the run is
- // delivered too
- LineageHub.getInstance().shutdownOnExit();
- }
- });
- }
+ // The JVM exit hook that shuts the lineage hub down is registered by the
hub itself, once it has
+ // a worker; see LineageHub#shutdownOnExit.
// Shutdown the Hop environment programmatically
public static void shutdown() {
diff --git
a/engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java
b/engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java
new file mode 100644
index 0000000000..261a5747dc
--- /dev/null
+++
b/engine/src/main/java/org/apache/hop/lineage/LineageActionStartListener.java
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.util.Utils;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.action.IAction;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+
+/**
+ * Emits an action's {@code STARTED} lineage event once the workflow has
cloned the action, so the
+ * event carries the run id of the action that actually executes. One listener
per workflow, added
+ * by {@link #register(IWorkflowEngine)}.
+ *
+ * <p>The listener remembers the action runs it started. An action whose
{@code execute()} throws
+ * never reaches the {@code WorkflowAfterActionExecution} extension point that
emits its {@code
+ * FINISHED} or {@code FAILED}, so {@link
#failUnfinishedActions(IWorkflowEngine)} closes those runs
+ * as {@code FAILED} when the workflow ends.
+ */
+public final class LineageActionStartListener implements
IActionListener<WorkflowMeta> {
+
+ /** The action runs that started and have no terminal event yet, by run id.
*/
+ private final Map<String, IAction> unfinishedActions = new
ConcurrentHashMap<>();
+
+ private LineageActionStartListener() {}
+
+ /** Adds the listener to a workflow, unless it is already there (a workflow
can run again). */
+ @SuppressWarnings("unchecked")
+ public static void register(IWorkflowEngine<WorkflowMeta> workflow) {
+ if (workflow == null || workflow.getActionListeners() == null ||
find(workflow) != null) {
+ return;
+ }
+ workflow.addActionListener(new LineageActionStartListener());
+ }
+
+ /** The action run ended with a {@code FINISHED} or {@code FAILED} event. */
+ public static void actionEnded(IWorkflowEngine<WorkflowMeta> workflow,
String actionRunId) {
+ LineageActionStartListener listener = find(workflow);
+ if (listener != null && !Utils.isEmpty(actionRunId)) {
+ listener.unfinishedActions.remove(actionRunId);
+ }
+ }
+
+ /**
+ * Emits {@code FAILED} for every action run of this workflow that started
but never ended: its
+ * {@code execute()} threw, so the workflow skipped the extension point that
ends it. Without this
+ * a sink keeps those runs open forever.
+ */
+ public static void failUnfinishedActions(IWorkflowEngine<WorkflowMeta>
workflow) {
+ LineageActionStartListener listener = find(workflow);
+ if (listener == null || listener.unfinishedActions.isEmpty()) {
+ return;
+ }
+ List<IAction> unfinished = new
ArrayList<>(listener.unfinishedActions.values());
+ listener.unfinishedActions.clear();
+ for (IAction action : unfinished) {
+ LineageRunLifecycleEmitter.emitActionFailedWithoutResult(workflow,
action);
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private static LineageActionStartListener find(IWorkflowEngine<WorkflowMeta>
workflow) {
+ List<IActionListener> listeners = workflow != null ?
workflow.getActionListeners() : null;
+ if (listeners != null) {
+ for (IActionListener listener : listeners) {
+ if (listener instanceof LineageActionStartListener lineageListener) {
+ return lineageListener;
+ }
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Runs inside the workflow's action execution, before the action's own
error handling: a lineage
+ * problem is logged and may never fail the action.
+ */
+ @Override
+ public void beforeExecution(
+ IWorkflowEngine<WorkflowMeta> workflow, ActionMeta actionMeta, IAction
action) {
+ try {
+ LineageRunLifecycleEmitter.emitActionStarted(workflow, action);
+ ILogChannel actionLog = action != null ? action.getLogChannel() : null;
+ String runId = actionLog != null ? actionLog.getLogChannelId() : null;
+ if (!Utils.isEmpty(runId)) {
+ unfinishedActions.put(runId, action);
+ }
+ } catch (RuntimeException e) {
+ ILogChannel log = workflow != null ? workflow.getLogChannel() : null;
+ if (log != null) {
+ log.logError(
+ "Unable to emit the lineage start event of action "
+ + (actionMeta != null ? actionMeta.getName() : null),
+ e);
+ }
+ }
+ }
+
+ @Override
+ public void afterExecution(
+ IWorkflowEngine<WorkflowMeta> workflow,
+ ActionMeta actionMeta,
+ IAction action,
+ Result result) {
+ // FINISHED and FAILED come from the WorkflowAfterActionExecution
extension point, which the
+ // workflow reaches whenever the action returned a result. An action that
threw is closed by
+ // failUnfinishedActions() at the end of the workflow.
+ }
+}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
b/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
index d9d45b8588..6656071fb6 100644
--- a/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
+++ b/engine/src/main/java/org/apache/hop/lineage/LineageFileIoEmitter.java
@@ -17,13 +17,18 @@
package org.apache.hop.lineage;
+import java.util.ArrayList;
+import java.util.List;
import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.util.Utils;
import org.apache.hop.lineage.context.LineageContext;
import org.apache.hop.lineage.hub.LineageHub;
import org.apache.hop.lineage.model.FileIoContentSchema;
import org.apache.hop.lineage.model.FileIoLineagePayload;
import org.apache.hop.lineage.model.FileIoOperation;
+import org.apache.hop.lineage.model.FileIoTabularColumn;
import org.apache.hop.lineage.model.LineageEvent;
import org.apache.hop.lineage.model.LineageEventKind;
import org.apache.hop.pipeline.transform.ITransform;
@@ -47,6 +52,34 @@ public final class LineageFileIoEmitter {
return LineageHub.getInstance().isEnabled();
}
+ /**
+ * Converts a Hop {@link IRowMeta} to the neutral {@link
FileIoContentSchema} a file event
+ * carries, so a sink can describe the file's columns (name and Hop type).
Returns {@code null}
+ * for a null/empty row shape. Columns have no path locator and no structure
tree here: a reader
+ * that knows a hierarchy derives its own.
+ *
+ * @param rowMeta the row shape read from or written to the file
+ * @param formatHint short format name ({@code csv}, {@code json}, …), or
null when unknown
+ */
+ public static FileIoContentSchema toContentSchema(IRowMeta rowMeta, String
formatHint) {
+ if (rowMeta == null || rowMeta.isEmpty()) {
+ return null;
+ }
+ List<FileIoTabularColumn> columns = new ArrayList<>(rowMeta.size());
+ for (IValueMeta valueMeta : rowMeta.getValueMetaList()) {
+ columns.add(
+ new FileIoTabularColumn(
+ valueMeta.getName(),
+ valueMeta.getTypeDesc(),
+ valueMeta.getLength(),
+ valueMeta.getPrecision(),
+ null,
+ null,
+ false));
+ }
+ return new FileIoContentSchema(formatHint == null ? "" : formatHint,
columns, List.of());
+ }
+
/**
* Records a file/object-store operation performed by a workflow action
during {@link
* IAction#execute}.
diff --git
a/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
b/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
index 8646f1583e..d1baa44224 100644
---
a/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
+++
b/engine/src/main/java/org/apache/hop/lineage/LineageRelationalIoEmitter.java
@@ -28,7 +28,6 @@ import org.apache.hop.core.util.Utils;
import org.apache.hop.lineage.context.LineageContext;
import org.apache.hop.lineage.hub.LineageHub;
import org.apache.hop.lineage.model.FileIoContentSchema;
-import org.apache.hop.lineage.model.FileIoTabularColumn;
import org.apache.hop.lineage.model.LineageEvent;
import org.apache.hop.lineage.model.LineageEventKind;
import org.apache.hop.lineage.model.RelationalIoLineagePayload;
@@ -462,22 +461,7 @@ public final class LineageRelationalIoEmitter {
* locator, so the structure-roots tree is empty.
*/
static FileIoContentSchema toContentSchema(IRowMeta rowMeta) {
- if (rowMeta == null || rowMeta.isEmpty()) {
- return null;
- }
- List<FileIoTabularColumn> columns = new ArrayList<>(rowMeta.size());
- for (IValueMeta valueMeta : rowMeta.getValueMetaList()) {
- columns.add(
- new FileIoTabularColumn(
- valueMeta.getName(),
- valueMeta.getTypeDesc(),
- valueMeta.getLength(),
- valueMeta.getPrecision(),
- null,
- null,
- false));
- }
- return new FileIoContentSchema(JDBC_FORMAT_HINT, columns, List.of());
+ return LineageFileIoEmitter.toContentSchema(rowMeta, JDBC_FORMAT_HINT);
}
/**
diff --git
a/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
b/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
index 53d1669793..010b557c91 100644
---
a/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
+++
b/engine/src/main/java/org/apache/hop/lineage/LineageRunLifecycleEmitter.java
@@ -40,6 +40,7 @@ import org.apache.hop.pipeline.engine.IPipelineEngine;
import org.apache.hop.pipeline.transform.ITransform;
import org.apache.hop.pipeline.transform.TransformMetaDataCombi;
import org.apache.hop.workflow.WorkflowExecutionExtension;
+import org.apache.hop.workflow.WorkflowHopMeta;
import org.apache.hop.workflow.WorkflowMeta;
import org.apache.hop.workflow.action.ActionMeta;
import org.apache.hop.workflow.action.IAction;
@@ -55,8 +56,164 @@ public final class LineageRunLifecycleEmitter {
public static final String FAILED = "failed";
public static final String SUCCESS = "success";
+ /**
+ * The run that started this pipeline or workflow, when the engine knows it:
the workflow action,
+ * or the transform of a parent pipeline (Pipeline Executor, mapping), that
ran it. {@code
+ * parentRunId} is that subject's log-channel id and {@code parentJobName}
is the job name the
+ * lineage mapper gives it (e.g. {@code my-workflow/Run orders} or {@code
my-pipeline/Run per
+ * row}); both are needed to form a ParentRunFacet. Sinks that write the job
tree themselves
+ * (DataHub) read the same pair.
+ */
+ public static final String ATTR_PARENT_RUN_ID = "parentRunId";
+
+ public static final String ATTR_PARENT_JOB_NAME = "parentJobName";
+
+ /**
+ * The parts {@link #ATTR_PARENT_JOB_NAME} is made of, for a sink that
models workflows and
+ * pipelines apart from the jobs inside them (DataHub): the workflow and
action that ran this run,
+ * or the pipeline and transform (Pipeline Executor, mapping) that did. A
job name cannot be split
+ * back reliably, since names may contain the separator.
+ */
+ public static final String ATTR_PARENT_WORKFLOW_NAME = "parentWorkflowName";
+
+ public static final String ATTR_PARENT_ACTION_NAME = "parentActionName";
+ public static final String ATTR_PARENT_PIPELINE_NAME = "parentPipelineName";
+ public static final String ATTR_PARENT_TRANSFORM_NAME =
"parentTransformName";
+
+ /** The description the subject's metadata gives it (pipeline, workflow,
transform, action). */
+ public static final String ATTR_DESCRIPTION = "description";
+
+ private static void fillDescription(LineageContext.Builder ctx, String
description) {
+ if (!Utils.isEmpty(description)) {
+ ctx.putAttribute(ATTR_DESCRIPTION, description);
+ }
+ }
+
private LineageRunLifecycleEmitter() {}
+ /**
+ * Records who started this run, when the engine knows: an action that ran a
pipeline or workflow,
+ * or a parent pipeline that ran a sub-pipeline. Left out entirely for a
top-level run.
+ */
+ private static void fillParentFields(
+ LineageContext.Builder ctx, String parentJobName, String parentRunId) {
+ if (!Utils.isEmpty(parentJobName)) {
+ ctx.putAttribute(ATTR_PARENT_JOB_NAME, parentJobName);
+ }
+ if (!Utils.isEmpty(parentRunId)) {
+ ctx.putAttribute(ATTR_PARENT_RUN_ID, parentRunId);
+ }
+ }
+
+ /**
+ * The run that started a pipeline: its sub-pipeline parent, or the workflow
action that ran it
+ * (with the workflow that owns the action). Package-private so the naming
rule can be checked
+ * without starting an engine.
+ */
+ static void fillPipelineParentFields(
+ LineageContext.Builder ctx, IPipelineEngine<PipelineMeta> pipeline) {
+ IPipelineEngine<PipelineMeta> parentPipeline =
pipeline.getParentPipeline();
+ if (parentPipeline != null) {
+ fillPipelineParentFields(ctx, parentPipeline, pipeline.getParent());
+ return;
+ }
+ if (pipeline.getParent() instanceof IAction parentAction) {
+ fillActionParentFields(ctx, parentAction, pipeline.getParentWorkflow());
+ }
+ }
+
+ /**
+ * A run started from a pipeline. When the transform that started it
(Pipeline Executor, mapping,
+ * Workflow Executor) is its logging parent, the parent is that transform's
run, named {@code
+ * <pipeline>/<transform>}: the same shape as a run started by a workflow
action. Only when the
+ * transform is unknown does the parent pipeline's own run stand in.
+ */
+ private static void fillPipelineParentFields(
+ LineageContext.Builder ctx, IPipelineEngine<PipelineMeta>
parentPipeline, Object parent) {
+ PipelineMeta parentMeta = parentPipeline.getPipelineMeta();
+ String parentPipelineName = parentMeta != null ? parentMeta.getName() :
null;
+ if (Utils.isEmpty(parentPipelineName)) {
+ fillParentFields(ctx, null, parentPipeline.getLogChannelId());
+ return;
+ }
+ ctx.putAttribute(ATTR_PARENT_PIPELINE_NAME, parentPipelineName);
+ if (parent instanceof ITransform transform &&
!Utils.isEmpty(transform.getTransformName())) {
+ ctx.putAttribute(ATTR_PARENT_TRANSFORM_NAME,
transform.getTransformName());
+ ILogChannel transformLog = transform.getLogChannel();
+ String transformRunId = transformLog != null ?
transformLog.getLogChannelId() : null;
+ if (!Utils.isEmpty(transformRunId)) {
+ fillParentFields(
+ ctx, parentPipelineName + "/" + transform.getTransformName(),
transformRunId);
+ return;
+ }
+ }
+ fillParentFields(ctx, parentPipelineName,
parentPipeline.getLogChannelId());
+ }
+
+ /**
+ * The run that started a workflow: the transform that ran it (Workflow
Executor), the workflow
+ * action that ran it (with the workflow that owns the action), or failing
that the parent
+ * workflow itself.
+ */
+ static void fillWorkflowParentFields(
+ LineageContext.Builder ctx, IWorkflowEngine<WorkflowMeta> workflow) {
+ IPipelineEngine<PipelineMeta> parentPipeline =
workflow.getParentPipeline();
+ if (parentPipeline != null) {
+ fillPipelineParentFields(ctx, parentPipeline, workflow.getParent());
+ return;
+ }
+ IWorkflowEngine<WorkflowMeta> parentWorkflow =
workflow.getParentWorkflow();
+ if (workflow.getParent() instanceof IAction parentAction) {
+ fillActionParentFields(ctx, parentAction, parentWorkflow);
+ return;
+ }
+ WorkflowMeta parentMeta = parentWorkflow != null ?
parentWorkflow.getWorkflowMeta() : null;
+ if (parentMeta != null) {
+ fillParentFields(ctx, parentMeta.getName(),
parentWorkflow.getLogChannelId());
+ }
+ }
+
+ /**
+ * Whether a pipeline is a top-level run: not a sub-pipeline, not started by
a workflow action.
+ * Only a top-level run always waits for the lineage hub to drain when it
completes. A nested run
+ * (a Pipeline Executor row, a mapping, a Pipeline action) waits only once
the queue is filling up
+ * ({@link LineageHub#flushIfBacklogged()}): a full drain after every nested
run would sit on the
+ * path of every caller waiting for it, and no drain at all would let the
queue overflow.
+ */
+ public static boolean isTopLevel(IPipelineEngine<PipelineMeta> pipeline) {
+ return pipeline != null
+ && pipeline.getParentPipeline() == null
+ && pipeline.getParentWorkflow() == null
+ && !(pipeline.getParent() instanceof IAction);
+ }
+
+ /** Whether a workflow is a top-level run; see {@link
#isTopLevel(IPipelineEngine)}. */
+ public static boolean isTopLevel(IWorkflowEngine<WorkflowMeta> workflow) {
+ return workflow != null
+ && workflow.getParentPipeline() == null
+ && workflow.getParentWorkflow() == null
+ && !(workflow.getParent() instanceof IAction);
+ }
+
+ /** An action is named {@code <workflow>/<action>}, the convention the
lineage mapper uses. */
+ private static void fillActionParentFields(
+ LineageContext.Builder ctx,
+ IAction parentAction,
+ IWorkflowEngine<WorkflowMeta> parentWorkflow) {
+ WorkflowMeta parentMeta = parentWorkflow != null ?
parentWorkflow.getWorkflowMeta() : null;
+ String parentWorkflowName = parentMeta != null ? parentMeta.getName() :
null;
+ String parentJobName =
+ !Utils.isEmpty(parentWorkflowName) &&
!Utils.isEmpty(parentAction.getName())
+ ? parentWorkflowName + "/" + parentAction.getName()
+ : null;
+ ILogChannel parentLog = parentAction.getLogChannel();
+ fillParentFields(ctx, parentJobName, parentLog != null ?
parentLog.getLogChannelId() : null);
+ if (parentJobName != null) {
+ ctx.putAttribute(ATTR_PARENT_WORKFLOW_NAME, parentWorkflowName);
+ ctx.putAttribute(ATTR_PARENT_ACTION_NAME, parentAction.getName());
+ }
+ }
+
private static void fillFilenameFields(
LineageContext.Builder ctx, String filename, IVariables variables) {
ctx.hopFilename(filename);
@@ -135,6 +292,7 @@ public final class LineageRunLifecycleEmitter {
if (!Utils.isEmpty(action.getPluginId())) {
ctx.putAttribute("actionPluginId", action.getPluginId());
}
+ fillDescription(ctx, action.getDescription());
return ctx;
}
@@ -147,6 +305,9 @@ public final class LineageRunLifecycleEmitter {
ctx.transformName(combi.transformName);
}
ctx.copyNr(String.valueOf(combi.copy));
+ if (combi.transformMeta != null) {
+ fillDescription(ctx, combi.transformMeta.getDescription());
+ }
return ctx;
}
@@ -168,6 +329,13 @@ public final class LineageRunLifecycleEmitter {
if (!Utils.isEmpty(pluginId)) {
ctx.putAttribute("pipelineEnginePluginId", pluginId);
}
+ // A pipeline run by a workflow action knows the action (its logging
parent) and the workflow
+ // that owns it; a sub-pipeline knows its parent pipeline and the
transform that ran it. Either
+ // way the parent is that action's or transform's run, named <owner>/<job>
as the mapper does.
+ fillPipelineParentFields(ctx, pipeline);
+ if (meta != null) {
+ fillDescription(ctx, meta.getDescription());
+ }
ctx.putAttribute("errors", String.valueOf(pipeline.getErrors()));
if (phase == RunLifecyclePhase.FINISHED || phase ==
RunLifecyclePhase.FAILED) {
ctx.putAttribute(OUTCOME, phase == RunLifecyclePhase.FAILED ? FAILED :
SUCCESS);
@@ -182,18 +350,16 @@ public final class LineageRunLifecycleEmitter {
// The hop graph is a fact about the pipeline, not about a phase: emit it
once, next to the
// STARTED event. Sinks that attribute I/O to individual transforms need
it to know which
- // transform fed which; see PipelineGraphLineagePayload.
+ // transform fed which; see PipelineGraphLineagePayload. A pipeline
without hops gets an empty
+ // graph: without one a sink cannot tell it from an engine that sends no
graph at all.
//
if (phase == RunLifecyclePhase.STARTED && meta != null) {
- List<Transition> transitions = pipelineTransitions(meta);
- if (!transitions.isEmpty()) {
- LineageHub.getInstance()
- .emit(
- LineageEvent.of(
- LineageEventKind.PIPELINE_GRAPH,
- context,
- new PipelineGraphLineagePayload(transitions)));
- }
+ LineageHub.getInstance()
+ .emit(
+ LineageEvent.of(
+ LineageEventKind.PIPELINE_GRAPH,
+ context,
+ new PipelineGraphLineagePayload(pipelineTransitions(meta))));
}
}
@@ -237,6 +403,10 @@ public final class LineageRunLifecycleEmitter {
.logChannelId(workflow.getLogChannelId())
.workflowName(name);
fillFilenameFields(ctx, filename, workflow);
+ fillWorkflowParentFields(ctx, workflow);
+ if (meta != null) {
+ fillDescription(ctx, meta.getDescription());
+ }
Result result = workflow.getResult();
ctx.putAttribute("errors", String.valueOf(result != null ?
result.getNrErrors() : 0));
if (result != null) {
@@ -246,12 +416,50 @@ public final class LineageRunLifecycleEmitter {
if (phase == RunLifecyclePhase.FINISHED || phase ==
RunLifecyclePhase.FAILED) {
ctx.putAttribute(OUTCOME, phase == RunLifecyclePhase.FAILED ? FAILED :
SUCCESS);
}
+ LineageContext context = ctx.build();
LineageHub.getInstance()
.emit(
LineageEvent.of(
LineageEventKind.RUN_LIFECYCLE,
- ctx.build(),
+ context,
new RunLifecycleLineagePayload(phase, detail)));
+
+ // Like a pipeline's, the hop graph is a fact about the workflow: emitted
once, next to STARTED,
+ // and empty when the workflow has no hops.
+ if (phase == RunLifecyclePhase.STARTED && meta != null) {
+ LineageHub.getInstance()
+ .emit(
+ LineageEvent.of(
+ LineageEventKind.WORKFLOW_GRAPH,
+ context,
+ new PipelineGraphLineagePayload(workflowTransitions(meta))));
+ }
+ }
+
+ /**
+ * The enabled hops of a workflow, in workflow order, as {@code from -> to}
transitions between
+ * action names. Every hop counts, whatever its condition: each is a way the
workflow can go.
+ */
+ public static List<Transition> workflowTransitions(WorkflowMeta meta) {
+ if (meta == null) {
+ return List.of();
+ }
+ List<Transition> transitions = new ArrayList<>();
+ for (int i = 0; i < meta.nrWorkflowHops(); i++) {
+ WorkflowHopMeta hop = meta.getWorkflowHop(i);
+ if (hop == null
+ || !hop.isEnabled()
+ || hop.getFromAction() == null
+ || hop.getToAction() == null) {
+ continue;
+ }
+ String from = hop.getFromAction().getName();
+ String to = hop.getToAction().getName();
+ if (!Utils.isEmpty(from) && !Utils.isEmpty(to)) {
+ transitions.add(new Transition(from, to));
+ }
+ }
+ return transitions;
}
/** Terminal failure for a workflow at {@code WorkflowFinish} time. */
@@ -294,9 +502,55 @@ public final class LineageRunLifecycleEmitter {
return combi != null && combi.transform != null &&
combi.transform.getErrors() > 0;
}
+ /**
+ * Action {@code STARTED}, emitted by {@link LineageActionStartListener}
once the workflow has
+ * cloned the action: the clone's log channel is the action run's identity,
the same one its
+ * {@code FINISHED} event, its I/O events and any pipeline it starts refer
to. The {@code
+ * WorkflowBeforeActionExecution} extension point fires before that clone
exists, which is why the
+ * start is not emitted from there.
+ */
+ public static void emitActionStarted(IWorkflowEngine<WorkflowMeta> wf,
IAction action) {
+ if (wf == null || action == null) {
+ return;
+ }
+ LineageContext.Builder ctx = actionContextBuilder(wf, action);
+ LineageHub.getInstance()
+ .emit(
+ LineageEvent.of(
+ LineageEventKind.RUN_LIFECYCLE,
+ ctx.build(),
+ new RunLifecycleLineagePayload(RunLifecyclePhase.STARTED,
null)));
+ }
+
+ /**
+ * Action {@code FAILED} for a run that started but whose {@code execute()}
threw: the workflow
+ * then skips the extension point that ends it. See {@link
+ * LineageActionStartListener#failUnfinishedActions(IWorkflowEngine)}.
+ */
+ static void emitActionFailedWithoutResult(IWorkflowEngine<WorkflowMeta> wf,
IAction action) {
+ LineageContext.Builder ctx = actionContextBuilder(wf, action);
+ if (ctx == null) {
+ return;
+ }
+ ctx.putAttribute(OUTCOME, FAILED);
+ LineageHub.getInstance()
+ .emit(
+ LineageEvent.of(
+ LineageEventKind.RUN_LIFECYCLE,
+ ctx.build(),
+ new RunLifecycleLineagePayload(
+ RunLifecyclePhase.FAILED, "The action ended without a
result")));
+ }
+
+ /**
+ * Action {@code FINISHED} or {@code FAILED}. An action the workflow skipped
(an extension point
+ * set {@code executeAction} to false) never ran and has no run of its own,
so it emits nothing.
+ */
public static void emitWorkflowAction(
WorkflowExecutionExtension extension, RunLifecyclePhase phase, String
detail) {
- if (extension == null || extension.workflow == null) {
+ if (extension == null
+ || extension.workflow == null
+ || Utils.isEmpty(extension.actionLogChannelId)) {
return;
}
IWorkflowEngine<WorkflowMeta> wf = extension.workflow;
@@ -309,26 +563,18 @@ public final class LineageRunLifecycleEmitter {
LineageContext.Builder ctx =
LineageContext.builder()
.subjectType(LineageSubjectType.ACTION)
+ .logChannelId(extension.actionLogChannelId)
.workflowName(workflowName)
.actionName(actionName);
fillFilenameFields(ctx, filename, wf);
ctx.putAttribute("workflowLogChannelId", wf.getLogChannelId());
- ctx.putAttribute("executeAction", String.valueOf(extension.executeAction));
if (am != null && am.getAction() != null && am.getAction().getPluginId()
!= null) {
ctx.putAttribute("actionPluginId", am.getAction().getPluginId());
}
-
- if (phase == RunLifecyclePhase.STARTED) {
- ctx.logChannelId(wf.getLogChannelId());
- if (am != null && am.getAction() != null) {
- ctx.putAttribute(
- "actionTemplateLogChannelId",
am.getAction().getLogChannel().getLogChannelId());
- }
- } else {
- String actionLogId = extension.actionLogChannelId;
- ctx.logChannelId(!Utils.isEmpty(actionLogId) ? actionLogId :
wf.getLogChannelId());
+ if (am != null) {
+ fillDescription(ctx, am.getDescription());
}
if (extension.actionExecutionResult != null) {
diff --git a/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
b/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
index 07133b3b85..8ad95039ec 100644
--- a/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
+++ b/engine/src/main/java/org/apache/hop/lineage/hub/LineageHub.java
@@ -30,6 +30,7 @@ import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.function.Consumer;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopPluginException;
import org.apache.hop.core.logging.ILogChannel;
@@ -62,6 +63,14 @@ public final class LineageHub {
private volatile BlockingQueue<Object> queue;
private final AtomicLong droppedEvents = new AtomicLong();
+
+ /**
+ * Whether the JVM exit hook is in place. A sink delivers asynchronously, so
a process that exits
+ * right after its last run (hop-run calls System.exit) would lose whatever
the sink still holds,
+ * typically the runs' terminal events, unless the hub shuts its sinks down
on the way out.
+ */
+ private final AtomicBoolean exitHookRegistered = new AtomicBoolean(false);
+
private final AtomicBoolean running = new AtomicBoolean(false);
/**
@@ -185,6 +194,23 @@ public final class LineageHub {
}
}
+ /**
+ * {@link #flushQuietly()}, but only once the queue is at least half full. A
nested run (a
+ * Pipeline Executor running a pipeline per row, say) calls this when it
ends: it does not wait
+ * for the sinks while they keep up, and waits for them before the queue
fills up and events get
+ * dropped.
+ */
+ public void flushIfBacklogged() {
+ BlockingQueue<Object> q = queue;
+ if (q == null || !running.get()) {
+ return;
+ }
+ int size = q.size();
+ if (size > 0 && size >= q.remainingCapacity()) {
+ flushQuietly();
+ }
+ }
+
/**
* Stop the worker thread and shut down sinks. Calls are serialized: a
second call waits until the
* first one has delivered the remaining events, instead of shutting the
sinks down under it. An
@@ -307,6 +333,25 @@ public final class LineageHub {
thread.setDaemon(true);
worker = thread;
thread.start();
+ if (this == INSTANCE) {
+ registerExitHook(Runtime.getRuntime()::addShutdownHook);
+ }
+ }
+ }
+
+ /**
+ * Register the JVM exit hook, once. It goes through {@link
#shutdownOnExit()}: the JVM starts
+ * every shutdown hook at once, and hop-run may still be stopping its run on
a termination signal.
+ */
+ void registerExitHook(Consumer<Thread> addShutdownHook) {
+ if (!exitHookRegistered.compareAndSet(false, true)) {
+ return;
+ }
+ try {
+ addShutdownHook.accept(new Thread(this::shutdownOnExit,
"Hop-LineageHub-Exit"));
+ } catch (IllegalStateException e) {
+ // The JVM is already exiting. Too late for a hook, but the event must
still be queued.
+ log.logDebug("The JVM is exiting, no lineage exit hook registered");
}
}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
b/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
index a81905eade..90a9284cd3 100644
--- a/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
+++ b/engine/src/main/java/org/apache/hop/lineage/model/LineageEventKind.java
@@ -37,5 +37,11 @@ public enum LineageEventKind {
* PipelineGraphLineagePayload}). Sinks need it to attribute a dataset to
the transform that
* touched it rather than to the whole pipeline.
*/
- PIPELINE_GRAPH
+ PIPELINE_GRAPH,
+ /**
+ * The workflow's hop graph, emitted once per run next to its start
(payload: {@link
+ * PipelineGraphLineagePayload}, its transitions naming actions). Sinks use
it to say which action
+ * follows which.
+ */
+ WORKFLOW_GRAPH
}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
b/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
index 13a837fb75..b739d89a7d 100644
---
a/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
+++
b/engine/src/main/java/org/apache/hop/lineage/model/PipelineGraphLineagePayload.java
@@ -31,7 +31,9 @@ import lombok.Getter;
* (its own reads plus what its upstream transforms produced) and its own
outputs.
*
* <p>Only enabled hops with two known ends are listed, in pipeline order.
Informational hops are
- * included: data flows through them, even though they do no I/O of their own.
+ * included: data flows through them, even though they do no I/O of their own.
A pipeline without
+ * such hops still gets the event, with no transitions, so a sink can tell it
from an engine that
+ * sends no graph.
*
* <p>The graph is emitted with the pipeline's {@code STARTED} event, which
fires after the
* transform threads have started. A transform's own events, including early
I/O, can therefore
diff --git
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
index de1e93201f..c17696df0a 100644
---
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
+++
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubPipelineCompletedXp.java
@@ -44,6 +44,12 @@ public class LineageHubPipelineCompletedXp
? RunLifecyclePhase.FAILED
: RunLifecyclePhase.FINISHED;
LineageRunLifecycleEmitter.emitPipeline(pipeline, phase, null);
- LineageHub.getInstance().flushQuietly();
+ // A top-level run waits for delivery. A nested run only waits once the
queue fills up: that
+ // keeps a run per row from dropping events, without a full drain after
every row.
+ if (LineageRunLifecycleEmitter.isTopLevel(pipeline)) {
+ LineageHub.getInstance().flushQuietly();
+ } else {
+ LineageHub.getInstance().flushIfBacklogged();
+ }
}
}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
index 1283e485c3..86af8bd70a 100644
---
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
+++
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionAfterXp.java
@@ -22,6 +22,7 @@ import org.apache.hop.core.extension.ExtensionPoint;
import org.apache.hop.core.extension.IExtensionPoint;
import org.apache.hop.core.logging.ILogChannel;
import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.lineage.LineageActionStartListener;
import org.apache.hop.lineage.LineageRunLifecycleEmitter;
import org.apache.hop.lineage.model.RunLifecyclePhase;
import org.apache.hop.workflow.WorkflowExecutionExtension;
@@ -41,5 +42,6 @@ public class LineageHubWorkflowActionAfterXp
? RunLifecyclePhase.FAILED
: RunLifecyclePhase.FINISHED;
LineageRunLifecycleEmitter.emitWorkflowAction(extension, phase, null);
+ LineageActionStartListener.actionEnded(extension.workflow,
extension.actionLogChannelId);
}
}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionBeforeXp.java
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionBeforeXp.java
deleted file mode 100644
index 833ccf9d02..0000000000
---
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowActionBeforeXp.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hop.lineage.xp;
-
-import org.apache.hop.core.exception.HopException;
-import org.apache.hop.core.extension.ExtensionPoint;
-import org.apache.hop.core.extension.IExtensionPoint;
-import org.apache.hop.core.logging.ILogChannel;
-import org.apache.hop.core.variables.IVariables;
-import org.apache.hop.lineage.LineageRunLifecycleEmitter;
-import org.apache.hop.lineage.model.RunLifecyclePhase;
-import org.apache.hop.workflow.WorkflowExecutionExtension;
-
-@ExtensionPoint(
- id = "LineageHubWorkflowActionBeforeXp",
- extensionPointId = "WorkflowBeforeActionExecution",
- description = "Emits lineage RUN_LIFECYCLE STARTED before a workflow
action runs")
-public class LineageHubWorkflowActionBeforeXp
- implements IExtensionPoint<WorkflowExecutionExtension> {
- @Override
- public void callExtensionPoint(
- ILogChannel log, IVariables variables, WorkflowExecutionExtension
extension)
- throws HopException {
- LineageRunLifecycleEmitter.emitWorkflowAction(extension,
RunLifecyclePhase.STARTED, null);
- }
-}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
index 780036e277..cd6225fa83 100644
---
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
+++
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowFinishXp.java
@@ -22,6 +22,7 @@ import org.apache.hop.core.extension.ExtensionPoint;
import org.apache.hop.core.extension.IExtensionPoint;
import org.apache.hop.core.logging.ILogChannel;
import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.lineage.LineageActionStartListener;
import org.apache.hop.lineage.LineageRunLifecycleEmitter;
import org.apache.hop.lineage.hub.LineageHub;
import org.apache.hop.lineage.model.RunLifecyclePhase;
@@ -38,11 +39,19 @@ public class LineageHubWorkflowFinishXp implements
IExtensionPoint<IWorkflowEngi
public void callExtensionPoint(
ILogChannel log, IVariables variables, IWorkflowEngine<WorkflowMeta>
workflow)
throws HopException {
+ // An action that threw never got its FINISHED or FAILED; close it before
the workflow ends.
+ LineageActionStartListener.failUnfinishedActions(workflow);
RunLifecyclePhase phase =
LineageRunLifecycleEmitter.workflowFailed(workflow,
workflow.getResult())
? RunLifecyclePhase.FAILED
: RunLifecyclePhase.FINISHED;
LineageRunLifecycleEmitter.emitWorkflow(workflow, phase, null);
- LineageHub.getInstance().flushQuietly();
+ // A top-level run waits for delivery. A nested run only waits once the
queue fills up: that
+ // keeps a run per row from dropping events, without a full drain after
every row.
+ if (LineageRunLifecycleEmitter.isTopLevel(workflow)) {
+ LineageHub.getInstance().flushQuietly();
+ } else {
+ LineageHub.getInstance().flushIfBacklogged();
+ }
}
}
diff --git
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
index ed6c2099b9..a4b071f78c 100644
---
a/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
+++
b/engine/src/main/java/org/apache/hop/lineage/xp/LineageHubWorkflowStartXp.java
@@ -22,7 +22,9 @@ import org.apache.hop.core.extension.ExtensionPoint;
import org.apache.hop.core.extension.IExtensionPoint;
import org.apache.hop.core.logging.ILogChannel;
import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.lineage.LineageActionStartListener;
import org.apache.hop.lineage.LineageRunLifecycleEmitter;
+import org.apache.hop.lineage.hub.LineageHub;
import org.apache.hop.lineage.model.RunLifecyclePhase;
import org.apache.hop.workflow.WorkflowMeta;
import org.apache.hop.workflow.engine.IWorkflowEngine;
@@ -30,12 +32,19 @@ import org.apache.hop.workflow.engine.IWorkflowEngine;
@ExtensionPoint(
id = "LineageHubWorkflowStartXp",
extensionPointId = "WorkflowStart",
- description = "Emits lineage RUN_LIFECYCLE STARTED for workflow
executions")
+ description =
+ "Emits lineage RUN_LIFECYCLE STARTED for workflow executions and
registers the action"
+ + " start listener")
public class LineageHubWorkflowStartXp implements
IExtensionPoint<IWorkflowEngine<WorkflowMeta>> {
@Override
public void callExtensionPoint(
ILogChannel log, IVariables variables, IWorkflowEngine<WorkflowMeta>
workflow)
throws HopException {
LineageRunLifecycleEmitter.emitWorkflow(workflow,
RunLifecyclePhase.STARTED, null);
+ // Action starts are emitted from an action listener: only there does the
cloned action, and
+ // with it the action run's id, exist.
+ if (LineageHub.getInstance().isEnabled()) {
+ LineageActionStartListener.register(workflow);
+ }
}
}
diff --git
a/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
b/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
index 22ad30841c..5d87e30c77 100644
---
a/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
+++
b/engine/src/main/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransform.java
@@ -19,6 +19,8 @@ package org.apache.hop.pipeline.transforms.file;
import java.util.Date;
import java.util.HashMap;
+import java.util.List;
+import java.util.Locale;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.FileSystemException;
import org.apache.hop.core.IRowSet;
@@ -26,6 +28,7 @@ import org.apache.hop.core.ResultFile;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.exception.HopFileException;
import org.apache.hop.core.row.IRowMeta;
+import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.row.value.ValueMetaFactory;
import org.apache.hop.core.vfs.HopVfs;
@@ -44,6 +47,9 @@ public abstract class BaseFileInputTransform<
private static final Class<?> PKG = BaseFileInputTransform.class;
+ /** The file schema for lineage, built on the first file this transform
opens. */
+ private IRowMeta lineageFileRowMeta;
+
/** Create reader for specific file. */
protected abstract IBaseFileInputReader createReader(Meta meta, Data data,
FileObject file)
throws Exception;
@@ -164,9 +170,12 @@ public abstract class BaseFileInputTransform<
/** Best-effort {@link FileIoOperation#READ} lineage for the file being
opened. */
protected void emitLineageFileRead(FileObject file) {
- if (file == null) {
+ if (file == null || !LineageFileIoEmitter.isEnabled()) {
return;
}
+ if (lineageFileRowMeta == null) {
+ lineageFileRowMeta = fileRowMeta();
+ }
Long size = null;
try {
if (file.getType().hasContent()) {
@@ -175,8 +184,50 @@ public abstract class BaseFileInputTransform<
} catch (Exception ignored) {
// optional metadata for lineage only
}
+ String extension = file.getName().getExtension();
LineageFileIoEmitter.emitTransformFileIo(
- this, FileIoOperation.READ, file, null, size, true, null);
+ this,
+ FileIoOperation.READ,
+ file,
+ null,
+ size,
+ true,
+ null,
+ LineageFileIoEmitter.toContentSchema(
+ lineageFileRowMeta, extension == null ? null :
extension.toLowerCase(Locale.ROOT)));
+ }
+
+ /**
+ * The names of the fields this transform reads from the file itself, in the
order it reads them,
+ * which make up the file's schema in its lineage; {@code null} when every
output field comes from
+ * the file. In the output row these fields follow the ones passed through
from the input rows,
+ * and precede the fields the transform adds (file name, row number, ...).
Asked once, on the
+ * first file the transform opens with lineage on.
+ */
+ protected List<String> fileFieldNames() {
+ return null;
+ }
+
+ /**
+ * The file's own fields: types as read, from their position in the output
row, and names as the
+ * file has them. The output row cannot be matched by name: a passed-through
field can have the
+ * name of a file field (which the row then renames to {@code name_1}), and
the file name can be
+ * prepended to every file field.
+ */
+ private IRowMeta fileRowMeta() {
+ IRowMeta output = data.outputRowMeta;
+ List<String> names = fileFieldNames();
+ if (output == null || names == null) {
+ return output;
+ }
+ int from = meta.getFileInput().isPassingThruFields() ?
data.nrPassThruFields : 0;
+ RowMeta fileRow = new RowMeta();
+ for (int i = 0; i < names.size() && from + i < output.size(); i++) {
+ IValueMeta valueMeta = output.getValueMeta(from + i).clone();
+ valueMeta.setName(names.get(i));
+ fileRow.addValueMeta(valueMeta);
+ }
+ return fileRow;
}
/** Read files from previous transform. */
diff --git a/engine/src/main/java/org/apache/hop/run/HopRunBase.java
b/engine/src/main/java/org/apache/hop/run/HopRunBase.java
index bf27b7229b..ce769999c5 100644
--- a/engine/src/main/java/org/apache/hop/run/HopRunBase.java
+++ b/engine/src/main/java/org/apache/hop/run/HopRunBase.java
@@ -202,8 +202,8 @@ public abstract class HopRunBase implements Runnable,
IHasHopMetadataProvider {
Duration.ofSeconds(Math.max(0L, shutdownTimeout)),
() -> LineageHub.getInstance().shutdown());
shutdownHookThread = new Thread(shutdownHook,
"hop-run-stop-on-shutdown");
- // The lineage shutdown hook of HopEnvironment waits for this run, so
that its end is
- // delivered as well.
+ // The exit hook of the lineage hub waits for this run, so that its end
is delivered as
+ // well.
LineageHub.getInstance().holdShutdownOnExit(shutdownHook.getRunOver());
Runtime.getRuntime().addShutdownHook(shutdownHookThread);
diff --git
a/engine/src/test/java/org/apache/hop/lineage/LineageActionLifecycleTest.java
b/engine/src/test/java/org/apache/hop/lineage/LineageActionLifecycleTest.java
new file mode 100644
index 0000000000..b39f4a2490
--- /dev/null
+++
b/engine/src/test/java/org/apache/hop/lineage/LineageActionLifecycleTest.java
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyString;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hop.core.Result;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.lineage.hub.LineageHub;
+import org.apache.hop.lineage.model.LineageEvent;
+import org.apache.hop.lineage.model.RunLifecycleLineagePayload;
+import org.apache.hop.lineage.model.RunLifecyclePhase;
+import org.apache.hop.workflow.IActionListener;
+import org.apache.hop.workflow.WorkflowExecutionExtension;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionBase;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+/**
+ * An action run is identified by the log channel of the action clone the
workflow executes. Its
+ * STARTED and FINISHED events have to carry that id, not the workflow's: the
action is a job of its
+ * own ({@code <workflow>/<action>}), and a pipeline it starts names that same
id as its parent.
+ */
+class LineageActionLifecycleTest {
+
+ @SuppressWarnings("unchecked")
+ private static IWorkflowEngine<WorkflowMeta> workflow() {
+ IWorkflowEngine<WorkflowMeta> wf = mock(IWorkflowEngine.class);
+ WorkflowMeta meta = new WorkflowMeta();
+ meta.setName("nightly-load");
+ when(wf.getWorkflowMeta()).thenReturn(meta);
+ when(wf.getLogChannelId()).thenReturn("workflow-run-id");
+ return wf;
+ }
+
+ private static ActionBase executingAction() {
+ ActionBase action = mock(ActionBase.class);
+ ILogChannel log = mock(ILogChannel.class);
+ when(log.getLogChannelId()).thenReturn("action-run-id");
+ when(action.getName()).thenReturn("Run orders");
+ when(action.getDescription()).thenReturn("Loads the orders");
+ when(action.getLogChannel()).thenReturn(log);
+ return action;
+ }
+
+ @Test
+ void actionStartCarriesTheExecutingActionsRunId() {
+ LineageHub hub = mock(LineageHub.class);
+ IWorkflowEngine<WorkflowMeta> wf = workflow();
+ ActionBase action = executingAction();
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ List<IActionListener> listeners = new ArrayList<>();
+ when(wf.getActionListeners()).thenReturn(listeners);
+ doAnswer(invocation -> listeners.add(invocation.getArgument(0)))
+ .when(wf)
+ .addActionListener(any());
+
+ LineageActionStartListener.register(wf);
+ listeners.get(0).beforeExecution(wf, new ActionMeta(action), action);
+ }
+
+ ArgumentCaptor<LineageEvent> event =
ArgumentCaptor.forClass(LineageEvent.class);
+ verify(hub).emit(event.capture());
+ assertEquals(
+ RunLifecyclePhase.STARTED,
+ ((RunLifecycleLineagePayload)
event.getValue().getPayload()).getPhase());
+ assertEquals("action-run-id",
event.getValue().getContext().getLogChannelId());
+ assertEquals("Run orders", event.getValue().getContext().getActionName());
+ assertEquals("nightly-load",
event.getValue().getContext().getWorkflowName());
+ assertEquals(
+ "Loads the orders",
+ event
+ .getValue()
+ .getContext()
+ .getAttributes()
+ .get(LineageRunLifecycleEmitter.ATTR_DESCRIPTION));
+ }
+
+ @Test
+ void actionStartListenerIsRegisteredOncePerWorkflow() {
+ IWorkflowEngine<WorkflowMeta> wf = workflow();
+ List<IActionListener> listeners = new ArrayList<>();
+ when(wf.getActionListeners()).thenReturn(listeners);
+ doAnswer(invocation -> listeners.add(invocation.getArgument(0)))
+ .when(wf)
+ .addActionListener(any());
+
+ LineageActionStartListener.register(wf);
+ LineageActionStartListener.register(wf);
+
+ verify(wf, times(1)).addActionListener(any());
+ }
+
+ @Test
+ void actionFinishCarriesTheExecutingActionsRunId() {
+ LineageHub hub = mock(LineageHub.class);
+ WorkflowExecutionExtension extension =
+ new WorkflowExecutionExtension(
+ workflow(), new Result(), new ActionMeta(executingAction()), true);
+ extension.actionLogChannelId = "action-run-id";
+ extension.actionExecutionResult = new Result();
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ LineageRunLifecycleEmitter.emitWorkflowAction(extension,
RunLifecyclePhase.FINISHED, null);
+ }
+
+ ArgumentCaptor<LineageEvent> event =
ArgumentCaptor.forClass(LineageEvent.class);
+ verify(hub).emit(event.capture());
+ assertEquals("action-run-id",
event.getValue().getContext().getLogChannelId());
+ }
+
+ @Test
+ void skippedActionEmitsNothing() {
+ LineageHub hub = mock(LineageHub.class);
+ // An extension point told the workflow not to execute the action: no
clone, no run id.
+ WorkflowExecutionExtension extension =
+ new WorkflowExecutionExtension(
+ workflow(), new Result(), new ActionMeta(executingAction()),
false);
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ LineageRunLifecycleEmitter.emitWorkflowAction(extension,
RunLifecyclePhase.FINISHED, null);
+ }
+
+ verify(hub, never()).emit(any());
+ }
+
+ private static IActionListener
registeredListener(IWorkflowEngine<WorkflowMeta> wf) {
+ List<IActionListener> listeners = new ArrayList<>();
+ when(wf.getActionListeners()).thenReturn(listeners);
+ doAnswer(invocation -> listeners.add(invocation.getArgument(0)))
+ .when(wf)
+ .addActionListener(any());
+ LineageActionStartListener.register(wf);
+ return listeners.get(0);
+ }
+
+ // The listener runs inside the workflow's action execution: a lineage
problem may not fail it.
+ @Test
+ void aFailingStartEmitDoesNotFailTheAction() {
+ LineageHub hub = mock(LineageHub.class);
+ doThrow(new IllegalStateException("sink exploded")).when(hub).emit(any());
+ IWorkflowEngine<WorkflowMeta> wf = workflow();
+ ILogChannel workflowLog = mock(ILogChannel.class);
+ when(wf.getLogChannel()).thenReturn(workflowLog);
+ ActionBase action = executingAction();
+ IActionListener listener = registeredListener(wf);
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ assertDoesNotThrow(() -> listener.beforeExecution(wf, new
ActionMeta(action), action));
+ }
+
+ verify(workflowLog).logError(anyString(), any(Throwable.class));
+ }
+
+ // execute() threw: the workflow skips WorkflowAfterActionExecution, so the
run is closed as
+ // FAILED when the workflow ends, under the run id its STARTED carried.
+ @Test
+ void anActionThatThrewIsClosedAsFailedWhenTheWorkflowEnds() {
+ LineageHub hub = mock(LineageHub.class);
+ IWorkflowEngine<WorkflowMeta> wf = workflow();
+ ActionBase action = executingAction();
+ IActionListener listener = registeredListener(wf);
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ listener.beforeExecution(wf, new ActionMeta(action), action);
+ LineageActionStartListener.failUnfinishedActions(wf);
+ // Closed once: a second end of the workflow has nothing left to close.
+ LineageActionStartListener.failUnfinishedActions(wf);
+ }
+
+ ArgumentCaptor<LineageEvent> events =
ArgumentCaptor.forClass(LineageEvent.class);
+ verify(hub, times(2)).emit(events.capture());
+ LineageEvent failed = events.getAllValues().get(1);
+ assertEquals(
+ RunLifecyclePhase.FAILED, ((RunLifecycleLineagePayload)
failed.getPayload()).getPhase());
+ assertEquals("action-run-id", failed.getContext().getLogChannelId());
+ assertEquals(
+ LineageRunLifecycleEmitter.FAILED,
+
failed.getContext().getAttributes().get(LineageRunLifecycleEmitter.OUTCOME));
+ }
+
+ @Test
+ void anActionThatEndedIsNotClosedAgain() {
+ LineageHub hub = mock(LineageHub.class);
+ IWorkflowEngine<WorkflowMeta> wf = workflow();
+ ActionBase action = executingAction();
+ IActionListener listener = registeredListener(wf);
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ listener.beforeExecution(wf, new ActionMeta(action), action);
+ LineageActionStartListener.actionEnded(wf, "action-run-id");
+ LineageActionStartListener.failUnfinishedActions(wf);
+ }
+
+ verify(hub, times(1)).emit(any());
+ }
+}
diff --git
a/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
b/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
index 7f7f3117ab..6f63f44a41 100644
--- a/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
+++ b/engine/src/test/java/org/apache/hop/lineage/LineageFileIoEmitterTest.java
@@ -17,6 +17,10 @@
package org.apache.hop.lineage;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.argThat;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.verify;
@@ -25,6 +29,9 @@ import static org.mockito.Mockito.when;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.VFS;
import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaInteger;
+import org.apache.hop.core.row.value.ValueMetaString;
import org.apache.hop.lineage.context.LineageSubjectType;
import org.apache.hop.lineage.hub.LineageHub;
import org.apache.hop.lineage.model.FileIoContentSchema;
@@ -249,6 +256,26 @@ class LineageFileIoEmitterTest {
}));
}
+ @Test
+ void toContentSchema_describesTheRowShapeWithTheFileFormat() {
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("customer"));
+ rowMeta.addValueMeta(new ValueMetaInteger("id"));
+
+ FileIoContentSchema schema = LineageFileIoEmitter.toContentSchema(rowMeta,
"csv");
+
+ assertEquals("csv", schema.getFormatHint());
+ assertEquals(2, schema.getColumns().size());
+ assertEquals("customer", schema.getColumns().get(0).getName());
+ assertEquals("id", schema.getColumns().get(1).getName());
+ // The type name comes from the value-meta plugin registry, which a bare
unit test does not
+ // load.
+ assertNotNull(schema.getColumns().get(0).getHopTypeName());
+ assertTrue(schema.getStructureRoots().isEmpty());
+ assertNull(LineageFileIoEmitter.toContentSchema(new RowMeta(), "csv"));
+ assertNull(LineageFileIoEmitter.toContentSchema(null, null));
+ }
+
@Test
void emitTransformFileIo_postsFileIoEventWithTransformContext() throws
Exception {
LineageHub hub = mock(LineageHub.class);
diff --git
a/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
index a197d70d9c..67d87a0215 100644
--- a/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
+++ b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineGraphTest.java
@@ -36,6 +36,11 @@ import org.apache.hop.pipeline.PipelineHopMeta;
import org.apache.hop.pipeline.PipelineMeta;
import org.apache.hop.pipeline.engine.IPipelineEngine;
import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.workflow.WorkflowHopMeta;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionMeta;
+import org.apache.hop.workflow.actions.dummy.ActionDummy;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
import org.mockito.MockedStatic;
@@ -160,14 +165,19 @@ class LineagePipelineGraphTest {
}
@Test
- void pipelineWithoutHopsEmitsNoGraph() {
+ void pipelineWithoutHopsEmitsAnEmptyGraph() {
+ // An empty graph, not none: a sink that gets no graph assumes an engine
that sends none and
+ // files the run's datasets under the pipeline instead of under its only
transform.
PipelineMeta meta = new PipelineMeta();
meta.getTransforms().add(transform("Alone"));
List<LineageEvent> events = emitPipeline(meta, RunLifecyclePhase.STARTED);
- assertEquals(1, events.size());
+ assertEquals(2, events.size());
assertEquals(LineageEventKind.RUN_LIFECYCLE, events.get(0).getKind());
+ assertEquals(LineageEventKind.PIPELINE_GRAPH, events.get(1).getKind());
+ assertTrue(
+ ((PipelineGraphLineagePayload)
events.get(1).getPayload()).getTransitions().isEmpty());
}
private static PipelineMeta chain() {
@@ -196,4 +206,60 @@ class LineagePipelineGraphTest {
verify(hub, atLeastOnce()).emit(captor.capture());
return captor.getAllValues();
}
+
+ // Like a pipeline's: a workflow without hops still tells the sinks so, with
an empty graph.
+ @Test
+ void workflowWithoutHopsEmitsAnEmptyGraph() {
+ WorkflowMeta meta = new WorkflowMeta();
+ meta.addAction(new ActionMeta(new ActionDummy("Alone")));
+
+ List<LineageEvent> events = emitWorkflow(meta, RunLifecyclePhase.STARTED);
+
+ assertEquals(2, events.size());
+ assertEquals(LineageEventKind.RUN_LIFECYCLE, events.get(0).getKind());
+ assertEquals(LineageEventKind.WORKFLOW_GRAPH, events.get(1).getKind());
+ assertTrue(
+ ((PipelineGraphLineagePayload)
events.get(1).getPayload()).getTransitions().isEmpty());
+ }
+
+ @SuppressWarnings("unchecked")
+ private static List<LineageEvent> emitWorkflow(WorkflowMeta meta,
RunLifecyclePhase phase) {
+ LineageHub hub = mock(LineageHub.class);
+ IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+ when(workflow.getWorkflowMeta()).thenReturn(meta);
+ when(workflow.getLogChannelId()).thenReturn("workflow-log");
+
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ LineageRunLifecycleEmitter.emitWorkflow(workflow, phase, null);
+ }
+
+ ArgumentCaptor<LineageEvent> captor =
ArgumentCaptor.forClass(LineageEvent.class);
+ verify(hub, atLeastOnce()).emit(captor.capture());
+ return captor.getAllValues();
+ }
+
+ // A workflow's hops say which action follows which, whatever their
condition; disabled hops are
+ // not part of what ran.
+ @Test
+ void workflowHopsAreReportedBetweenActions() {
+ WorkflowMeta meta = new WorkflowMeta();
+ ActionMeta load = new ActionMeta(new ActionDummy("Load"));
+ ActionMeta report = new ActionMeta(new ActionDummy("Report"));
+ ActionMeta alert = new ActionMeta(new ActionDummy("Alert"));
+ meta.addAction(load);
+ meta.addAction(report);
+ meta.addAction(alert);
+ meta.addWorkflowHop(new WorkflowHopMeta(load, report));
+ WorkflowHopMeta onFailure = new WorkflowHopMeta(load, alert);
+ onFailure.setEvaluation(false);
+ meta.addWorkflowHop(onFailure);
+ WorkflowHopMeta disabled = new WorkflowHopMeta(report, alert);
+ disabled.setEnabled(false);
+ meta.addWorkflowHop(disabled);
+
+ assertEquals(
+ List.of(new Transition("Load", "Report"), new Transition("Load",
"Alert")),
+ LineageRunLifecycleEmitter.workflowTransitions(meta));
+ }
}
diff --git
a/engine/src/test/java/org/apache/hop/lineage/LineagePipelineParentTest.java
b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineParentTest.java
new file mode 100644
index 0000000000..0755a32c2d
--- /dev/null
+++ b/engine/src/test/java/org/apache/hop/lineage/LineagePipelineParentTest.java
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.lineage;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.lineage.context.LineageContext;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.engine.IPipelineEngine;
+import org.apache.hop.pipeline.transform.BaseTransform;
+import org.apache.hop.workflow.WorkflowMeta;
+import org.apache.hop.workflow.action.ActionBase;
+import org.apache.hop.workflow.engine.IWorkflowEngine;
+import org.junit.jupiter.api.Test;
+
+/**
+ * A pipeline's lineage run has to say which run started it. The engine knows
either the action that
+ * ran it (inside its workflow) or the parent pipeline of a sub-pipeline; the
job name it reports is
+ * the one the OpenLineage mapper gives that parent, so a collector can draw
the hierarchy.
+ */
+class LineagePipelineParentTest {
+
+ private static LineageContext.Builder context() {
+ return LineageContext.builder();
+ }
+
+ @Test
+ void pipelineRunByAWorkflowActionNamesTheActionAsItsParent() {
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+ @SuppressWarnings("unchecked")
+ IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+ ActionBase action = mock(ActionBase.class);
+ ILogChannel actionLog = mock(ILogChannel.class);
+ WorkflowMeta workflowMeta = new WorkflowMeta();
+ workflowMeta.setName("nightly-load");
+ when(pipeline.getParent()).thenReturn(action);
+ when(pipeline.getParentWorkflow()).thenReturn(workflow);
+ when(workflow.getWorkflowMeta()).thenReturn(workflowMeta);
+ when(action.getName()).thenReturn("Run orders pipeline");
+ when(action.getLogChannel()).thenReturn(actionLog);
+ when(actionLog.getLogChannelId()).thenReturn("action-run-id");
+
+ LineageContext.Builder ctx = context();
+ LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+ LineageContext built = ctx.build();
+
+ assertEquals(
+ "nightly-load/Run orders pipeline",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+ assertEquals(
+ "nightly-load",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_WORKFLOW_NAME));
+ assertEquals(
+ "Run orders pipeline",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_ACTION_NAME));
+ assertEquals(
+ "action-run-id",
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+ }
+
+ @Test
+ void subPipelineNamesTheParentPipelineAsItsParent() {
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> parentPipeline = mock(IPipelineEngine.class);
+ PipelineMeta parentMeta = new PipelineMeta();
+ parentMeta.setName("orders-parent");
+ when(pipeline.getParentPipeline()).thenReturn(parentPipeline);
+ when(parentPipeline.getPipelineMeta()).thenReturn(parentMeta);
+ when(parentPipeline.getLogChannelId()).thenReturn("parent-run-id");
+
+ LineageContext.Builder ctx = context();
+ LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+ LineageContext built = ctx.build();
+
+ assertEquals(
+ "orders-parent",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+ assertEquals(
+ "orders-parent",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_PIPELINE_NAME));
+ assertEquals(
+ "parent-run-id",
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+ }
+
+ @Test
+ void workflowRunByAWorkflowActionNamesTheActionAsItsParent() {
+ @SuppressWarnings("unchecked")
+ IWorkflowEngine<WorkflowMeta> child = mock(IWorkflowEngine.class);
+ @SuppressWarnings("unchecked")
+ IWorkflowEngine<WorkflowMeta> parent = mock(IWorkflowEngine.class);
+ ActionBase action = mock(ActionBase.class);
+ ILogChannel actionLog = mock(ILogChannel.class);
+ WorkflowMeta parentMeta = new WorkflowMeta();
+ parentMeta.setName("nightly-load");
+ when(child.getParent()).thenReturn(action);
+ when(child.getParentWorkflow()).thenReturn(parent);
+ when(parent.getWorkflowMeta()).thenReturn(parentMeta);
+ when(parent.getLogChannelId()).thenReturn("parent-workflow-run-id");
+ when(action.getName()).thenReturn("Run sub-workflow");
+ when(action.getLogChannel()).thenReturn(actionLog);
+ when(actionLog.getLogChannelId()).thenReturn("action-run-id");
+
+ LineageContext.Builder ctx = context();
+ LineageRunLifecycleEmitter.fillWorkflowParentFields(ctx, child);
+ LineageContext built = ctx.build();
+
+ assertEquals(
+ "nightly-load/Run sub-workflow",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+ assertEquals(
+ "action-run-id",
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+ }
+
+ // A Pipeline Executor or mapping runs its pipeline as its logging parent.
The parent is that
+ // transform's run, <pipeline>/<transform>: the same shape as a pipeline run
by a workflow action.
+ @Test
+ void subPipelineRunByATransformNamesThatTransform() {
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> parentPipeline = mock(IPipelineEngine.class);
+ BaseTransform<?, ?> executor = mock(BaseTransform.class);
+ ILogChannel executorLog = mock(ILogChannel.class);
+ PipelineMeta parentMeta = new PipelineMeta();
+ parentMeta.setName("orders-parent");
+ when(pipeline.getParentPipeline()).thenReturn(parentPipeline);
+ when(pipeline.getParent()).thenReturn(executor);
+ when(parentPipeline.getPipelineMeta()).thenReturn(parentMeta);
+ when(parentPipeline.getLogChannelId()).thenReturn("parent-run-id");
+ when(executor.getTransformName()).thenReturn("Run per order");
+ when(executor.getLogChannel()).thenReturn(executorLog);
+ when(executorLog.getLogChannelId()).thenReturn("executor-run-id");
+
+ LineageContext.Builder ctx = context();
+ LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+ LineageContext built = ctx.build();
+
+ assertEquals(
+ "orders-parent/Run per order",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+ assertEquals(
+ "executor-run-id",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+ assertEquals(
+ "orders-parent",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_PIPELINE_NAME));
+ assertEquals(
+ "Run per order",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_TRANSFORM_NAME));
+ }
+
+ // A Workflow Executor is the logging parent of the workflow it runs: same
shape.
+ @Test
+ void workflowRunByATransformNamesThatTransform() {
+ @SuppressWarnings("unchecked")
+ IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> parentPipeline = mock(IPipelineEngine.class);
+ BaseTransform<?, ?> executor = mock(BaseTransform.class);
+ ILogChannel executorLog = mock(ILogChannel.class);
+ PipelineMeta parentMeta = new PipelineMeta();
+ parentMeta.setName("orders-parent");
+ when(workflow.getParentPipeline()).thenReturn(parentPipeline);
+ when(workflow.getParent()).thenReturn(executor);
+ when(parentPipeline.getPipelineMeta()).thenReturn(parentMeta);
+ when(executor.getTransformName()).thenReturn("Run workflow per order");
+ when(executor.getLogChannel()).thenReturn(executorLog);
+ when(executorLog.getLogChannelId()).thenReturn("executor-run-id");
+
+ LineageContext.Builder ctx = context();
+ LineageRunLifecycleEmitter.fillWorkflowParentFields(ctx, workflow);
+ LineageContext built = ctx.build();
+
+ assertEquals(
+ "orders-parent/Run workflow per order",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+ assertEquals(
+ "executor-run-id",
+
built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+ }
+
+ // Only a top-level run flushes the lineage hub when it completes.
+ @Test
+ void onlyRunsWithoutAParentAreTopLevel() {
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> topLevel = mock(IPipelineEngine.class);
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> subPipeline = mock(IPipelineEngine.class);
+ when(subPipeline.getParentPipeline()).thenReturn(topLevel);
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> runByAction = mock(IPipelineEngine.class);
+ when(runByAction.getParent()).thenReturn(mock(ActionBase.class));
+ @SuppressWarnings("unchecked")
+ IWorkflowEngine<WorkflowMeta> workflow = mock(IWorkflowEngine.class);
+ @SuppressWarnings("unchecked")
+ IWorkflowEngine<WorkflowMeta> childWorkflow = mock(IWorkflowEngine.class);
+ when(childWorkflow.getParentWorkflow()).thenReturn(workflow);
+
+ assertTrue(LineageRunLifecycleEmitter.isTopLevel(topLevel));
+ assertFalse(LineageRunLifecycleEmitter.isTopLevel(subPipeline));
+ assertFalse(LineageRunLifecycleEmitter.isTopLevel(runByAction));
+ assertTrue(LineageRunLifecycleEmitter.isTopLevel(workflow));
+ assertFalse(LineageRunLifecycleEmitter.isTopLevel(childWorkflow));
+ }
+
+ @Test
+ void topLevelPipelineHasNoParentFields() {
+ @SuppressWarnings("unchecked")
+ IPipelineEngine<PipelineMeta> pipeline = mock(IPipelineEngine.class);
+
+ LineageContext.Builder ctx = context();
+ LineageRunLifecycleEmitter.fillPipelineParentFields(ctx, pipeline);
+ LineageContext built = ctx.build();
+
+
assertNull(built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_JOB_NAME));
+
assertNull(built.getAttributes().get(LineageRunLifecycleEmitter.ATTR_PARENT_RUN_ID));
+ }
+}
diff --git
a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
index 9fcaf7c0a8..0b8ef189dd 100644
---
a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
+++
b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubConcurrentShutdownTest.java
@@ -17,10 +17,12 @@
package org.apache.hop.lineage.hub;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
@@ -160,8 +162,8 @@ class LineageHubConcurrentShutdownTest {
}
/**
- * The JVM runs the shutdown hook of HopEnvironment at the same time as the
one of hop-run. While
- * hop-run holds the hub, that hook may not shut it down: the run still has
to emit its end.
+ * The JVM runs the exit hook of the hub at the same time as the one of
hop-run. While hop-run
+ * holds the hub, that hook may not shut it down: the run still has to emit
its end.
*/
@Test
void shutdownOnExitWaitsForTheRunThatHoldsIt() throws Exception {
@@ -206,6 +208,47 @@ class LineageHubConcurrentShutdownTest {
assertEquals(1, sinkShutdowns.get());
}
+ /** The exit hook the hub registers is the one that waits for hop-run, and
it is added once. */
+ @Test
+ void theRegisteredExitHookWaitsForTheRunThatHoldsIt() throws Exception {
+ AtomicInteger sinkShutdowns = new AtomicInteger();
+ LineageHub hub =
+ LineageHub.newIsolatedForTesting(
+ configuration(), List.of(sink(new AtomicInteger(),
sinkShutdowns)));
+ hub.emit(event());
+ CountDownLatch runOver = new CountDownLatch(1);
+ hub.holdShutdownOnExit(runOver);
+ List<Thread> hooks = new ArrayList<>();
+ hub.registerExitHook(hooks::add);
+ hub.registerExitHook(hooks::add);
+ assertEquals(1, hooks.size(), "The exit hook should be registered once");
+
+ Thread exitHook = hooks.get(0);
+ exitHook.start();
+ Thread.sleep(300);
+ assertTrue(exitHook.isAlive(), "The exit hook shut the hub down while a
run held it");
+ assertEquals(0, sinkShutdowns.get());
+
+ runOver.countDown();
+ exitHook.join(10_000);
+ assertFalse(exitHook.isAlive());
+ assertEquals(1, sinkShutdowns.get());
+ }
+
+ /** The first event can come in while the JVM is already exiting: it may not
fail the emit. */
+ @Test
+ void aJvmThatIsAlreadyExitingDoesNotFailTheRegistration() {
+ LineageHub hub =
+ LineageHub.newIsolatedForTesting(
+ configuration(), List.of(sink(new AtomicInteger(), new
AtomicInteger())));
+ assertDoesNotThrow(
+ () ->
+ hub.registerExitHook(
+ hook -> {
+ throw new IllegalStateException("Shutdown in progress");
+ }));
+ }
+
private static LineageConfiguration configuration() {
return LineageConfiguration.forTesting(true, 100, 1, 5000L, Set.of());
}
diff --git
a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
index b1d9fdd7ca..f049c4b40e 100644
--- a/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
+++ b/engine/src/test/java/org/apache/hop/lineage/hub/LineageHubTest.java
@@ -18,6 +18,7 @@
package org.apache.hop.lineage.hub;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.util.ArrayList;
@@ -539,6 +540,56 @@ class LineageHubTest {
assertEquals(2, delivered.get());
}
+ /**
+ * A nested run (one per row of a Pipeline Executor, say) does not wait for
the sinks while the
+ * queue has room, and waits for them once it is half full, before events
get dropped.
+ */
+ @Test
+ void flushIfBackloggedWaitsOnlyOnceTheQueueIsHalfFull() throws Exception {
+ LineageConfiguration cfg = LineageConfiguration.forTesting(true, 10, 1,
5000L, Set.of());
+ CountDownLatch inSink = new CountDownLatch(1);
+ CountDownLatch release = new CountDownLatch(1);
+ AtomicInteger delivered = new AtomicInteger();
+ ILineageSink sink =
+ batch -> {
+ inSink.countDown();
+ try {
+ release.await(10, TimeUnit.SECONDS);
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ delivered.addAndGet(batch.size());
+ };
+ LineageHub hub = LineageHub.newIsolatedForTesting(cfg, List.of(sink));
+ hub.emit(sampleEvent());
+ assertTrue(inSink.await(5, TimeUnit.SECONDS), "the sink never got the
first event");
+
+ // The worker is busy in the sink, so these stay queued: 3 of 10.
+ for (int i = 0; i < 3; i++) {
+ hub.emit(sampleEvent());
+ }
+ Thread belowHalf = new Thread(hub::flushIfBacklogged);
+ belowHalf.start();
+ belowHalf.join(2000);
+ assertFalse(belowHalf.isAlive(), "a nested run waited for the sinks with
the queue 3/10 full");
+
+ // 5 of 10 queued: now it waits until the sinks have caught up.
+ for (int i = 0; i < 2; i++) {
+ hub.emit(sampleEvent());
+ }
+ Thread halfFull = new Thread(hub::flushIfBacklogged);
+ halfFull.start();
+ halfFull.join(300);
+ assertTrue(halfFull.isAlive(), "a nested run did not wait with the queue
half full");
+
+ release.countDown();
+ halfFull.join(5000);
+ assertFalse(halfFull.isAlive());
+ assertEquals(6, delivered.get());
+ assertEquals(0, hub.getDroppedEventCount());
+ hub.shutdown();
+ }
+
private static LineageEvent sampleEvent() {
return LineageEvent.of(LineageEventKind.RUN_LIFECYCLE,
LineageContext.empty(), null);
}
diff --git
a/engine/src/test/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransformLineageTest.java
b/engine/src/test/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransformLineageTest.java
new file mode 100644
index 0000000000..74dda8544a
--- /dev/null
+++
b/engine/src/test/java/org/apache/hop/pipeline/transforms/file/BaseFileInputTransformLineageTest.java
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.file;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.OutputStream;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.vfs2.FileObject;
+import org.apache.hop.core.HopEnvironment;
+import org.apache.hop.core.logging.ILoggingObject;
+import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.value.ValueMetaInteger;
+import org.apache.hop.core.row.value.ValueMetaNumber;
+import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.vfs.HopVfs;
+import org.apache.hop.lineage.hub.LineageHub;
+import org.apache.hop.lineage.model.FileIoContentSchema;
+import org.apache.hop.lineage.model.FileIoLineagePayload;
+import org.apache.hop.lineage.model.FileIoTabularColumn;
+import org.apache.hop.lineage.model.LineageEvent;
+import org.apache.hop.pipeline.Pipeline;
+import org.apache.hop.pipeline.PipelineMeta;
+import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transforms.mock.TransformMockHelper;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeAll;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.ArgumentCaptor;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+/** The schema a file input transform reports for the file it reads, in its
FILE_IO lineage. */
+@SuppressWarnings({"rawtypes", "unchecked"})
+class BaseFileInputTransformLineageTest {
+
+ /** A file input that reads the fields "tag" and "amount" from its files. */
+ static final class FileInput
+ extends BaseFileInputTransform<BaseFileInputMeta,
BaseFileInputTransformData> {
+ final AtomicInteger fileFieldNamesCalls = new AtomicInteger();
+
+ FileInput(
+ TransformMeta transformMeta,
+ BaseFileInputMeta meta,
+ BaseFileInputTransformData data,
+ PipelineMeta pipelineMeta,
+ Pipeline pipeline) {
+ super(transformMeta, meta, data, 0, pipelineMeta, pipeline);
+ }
+
+ @Override
+ protected IBaseFileInputReader createReader(
+ BaseFileInputMeta meta, BaseFileInputTransformData data, FileObject
file) {
+ return null;
+ }
+
+ @Override
+ public boolean failAfterBadFile(String errorMsg, boolean errorIgnored,
boolean skipBadFiles) {
+ return true;
+ }
+
+ @Override
+ protected List<String> fileFieldNames() {
+ fileFieldNamesCalls.incrementAndGet();
+ return List.of("tag", "amount");
+ }
+ }
+
+ private TransformMockHelper<BaseFileInputMeta, BaseFileInputTransformData>
helper;
+ private FileInput transform;
+ private FileObject file;
+
+ @BeforeAll
+ static void initHop() throws Exception {
+ HopEnvironment.init();
+ }
+
+ @BeforeEach
+ void setUp() throws Exception {
+ helper =
+ new TransformMockHelper<>(
+ "File input", BaseFileInputMeta.class,
BaseFileInputTransformData.class);
+ when(helper.logChannelFactory.create(any(), any(ILoggingObject.class)))
+ .thenReturn(helper.iLogChannel);
+ BaseFileInput fileInput = new BaseFileInput();
+ fileInput.setPassingThruFields(true);
+ when(helper.iTransformMeta.getFileInput()).thenReturn(fileInput);
+ BaseFileInputTransformData data = new BaseFileInputTransformData();
+ transform =
+ new FileInput(
+ helper.transformMeta,
+ helper.iTransformMeta,
+ data,
+ helper.pipelineMeta,
+ helper.pipeline);
+
+ // Passed through from the filename rows: "filename" and a "tag" of their
own. The file's "tag"
+ // comes after them and is renamed "tag_1" in the row; "rownum" is added
by the transform.
+ RowMeta passedThrough = new RowMeta();
+ passedThrough.addValueMeta(new ValueMetaString("filename"));
+ passedThrough.addValueMeta(new ValueMetaString("tag"));
+ RowMeta output = new RowMeta();
+ output.mergeRowMeta(passedThrough, "File input");
+ output.addValueMeta(new ValueMetaInteger("tag"));
+ output.addValueMeta(new ValueMetaNumber("amount"));
+ output.addValueMeta(new ValueMetaInteger("rownum"));
+ data.outputRowMeta = output;
+ data.nrPassThruFields = passedThrough.size();
+
+ file = HopVfs.getFileObject("ram:///lineage/orders.csv");
+ try (OutputStream out = file.getContent().getOutputStream()) {
+ out.write("7,12.5\n".getBytes(StandardCharsets.UTF_8));
+ }
+ }
+
+ @AfterEach
+ void tearDown() throws Exception {
+ file.delete();
+ helper.cleanUp();
+ }
+
+ @Test
+ void theFileSchemaIsTheFilesOwnFieldsWithTheirTypes() {
+ LineageHub hub = mock(LineageHub.class);
+ when(hub.isEnabled()).thenReturn(true);
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ transform.emitLineageFileRead(file);
+ transform.emitLineageFileRead(file);
+ }
+
+ ArgumentCaptor<LineageEvent> events =
ArgumentCaptor.forClass(LineageEvent.class);
+ verify(hub, times(2)).emit(events.capture());
+ FileIoContentSchema schema =
+ ((FileIoLineagePayload)
events.getValue().getPayload()).getContentSchema();
+ List<FileIoTabularColumn> columns = schema.getColumns();
+ assertEquals(2, columns.size());
+ assertEquals("tag", columns.get(0).getName());
+ assertEquals("Integer", columns.get(0).getHopTypeName());
+ assertEquals("amount", columns.get(1).getName());
+ assertEquals("Number", columns.get(1).getHopTypeName());
+ assertEquals("csv", schema.getFormatHint());
+ // Worked out once, not on every file.
+ assertEquals(1, transform.fileFieldNamesCalls.get());
+ }
+
+ @Test
+ void nothingIsWorkedOutWithLineageOff() {
+ LineageHub hub = mock(LineageHub.class);
+ when(hub.isEnabled()).thenReturn(false);
+ try (MockedStatic<LineageHub> staticHub =
Mockito.mockStatic(LineageHub.class)) {
+ staticHub.when(LineageHub::getInstance).thenReturn(hub);
+ transform.emitLineageFileRead(file);
+ }
+
+ verify(hub, never()).emit(any());
+ assertEquals(0, transform.fileFieldNamesCalls.get());
+ }
+}
diff --git
a/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
b/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
index e8326cc085..4b41d94896 100644
---
a/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
+++
b/plugins/tech/openlineage/src/main/java/org/apache/hop/lineage/openlineage/OpenLineageEventMapper.java
@@ -135,6 +135,7 @@ public final class OpenLineageEventMapper {
case TRANSFORM_SCHEMA -> Optional.empty();
case HTTP_IO -> Optional.empty();
case PIPELINE_GRAPH -> Optional.empty();
+ default -> Optional.empty();
};
}
diff --git
a/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
b/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
index 7eaaf125d2..643c866b68 100644
---
a/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
+++
b/plugins/transforms/textfile/src/main/java/org/apache/hop/pipeline/transforms/fileinput/text/TextFileInput.java
@@ -623,4 +623,14 @@ public class TextFileInput extends
BaseFileInputTransform<TextFileInputMeta, Tex
}
}
}
+
+ /** The reader fills one output field per input field, right after the
passed-through ones. */
+ @Override
+ protected List<String> fileFieldNames() {
+ List<String> names = new ArrayList<>();
+ for (TextFileInputField field : meta.getInputFields()) {
+ names.add(field.getName());
+ }
+ return names;
+ }
}