This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hop-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8973cfbf64 Website updated to ce72275
8973cfbf64 is described below
commit 8973cfbf64d855021b40d9aade1d8672882c730c
Author: jenkins <[email protected]>
AuthorDate: Mon Nov 16 19:47:53 2026 +0000
Website updated to ce72275
---
.../next/plugin-types/metadata-configuration.html | 5 +-
manual/next/metadata-types/data-catalog.html | 9 ++-
manual/next/pipeline/transforms/delay.html | 8 +++
manual/next/pipeline/transforms/delete.html | 8 +++
.../pipeline/transforms/detectemptystream.html | 9 +++
.../next/pipeline/transforms/detectlanguage.html | 66 ++++++++++++++++++++++
.../next/pipeline/transforms/dimensionlookup.html | 9 +++
manual/next/pipeline/transforms/dynamicsqlrow.html | 8 +++
manual/next/pipeline/transforms/edi2xml.html | 8 +++
manual/next/pipeline/transforms/emailinput.html | 9 +++
.../pipeline/transforms/enhancedjsonoutput.html | 8 +++
manual/next/pipeline/transforms/excelinput.html | 18 ++++++
manual/next/pipeline/transforms/excelwriter.html | 8 +++
manual/next/pipeline/transforms/execinfo.html | 8 +++
manual/next/pipeline/transforms/execprocess.html | 8 +++
manual/next/pipeline/transforms/execsql.html | 8 +++
manual/next/pipeline/transforms/execsqlrow.html | 8 +++
manual/next/pipeline/transforms/exectests.html | 8 +++
manual/next/pipeline/transforms/fake.html | 8 +++
manual/next/pipeline/transforms/fileexists.html | 8 +++
manual/next/pipeline/transforms/maskfields.html | 41 ++++++++++++++
manual/next/technology/datahub/index.html | 17 +++++-
manual/next/technology/openlineage/index.html | 41 ++++++++++++++
23 files changed, 323 insertions(+), 5 deletions(-)
diff --git a/dev-manual/next/plugin-types/metadata-configuration.html
b/dev-manual/next/plugin-types/metadata-configuration.html
index e0210dc8cf..aff9fbc0d6 100644
--- a/dev-manual/next/plugin-types/metadata-configuration.html
+++ b/dev-manual/next/plugin-types/metadata-configuration.html
@@ -795,7 +795,7 @@ The API is experimental.</p>
</dd>
<dt class="hdlist1">Implement</dt>
<dd>
-<p><code>IDataCatalog</code>, plus the <code>ICatalogCapability</code>
interfaces for what the catalog can answer:
<code>IDeprecationCapability</code>, <code>IFreshnessCapability</code>,
<code>IAssertionCapability</code></p>
+<p><code>IDataCatalog</code>, plus the <code>ICatalogCapability</code>
interfaces for what the catalog can answer:
<code>IDeprecationCapability</code>, <code>IFreshnessCapability</code>,
<code>IAssertionCapability</code>, <code>ISchemaCapability</code></p>
</dd>
<dt class="hdlist1">Plugin type</dt>
<dd>
@@ -820,7 +820,8 @@ A catalog that can’t answer is a normal outcome, not
an exception; the cal
<div class="paragraph">
<p>A dataset is a <code>DatasetRef</code>.
Build it with <code>DatasetRef.ofTable(…​)</code> or
<code>ofFile(…​)</code>: those give a table or a file the
OpenLineage namespace and name that lineage gives it, so <code>resolve()</code>
finds the dataset lineage wrote.
-<code>ofCatalogId(…​)</code> names a dataset by the
catalog’s own id.</p>
+<code>ofCatalogId(…​)</code> names a dataset by the
catalog’s own id.
+A transform or action that lets the user pick a dataset stores a
<code>CatalogDataset</code> (type, connection, schema, name) and calls
<code>toRef(variables, metadataProvider)</code>, which also reports a dataset
that can’t be named.</p>
</div>
</div>
</div>
diff --git a/manual/next/metadata-types/data-catalog.html
b/manual/next/metadata-types/data-catalog.html
index 2813595ba6..12abae4029 100644
--- a/manual/next/metadata-types/data-catalog.html
+++ b/manual/next/metadata-types/data-catalog.html
@@ -3006,7 +3006,8 @@
The catalog stays the owner of that metadata. Hop only reads it.</p>
</div>
<div class="paragraph">
-<p>The <a href="../workflow/actions/checkdatacatalog.html" class="xref
page">Check data catalog</a> action uses a Data Catalog to check datasets
before a workflow goes on.</p>
+<p>The <a href="../workflow/actions/checkdatacatalog.html" class="xref
page">Check data catalog</a> action uses a Data Catalog to check datasets
before a workflow goes on.
+The <a href="../pipeline/transforms/maskfields.html" class="xref page">Mask
fields</a> transform can use one to mask the fields that carry a tag or
glossary term, such as <code>PII</code>.</p>
</div>
<div class="paragraph">
<p>This metadata type is experimental: its options and what the catalog types
can answer may still change.</p>
@@ -3147,12 +3148,16 @@ Use the same one as the DataHub lineage sink
(<code>HOP_LINEAGE_DATAHUB_ENV</cod
<li>
<p><strong>Last change</strong>: the newest operation DataHub has for the
dataset.
DataHub learns about operations from the sources that report them, such as the
usage ingestion of some warehouses or dbt.
-The DataHub lineage sink of Hop doesn’t report them yet, so a table that
only Hop writes has none.</p>
+The <a href="../technology/datahub/index.html" class="xref page">DataHub
lineage sink</a> reports one for each file and table a successful Hop run
writes, so a dataset that only Hop writes has a last change too.</p>
</li>
<li>
<p><strong>Assertions</strong>: the latest run of each assertion on the
dataset, up to 100 assertions.
The assertions themselves are defined and run in DataHub.</p>
</li>
+<li>
+<p><strong>Fields</strong>: the fields of the dataset’s schema, with the
tags and glossary terms on them, both the ones that came with the ingested
schema and the ones added in DataHub.
+The <a href="../pipeline/transforms/maskfields.html" class="xref page">Mask
fields</a> transform uses them.</p>
+</li>
</ul>
</div>
</div>
diff --git a/manual/next/pipeline/transforms/delay.html
b/manual/next/pipeline/transforms/delay.html
index 9877875099..ab04d3c650 100644
--- a/manual/next/pipeline/transforms/delay.html
+++ b/manual/next/pipeline/transforms/delay.html
@@ -3110,6 +3110,14 @@
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/delay-row-basic.hpl</code>, which
delays each of 10 generated rows by 5 seconds, so a line appears in the log
every 5 seconds.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/delete.html
b/manual/next/pipeline/transforms/delete.html
index 622406ece6..bab48158b4 100644
--- a/manual/next/pipeline/transforms/delete.html
+++ b/manual/next/pipeline/transforms/delete.html
@@ -3121,6 +3121,14 @@ This is important for data sources that allow for table
names with periods in th
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/delete.hpl</code>, which deletes
the employees with <code>id</code> 1 and 2 from the <code>employees</code>
table of the <code>hop-samples</code> connection.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/detectemptystream.html
b/manual/next/pipeline/transforms/detectemptystream.html
index b71efa4441..9bab6b83af 100644
--- a/manual/next/pipeline/transforms/detectemptystream.html
+++ b/manual/next/pipeline/transforms/detectemptystream.html
@@ -3096,6 +3096,15 @@ This name must be unique throughout the
pipeline.</p></td>
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has
<code>transforms/detect-empty-stream-basic.hpl</code>: Get file names looks for
a file that doesn’t exist, so Detect empty stream passes on one row with
null values for the Get file names fields.
+The <strong>No files found!</strong> error in the log comes from Get file
names and doesn’t stop the pipeline.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/detectlanguage.html
b/manual/next/pipeline/transforms/detectlanguage.html
index 2285989a6c..241d6745c0 100644
--- a/manual/next/pipeline/transforms/detectlanguage.html
+++ b/manual/next/pipeline/transforms/detectlanguage.html
@@ -3104,11 +3104,77 @@
<td class="tableblock halign-left valign-top"><p class="tableblock">Name of
the transform.
This name must be unique throughout the pipeline.</p></td>
</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Corpus
text field</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The field
with the text to examine.
+A variable is allowed.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use
Parallelism</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Detect the
language of several rows at the same time.
+The rows can leave the transform in another order than they came in.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Low
accuracy mode</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use only
the three-letter language models.
+This is faster and uses far less memory, but it is less accurate: short texts
of a few words are often given the wrong language.
+Use it for longer texts.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Languages
to detect</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The
languages the text can be in, by their Lingua name (<code>ENGLISH</code>,
<code>FRENCH</code>, …​).
+A variable is allowed.
+Leave the list empty to detect every spoken language.
+List at least two languages.</p></td>
+</tr>
</tbody>
</table>
</div>
</div>
<div class="sect1">
+<h2 id="_output_fields"><a class="link" href="#_output_fields">Output
fields</a></h2>
+<div class="sectionbody">
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Field</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">detected_language</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The name
of the detected language, for example <code>ENGLISH</code>, or
<code>UNKNOWN</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">detected_language_confidence</p></td>
+<td class="tableblock halign-left valign-top"><p
class="tableblock">Lingua’s score for that language, relative to the
other languages.
+The detected language is the one with the highest score, so the value is
always 1: it is not a probability and does not tell how sure the detection is.
+Empty when the language is <code>UNKNOWN</code>.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>A row with an empty text passes with <code>UNKNOWN</code> and no confidence.
+The language is also <code>UNKNOWN</code> when the two most likely languages
score exactly the same.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_performance"><a class="link" href="#_performance">Performance</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Detecting every spoken language loads the models of all 74 spoken
languages: about 1.4 GB of memory, released when the transform finishes.
+When you know which languages the data can contain, list them: with five
languages the transform is about 2.5 times as fast and needs about 175 MB.
+A text in a language that is not listed is given one of the listed languages,
or <code>UNKNOWN</code>.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
<h2 id="_installation"><a class="link"
href="#_installation">Installation</a></h2>
<div class="sectionbody">
<div class="paragraph">
diff --git a/manual/next/pipeline/transforms/dimensionlookup.html
b/manual/next/pipeline/transforms/dimensionlookup.html
index 5534695376..3657ac78b0 100644
--- a/manual/next/pipeline/transforms/dimensionlookup.html
+++ b/manual/next/pipeline/transforms/dimensionlookup.html
@@ -3470,6 +3470,15 @@ The result can be one of the following situations:</p>
</div>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/dimensionlookup-basic.hpl</code>,
which loads five rows for three customers into the <code>dim_sample</code>
dimension of the <code>hop-samples</code> connection.
+A new <code>street</code>, <code>number</code> or <code>zip</code> adds a
version, a new <code>name</code> is written to every version (punch through),
and a new <code>city</code> only changes the latest version.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/dynamicsqlrow.html
b/manual/next/pipeline/transforms/dynamicsqlrow.html
index a0c1a86626..bd9ae471bc 100644
--- a/manual/next/pipeline/transforms/dynamicsqlrow.html
+++ b/manual/next/pipeline/transforms/dynamicsqlrow.html
@@ -3123,6 +3123,14 @@ I mean any statement that returns the same row
structure.</p></td>
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/dynamic-sql-row.hpl</code>, which
finds the tables with an <code>ID</code> column in the <code>PUBLIC</code>
schema of the <code>hop-samples</code> connection, builds a query for each
table in a JavaScript transform, and writes each table name and id to the
log.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/edi2xml.html
b/manual/next/pipeline/transforms/edi2xml.html
index 7eb5470692..4efff7c526 100644
--- a/manual/next/pipeline/transforms/edi2xml.html
+++ b/manual/next/pipeline/transforms/edi2xml.html
@@ -3116,6 +3116,14 @@ If the output field name is left empty, the EDI text is
going to be replaced by
</ul>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/edi-to-xml.hpl</code>, which loads
<code>files/edi-to-xml.txt</code> into a field and converts that EDI message to
XML in the field <code>xml</code>.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/emailinput.html
b/manual/next/pipeline/transforms/emailinput.html
index 591509aea0..b286995e5c 100644
--- a/manual/next/pipeline/transforms/emailinput.html
+++ b/manual/next/pipeline/transforms/emailinput.html
@@ -3284,6 +3284,15 @@ Mail for which the header matches your filter will be
retrieved, the rest is ign
</table>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/email-messages-input.hpl</code>,
which reads the messages in the <code>INBOX</code> folder of a mail server and
its subfolders.
+Create a mail server connection named <code>sample-mail</code> before you run
it.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/enhancedjsonoutput.html
b/manual/next/pipeline/transforms/enhancedjsonoutput.html
index 910a6df96c..baf606f898 100644
--- a/manual/next/pipeline/transforms/enhancedjsonoutput.html
+++ b/manual/next/pipeline/transforms/enhancedjsonoutput.html
@@ -3369,6 +3369,14 @@ The Group Key section above shows that file shape.</p>
</div>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has
<code>transforms/json-output-generate-nested-structure.hpl</code>: a first
Enhanced JSON output groups the <code>field2</code> and <code>field3</code>
values of nine rows by <code>field1</code>, and a second one puts those groups
as <code>details</code> in a single <code>results</code> JSON value.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/excelinput.html
b/manual/next/pipeline/transforms/excelinput.html
index 5a957015ac..d08e5e5256 100644
--- a/manual/next/pipeline/transforms/excelinput.html
+++ b/manual/next/pipeline/transforms/excelinput.html
@@ -3396,6 +3396,24 @@ Some of these are missing.</p>
</table>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><code>transforms/microsoft-excel-input.hpl</code>, which reads the
<code>id</code>, <code>title</code> and <code>author</code> of two books from
the <code>Books</code> sheet of
<code>files/microsoft-excel-input.xlsx</code>.</p>
+</li>
+<li>
+<p><code>pipelines/read-transactions.hpl</code>, which reads the files in
<code>pipelines/input/</code> that match the regular expression
<code>file-.*.xlsx</code>, and uses the sheet row number to split the header
row of each file from its data rows.</p>
+</li>
+</ul>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/excelwriter.html
b/manual/next/pipeline/transforms/excelwriter.html
index dd4932e8cf..e7722bd2c6 100644
--- a/manual/next/pipeline/transforms/excelwriter.html
+++ b/manual/next/pipeline/transforms/excelwriter.html
@@ -3437,6 +3437,14 @@ Known differences:</p>
</ul>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/microsoft-excel-writer.hpl</code>,
which writes two books from a Data grid to the <code>Books</code> sheet of
<code>output/microsoft-excel-writer.xlsx</code>.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execinfo.html
b/manual/next/pipeline/transforms/execinfo.html
index 9f2113f534..475f7b4b00 100644
--- a/manual/next/pipeline/transforms/execinfo.html
+++ b/manual/next/pipeline/transforms/execinfo.html
@@ -3255,6 +3255,14 @@ First use Generate rows to create your fields and values
to use (children: Boole
<p><strong>Find previous successful execution</strong> is the operation to use
for incremental extracts: it answers “when did this workflow or pipeline last
succeed?” so a <a href="tableinput.html" class="xref page">Table Input</a> can
filter on a timestamp or increasing ID. How to wire that (or write the
watermark yourself on the success path) is in <a
href="../../how-to-guides/cdc-filtered-selection.html" class="xref page">CDC:
filtered selection</a>.</p>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execution-information.hpl</code>,
which lists up to 10 execution IDs from the execution information location
<code>local</code>.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execprocess.html
b/manual/next/pipeline/transforms/execprocess.html
index 987be1541e..59e7e0bedd 100644
--- a/manual/next/pipeline/transforms/execprocess.html
+++ b/manual/next/pipeline/transforms/execprocess.html
@@ -3257,6 +3257,14 @@ This is the most reliable option for values that come
from your data, such as fi
</div>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-a-process.hpl</code>,
which runs <code>echo</code> with the arguments <code>hello</code> and
<code>world</code> from two fields and puts <code>hello world</code> in the
field <code>output</code> and the exit value in <code>Exit value</code>.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execsql.html
b/manual/next/pipeline/transforms/execsql.html
index 7cc4c4d9de..30b788ff95 100644
--- a/manual/next/pipeline/transforms/execsql.html
+++ b/manual/next/pipeline/transforms/execsql.html
@@ -3244,6 +3244,14 @@ So the first question mark will be replaced by the first
parameter, the second q
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-sql-script.hpl</code>,
which runs a <code>CREATE TABLE IF NOT EXISTS employees</code> statement once
through the <code>hop-samples</code> connection.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execsqlrow.html
b/manual/next/pipeline/transforms/execsqlrow.html
index 9f18486e4a..dc6c427463 100644
--- a/manual/next/pipeline/transforms/execsqlrow.html
+++ b/manual/next/pipeline/transforms/execsqlrow.html
@@ -3142,6 +3142,14 @@
For good performance turn to dedicated transforms like Table Output (insert
into), Table Input (Select), Update, Delete, etc.</p>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-row-sql-script.hpl</code>,
which runs the <code>DELETE</code> and the <code>UPDATE</code> statement in the
two rows of a Data grid against the <code>employees</code> table of the
<code>hop-samples</code> connection, and adds to each row the number of rows
the statement updated, inserted, deleted and read.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/exectests.html
b/manual/next/pipeline/transforms/exectests.html
index 8e9cb2823a..2dc738f7b3 100644
--- a/manual/next/pipeline/transforms/exectests.html
+++ b/manual/next/pipeline/transforms/exectests.html
@@ -3124,6 +3124,14 @@ This option is only available when the transform
receives input.</p></td>
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-unit-tests.hpl</code>,
which runs the unit test <code>run-pipeline-tests-unit</code> and returns one
row for the check against its golden data set and one for the test as a
whole.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/fake.html
b/manual/next/pipeline/transforms/fake.html
index a3e91305c1..ac39c7029b 100644
--- a/manual/next/pipeline/transforms/fake.html
+++ b/manual/next/pipeline/transforms/fake.html
@@ -4505,6 +4505,14 @@
</ul>
</div>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has
<code>transforms/fake-data-generate-person-record.hpl</code>, which generates
1,000 fake person records: a name, a birthday, a blood group, a credit card
number, a cell phone number, a favourite book, a workplace, a programming
language, a digit and a random number.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/fileexists.html
b/manual/next/pipeline/transforms/fileexists.html
index 461812b3c0..93c22c2445 100644
--- a/manual/next/pipeline/transforms/fileexists.html
+++ b/manual/next/pipeline/transforms/fileexists.html
@@ -3114,6 +3114,14 @@
</tbody>
</table>
</div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/file-exists-basic.hpl</code>,
which checks whether <code>files/customers-100.txt</code> exists, adds the
result and the file type, and writes the row to the log through <strong>log
exists</strong> or <strong>log does not exist</strong>.</p>
+</div>
+</div>
</div>
</article>
<nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/maskfields.html
b/manual/next/pipeline/transforms/maskfields.html
index eff1d2a7a1..ae6345226a 100644
--- a/manual/next/pipeline/transforms/maskfields.html
+++ b/manual/next/pipeline/transforms/maskfields.html
@@ -3123,6 +3123,47 @@
<div class="paragraph">
<p>When a field cannot be masked and the transform has error handling, the
error row has that field and every masked field after it set to null.</p>
</div>
+<div class="sect2">
+<h3 id="_data_catalog"><a class="link" href="#_data_catalog">Data
catalog</a></h3>
+<div class="paragraph">
+<p>These options mask fields by the tags and glossary terms a <a
href="../../metadata-types/data-catalog.html" class="xref page">Data
Catalog</a>, such as DataHub, has on them. They are experimental. Leave
<strong>Data catalog</strong> empty not to use them.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Option</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Data
catalog</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The data
catalog that says which fields carry a tag or term.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Dataset
type, Dataset connection, Dataset schema, Dataset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The
dataset whose fields the catalog labels: a table of a relational connection, a
file, or the id of the dataset in the catalog. A table and a file are found the
way <a href="../../metadata-types/data-catalog.html#_how_datasets_are_found"
class="xref page">a data catalog finds datasets</a>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Tag or
term, Masking rule</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Every
incoming field that the catalog labels with the tag or glossary term on a row
is masked with that row’s Masking pattern. A tag is matched by its name
or URN (<code>PII</code>, <code>urn:li:tag:PII</code>), a term by its name, its
URN or its path in the glossary, the term groups above it followed by its name
(<code>Personal</code>, <code>Classification.Personal</code>), without regard
to case. The first row th [...]
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>When the transform starts, it reads the fields of the dataset from the
catalog. When the first row arrives, it masks every incoming field whose name
matches a labelled field, without regard to case: <code>EMAIL</code> in the
stream matches <code>email</code> in the catalog. A field that only shows up at
run time is masked too. The log has a line for every field masked this way,
with its tag or term.</p>
+</div>
+<div class="paragraph">
+<p>A field that also has a row in the fields table keeps the pattern of that
row.</p>
+</div>
+<div class="paragraph">
+<p>When the catalog can’t be reached, refuses the token, doesn’t
have the dataset or has no fields for it, or a tag or term holds a variable
that is not set, the transform doesn’t start: it doesn’t guess
which fields to mask.</p>
+</div>
+</div>
</div>
</div>
<div class="sect1">
diff --git a/manual/next/technology/datahub/index.html
b/manual/next/technology/datahub/index.html
index ab1cd17bb9..84cd5ce045 100644
--- a/manual/next/technology/datahub/index.html
+++ b/manual/next/technology/datahub/index.html
@@ -3066,7 +3066,8 @@ neither the tasks nor what they read and write.</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">A run of
any of these</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">A
<strong>run</strong> (DataProcessInstance) named <code><name>
<time></code>, for example <code>nightly/Load orders
2026-10-18T07:34Z</code>, with its start, its end, its outcome (success or
failure) and the run that
-started it. A run whose end Hop never reported, such as an action that stopped
with an error, is
+started it. A transform’s run lists its row counts
(<code>linesInput</code>, <code>linesOutput</code>, <code>linesUpdated</code>,
+<code>linesRejected</code>, …​) in its properties, a
pipeline’s run the totals of its transforms. A run whose end Hop never
reported, such as an action that stopped with an error, is
ended as failed when the sink shuts down.</p></td>
</tr>
<tr>
@@ -3087,6 +3088,20 @@ a local file as its path on <code>file</code>.</p></td>
<td class="tableblock halign-left valign-top"><p
class="tableblock">Column-level lineage, on the task and on the
dataset.</p></td>
</tr>
<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">A file or
table a run wrote</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">An
<strong>operation</strong> on the dataset when the run succeeds:
<code>INSERT</code>, <code>UPDATE</code> or <code>DELETE</code>, or
<code>CREATE</code>,
+<code>DROP</code> or <code>ALTER</code> for a dataset created, deleted, moved
or renamed, with the number of rows when Hop
+knows them (see <a href="../openlineage/index.html#_row_counts" class="xref
page">Row counts</a>). A pipeline reports
+one operation per dataset when it completes, with the rows of all the
transforms and copies that
+wrote it; an action reports the files it wrote when it completes. DataHub
shows when a dataset last
+changed from its operations, and the freshness check of a
+<a href="../../workflow/actions/checkdatacatalog.html" class="xref page">Check
data catalog</a> action reads them, so a dataset that
+only Hop writes has a freshness too. A failed or stopped run reports no
operation: a half-written table
+doesn’t look up to date. Neither does a run that inserted, updated or
deleted no rows; a table
+truncated or a file rewritten with nothing written was emptied, and gets a
<code>DELETE</code> without a row
+count.</p></td>
+</tr>
+<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">A file
copied or moved (Copy files, Move files)</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">The copy
is linked to its own source as a <code>COPY</code> upstream, so copying several
files does not pair
every source with every copy. The run of the action still lists all of
them.</p></td>
diff --git a/manual/next/technology/openlineage/index.html
b/manual/next/technology/openlineage/index.html
index 4675e7d1f8..08f0cf3785 100644
--- a/manual/next/technology/openlineage/index.html
+++ b/manual/next/technology/openlineage/index.html
@@ -3306,6 +3306,46 @@ job facet with the subject type and the pipeline,
workflow, transform or action
A job name such as <code>nightly/Load orders</code> cannot be split back
reliably, so a collector that models
pipelines and transforms separately reads those names from the facet.</p>
</div>
+<div class="sect2">
+<h3 id="_row_counts"><a class="link" href="#_row_counts">Row counts</a></h3>
+<div class="paragraph">
+<p>A transform’s <code>COMPLETE</code> or <code>FAIL</code> event
carries the row counters of that transform copy in a
+<code>hopRowCounts</code> run facet: <code>linesRead</code> and
<code>linesWritten</code> (rows from and to other transforms),
+<code>linesInput</code> and <code>linesOutput</code> (rows read from and
written to files and tables), <code>linesUpdated</code> and
+<code>linesRejected</code>. These are the transform’s own counters, the
numbers it logs as <code>I=</code>, <code>O=</code>, <code>R=</code>,
+<code>W=</code> and <code>U=</code>; Text File Output counts its header and
footer lines as output too, so a file with a
+header shows one line more than it has rows. A pipeline’s event carries
the sum over all its
+transforms and copies of the counters that add up: <code>linesInput</code>,
<code>linesOutput</code>, <code>linesUpdated</code> and
+<code>linesRejected</code>. That sum is not the pipeline result Hop logs,
which takes the highest value of each
+counter, and a Metadata Injection transform counts the rows of the pipeline it
runs too. Only the
+local pipeline engine sends counters; Beam, Spark and remote runs don’t
have final ones when their
+run ends.</p>
+</div>
+<div class="paragraph">
+<p>A dataset gets the standard <code>inputStatistics</code> or
<code>outputStatistics</code> facet with its <code>rowCount</code> when
+the count is known to be its own: the transforms that read (or wrote) it read
(or wrote) nothing
+else. The rows written are the rows output plus the rows updated. A transform
that writes several
+files or tables, such as Text File Output with a file per copy or Table Output
with the table in a
+field, gives them no row count rather than a wrong one. On a pipeline’s
event a dataset’s count is
+the sum over its transforms and their copies; with transform jobs on, each
copy’s event counts the
+rows of that copy. Rows a transform deleted from a table (Delete, an Execute
SQL with a single
+<code>DELETE</code> statement) are not rows written; a table emptied and
reloaded in one run counts the rows
+loaded. An Execute SQL script of several statements gives no count, as its
counters add all of them
+up, and neither does a transform that counts 0 while it does handle rows (not
every transform counts
+what it reads or writes).</p>
+</div>
+<div class="paragraph">
+<p>A pipeline’s <code>COMPLETE</code> or <code>FAIL</code> event also
lists every dataset the run wrote, once, in a
+<code>hopWrites</code> run facet: its <code>namespace</code> and
<code>name</code>, its <code>operationType</code> (<code>INSERT</code>,
<code>UPDATE</code>,
+<code>DELETE</code>, <code>CREATE</code>, <code>DROP</code> or
<code>ALTER</code>) and, when known, its <code>rows</code>. The DataHub sink
turns these
+into operations. A pipeline that was stopped (Stop, Abort, or a failing parent
pipeline stopping
+it) says so with <code>stopped</code> in its <code>hopRowCounts</code> facet
and lists no writes.</p>
+</div>
+<div class="paragraph">
+<p>Apache Beam and Spark run the transforms of a pipeline in small local
pipelines on their workers.
+With lineage on in the JVM of those workers, as with the Beam DirectRunner or
Spark in local mode,
+each of those reports a run of its own, with the counts and writes of its
bundle or partition.</p>
+</div>
<div class="paragraph">
<p>The run id is the executing subject’s log-channel id. By default file
I/O is correlated to
the parent pipeline run, so inputs and outputs appear on the pipeline job in
Marquez. Every run also
@@ -3314,6 +3354,7 @@ its workflow, and a pipeline’s at the action that
ran it (or at the parent
sub-pipeline). Actions are jobs of their own, named
<code><workflow>/<action></code>, so a workflow with several
actions no longer files them all under the workflow’s job.</p>
</div>
+</div>
<div class="sect2">
<h3 id="job-hierarchy"><a class="link" href="#job-hierarchy">Job
hierarchy</a></h3>
<div class="paragraph">