This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hop-website.git


The following commit(s) were added to refs/heads/asf-site by this push:
     new 8973cfbf64 Website updated to ce72275
8973cfbf64 is described below

commit 8973cfbf64d855021b40d9aade1d8672882c730c
Author: jenkins <[email protected]>
AuthorDate: Mon Nov 16 19:47:53 2026 +0000

    Website updated to ce72275
---
 .../next/plugin-types/metadata-configuration.html  |  5 +-
 manual/next/metadata-types/data-catalog.html       |  9 ++-
 manual/next/pipeline/transforms/delay.html         |  8 +++
 manual/next/pipeline/transforms/delete.html        |  8 +++
 .../pipeline/transforms/detectemptystream.html     |  9 +++
 .../next/pipeline/transforms/detectlanguage.html   | 66 ++++++++++++++++++++++
 .../next/pipeline/transforms/dimensionlookup.html  |  9 +++
 manual/next/pipeline/transforms/dynamicsqlrow.html |  8 +++
 manual/next/pipeline/transforms/edi2xml.html       |  8 +++
 manual/next/pipeline/transforms/emailinput.html    |  9 +++
 .../pipeline/transforms/enhancedjsonoutput.html    |  8 +++
 manual/next/pipeline/transforms/excelinput.html    | 18 ++++++
 manual/next/pipeline/transforms/excelwriter.html   |  8 +++
 manual/next/pipeline/transforms/execinfo.html      |  8 +++
 manual/next/pipeline/transforms/execprocess.html   |  8 +++
 manual/next/pipeline/transforms/execsql.html       |  8 +++
 manual/next/pipeline/transforms/execsqlrow.html    |  8 +++
 manual/next/pipeline/transforms/exectests.html     |  8 +++
 manual/next/pipeline/transforms/fake.html          |  8 +++
 manual/next/pipeline/transforms/fileexists.html    |  8 +++
 manual/next/pipeline/transforms/maskfields.html    | 41 ++++++++++++++
 manual/next/technology/datahub/index.html          | 17 +++++-
 manual/next/technology/openlineage/index.html      | 41 ++++++++++++++
 23 files changed, 323 insertions(+), 5 deletions(-)

diff --git a/dev-manual/next/plugin-types/metadata-configuration.html 
b/dev-manual/next/plugin-types/metadata-configuration.html
index e0210dc8cf..aff9fbc0d6 100644
--- a/dev-manual/next/plugin-types/metadata-configuration.html
+++ b/dev-manual/next/plugin-types/metadata-configuration.html
@@ -795,7 +795,7 @@ The API is experimental.</p>
 </dd>
 <dt class="hdlist1">Implement</dt>
 <dd>
-<p><code>IDataCatalog</code>, plus the <code>ICatalogCapability</code> 
interfaces for what the catalog can answer: 
<code>IDeprecationCapability</code>, <code>IFreshnessCapability</code>, 
<code>IAssertionCapability</code></p>
+<p><code>IDataCatalog</code>, plus the <code>ICatalogCapability</code> 
interfaces for what the catalog can answer: 
<code>IDeprecationCapability</code>, <code>IFreshnessCapability</code>, 
<code>IAssertionCapability</code>, <code>ISchemaCapability</code></p>
 </dd>
 <dt class="hdlist1">Plugin type</dt>
 <dd>
@@ -820,7 +820,8 @@ A catalog that can&#8217;t answer is a normal outcome, not 
an exception; the cal
 <div class="paragraph">
 <p>A dataset is a <code>DatasetRef</code>.
 Build it with <code>DatasetRef.ofTable(&#8230;&#8203;)</code> or 
<code>ofFile(&#8230;&#8203;)</code>: those give a table or a file the 
OpenLineage namespace and name that lineage gives it, so <code>resolve()</code> 
finds the dataset lineage wrote.
-<code>ofCatalogId(&#8230;&#8203;)</code> names a dataset by the 
catalog&#8217;s own id.</p>
+<code>ofCatalogId(&#8230;&#8203;)</code> names a dataset by the 
catalog&#8217;s own id.
+A transform or action that lets the user pick a dataset stores a 
<code>CatalogDataset</code> (type, connection, schema, name) and calls 
<code>toRef(variables, metadataProvider)</code>, which also reports a dataset 
that can&#8217;t be named.</p>
 </div>
 </div>
 </div>
diff --git a/manual/next/metadata-types/data-catalog.html 
b/manual/next/metadata-types/data-catalog.html
index 2813595ba6..12abae4029 100644
--- a/manual/next/metadata-types/data-catalog.html
+++ b/manual/next/metadata-types/data-catalog.html
@@ -3006,7 +3006,8 @@
 The catalog stays the owner of that metadata. Hop only reads it.</p>
 </div>
 <div class="paragraph">
-<p>The <a href="../workflow/actions/checkdatacatalog.html" class="xref 
page">Check data catalog</a> action uses a Data Catalog to check datasets 
before a workflow goes on.</p>
+<p>The <a href="../workflow/actions/checkdatacatalog.html" class="xref 
page">Check data catalog</a> action uses a Data Catalog to check datasets 
before a workflow goes on.
+The <a href="../pipeline/transforms/maskfields.html" class="xref page">Mask 
fields</a> transform can use one to mask the fields that carry a tag or 
glossary term, such as <code>PII</code>.</p>
 </div>
 <div class="paragraph">
 <p>This metadata type is experimental: its options and what the catalog types 
can answer may still change.</p>
@@ -3147,12 +3148,16 @@ Use the same one as the DataHub lineage sink 
(<code>HOP_LINEAGE_DATAHUB_ENV</cod
 <li>
 <p><strong>Last change</strong>: the newest operation DataHub has for the 
dataset.
 DataHub learns about operations from the sources that report them, such as the 
usage ingestion of some warehouses or dbt.
-The DataHub lineage sink of Hop doesn&#8217;t report them yet, so a table that 
only Hop writes has none.</p>
+The <a href="../technology/datahub/index.html" class="xref page">DataHub 
lineage sink</a> reports one for each file and table a successful Hop run 
writes, so a dataset that only Hop writes has a last change too.</p>
 </li>
 <li>
 <p><strong>Assertions</strong>: the latest run of each assertion on the 
dataset, up to 100 assertions.
 The assertions themselves are defined and run in DataHub.</p>
 </li>
+<li>
+<p><strong>Fields</strong>: the fields of the dataset&#8217;s schema, with the 
tags and glossary terms on them, both the ones that came with the ingested 
schema and the ones added in DataHub.
+The <a href="../pipeline/transforms/maskfields.html" class="xref page">Mask 
fields</a> transform uses them.</p>
+</li>
 </ul>
 </div>
 </div>
diff --git a/manual/next/pipeline/transforms/delay.html 
b/manual/next/pipeline/transforms/delay.html
index 9877875099..ab04d3c650 100644
--- a/manual/next/pipeline/transforms/delay.html
+++ b/manual/next/pipeline/transforms/delay.html
@@ -3110,6 +3110,14 @@
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/delay-row-basic.hpl</code>, which 
delays each of 10 generated rows by 5 seconds, so a line appears in the log 
every 5 seconds.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/delete.html 
b/manual/next/pipeline/transforms/delete.html
index 622406ece6..bab48158b4 100644
--- a/manual/next/pipeline/transforms/delete.html
+++ b/manual/next/pipeline/transforms/delete.html
@@ -3121,6 +3121,14 @@ This is important for data sources that allow for table 
names with periods in th
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/delete.hpl</code>, which deletes 
the employees with <code>id</code> 1 and 2 from the <code>employees</code> 
table of the <code>hop-samples</code> connection.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/detectemptystream.html 
b/manual/next/pipeline/transforms/detectemptystream.html
index b71efa4441..9bab6b83af 100644
--- a/manual/next/pipeline/transforms/detectemptystream.html
+++ b/manual/next/pipeline/transforms/detectemptystream.html
@@ -3096,6 +3096,15 @@ This name must be unique throughout the 
pipeline.</p></td>
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has 
<code>transforms/detect-empty-stream-basic.hpl</code>: Get file names looks for 
a file that doesn&#8217;t exist, so Detect empty stream passes on one row with 
null values for the Get file names fields.
+The <strong>No files found!</strong> error in the log comes from Get file 
names and doesn&#8217;t stop the pipeline.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/detectlanguage.html 
b/manual/next/pipeline/transforms/detectlanguage.html
index 2285989a6c..241d6745c0 100644
--- a/manual/next/pipeline/transforms/detectlanguage.html
+++ b/manual/next/pipeline/transforms/detectlanguage.html
@@ -3104,11 +3104,77 @@
 <td class="tableblock halign-left valign-top"><p class="tableblock">Name of 
the transform.
 This name must be unique throughout the pipeline.</p></td>
 </tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Corpus 
text field</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The field 
with the text to examine.
+A variable is allowed.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use 
Parallelism</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Detect the 
language of several rows at the same time.
+The rows can leave the transform in another order than they came in.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Low 
accuracy mode</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Use only 
the three-letter language models.
+This is faster and uses far less memory, but it is less accurate: short texts 
of a few words are often given the wrong language.
+Use it for longer texts.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Languages 
to detect</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The 
languages the text can be in, by their Lingua name (<code>ENGLISH</code>, 
<code>FRENCH</code>, &#8230;&#8203;).
+A variable is allowed.
+Leave the list empty to detect every spoken language.
+List at least two languages.</p></td>
+</tr>
 </tbody>
 </table>
 </div>
 </div>
 <div class="sect1">
+<h2 id="_output_fields"><a class="link" href="#_output_fields">Output 
fields</a></h2>
+<div class="sectionbody">
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Field</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p 
class="tableblock">detected_language</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The name 
of the detected language, for example <code>ENGLISH</code>, or 
<code>UNKNOWN</code>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p 
class="tableblock">detected_language_confidence</p></td>
+<td class="tableblock halign-left valign-top"><p 
class="tableblock">Lingua&#8217;s score for that language, relative to the 
other languages.
+The detected language is the one with the highest score, so the value is 
always 1: it is not a probability and does not tell how sure the detection is.
+Empty when the language is <code>UNKNOWN</code>.</p></td>
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>A row with an empty text passes with <code>UNKNOWN</code> and no confidence.
+The language is also <code>UNKNOWN</code> when the two most likely languages 
score exactly the same.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
+<h2 id="_performance"><a class="link" href="#_performance">Performance</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>Detecting every spoken language loads the models of all 74 spoken 
languages: about 1.4 GB of memory, released when the transform finishes.
+When you know which languages the data can contain, list them: with five 
languages the transform is about 2.5 times as fast and needs about 175 MB.
+A text in a language that is not listed is given one of the listed languages, 
or <code>UNKNOWN</code>.</p>
+</div>
+</div>
+</div>
+<div class="sect1">
 <h2 id="_installation"><a class="link" 
href="#_installation">Installation</a></h2>
 <div class="sectionbody">
 <div class="paragraph">
diff --git a/manual/next/pipeline/transforms/dimensionlookup.html 
b/manual/next/pipeline/transforms/dimensionlookup.html
index 5534695376..3657ac78b0 100644
--- a/manual/next/pipeline/transforms/dimensionlookup.html
+++ b/manual/next/pipeline/transforms/dimensionlookup.html
@@ -3470,6 +3470,15 @@ The result can be one of the following situations:</p>
 </div>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/dimensionlookup-basic.hpl</code>, 
which loads five rows for three customers into the <code>dim_sample</code> 
dimension of the <code>hop-samples</code> connection.
+A new <code>street</code>, <code>number</code> or <code>zip</code> adds a 
version, a new <code>name</code> is written to every version (punch through), 
and a new <code>city</code> only changes the latest version.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/dynamicsqlrow.html 
b/manual/next/pipeline/transforms/dynamicsqlrow.html
index a0c1a86626..bd9ae471bc 100644
--- a/manual/next/pipeline/transforms/dynamicsqlrow.html
+++ b/manual/next/pipeline/transforms/dynamicsqlrow.html
@@ -3123,6 +3123,14 @@ I mean any statement that returns the same row 
structure.</p></td>
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/dynamic-sql-row.hpl</code>, which 
finds the tables with an <code>ID</code> column in the <code>PUBLIC</code> 
schema of the <code>hop-samples</code> connection, builds a query for each 
table in a JavaScript transform, and writes each table name and id to the 
log.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/edi2xml.html 
b/manual/next/pipeline/transforms/edi2xml.html
index 7eb5470692..4efff7c526 100644
--- a/manual/next/pipeline/transforms/edi2xml.html
+++ b/manual/next/pipeline/transforms/edi2xml.html
@@ -3116,6 +3116,14 @@ If the output field name is left empty, the EDI text is 
going to be replaced by
 </ul>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/edi-to-xml.hpl</code>, which loads 
<code>files/edi-to-xml.txt</code> into a field and converts that EDI message to 
XML in the field <code>xml</code>.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/emailinput.html 
b/manual/next/pipeline/transforms/emailinput.html
index 591509aea0..b286995e5c 100644
--- a/manual/next/pipeline/transforms/emailinput.html
+++ b/manual/next/pipeline/transforms/emailinput.html
@@ -3284,6 +3284,15 @@ Mail for which the header matches your filter will be 
retrieved, the rest is ign
 </table>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/email-messages-input.hpl</code>, 
which reads the messages in the <code>INBOX</code> folder of a mail server and 
its subfolders.
+Create a mail server connection named <code>sample-mail</code> before you run 
it.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/enhancedjsonoutput.html 
b/manual/next/pipeline/transforms/enhancedjsonoutput.html
index 910a6df96c..baf606f898 100644
--- a/manual/next/pipeline/transforms/enhancedjsonoutput.html
+++ b/manual/next/pipeline/transforms/enhancedjsonoutput.html
@@ -3369,6 +3369,14 @@ The Group Key section above shows that file shape.</p>
 </div>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has 
<code>transforms/json-output-generate-nested-structure.hpl</code>: a first 
Enhanced JSON output groups the <code>field2</code> and <code>field3</code> 
values of nine rows by <code>field1</code>, and a second one puts those groups 
as <code>details</code> in a single <code>results</code> JSON value.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/excelinput.html 
b/manual/next/pipeline/transforms/excelinput.html
index 5a957015ac..d08e5e5256 100644
--- a/manual/next/pipeline/transforms/excelinput.html
+++ b/manual/next/pipeline/transforms/excelinput.html
@@ -3396,6 +3396,24 @@ Some of these are missing.</p>
 </table>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has:</p>
+</div>
+<div class="ulist">
+<ul>
+<li>
+<p><code>transforms/microsoft-excel-input.hpl</code>, which reads the 
<code>id</code>, <code>title</code> and <code>author</code> of two books from 
the <code>Books</code> sheet of 
<code>files/microsoft-excel-input.xlsx</code>.</p>
+</li>
+<li>
+<p><code>pipelines/read-transactions.hpl</code>, which reads the files in 
<code>pipelines/input/</code> that match the regular expression 
<code>file-.*.xlsx</code>, and uses the sheet row number to split the header 
row of each file from its data rows.</p>
+</li>
+</ul>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/excelwriter.html 
b/manual/next/pipeline/transforms/excelwriter.html
index dd4932e8cf..e7722bd2c6 100644
--- a/manual/next/pipeline/transforms/excelwriter.html
+++ b/manual/next/pipeline/transforms/excelwriter.html
@@ -3437,6 +3437,14 @@ Known differences:</p>
 </ul>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/microsoft-excel-writer.hpl</code>, 
which writes two books from a Data grid to the <code>Books</code> sheet of 
<code>output/microsoft-excel-writer.xlsx</code>.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execinfo.html 
b/manual/next/pipeline/transforms/execinfo.html
index 9f2113f534..475f7b4b00 100644
--- a/manual/next/pipeline/transforms/execinfo.html
+++ b/manual/next/pipeline/transforms/execinfo.html
@@ -3255,6 +3255,14 @@ First use Generate rows to create your fields and values 
to use (children: Boole
 <p><strong>Find previous successful execution</strong> is the operation to use 
for incremental extracts: it answers “when did this workflow or pipeline last 
succeed?” so a <a href="tableinput.html" class="xref page">Table Input</a> can 
filter on a timestamp or increasing ID. How to wire that (or write the 
watermark yourself on the success path) is in <a 
href="../../how-to-guides/cdc-filtered-selection.html" class="xref page">CDC: 
filtered selection</a>.</p>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execution-information.hpl</code>, 
which lists up to 10 execution IDs from the execution information location 
<code>local</code>.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execprocess.html 
b/manual/next/pipeline/transforms/execprocess.html
index 987be1541e..59e7e0bedd 100644
--- a/manual/next/pipeline/transforms/execprocess.html
+++ b/manual/next/pipeline/transforms/execprocess.html
@@ -3257,6 +3257,14 @@ This is the most reliable option for values that come 
from your data, such as fi
 </div>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-a-process.hpl</code>, 
which runs <code>echo</code> with the arguments <code>hello</code> and 
<code>world</code> from two fields and puts <code>hello world</code> in the 
field <code>output</code> and the exit value in <code>Exit value</code>.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execsql.html 
b/manual/next/pipeline/transforms/execsql.html
index 7cc4c4d9de..30b788ff95 100644
--- a/manual/next/pipeline/transforms/execsql.html
+++ b/manual/next/pipeline/transforms/execsql.html
@@ -3244,6 +3244,14 @@ So the first question mark will be replaced by the first 
parameter, the second q
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-sql-script.hpl</code>, 
which runs a <code>CREATE TABLE IF NOT EXISTS employees</code> statement once 
through the <code>hop-samples</code> connection.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/execsqlrow.html 
b/manual/next/pipeline/transforms/execsqlrow.html
index 9f18486e4a..dc6c427463 100644
--- a/manual/next/pipeline/transforms/execsqlrow.html
+++ b/manual/next/pipeline/transforms/execsqlrow.html
@@ -3142,6 +3142,14 @@
 For good performance turn to dedicated transforms like Table Output (insert 
into), Table Input (Select), Update, Delete, etc.</p>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-row-sql-script.hpl</code>, 
which runs the <code>DELETE</code> and the <code>UPDATE</code> statement in the 
two rows of a Data grid against the <code>employees</code> table of the 
<code>hop-samples</code> connection, and adds to each row the number of rows 
the statement updated, inserted, deleted and read.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/exectests.html 
b/manual/next/pipeline/transforms/exectests.html
index 8e9cb2823a..2dc738f7b3 100644
--- a/manual/next/pipeline/transforms/exectests.html
+++ b/manual/next/pipeline/transforms/exectests.html
@@ -3124,6 +3124,14 @@ This option is only available when the transform 
receives input.</p></td>
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/execute-unit-tests.hpl</code>, 
which runs the unit test <code>run-pipeline-tests-unit</code> and returns one 
row for the check against its golden data set and one for the test as a 
whole.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/fake.html 
b/manual/next/pipeline/transforms/fake.html
index a3e91305c1..ac39c7029b 100644
--- a/manual/next/pipeline/transforms/fake.html
+++ b/manual/next/pipeline/transforms/fake.html
@@ -4505,6 +4505,14 @@
 </ul>
 </div>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has 
<code>transforms/fake-data-generate-person-record.hpl</code>, which generates 
1,000 fake person records: a name, a birthday, a blood group, a credit card 
number, a cell phone number, a favourite book, a workplace, a programming 
language, a digit and a random number.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/fileexists.html 
b/manual/next/pipeline/transforms/fileexists.html
index 461812b3c0..93c22c2445 100644
--- a/manual/next/pipeline/transforms/fileexists.html
+++ b/manual/next/pipeline/transforms/fileexists.html
@@ -3114,6 +3114,14 @@
 </tbody>
 </table>
 </div>
+</div>
+<div class="sect1">
+<h2 id="_samples"><a class="link" href="#_samples">Samples</a></h2>
+<div class="sectionbody">
+<div class="paragraph">
+<p>The samples project has <code>transforms/file-exists-basic.hpl</code>, 
which checks whether <code>files/customers-100.txt</code> exists, adds the 
result and the file type, and writes the row to the log through <strong>log 
exists</strong> or <strong>log does not exist</strong>.</p>
+</div>
+</div>
 </div>
     </article>
 <nav class="pagination" data-pagefind-ignore aria-label="Page">
diff --git a/manual/next/pipeline/transforms/maskfields.html 
b/manual/next/pipeline/transforms/maskfields.html
index eff1d2a7a1..ae6345226a 100644
--- a/manual/next/pipeline/transforms/maskfields.html
+++ b/manual/next/pipeline/transforms/maskfields.html
@@ -3123,6 +3123,47 @@
 <div class="paragraph">
 <p>When a field cannot be masked and the transform has error handling, the 
error row has that field and every masked field after it set to null.</p>
 </div>
+<div class="sect2">
+<h3 id="_data_catalog"><a class="link" href="#_data_catalog">Data 
catalog</a></h3>
+<div class="paragraph">
+<p>These options mask fields by the tags and glossary terms a <a 
href="../../metadata-types/data-catalog.html" class="xref page">Data 
Catalog</a>, such as DataHub, has on them. They are experimental. Leave 
<strong>Data catalog</strong> empty not to use them.</p>
+</div>
+<table class="tableblock frame-all grid-all stretch">
+<colgroup>
+<col style="width: 50%;">
+<col style="width: 50%;">
+</colgroup>
+<thead>
+<tr>
+<th class="tableblock halign-left valign-top">Option</th>
+<th class="tableblock halign-left valign-top">Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Data 
catalog</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The data 
catalog that says which fields carry a tag or term.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Dataset 
type, Dataset connection, Dataset schema, Dataset</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">The 
dataset whose fields the catalog labels: a table of a relational connection, a 
file, or the id of the dataset in the catalog. A table and a file are found the 
way <a href="../../metadata-types/data-catalog.html#_how_datasets_are_found" 
class="xref page">a data catalog finds datasets</a>.</p></td>
+</tr>
+<tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Tag or 
term, Masking rule</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">Every 
incoming field that the catalog labels with the tag or glossary term on a row 
is masked with that row&#8217;s Masking pattern. A tag is matched by its name 
or URN (<code>PII</code>, <code>urn:li:tag:PII</code>), a term by its name, its 
URN or its path in the glossary, the term groups above it followed by its name 
(<code>Personal</code>, <code>Classification.Personal</code>), without regard 
to case. The first row th [...]
+</tr>
+</tbody>
+</table>
+<div class="paragraph">
+<p>When the transform starts, it reads the fields of the dataset from the 
catalog. When the first row arrives, it masks every incoming field whose name 
matches a labelled field, without regard to case: <code>EMAIL</code> in the 
stream matches <code>email</code> in the catalog. A field that only shows up at 
run time is masked too. The log has a line for every field masked this way, 
with its tag or term.</p>
+</div>
+<div class="paragraph">
+<p>A field that also has a row in the fields table keeps the pattern of that 
row.</p>
+</div>
+<div class="paragraph">
+<p>When the catalog can&#8217;t be reached, refuses the token, doesn&#8217;t 
have the dataset or has no fields for it, or a tag or term holds a variable 
that is not set, the transform doesn&#8217;t start: it doesn&#8217;t guess 
which fields to mask.</p>
+</div>
+</div>
 </div>
 </div>
 <div class="sect1">
diff --git a/manual/next/technology/datahub/index.html 
b/manual/next/technology/datahub/index.html
index ab1cd17bb9..84cd5ce045 100644
--- a/manual/next/technology/datahub/index.html
+++ b/manual/next/technology/datahub/index.html
@@ -3066,7 +3066,8 @@ neither the tasks nor what they read and write.</p></td>
 <td class="tableblock halign-left valign-top"><p class="tableblock">A run of 
any of these</p></td>
 <td class="tableblock halign-left valign-top"><p class="tableblock">A 
<strong>run</strong> (DataProcessInstance) named <code>&lt;name&gt; 
&lt;time&gt;</code>, for example <code>nightly/Load orders
 2026-10-18T07:34Z</code>, with its start, its end, its outcome (success or 
failure) and the run that
-started it. A run whose end Hop never reported, such as an action that stopped 
with an error, is
+started it. A transform&#8217;s run lists its row counts 
(<code>linesInput</code>, <code>linesOutput</code>, <code>linesUpdated</code>,
+<code>linesRejected</code>, &#8230;&#8203;) in its properties, a 
pipeline&#8217;s run the totals of its transforms. A run whose end Hop never 
reported, such as an action that stopped with an error, is
 ended as failed when the sink shuts down.</p></td>
 </tr>
 <tr>
@@ -3087,6 +3088,20 @@ a local file as its path on <code>file</code>.</p></td>
 <td class="tableblock halign-left valign-top"><p 
class="tableblock">Column-level lineage, on the task and on the 
dataset.</p></td>
 </tr>
 <tr>
+<td class="tableblock halign-left valign-top"><p class="tableblock">A file or 
table a run wrote</p></td>
+<td class="tableblock halign-left valign-top"><p class="tableblock">An 
<strong>operation</strong> on the dataset when the run succeeds: 
<code>INSERT</code>, <code>UPDATE</code> or <code>DELETE</code>, or 
<code>CREATE</code>,
+<code>DROP</code> or <code>ALTER</code> for a dataset created, deleted, moved 
or renamed, with the number of rows when Hop
+knows them (see <a href="../openlineage/index.html#_row_counts" class="xref 
page">Row counts</a>). A pipeline reports
+one operation per dataset when it completes, with the rows of all the 
transforms and copies that
+wrote it; an action reports the files it wrote when it completes. DataHub 
shows when a dataset last
+changed from its operations, and the freshness check of a
+<a href="../../workflow/actions/checkdatacatalog.html" class="xref page">Check 
data catalog</a> action reads them, so a dataset that
+only Hop writes has a freshness too. A failed or stopped run reports no 
operation: a half-written table
+doesn&#8217;t look up to date. Neither does a run that inserted, updated or 
deleted no rows; a table
+truncated or a file rewritten with nothing written was emptied, and gets a 
<code>DELETE</code> without a row
+count.</p></td>
+</tr>
+<tr>
 <td class="tableblock halign-left valign-top"><p class="tableblock">A file 
copied or moved (Copy files, Move files)</p></td>
 <td class="tableblock halign-left valign-top"><p class="tableblock">The copy 
is linked to its own source as a <code>COPY</code> upstream, so copying several 
files does not pair
 every source with every copy. The run of the action still lists all of 
them.</p></td>
diff --git a/manual/next/technology/openlineage/index.html 
b/manual/next/technology/openlineage/index.html
index 4675e7d1f8..08f0cf3785 100644
--- a/manual/next/technology/openlineage/index.html
+++ b/manual/next/technology/openlineage/index.html
@@ -3306,6 +3306,46 @@ job facet with the subject type and the pipeline, 
workflow, transform or action
 A job name such as <code>nightly/Load orders</code> cannot be split back 
reliably, so a collector that models
 pipelines and transforms separately reads those names from the facet.</p>
 </div>
+<div class="sect2">
+<h3 id="_row_counts"><a class="link" href="#_row_counts">Row counts</a></h3>
+<div class="paragraph">
+<p>A transform&#8217;s <code>COMPLETE</code> or <code>FAIL</code> event 
carries the row counters of that transform copy in a
+<code>hopRowCounts</code> run facet: <code>linesRead</code> and 
<code>linesWritten</code> (rows from and to other transforms),
+<code>linesInput</code> and <code>linesOutput</code> (rows read from and 
written to files and tables), <code>linesUpdated</code> and
+<code>linesRejected</code>. These are the transform&#8217;s own counters, the 
numbers it logs as <code>I=</code>, <code>O=</code>, <code>R=</code>,
+<code>W=</code> and <code>U=</code>; Text File Output counts its header and 
footer lines as output too, so a file with a
+header shows one line more than it has rows. A pipeline&#8217;s event carries 
the sum over all its
+transforms and copies of the counters that add up: <code>linesInput</code>, 
<code>linesOutput</code>, <code>linesUpdated</code> and
+<code>linesRejected</code>. That sum is not the pipeline result Hop logs, 
which takes the highest value of each
+counter, and a Metadata Injection transform counts the rows of the pipeline it 
runs too. Only the
+local pipeline engine sends counters; Beam, Spark and remote runs don&#8217;t 
have final ones when their
+run ends.</p>
+</div>
+<div class="paragraph">
+<p>A dataset gets the standard <code>inputStatistics</code> or 
<code>outputStatistics</code> facet with its <code>rowCount</code> when
+the count is known to be its own: the transforms that read (or wrote) it read 
(or wrote) nothing
+else. The rows written are the rows output plus the rows updated. A transform 
that writes several
+files or tables, such as Text File Output with a file per copy or Table Output 
with the table in a
+field, gives them no row count rather than a wrong one. On a pipeline&#8217;s 
event a dataset&#8217;s count is
+the sum over its transforms and their copies; with transform jobs on, each 
copy&#8217;s event counts the
+rows of that copy. Rows a transform deleted from a table (Delete, an Execute 
SQL with a single
+<code>DELETE</code> statement) are not rows written; a table emptied and 
reloaded in one run counts the rows
+loaded. An Execute SQL script of several statements gives no count, as its 
counters add all of them
+up, and neither does a transform that counts 0 while it does handle rows (not 
every transform counts
+what it reads or writes).</p>
+</div>
+<div class="paragraph">
+<p>A pipeline&#8217;s <code>COMPLETE</code> or <code>FAIL</code> event also 
lists every dataset the run wrote, once, in a
+<code>hopWrites</code> run facet: its <code>namespace</code> and 
<code>name</code>, its <code>operationType</code> (<code>INSERT</code>, 
<code>UPDATE</code>,
+<code>DELETE</code>, <code>CREATE</code>, <code>DROP</code> or 
<code>ALTER</code>) and, when known, its <code>rows</code>. The DataHub sink 
turns these
+into operations. A pipeline that was stopped (Stop, Abort, or a failing parent 
pipeline stopping
+it) says so with <code>stopped</code> in its <code>hopRowCounts</code> facet 
and lists no writes.</p>
+</div>
+<div class="paragraph">
+<p>Apache Beam and Spark run the transforms of a pipeline in small local 
pipelines on their workers.
+With lineage on in the JVM of those workers, as with the Beam DirectRunner or 
Spark in local mode,
+each of those reports a run of its own, with the counts and writes of its 
bundle or partition.</p>
+</div>
 <div class="paragraph">
 <p>The run id is the executing subject&#8217;s log-channel id. By default file 
I/O is correlated to
 the parent pipeline run, so inputs and outputs appear on the pipeline job in 
Marquez. Every run also
@@ -3314,6 +3354,7 @@ its workflow, and a pipeline&#8217;s at the action that 
ran it (or at the parent
 sub-pipeline). Actions are jobs of their own, named 
<code>&lt;workflow&gt;/&lt;action&gt;</code>, so a workflow with several
 actions no longer files them all under the workflow&#8217;s job.</p>
 </div>
+</div>
 <div class="sect2">
 <h3 id="job-hierarchy"><a class="link" href="#job-hierarchy">Job 
hierarchy</a></h3>
 <div class="paragraph">

Reply via email to