This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new d33e782ba4 Mask fields masks the fields a data catalog such as DataHub
labels with a tag or glossary term, part of #9323 (#9471)
d33e782ba4 is described below
commit d33e782ba4f1ed278299ff988cd6b59746c57c1b
Author: Bart Maertens <[email protected]>
AuthorDate: Mon Nov 16 10:10:07 2026 +0100
Mask fields masks the fields a data catalog such as DataHub labels with a
tag or glossary term, part of #9323 (#9471)
* Mask fields masks the fields a data catalog such as DataHub labels with a
tag or glossary term, part of #9323
* Mask fields: tag rules get their own injection keys, a tag whose variable
is not set stops the transform, #9323
---
.../pages/plugin-types/metadata-configuration.adoc | 3 +-
.../ROOT/pages/metadata-types/data-catalog.adoc | 3 +
.../ROOT/pages/pipeline/transforms/maskfields.adoc | 24 ++
.../org/apache/hop/datacatalog/CatalogDataset.java | 109 +++++++++
.../hop/datacatalog}/CatalogDatasetType.java | 7 +-
.../apache/hop/datacatalog/ISchemaCapability.java | 56 +++++
.../datacatalog/messages/messages_en_US.properties | 6 +
.../0002-check-and-stop-datahub-stub.sh | 46 ++++
integration-tests/datacatalog/0002-mask-by-tag.hpl | 168 ++++++++++++++
.../datacatalog/0002-mask-unreachable.hpl | 168 ++++++++++++++
integration-tests/datacatalog/datahub-stub.py | 32 +++
.../datasets/golden-0002-mask-by-tag.csv | 3 +
.../datacatalog/main-0002-mask-by-tag.hwf | 247 +++++++++++++++++++++
.../metadata/dataset/golden-0002-mask-by-tag.json | 12 +
.../metadata/masking-pattern/hidden.json | 6 +
.../metadata/masking-pattern/masked-email.json | 14 ++
.../metadata/unit-test/0002-mask-by-tag UNIT.json | 26 +++
.../checkdatacatalog/ActionCheckDataCatalog.java | 31 +--
.../ActionCheckDataCatalogDialog.java | 2 +
.../actions/checkdatacatalog/CatalogDataset.java | 59 -----
.../messages/messages_en_US.properties | 6 -
.../ActionCheckDataCatalogTest.java | 2 +
.../hop/lineage/datahub/DataHubDataCatalog.java | 139 +++++++++++-
.../lineage/datahub/DataHubDataCatalogTest.java | 68 ++++++
.../maskfields/CatalogFieldResolver.java | 164 ++++++++++++++
.../pipeline/transforms/maskfields/MaskFields.java | 102 +++++++--
.../transforms/maskfields/MaskFieldsData.java | 10 +
.../transforms/maskfields/MaskFieldsDialog.java | 12 +-
.../transforms/maskfields/MaskFieldsMeta.java | 136 ++++++++++++
.../transforms/maskfields/MaskTagRule.java | 68 ++++++
.../maskfields/messages/messages_en_US.properties | 27 +++
.../transforms/maskfields/MaskFieldsMetaTest.java | 78 +++++++
.../transforms/maskfields/MaskFieldsTest.java | 175 +++++++++++++++
.../resources/mask-fields-catalog-transform.xml | 41 ++++
34 files changed, 1924 insertions(+), 126 deletions(-)
diff --git
a/docs/hop-dev-manual/modules/ROOT/pages/plugin-types/metadata-configuration.adoc
b/docs/hop-dev-manual/modules/ROOT/pages/plugin-types/metadata-configuration.adoc
index cfb5e1bf2b..33ef98d202 100644
---
a/docs/hop-dev-manual/modules/ROOT/pages/plugin-types/metadata-configuration.adoc
+++
b/docs/hop-dev-manual/modules/ROOT/pages/plugin-types/metadata-configuration.adoc
@@ -89,7 +89,7 @@ Hop reads dataset metadata from it; writing to a catalog
stays with the lineage
The API is experimental.
Annotation:: `@DataCatalogPlugin`, together with `@GuiPlugin` for the options
-Implement:: `IDataCatalog`, plus the `ICatalogCapability` interfaces for what
the catalog can answer: `IDeprecationCapability`, `IFreshnessCapability`,
`IAssertionCapability`
+Implement:: `IDataCatalog`, plus the `ICatalogCapability` interfaces for what
the catalog can answer: `IDeprecationCapability`, `IFreshnessCapability`,
`IAssertionCapability`, `ISchemaCapability`
Plugin type:: `DataCatalogPluginType`, id `DATA_CATALOGS`
Examples:: `plugins/tech/datahub` (`DataHubDataCatalog`)
@@ -104,6 +104,7 @@ A catalog that can't answer is a normal outcome, not an
exception; the caller de
A dataset is a `DatasetRef`.
Build it with `DatasetRef.ofTable(...)` or `ofFile(...)`: those give a table
or a file the OpenLineage namespace and name that lineage gives it, so
`resolve()` finds the dataset lineage wrote.
`ofCatalogId(...)` names a dataset by the catalog's own id.
+A transform or action that lets the user pick a dataset stores a
`CatalogDataset` (type, connection, schema, name) and calls `toRef(variables,
metadataProvider)`, which also reports a dataset that can't be named.
== Configuration options
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/data-catalog.adoc
b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/data-catalog.adoc
index 21f3c3bc89..418b80326e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/metadata-types/data-catalog.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/metadata-types/data-catalog.adoc
@@ -26,6 +26,7 @@ A Data Catalog is a catalog, such as DataHub, that Hop reads
dataset metadata fr
The catalog stays the owner of that metadata. Hop only reads it.
The xref:workflow/actions/checkdatacatalog.adoc[Check data catalog] action
uses a Data Catalog to check datasets before a workflow goes on.
+The xref:pipeline/transforms/maskfields.adoc[Mask fields] transform can use
one to mask the fields that carry a tag or glossary term, such as `PII`.
This metadata type is experimental: its options and what the catalog types can
answer may still change.
@@ -109,3 +110,5 @@ DataHub learns about operations from the sources that
report them, such as the u
The DataHub lineage sink of Hop doesn't report them yet, so a table that only
Hop writes has none.
* *Assertions*: the latest run of each assertion on the dataset, up to 100
assertions.
The assertions themselves are defined and run in DataHub.
+* *Fields*: the fields of the dataset's schema, with the tags and glossary
terms on them, both the ones that came with the ingested schema and the ones
added in DataHub.
+The xref:pipeline/transforms/maskfields.adoc[Mask fields] transform uses them.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/maskfields.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/maskfields.adoc
index 09994a8520..abdf4d4249 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/maskfields.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/maskfields.adoc
@@ -57,6 +57,30 @@ image::transforms/maskfields/maskfields-dialog.png[Mask
fields dialog,width="60%
When a field cannot be masked and the transform has error handling, the error
row has that field and every masked field after it set to null.
+=== Data catalog
+
+These options mask fields by the tags and glossary terms a
xref:metadata-types/data-catalog.adoc[Data Catalog], such as DataHub, has on
them. They are experimental. Leave *Data catalog* empty not to use them.
+
+[options="header"]
+|===
+|Option|Description
+
+|Data catalog
+|The data catalog that says which fields carry a tag or term.
+
+|Dataset type, Dataset connection, Dataset schema, Dataset
+|The dataset whose fields the catalog labels: a table of a relational
connection, a file, or the id of the dataset in the catalog. A table and a file
are found the way
xref:metadata-types/data-catalog.adoc#_how_datasets_are_found[a data catalog
finds datasets].
+
+|Tag or term, Masking rule
+|Every incoming field that the catalog labels with the tag or glossary term on
a row is masked with that row's Masking pattern. A tag is matched by its name
or URN (`PII`, `urn:li:tag:PII`), a term by its name, its URN or its path in
the glossary, the term groups above it followed by its name (`Personal`,
`Classification.Personal`), without regard to case. The first row that matches
a field wins.
+|===
+
+When the transform starts, it reads the fields of the dataset from the
catalog. When the first row arrives, it masks every incoming field whose name
matches a labelled field, without regard to case: `EMAIL` in the stream matches
`email` in the catalog. A field that only shows up at run time is masked too.
The log has a line for every field masked this way, with its tag or term.
+
+A field that also has a row in the fields table keeps the pattern of that row.
+
+When the catalog can't be reached, refuses the token, doesn't have the dataset
or has no fields for it, or a tag or term holds a variable that is not set, the
transform doesn't start: it doesn't guess which fields to mask.
+
== What a pattern writes
*SYNTHETIC* builds `prefix + token + suffix` for a string. The token is a
*SEQUENCE* (an integer that starts at *Sequence start* and steps by one) or a
*UUID*. A number field can take a sequence only, with an empty prefix and
suffix. A date cannot take a generated token.
diff --git
a/engine/src/main/java/org/apache/hop/datacatalog/CatalogDataset.java
b/engine/src/main/java/org/apache/hop/datacatalog/CatalogDataset.java
new file mode 100644
index 0000000000..29007141be
--- /dev/null
+++ b/engine/src/main/java/org/apache/hop/datacatalog/CatalogDataset.java
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.datacatalog;
+
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hop.core.database.DatabaseMeta;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.HopMetadataPropertyType;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+
+/**
+ * A dataset as a user names it in a transform or action: a table of a
relational connection, a
+ * file, or the id the catalog gives it. {@link #toRef} turns it into the
{@link DatasetRef} a
+ * catalog resolves. Experimental.
+ */
+@Getter
+@Setter
+public class CatalogDataset {
+ private static final Class<?> PKG = CatalogDataset.class;
+
+ @HopMetadataProperty(key = "type")
+ private CatalogDatasetType type;
+
+ /** The relational connection of a table. */
+ @HopMetadataProperty(
+ key = "connection",
+ hopMetadataPropertyType = HopMetadataPropertyType.RDBMS_CONNECTION)
+ private String connection;
+
+ /** The schema of a table; empty for the connection's preferred schema. */
+ @HopMetadataProperty(
+ key = "schema",
+ hopMetadataPropertyType = HopMetadataPropertyType.RDBMS_SCHEMA)
+ private String schema;
+
+ /** The table, the file name, or the catalog id. */
+ @HopMetadataProperty(key = "name")
+ private String name;
+
+ public CatalogDataset() {
+ type = CatalogDatasetType.TABLE;
+ }
+
+ public CatalogDataset(CatalogDatasetType type, String connection, String
schema, String name) {
+ this.type = type;
+ this.connection = connection;
+ this.schema = schema;
+ this.name = name;
+ }
+
+ /**
+ * The dataset as Hop identifies it, so a catalog finds the dataset lineage
wrote.
+ *
+ * @param variables to resolve the name, the connection and the schema
+ * @param metadataProvider to load the connection of a table
+ * @throws HopException when there is no name, the connection doesn't exist,
or the dataset can't
+ * be named for a catalog (a table of a connection without a host)
+ */
+ public DatasetRef toRef(IVariables variables, IHopMetadataProvider
metadataProvider)
+ throws HopException {
+ String resolvedName = variables.resolve(name);
+ if (StringUtils.isBlank(resolvedName)) {
+ throw new HopException(BaseMessages.getString(PKG,
"CatalogDataset.Error.NoName"));
+ }
+ CatalogDatasetType activeType = type == null ? CatalogDatasetType.TABLE :
type;
+ DatasetRef ref =
+ switch (activeType) {
+ case CATALOG_ID -> DatasetRef.ofCatalogId(resolvedName.trim());
+ case FILE -> DatasetRef.ofFile(resolvedName, variables);
+ case TABLE -> {
+ String connectionName = variables.resolve(connection);
+ DatabaseMeta databaseMeta =
DatabaseMeta.loadDatabase(metadataProvider, connectionName);
+ if (databaseMeta == null) {
+ throw new HopException(
+ BaseMessages.getString(
+ PKG, "CatalogDataset.Error.MissingConnection",
connectionName, resolvedName));
+ }
+ yield DatasetRef.ofTable(
+ databaseMeta, variables, variables.resolve(schema),
resolvedName.trim());
+ }
+ };
+ if (!ref.isComplete()) {
+ // A connection without a host (an embedded database, a manual URL) has
no name to look for
+ throw new HopException(
+ BaseMessages.getString(PKG, "CatalogDataset.Error.NoIdentity",
resolvedName));
+ }
+ return ref;
+ }
+}
diff --git
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/CatalogDatasetType.java
b/engine/src/main/java/org/apache/hop/datacatalog/CatalogDatasetType.java
similarity index 84%
rename from
plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/CatalogDatasetType.java
rename to
engine/src/main/java/org/apache/hop/datacatalog/CatalogDatasetType.java
index 281509d8ba..33fce14cf3 100644
---
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/CatalogDatasetType.java
+++ b/engine/src/main/java/org/apache/hop/datacatalog/CatalogDatasetType.java
@@ -15,12 +15,12 @@
* limitations under the License.
*/
-package org.apache.hop.workflow.actions.checkdatacatalog;
+package org.apache.hop.datacatalog;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.api.IEnumHasCodeAndDescription;
-/** How a dataset of {@link ActionCheckDataCatalog} is named. */
+/** How a {@link CatalogDataset} names its dataset. Experimental. */
public enum CatalogDatasetType implements IEnumHasCodeAndDescription {
/** A table of a relational connection. */
TABLE,
@@ -36,7 +36,6 @@ public enum CatalogDatasetType implements
IEnumHasCodeAndDescription {
@Override
public String getDescription() {
- return BaseMessages.getString(
- ActionCheckDataCatalog.class, "ActionCheckDataCatalog.DatasetType." +
name());
+ return BaseMessages.getString(CatalogDatasetType.class,
"CatalogDatasetType." + name());
}
}
diff --git
a/engine/src/main/java/org/apache/hop/datacatalog/ISchemaCapability.java
b/engine/src/main/java/org/apache/hop/datacatalog/ISchemaCapability.java
new file mode 100644
index 0000000000..b42307b95d
--- /dev/null
+++ b/engine/src/main/java/org/apache/hop/datacatalog/ISchemaCapability.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.datacatalog;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+
+/** The fields of a dataset as the catalog knows them, with their tags and
terms. Experimental. */
+public interface ISchemaCapability extends ICatalogCapability {
+
+ /**
+ * One field of a dataset.
+ *
+ * @param name the field name; for a nested field its path, for example
{@code address.city}
+ * @param nativeType the type in the source, for example {@code
VARCHAR(255)}, or null
+ * @param type the catalog's own type, for example {@code STRING} or {@code
NUMBER}, or null
+ * @param nullable whether the field can be empty, or null when not known
+ * @param tags every way the catalog names the tags on the field, for
example both {@code PII} and
+ * {@code urn:li:tag:PII}
+ * @param terms every way the catalog names the glossary terms on the field
+ */
+ record CatalogField(
+ String name,
+ String nativeType,
+ String type,
+ Boolean nullable,
+ Set<String> tags,
+ Set<String> terms) {
+
+ /** Whether the field carries this tag or term, compared without regard to
case. */
+ public boolean isLabelledWith(String tagOrTerm) {
+ String wanted = tagOrTerm.trim().toLowerCase(Locale.ROOT);
+ return tags.stream().anyMatch(t ->
t.toLowerCase(Locale.ROOT).equals(wanted))
+ || terms.stream().anyMatch(t ->
t.toLowerCase(Locale.ROOT).equals(wanted));
+ }
+ }
+
+ /** Not found means the catalog has no schema for the dataset. */
+ CatalogAnswer<List<CatalogField>> schema(CatalogEntityId id);
+}
diff --git
a/engine/src/main/resources/org/apache/hop/datacatalog/messages/messages_en_US.properties
b/engine/src/main/resources/org/apache/hop/datacatalog/messages/messages_en_US.properties
index 7c2afc3afc..bc57b947df 100644
---
a/engine/src/main/resources/org/apache/hop/datacatalog/messages/messages_en_US.properties
+++
b/engine/src/main/resources/org/apache/hop/datacatalog/messages/messages_en_US.properties
@@ -17,3 +17,9 @@
DataCatalog.name=Data Catalog
DataCatalog.description=A data catalog, such as DataHub, that Hop reads
dataset metadata from
+CatalogDatasetType.TABLE=Table
+CatalogDatasetType.FILE=File
+CatalogDatasetType.CATALOG_ID=Catalog id
+CatalogDataset.Error.NoName=A dataset has no table, file or id
+CatalogDataset.Error.MissingConnection=Relational connection ''{0}'' of table
''{1}'' can''t be found
+CatalogDataset.Error.NoIdentity=Hop can''t name dataset ''{0}'' for the
catalog: a table of a connection without a host (an embedded database or a
manual URL) has no name to look for. Use its catalog id instead.
diff --git a/integration-tests/datacatalog/0002-check-and-stop-datahub-stub.sh
b/integration-tests/datacatalog/0002-check-and-stop-datahub-stub.sh
new file mode 100755
index 0000000000..24f676e79f
--- /dev/null
+++ b/integration-tests/datacatalog/0002-check-and-stop-datahub-stub.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Checks that Mask fields in 0002-mask-by-tag.hpl asked the DataHub stub for
the customers dataset and
+# its schema (issue #9323), then stops the stub.
+
+LOG=/tmp/datahub-stub-requests.log
+status=0
+
+expect() {
+ if grep -q -E "$1" "${LOG}"; then
+ echo "OK: $2"
+ else
+ echo "FAILED: $2"
+ status=1
+ fi
+}
+
+expect '^hopDatasetExists
urn:li:dataset:\(urn:li:dataPlatform:hive,it\.customers,PROD\)$' \
+ "Mask fields looked for the customers dataset"
+expect '^hopDatasetSchema
urn:li:dataset:\(urn:li:dataPlatform:hive,it\.customers,PROD\)$' \
+ "Mask fields asked for the schema of the customers dataset"
+
+if [ "${status}" != "0" ]; then
+ echo "Requests the stub received:"
+ cat "${LOG}"
+fi
+
+kill "$(cat /tmp/datahub-stub.pid)" 2>/dev/null
+exit ${status}
diff --git a/integration-tests/datacatalog/0002-mask-by-tag.hpl
b/integration-tests/datacatalog/0002-mask-by-tag.hpl
new file mode 100644
index 0000000000..38e2b1127a
--- /dev/null
+++ b/integration-tests/datacatalog/0002-mask-by-tag.hpl
@@ -0,0 +1,168 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>0002-mask-by-tag</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Mask fields with only tag and term rules: the data catalog
'datahub-stub' says which fields to mask (issue #9323)</description>
+ <created_user>-</created_user>
+ <created_date>2026/11/13 09:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/13 09:00:00.000</modified_date>
+ <pipeline_status>0</pipeline_status>
+ <parameters/>
+ </info>
+ <notepads/>
+ <order>
+ <hop>
+ <from>customers</from>
+ <to>Mask by tag</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Mask by tag</from>
+ <to>Result</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <transform>
+ <type>DataGrid</type>
+ <name>customers</name>
+ <fields>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>id</name>
+ <type>Integer</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>EMAIL</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>name</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>city</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <data>
+ <line>
+ <item>1</item>
+ <item>[email protected]</item>
+ <item>Ann</item>
+ <item>Gent</item>
+ </line>
+ <line>
+ <item>2</item>
+ <item>[email protected]</item>
+ <item>Bob</item>
+ <item>Brugge</item>
+ </line>
+ </data>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>MaskFields</type>
+ <name>Mask by tag</name>
+ <fields/>
+ <data_catalog>datahub-stub</data_catalog>
+ <dataset_type>CATALOG_ID</dataset_type>
+ <dataset_connection/>
+ <dataset_schema/>
+
<dataset_name>urn:li:dataset:(urn:li:dataPlatform:hive,it.customers,PROD)</dataset_name>
+ <tag_rules>
+ <tag_rule>
+ <tag>pii</tag>
+ <pattern>masked-email</pattern>
+ </tag_rule>
+ <tag_rule>
+ <tag>Classification.Personal</tag>
+ <pattern>hidden</pattern>
+ </tag_rule>
+ </tag_rules>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>288</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Dummy</type>
+ <name>Result</name>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>480</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform_error_handling/>
+ <attributes/>
+</pipeline>
diff --git a/integration-tests/datacatalog/0002-mask-unreachable.hpl
b/integration-tests/datacatalog/0002-mask-unreachable.hpl
new file mode 100644
index 0000000000..7bd5e953ac
--- /dev/null
+++ b/integration-tests/datacatalog/0002-mask-unreachable.hpl
@@ -0,0 +1,168 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<pipeline>
+ <info>
+ <name>0002-mask-unreachable</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Mask fields with only tag and term rules: the data catalog
'datahub-unreachable' says which fields to mask (issue #9323)</description>
+ <created_user>-</created_user>
+ <created_date>2026/11/13 09:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/13 09:00:00.000</modified_date>
+ <pipeline_status>0</pipeline_status>
+ <parameters/>
+ </info>
+ <notepads/>
+ <order>
+ <hop>
+ <from>customers</from>
+ <to>Mask by tag</to>
+ <enabled>Y</enabled>
+ </hop>
+ <hop>
+ <from>Mask by tag</from>
+ <to>Result</to>
+ <enabled>Y</enabled>
+ </hop>
+ </order>
+ <transform>
+ <type>DataGrid</type>
+ <name>customers</name>
+ <fields>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>id</name>
+ <type>Integer</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>EMAIL</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>name</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ <field>
+ <currency/>
+ <decimal/>
+ <group/>
+ <name>city</name>
+ <type>String</type>
+ <format/>
+ <length>-1</length>
+ <precision>-1</precision>
+ <set_empty_string>N</set_empty_string>
+ </field>
+ </fields>
+ <data>
+ <line>
+ <item>1</item>
+ <item>[email protected]</item>
+ <item>Ann</item>
+ <item>Gent</item>
+ </line>
+ <line>
+ <item>2</item>
+ <item>[email protected]</item>
+ <item>Bob</item>
+ <item>Brugge</item>
+ </line>
+ </data>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>96</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>MaskFields</type>
+ <name>Mask by tag</name>
+ <fields/>
+ <data_catalog>datahub-unreachable</data_catalog>
+ <dataset_type>CATALOG_ID</dataset_type>
+ <dataset_connection/>
+ <dataset_schema/>
+
<dataset_name>urn:li:dataset:(urn:li:dataPlatform:hive,it.customers,PROD)</dataset_name>
+ <tag_rules>
+ <tag_rule>
+ <tag>pii</tag>
+ <pattern>masked-email</pattern>
+ </tag_rule>
+ <tag_rule>
+ <tag>Classification.Personal</tag>
+ <pattern>hidden</pattern>
+ </tag_rule>
+ </tag_rules>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>288</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform>
+ <type>Dummy</type>
+ <name>Result</name>
+ <distribute>Y</distribute>
+ <copies>1</copies>
+ <GUI>
+ <xloc>480</xloc>
+ <yloc>96</yloc>
+ </GUI>
+ <partitioning>
+ <method>none</method>
+ <schema_name/>
+ </partitioning>
+ <attributes/>
+ </transform>
+ <transform_error_handling/>
+ <attributes/>
+</pipeline>
diff --git a/integration-tests/datacatalog/datahub-stub.py
b/integration-tests/datacatalog/datahub-stub.py
index cf34be3b0d..3f716c7656 100644
--- a/integration-tests/datacatalog/datahub-stub.py
+++ b/integration-tests/datacatalog/datahub-stub.py
@@ -31,6 +31,35 @@ from http.server import BaseHTTPRequestHandler,
ThreadingHTTPServer
TOKEN = "Bearer it-token"
HOUR_MS = 3600 * 1000
+CUSTOMERS = "urn:li:dataset:(urn:li:dataPlatform:hive,it.customers,PROD)"
+
+
+def tags(*names):
+ return {"tags": [{"tag": {"urn": "urn:li:tag:" + n, "name": n,
"properties": None}} for n in names]}
+
+
+def terms(path):
+ """A term created in the DataHub UI: a generated id in its URN, its path
in the parent nodes."""
+ *parents, name = path.split(".")
+ term = {"urn": "urn:li:glossaryTerm:4f1c", "name": "4f1c",
"hierarchicalName": "4f1c",
+ "properties": {"name": name},
+ "parentNodes": {"nodes": [{"properties": {"name": p}} for p in
reversed(parents)]}}
+ return {"terms": [{"term": term}]}
+
+
+# The customers table: email is tagged PII in the ingested schema, name has a
glossary term added
+# in the DataHub UI (a v2 field path), id and city carry nothing.
+CUSTOMERS_SCHEMA = {
+ "schemaMetadata": {"fields": [
+ {"fieldPath": "id", "nativeDataType": "INTEGER", "type": "NUMBER",
"nullable": False, "globalTags": None, "glossaryTerms": None},
+ {"fieldPath": "email", "nativeDataType": "VARCHAR(255)", "type":
"STRING", "nullable": True, "globalTags": tags("PII"), "glossaryTerms": None},
+ {"fieldPath": "name", "nativeDataType": "VARCHAR(100)", "type":
"STRING", "nullable": True, "globalTags": None, "glossaryTerms": None},
+ {"fieldPath": "city", "nativeDataType": "VARCHAR(100)", "type":
"STRING", "nullable": True, "globalTags": None, "glossaryTerms": None},
+ ]},
+ "editableSchemaMetadata": {"editableSchemaFieldInfo": [
+ {"fieldPath": "[version=2.0].[type=string].name", "globalTags": None,
"glossaryTerms": terms("Classification.Personal")},
+ ]},
+}
TEN_MINUTES_MS = 10 * 60 * 1000
@@ -48,6 +77,7 @@ def dataset(urn):
"urn:li:dataset:(urn:li:dataPlatform:hive,it.deprecated,PROD)": (True,
TEN_MINUTES_MS, "SUCCESS"),
"urn:li:dataset:(urn:li:dataPlatform:hive,it.stale,PROD)": (False, 72
* HOUR_MS, "SUCCESS"),
"urn:li:dataset:(urn:li:dataPlatform:hive,it.failing,PROD)": (False,
TEN_MINUTES_MS, "FAILURE"),
+ CUSTOMERS: healthy,
}.get(urn)
@@ -66,6 +96,8 @@ def answer(query, urn):
return {"dataset": {"deprecation": deprecation if deprecated else
None}}
if "hopDatasetOperations" in query:
return {"dataset": {"operations": [{"lastUpdatedTimestamp": now -
age}]}}
+ if "hopDatasetSchema" in query:
+ return {"dataset": CUSTOMERS_SCHEMA if urn == CUSTOMERS else
{"schemaMetadata": None}}
if "hopDatasetAssertions" in query:
run = {"timestampMillis": now - HOUR_MS, "result": {"type": result}}
assertion = {
diff --git a/integration-tests/datacatalog/datasets/golden-0002-mask-by-tag.csv
b/integration-tests/datacatalog/datasets/golden-0002-mask-by-tag.csv
new file mode 100644
index 0000000000..47601abcb3
--- /dev/null
+++ b/integration-tests/datacatalog/datasets/golden-0002-mask-by-tag.csv
@@ -0,0 +1,3 @@
+id,EMAIL,name,city
+1,email-1,,Gent
+2,email-2,,Brugge
diff --git a/integration-tests/datacatalog/main-0002-mask-by-tag.hwf
b/integration-tests/datacatalog/main-0002-mask-by-tag.hwf
new file mode 100644
index 0000000000..bf4ac81898
--- /dev/null
+++ b/integration-tests/datacatalog/main-0002-mask-by-tag.hwf
@@ -0,0 +1,247 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+-->
+<workflow>
+ <name>main-0002-mask-by-tag</name>
+ <name_sync_with_filename>Y</name_sync_with_filename>
+ <description>Mask fields masks the fields a data catalog tags or terms, and
doesn't start without an answer from the catalog (issue #9323)</description>
+ <extended_description/>
+ <workflow_version/>
+ <created_user>-</created_user>
+ <created_date>2026/11/13 09:00:00.000</created_date>
+ <modified_user>-</modified_user>
+ <modified_date>2026/11/13 09:00:00.000</modified_date>
+ <parameters>
+ </parameters>
+ <actions>
+ <action>
+ <name>Start</name>
+ <description></description>
+ <type>SPECIAL</type>
+ <attributes/>
+ <DayOfMonth>1</DayOfMonth>
+ <doNotWaitOnFirstExecution>N</doNotWaitOnFirstExecution>
+ <hour>12</hour>
+ <intervalMinutes>60</intervalMinutes>
+ <intervalSeconds>0</intervalSeconds>
+ <minutes>0</minutes>
+ <repeat>N</repeat>
+ <schedulerType>0</schedulerType>
+ <weekDay>1</weekDay>
+ <parallel>N</parallel>
+ <xloc>64</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Start the DataHub stub</name>
+ <description>Starts a stand-in for the DataHub GraphQL API in the test
container</description>
+ <type>SHELL</type>
+ <attributes/>
+ <filename>${PROJECT_HOME}/0001-start-datahub-stub.sh</filename>
+ <work_directory>${PROJECT_HOME}</work_directory>
+ <arg_from_previous>N</arg_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <set_logfile>N</set_logfile>
+ <logfile/>
+ <set_append_logfile>N</set_append_logfile>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <insertScript>N</insertScript>
+ <script/>
+ <loglevel>Basic</loglevel>
+ <parallel>N</parallel>
+ <xloc>256</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Mask by tag</name>
+ <description>Unit test of 0002-mask-by-tag.hpl: EMAIL is tagged PII,
name has the term Classification.Personal, id and city carry
nothing</description>
+ <type>RunPipelineTests</type>
+ <attributes/>
+ <test_names>
+ <test_name>
+ <name>0002-mask-by-tag UNIT</name>
+ </test_name>
+ </test_names>
+ <parallel>N</parallel>
+ <xloc>480</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>An unreachable catalog stops the masking</name>
+ <description>The same pipeline with a data catalog nothing listens on:
Mask fields must not start</description>
+ <type>PIPELINE</type>
+ <attributes/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <clear_files>N</clear_files>
+ <clear_rows>N</clear_rows>
+ <create_parent_folder>N</create_parent_folder>
+ <exec_per_row>N</exec_per_row>
+ <filename>${PROJECT_HOME}/0002-mask-unreachable.hpl</filename>
+ <loglevel>Basic</loglevel>
+ <parameters>
+ <pass_all_parameters>Y</pass_all_parameters>
+ </parameters>
+ <params_from_previous>N</params_from_previous>
+ <run_configuration>local</run_configuration>
+ <set_append_logfile>N</set_append_logfile>
+ <set_logfile>N</set_logfile>
+ <wait_until_finished>Y</wait_until_finished>
+ <parallel>N</parallel>
+ <xloc>704</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Check the requests</name>
+ <description>Checks that Mask fields asked the stub for the schema of
it.customers, then stops it</description>
+ <type>SHELL</type>
+ <attributes/>
+ <filename>${PROJECT_HOME}/0002-check-and-stop-datahub-stub.sh</filename>
+ <work_directory>${PROJECT_HOME}</work_directory>
+ <arg_from_previous>N</arg_from_previous>
+ <exec_per_row>N</exec_per_row>
+ <set_logfile>N</set_logfile>
+ <logfile/>
+ <set_append_logfile>N</set_append_logfile>
+ <logext/>
+ <add_date>N</add_date>
+ <add_time>N</add_time>
+ <insertScript>N</insertScript>
+ <script/>
+ <loglevel>Basic</loglevel>
+ <parallel>N</parallel>
+ <xloc>928</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Success</name>
+ <description></description>
+ <type>SUCCESS</type>
+ <attributes/>
+ <parallel>N</parallel>
+ <xloc>1152</xloc>
+ <yloc>96</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Masking by tag failed</name>
+ <description></description>
+ <type>ABORT</type>
+ <attributes/>
+ <always_log_rows>N</always_log_rows>
+ <message>The masked rows are not the golden ones, see the unit test
above</message>
+ <parallel>N</parallel>
+ <xloc>480</xloc>
+ <yloc>240</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Unexpected: an unreachable catalog</name>
+ <description></description>
+ <type>ABORT</type>
+ <attributes/>
+ <always_log_rows>N</always_log_rows>
+ <message>Mask fields started without an answer from the data
catalog</message>
+ <parallel>N</parallel>
+ <xloc>704</xloc>
+ <yloc>240</yloc>
+ <attributes_hac/>
+ </action>
+ <action>
+ <name>Abort</name>
+ <description></description>
+ <type>ABORT</type>
+ <attributes/>
+ <always_log_rows>N</always_log_rows>
+ <message>Mask fields did not ask the stub for the schema, see the output
of the script above</message>
+ <parallel>N</parallel>
+ <xloc>928</xloc>
+ <yloc>240</yloc>
+ <attributes_hac/>
+ </action>
+ </actions>
+ <hops>
+ <hop>
+ <from>Start</from>
+ <to>Start the DataHub stub</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>Y</unconditional>
+ </hop>
+ <hop>
+ <from>Start the DataHub stub</from>
+ <to>Mask by tag</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Mask by tag</from>
+ <to>An unreachable catalog stops the masking</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Mask by tag</from>
+ <to>Masking by tag failed</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>An unreachable catalog stops the masking</from>
+ <to>Check the requests</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>An unreachable catalog stops the masking</from>
+ <to>Unexpected: an unreachable catalog</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Check the requests</from>
+ <to>Success</to>
+ <enabled>Y</enabled>
+ <evaluation>Y</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ <hop>
+ <from>Check the requests</from>
+ <to>Abort</to>
+ <enabled>Y</enabled>
+ <evaluation>N</evaluation>
+ <unconditional>N</unconditional>
+ </hop>
+ </hops>
+ <notepads>
+ </notepads>
+ <attributes/>
+</workflow>
diff --git
a/integration-tests/datacatalog/metadata/dataset/golden-0002-mask-by-tag.json
b/integration-tests/datacatalog/metadata/dataset/golden-0002-mask-by-tag.json
new file mode 100644
index 0000000000..385733fb34
--- /dev/null
+++
b/integration-tests/datacatalog/metadata/dataset/golden-0002-mask-by-tag.json
@@ -0,0 +1,12 @@
+{
+ "base_filename": "golden-0002-mask-by-tag.csv",
+ "name": "golden-0002-mask-by-tag",
+ "description": "The customers of 0002-mask-by-tag after Mask fields, masked
by their tags and terms in the data catalog",
+ "dataset_fields": [
+ { "field_comment": "", "field_length": -1, "field_type": 5,
"field_precision": -1, "field_format": "", "field_name": "id" },
+ { "field_comment": "", "field_length": -1, "field_type": 2,
"field_precision": -1, "field_format": "", "field_name": "EMAIL" },
+ { "field_comment": "", "field_length": -1, "field_type": 2,
"field_precision": -1, "field_format": "", "field_name": "name" },
+ { "field_comment": "", "field_length": -1, "field_type": 2,
"field_precision": -1, "field_format": "", "field_name": "city" }
+ ],
+ "folder_name": ""
+}
diff --git a/integration-tests/datacatalog/metadata/masking-pattern/hidden.json
b/integration-tests/datacatalog/metadata/masking-pattern/hidden.json
new file mode 100644
index 0000000000..c0a3c5be11
--- /dev/null
+++ b/integration-tests/datacatalog/metadata/masking-pattern/hidden.json
@@ -0,0 +1,6 @@
+{
+ "name": "hidden",
+ "description": "Sets the value to null",
+ "valueSource": "SET_NULL",
+ "storage": "NONE"
+}
diff --git
a/integration-tests/datacatalog/metadata/masking-pattern/masked-email.json
b/integration-tests/datacatalog/metadata/masking-pattern/masked-email.json
new file mode 100644
index 0000000000..cb0d9757eb
--- /dev/null
+++ b/integration-tests/datacatalog/metadata/masking-pattern/masked-email.json
@@ -0,0 +1,14 @@
+{
+ "name": "masked-email",
+ "description": "Replaces an email with email- and a sequence",
+ "piiClassification": "Direct identifier",
+ "valueSource": "SYNTHETIC",
+ "token": "SEQUENCE",
+ "prefix": "email-",
+ "suffix": "",
+ "sequenceStart": "1",
+ "storage": "NONE",
+ "connection": "",
+ "schemaName": "",
+ "tableName": ""
+}
diff --git a/integration-tests/datacatalog/metadata/unit-test/0002-mask-by-tag
UNIT.json b/integration-tests/datacatalog/metadata/unit-test/0002-mask-by-tag
UNIT.json
new file mode 100644
index 0000000000..1ce3cafe53
--- /dev/null
+++ b/integration-tests/datacatalog/metadata/unit-test/0002-mask-by-tag
UNIT.json
@@ -0,0 +1,26 @@
+{
+ "database_replacements": [],
+ "autoOpening": false,
+ "description": "Mask fields masks the fields the data catalog tags PII or
terms Classification.Personal, and leaves the others",
+ "persist_filename": "",
+ "test_type": "UNIT_TEST",
+ "variableValues": [],
+ "basePath": "",
+ "golden_data_sets": [
+ {
+ "field_mappings": [
+ { "transform_field": "id", "data_set_field": "id" },
+ { "transform_field": "EMAIL", "data_set_field": "EMAIL" },
+ { "transform_field": "name", "data_set_field": "name" },
+ { "transform_field": "city", "data_set_field": "city" }
+ ],
+ "field_order": [ "id" ],
+ "data_set_name": "golden-0002-mask-by-tag",
+ "transform_name": "Result"
+ }
+ ],
+ "input_data_sets": [],
+ "name": "0002-mask-by-tag UNIT",
+ "trans_test_tweaks": [],
+ "pipeline_filename": "./0002-mask-by-tag.hpl"
+}
diff --git
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalog.java
b/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalog.java
index b296bd6af0..cf9740ba08 100644
---
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalog.java
+++
b/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalog.java
@@ -32,7 +32,6 @@ import org.apache.hop.core.CheckResult;
import org.apache.hop.core.ICheckResult;
import org.apache.hop.core.Result;
import org.apache.hop.core.annotations.Action;
-import org.apache.hop.core.database.DatabaseMeta;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.gui.plugin.GuiElementType;
import org.apache.hop.core.gui.plugin.GuiPlugin;
@@ -40,6 +39,7 @@ import org.apache.hop.core.gui.plugin.GuiWidgetElement;
import org.apache.hop.core.gui.plugin.GuiWidgetGroupType;
import org.apache.hop.core.variables.IVariables;
import org.apache.hop.datacatalog.CatalogAnswer;
+import org.apache.hop.datacatalog.CatalogDataset;
import org.apache.hop.datacatalog.CatalogEntityId;
import org.apache.hop.datacatalog.DataCatalog;
import org.apache.hop.datacatalog.DatasetRef;
@@ -298,34 +298,7 @@ public class ActionCheckDataCatalog extends ActionBase
implements IAction {
/** The dataset as Hop identifies it, so the catalog finds the dataset
lineage wrote. */
DatasetRef datasetRef(CatalogDataset dataset) throws HopException {
- String name = resolve(dataset.getName());
- if (StringUtils.isBlank(name)) {
- throw new HopException(BaseMessages.getString(PKG,
"ActionCheckDataCatalog.Error.NoName"));
- }
- CatalogDatasetType type =
- dataset.getType() == null ? CatalogDatasetType.TABLE :
dataset.getType();
- DatasetRef ref =
- switch (type) {
- case CATALOG_ID -> DatasetRef.ofCatalogId(name.trim());
- case FILE -> DatasetRef.ofFile(name, this);
- case TABLE -> {
- String connectionName = resolve(dataset.getConnection());
- DatabaseMeta databaseMeta =
- DatabaseMeta.loadDatabase(getMetadataProvider(),
connectionName);
- if (databaseMeta == null) {
- throw new HopException(
- BaseMessages.getString(
- PKG, "ActionCheckDataCatalog.Error.MissingConnection",
connectionName, name));
- }
- yield DatasetRef.ofTable(databaseMeta, this,
resolve(dataset.getSchema()), name.trim());
- }
- };
- if (!ref.isComplete()) {
- // A connection without a host (an embedded database, a manual URL) has
no name to look for
- throw new HopException(
- BaseMessages.getString(PKG,
"ActionCheckDataCatalog.Error.NoIdentity", name));
- }
- return ref;
+ return dataset.toRef(this, getMetadataProvider());
}
/** How many checks the last {@link #check} skipped because the catalog
could not answer. */
diff --git
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogDialog.java
b/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogDialog.java
index 0b0116d024..a7a38ad4e5 100644
---
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogDialog.java
+++
b/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogDialog.java
@@ -22,6 +22,8 @@ import org.apache.hop.core.Const;
import org.apache.hop.core.database.DatabaseMeta;
import org.apache.hop.core.util.Utils;
import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.datacatalog.CatalogDataset;
+import org.apache.hop.datacatalog.CatalogDatasetType;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.api.IEnumHasCodeAndDescription;
import org.apache.hop.ui.core.PropsUi;
diff --git
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/CatalogDataset.java
b/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/CatalogDataset.java
deleted file mode 100644
index 27b3bdbd78..0000000000
---
a/plugins/actions/checkdatacatalog/src/main/java/org/apache/hop/workflow/actions/checkdatacatalog/CatalogDataset.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hop.workflow.actions.checkdatacatalog;
-
-import lombok.Getter;
-import lombok.Setter;
-import org.apache.hop.metadata.api.HopMetadataProperty;
-import org.apache.hop.metadata.api.HopMetadataPropertyType;
-
-/** One dataset {@link ActionCheckDataCatalog} checks. */
-@Getter
-@Setter
-public class CatalogDataset {
-
- @HopMetadataProperty(key = "type")
- private CatalogDatasetType type;
-
- /** The relational connection of a table. */
- @HopMetadataProperty(
- key = "connection",
- hopMetadataPropertyType = HopMetadataPropertyType.RDBMS_CONNECTION)
- private String connection;
-
- /** The schema of a table; empty for the connection's preferred schema. */
- @HopMetadataProperty(
- key = "schema",
- hopMetadataPropertyType = HopMetadataPropertyType.RDBMS_SCHEMA)
- private String schema;
-
- /** The table, the file name, or the catalog id. */
- @HopMetadataProperty(key = "name")
- private String name;
-
- public CatalogDataset() {
- type = CatalogDatasetType.TABLE;
- }
-
- public CatalogDataset(CatalogDatasetType type, String connection, String
schema, String name) {
- this.type = type;
- this.connection = connection;
- this.schema = schema;
- this.name = name;
- }
-}
diff --git
a/plugins/actions/checkdatacatalog/src/main/resources/org/apache/hop/workflow/actions/checkdatacatalog/messages/messages_en_US.properties
b/plugins/actions/checkdatacatalog/src/main/resources/org/apache/hop/workflow/actions/checkdatacatalog/messages/messages_en_US.properties
index 8b63403bbc..3823bbf944 100644
---
a/plugins/actions/checkdatacatalog/src/main/resources/org/apache/hop/workflow/actions/checkdatacatalog/messages/messages_en_US.properties
+++
b/plugins/actions/checkdatacatalog/src/main/resources/org/apache/hop/workflow/actions/checkdatacatalog/messages/messages_en_US.properties
@@ -37,9 +37,6 @@ ActionCheckDataCatalog.UnknownAnswer.IGNORE=Skip the check
and log it
ActionCheckDataCatalog.AgeUnit.MINUTES=Minutes
ActionCheckDataCatalog.AgeUnit.HOURS=Hours
ActionCheckDataCatalog.AgeUnit.DAYS=Days
-ActionCheckDataCatalog.DatasetType.TABLE=Table
-ActionCheckDataCatalog.DatasetType.FILE=File
-ActionCheckDataCatalog.DatasetType.CATALOG_ID=Catalog id
ActionCheckDataCatalog.Column.Type=Type
ActionCheckDataCatalog.Column.Type.Tooltip=A table of a relational connection,
a file, or the id of the dataset in the catalog
ActionCheckDataCatalog.Column.Connection=Connection
@@ -67,8 +64,5 @@ ActionCheckDataCatalog.Log.Summary={0} of {1} datasets passed
the checks in data
ActionCheckDataCatalog.Error.NoCatalog=No data catalog is selected
ActionCheckDataCatalog.Error.MissingCatalog=Data catalog ''{0}'' can''t be
found
ActionCheckDataCatalog.Error.NoDatasets=No datasets to check
-ActionCheckDataCatalog.Error.NoName=A dataset has no table, file or id
-ActionCheckDataCatalog.Error.NoIdentity=Hop can''t name dataset ''{0}'' for
the catalog: a table of a connection without a host (an embedded database or a
manual URL) has no name to look for. Use its catalog id instead.
-ActionCheckDataCatalog.Error.MissingConnection=Relational connection ''{0}''
of table ''{1}'' can''t be found
ActionCheckDataCatalog.Error.MaxAge=The maximum age must be a whole number
above 0, was ''{0}''
ActionCheckDataCatalog.Error.Failed=The datasets could not be checked
diff --git
a/plugins/actions/checkdatacatalog/src/test/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogTest.java
b/plugins/actions/checkdatacatalog/src/test/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogTest.java
index b36b74c5ad..9a6841740d 100644
---
a/plugins/actions/checkdatacatalog/src/test/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogTest.java
+++
b/plugins/actions/checkdatacatalog/src/test/java/org/apache/hop/workflow/actions/checkdatacatalog/ActionCheckDataCatalogTest.java
@@ -38,6 +38,8 @@ import org.apache.hop.core.variables.IVariables;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.datacatalog.CatalogAnswer;
import org.apache.hop.datacatalog.CatalogAnswer.Reason;
+import org.apache.hop.datacatalog.CatalogDataset;
+import org.apache.hop.datacatalog.CatalogDatasetType;
import org.apache.hop.datacatalog.CatalogEntityId;
import org.apache.hop.datacatalog.DataCatalog;
import org.apache.hop.datacatalog.DatasetRef;
diff --git
a/plugins/tech/datahub/src/main/java/org/apache/hop/lineage/datahub/DataHubDataCatalog.java
b/plugins/tech/datahub/src/main/java/org/apache/hop/lineage/datahub/DataHubDataCatalog.java
index 15d72d3078..638d576318 100644
---
a/plugins/tech/datahub/src/main/java/org/apache/hop/lineage/datahub/DataHubDataCatalog.java
+++
b/plugins/tech/datahub/src/main/java/org/apache/hop/lineage/datahub/DataHubDataCatalog.java
@@ -23,8 +23,13 @@ import com.fasterxml.jackson.databind.node.ObjectNode;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.regex.Pattern;
import lombok.AccessLevel;
import lombok.Getter;
import lombok.Setter;
@@ -45,6 +50,7 @@ import org.apache.hop.datacatalog.IAssertionCapability;
import org.apache.hop.datacatalog.IDataCatalog;
import org.apache.hop.datacatalog.IDeprecationCapability;
import org.apache.hop.datacatalog.IFreshnessCapability;
+import org.apache.hop.datacatalog.ISchemaCapability;
import org.apache.hop.datacatalog.plugin.DataCatalogPlugin;
import org.apache.hop.metadata.api.HopMetadataProperty;
import org.apache.hop.metadata.api.IHopMetadataProvider;
@@ -62,7 +68,11 @@ import org.apache.hop.metadata.api.IHopMetadataProvider;
@Getter
@Setter
public class DataHubDataCatalog
- implements IDataCatalog, IDeprecationCapability, IFreshnessCapability,
IAssertionCapability {
+ implements IDataCatalog,
+ IDeprecationCapability,
+ IFreshnessCapability,
+ IAssertionCapability,
+ ISchemaCapability {
static final String DEFAULT_ENVIRONMENT = "PROD";
static final String DEFAULT_CONNECT_TIMEOUT_MS = "5000";
@@ -88,6 +98,21 @@ public class DataHubDataCatalog
"query hopDatasetOperations($urn: String!, $limit: Int) {"
+ " dataset(urn: $urn) { operations(limit: $limit) {
lastUpdatedTimestamp } } }";
+ /** The tags and terms of a field, with every name DataHub gives them. */
+ private static final String LABELS =
+ "globalTags { tags { tag { urn name properties { name } } } }"
+ + " glossaryTerms { terms { term { urn name hierarchicalName
properties { name }"
+ + " parentNodes { nodes { properties { name } } } } } }";
+
+ private static final String SCHEMA_QUERY =
+ "query hopDatasetSchema($urn: String!) { dataset(urn: $urn) {"
+ + " schemaMetadata { fields { fieldPath nativeDataType type nullable
"
+ + LABELS
+ + " } }"
+ + " editableSchemaMetadata { editableSchemaFieldInfo { fieldPath "
+ + LABELS
+ + " } } } }";
+
private static final String ASSERTIONS_QUERY =
"query hopDatasetAssertions($urn: String!, $count: Int) {"
+ " dataset(urn: $urn) { assertions(start: 0, count: $count) { total
assertions {"
@@ -97,6 +122,9 @@ public class DataHubDataCatalog
private static final ObjectMapper MAPPER = new ObjectMapper();
+ /** A type or version part of a DataHub v2 field path, with the dot after
it. */
+ private static final Pattern V2_PATH_PART =
Pattern.compile("\\[[^\\]]*\\]\\.?");
+
private String pluginId;
private String pluginName;
@@ -327,6 +355,115 @@ public class DataHubDataCatalog
return last == null ? CatalogAnswer.notFound() : CatalogAnswer.known(last);
}
+ @Override
+ public CatalogAnswer<List<CatalogField>> schema(CatalogEntityId id) {
+ CatalogAnswer<JsonNode> answer = datasetQuery(SCHEMA_QUERY, id.value(),
null, 0);
+ if (!(answer instanceof CatalogAnswer.Known<JsonNode> known)) {
+ return answer.unknownAs();
+ }
+ JsonNode fields = known.value().path("schemaMetadata").path("fields");
+ if (!fields.isArray() || fields.isEmpty()) {
+ return CatalogAnswer.notFound();
+ }
+ // Tags and terms added in the DataHub UI live apart from the ingested
schema
+ // and can be stored under a v1 and a v2 path of the same field
+ Map<String, List<JsonNode>> edited = new LinkedHashMap<>();
+ for (JsonNode info :
+
known.value().path("editableSchemaMetadata").path("editableSchemaFieldInfo")) {
+ edited
+ .computeIfAbsent(fieldName(info.path("fieldPath").asText("")), k ->
new ArrayList<>())
+ .add(info);
+ }
+ List<CatalogField> result = new ArrayList<>();
+ for (JsonNode field : fields) {
+ String name = fieldName(field.path("fieldPath").asText(""));
+ if (name.isEmpty()) {
+ continue;
+ }
+ Set<String> tags = new LinkedHashSet<>();
+ Set<String> terms = new LinkedHashSet<>();
+ addLabels(field, tags, terms);
+ for (JsonNode editedField : edited.getOrDefault(name, List.of())) {
+ addLabels(editedField, tags, terms);
+ }
+ result.add(
+ new CatalogField(
+ name,
+
StringUtils.trimToNull(field.path("nativeDataType").asText(null)),
+ StringUtils.trimToNull(field.path("type").asText(null)),
+ field.path("nullable").isBoolean() ?
field.path("nullable").asBoolean() : null,
+ Set.copyOf(tags),
+ Set.copyOf(terms)));
+ }
+ return CatalogAnswer.known(result);
+ }
+
+ /**
+ * The name of a field: its path, without the type and version parts of a
DataHub v2 path ({@code
+ * [version=2.0].[type=string].email} is {@code email}).
+ */
+ static String fieldName(String fieldPath) {
+ if (!fieldPath.startsWith("[")) {
+ return fieldPath;
+ }
+ // The bracketed parts can hold dots of their own: [version=2.0]
+ return StringUtils.strip(V2_PATH_PART.matcher(fieldPath).replaceAll(""),
".");
+ }
+
+ private static void addLabels(JsonNode field, Set<String> tags, Set<String>
terms) {
+ for (JsonNode association : field.path("globalTags").path("tags")) {
+ JsonNode tag = association.path("tag");
+ addNames(tags, tag.path("urn"), tag.path("name"),
tag.path("properties").path("name"));
+ }
+ for (JsonNode association : field.path("glossaryTerms").path("terms")) {
+ JsonNode term = association.path("term");
+ addNames(
+ terms,
+ term.path("urn"),
+ term.path("name"),
+ term.path("hierarchicalName"),
+ term.path("properties").path("name"));
+ String path = glossaryPath(term);
+ if (path != null) {
+ terms.add(path);
+ }
+ }
+ }
+
+ /**
+ * The path of a term in the glossary, from the top node down: {@code
Classification.Personal}.
+ * DataHub's hierarchicalName is the id in the term's URN, which is only a
path for terms ingested
+ * from a business glossary file. A term created in the DataHub UI has a
generated id.
+ */
+ private static String glossaryPath(JsonNode term) {
+ String name =
StringUtils.trimToNull(term.path("properties").path("name").asText(null));
+ JsonNode parents = term.path("parentNodes").path("nodes");
+ if (name == null || !parents.isArray() || parents.isEmpty()) {
+ return null;
+ }
+ // DataHub lists the direct parent first
+ List<String> parts = new ArrayList<>();
+ for (JsonNode parent : parents) {
+ String parentName =
+
StringUtils.trimToNull(parent.path("properties").path("name").asText(null));
+ if (parentName == null) {
+ return null;
+ }
+ parts.add(0, parentName);
+ }
+ parts.add(name);
+ return String.join(".", parts);
+ }
+
+ private static void addNames(Set<String> names, JsonNode... values) {
+ for (JsonNode value : values) {
+ String text = StringUtils.trimToNull(value.asText(null));
+ if (text != null) {
+ names.add(text);
+ }
+ }
+ }
+
@Override
public CatalogAnswer<List<AssertionResult>>
latestAssertionResults(CatalogEntityId id) {
CatalogAnswer<JsonNode> answer =
diff --git
a/plugins/tech/datahub/src/test/java/org/apache/hop/lineage/datahub/DataHubDataCatalogTest.java
b/plugins/tech/datahub/src/test/java/org/apache/hop/lineage/datahub/DataHubDataCatalogTest.java
index 2cf530b4a7..5d868a81c0 100644
---
a/plugins/tech/datahub/src/test/java/org/apache/hop/lineage/datahub/DataHubDataCatalogTest.java
+++
b/plugins/tech/datahub/src/test/java/org/apache/hop/lineage/datahub/DataHubDataCatalogTest.java
@@ -18,7 +18,9 @@
package org.apache.hop.lineage.datahub;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -35,6 +37,7 @@ import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import org.apache.hop.core.HopClientEnvironment;
@@ -49,6 +52,7 @@ import org.apache.hop.datacatalog.DatasetRef;
import org.apache.hop.datacatalog.IAssertionCapability.AssertionResult;
import org.apache.hop.datacatalog.IAssertionCapability.Outcome;
import org.apache.hop.datacatalog.IDeprecationCapability.Deprecation;
+import org.apache.hop.datacatalog.ISchemaCapability.CatalogField;
import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeAll;
@@ -181,6 +185,70 @@ class DataHubDataCatalogTest {
assertTrue(requests.isEmpty());
}
+ @Test
+ void theSchemaHasTheTagsAndTermsOfTheIngestedSchemaAndOfTheUi() throws
Exception {
+ answer(
+ "hopDatasetSchema",
+ "{\"dataset\":{\"schemaMetadata\":{\"fields\":["
+ +
"{\"fieldPath\":\"id\",\"nativeDataType\":\"INTEGER\",\"type\":\"NUMBER\","
+ + "\"nullable\":false,\"globalTags\":null,\"glossaryTerms\":null},"
+ +
"{\"fieldPath\":\"email\",\"nativeDataType\":\"VARCHAR(255)\",\"type\":\"STRING\","
+ +
"\"nullable\":true,\"globalTags\":{\"tags\":[{\"tag\":{\"urn\":\"urn:li:tag:PII\","
+ +
"\"name\":\"PII\",\"properties\":null}}]},\"glossaryTerms\":null},"
+ +
"{\"fieldPath\":\"[version=2.0].[type=struct].address.[type=string].city\","
+ + "\"nativeDataType\":\"string\",\"type\":\"STRING\"}]},"
+ + "\"editableSchemaMetadata\":{\"editableSchemaFieldInfo\":["
+ + "{\"fieldPath\":\"email\",\"globalTags\":{\"tags\":[{\"tag\":"
+ +
"{\"urn\":\"urn:li:tag:Legacy\",\"name\":\"Legacy\",\"properties\":null}}]},"
+ + "\"glossaryTerms\":null},"
+ +
"{\"fieldPath\":\"[version=2.0].[type=string].email\",\"globalTags\":{\"tags\":[{\"tag\":"
+ +
"{\"urn\":\"urn:li:tag:Contact\",\"name\":\"Contact\",\"properties\":{\"name\":\"Contact\"}}}]},"
+ +
"\"glossaryTerms\":{\"terms\":[{\"term\":{\"urn\":\"urn:li:glossaryTerm:7d4e\","
+ + "\"name\":\"7d4e\",\"hierarchicalName\":\"7d4e\","
+ +
"\"properties\":{\"name\":\"Personal\"},\"parentNodes\":{\"nodes\":["
+ + "{\"properties\":{\"name\":\"Classification\"}},"
+ + "{\"properties\":{\"name\":\"Governance\"}}]}}}]}}]}}}");
+
+ CatalogAnswer<List<CatalogField>> answer = connect(url()).schema(new
CatalogEntityId(ORDERS));
+
+ List<CatalogField> fields =
+ assertInstanceOf(CatalogAnswer.Known.class, answer).value() instanceof
List<?> list
+ ? list.stream().map(CatalogField.class::cast).toList()
+ : List.of();
+ assertEquals(
+ List.of("id", "email", "address.city"),
fields.stream().map(CatalogField::name).toList());
+ CatalogField id = fields.get(0);
+ assertEquals("INTEGER", id.nativeType());
+ assertEquals("NUMBER", id.type());
+ assertEquals(Boolean.FALSE, id.nullable());
+ assertTrue(id.tags().isEmpty());
+ CatalogField email = fields.get(1);
+ assertEquals(
+ Set.of(
+ "urn:li:tag:PII",
+ "PII",
+ "urn:li:tag:Legacy",
+ "Legacy",
+ "urn:li:tag:Contact",
+ "Contact"),
+ email.tags());
+ assertTrue(email.isLabelledWith("pii"));
+ assertTrue(email.isLabelledWith("Governance.Classification.Personal"));
+ assertTrue(email.isLabelledWith("urn:li:glossaryTerm:7d4e"));
+ assertFalse(email.isLabelledWith("Classification.Personal"));
+ assertTrue(email.isLabelledWith("Personal"));
+ assertFalse(fields.get(2).isLabelledWith("PII"));
+ assertNull(fields.get(2).nullable());
+ }
+
+ @Test
+ void aDatasetWithoutASchemaHasNone() throws Exception {
+ answer("hopDatasetSchema", "{\"dataset\":{\"schemaMetadata\":null}}");
+
+ assertInstanceOf(
+ CatalogAnswer.NotFound.class, connect(url()).schema(new
CatalogEntityId(ORDERS)));
+ }
+
@Test
void theDeprecationIsRead() throws Exception {
answer(
diff --git
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/CatalogFieldResolver.java
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/CatalogFieldResolver.java
new file mode 100644
index 0000000000..33f3b7fd95
--- /dev/null
+++
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/CatalogFieldResolver.java
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.maskfields;
+
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hop.core.exception.HopException;
+import org.apache.hop.core.logging.ILogChannel;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.datacatalog.CatalogAnswer;
+import org.apache.hop.datacatalog.CatalogEntityId;
+import org.apache.hop.datacatalog.DataCatalog;
+import org.apache.hop.datacatalog.DatasetRef;
+import org.apache.hop.datacatalog.IDataCatalog;
+import org.apache.hop.datacatalog.ISchemaCapability;
+import org.apache.hop.datacatalog.ISchemaCapability.CatalogField;
+import org.apache.hop.i18n.BaseMessages;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
+
+/**
+ * Asks the data catalog of a Mask fields transform which fields of its
dataset carry the tags and
+ * terms of its rules. When the catalog can't give a clear answer, nothing is
guessed: the transform
+ * does not start, so no field it should mask goes through unmasked.
+ */
+public class CatalogFieldResolver {
+ private static final Class<?> PKG = CatalogFieldResolver.class;
+
+ /**
+ * A field the catalog labels with a tag or term of a rule.
+ *
+ * @param fieldName the field name in the catalog
+ * @param patternName the masking pattern of the first rule whose tag or
term it carries
+ * @param label that tag or term
+ */
+ public record Match(String fieldName, String patternName, String label) {}
+
+ private final IVariables variables;
+ private final ILogChannel log;
+ private final MaskFieldsMeta meta;
+ private final IHopMetadataProvider metadataProvider;
+
+ public CatalogFieldResolver(
+ IVariables variables,
+ ILogChannel log,
+ MaskFieldsMeta meta,
+ IHopMetadataProvider metadataProvider) {
+ this.variables = variables;
+ this.log = log;
+ this.meta = meta;
+ this.metadataProvider = metadataProvider;
+ }
+
+ /**
+ * The labelled fields by lower-case name.
+ *
+ * @throws HopException when there is no catalog or dataset, or the catalog
doesn't answer
+ */
+ public Map<String, Match> resolve() throws HopException {
+ String catalogName = variables.resolve(meta.getCatalogName());
+ if (StringUtils.isBlank(catalogName)) {
+ throw new HopException(
+ BaseMessages.getString(PKG,
"MaskFields.Check.TagRulesWithoutCatalog"));
+ }
+ DataCatalog dataCatalog =
+ metadataProvider == null
+ ? null
+ :
metadataProvider.getSerializer(DataCatalog.class).load(catalogName);
+ if (dataCatalog == null) {
+ throw new HopException(
+ BaseMessages.getString(PKG, "MaskFields.Error.MissingCatalog",
catalogName));
+ }
+ // A tag whose variable isn't set would match nothing and leave every
field unmasked
+ List<MaskTagRule> rules = meta.activeTagRules();
+ List<String> labels = new ArrayList<>();
+ for (MaskTagRule rule : rules) {
+ String label = resolvedTag(variables, rule);
+ if (label == null) {
+ throw new HopException(
+ BaseMessages.getString(PKG, "MaskFields.Error.UnresolvedTag",
rule.getTag()));
+ }
+ labels.add(label);
+ }
+ DatasetRef ref = meta.catalogDataset().toRef(variables, metadataProvider);
+ try (IDataCatalog catalog = dataCatalog.connect(variables,
metadataProvider)) {
+ CatalogEntityId id = known(catalog.resolve(ref), ref, catalogName);
+ ISchemaCapability schemaCapability =
+ catalog
+ .capability(ISchemaCapability.class)
+ .orElseThrow(
+ () ->
+ new HopException(
+ BaseMessages.getString(
+ PKG, "MaskFields.Error.NoSchemaSupport",
catalogName)));
+ List<CatalogField> fields = known(schemaCapability.schema(id), ref,
catalogName);
+
+ Map<String, Match> matches = new LinkedHashMap<>();
+ for (CatalogField field : fields) {
+ for (int i = 0; i < rules.size(); i++) {
+ if (field.isLabelledWith(labels.get(i))) {
+ matches.putIfAbsent(
+ field.name().toLowerCase(Locale.ROOT),
+ new Match(field.name(), rules.get(i).getPatternName(),
labels.get(i)));
+ break;
+ }
+ }
+ }
+ log.logBasic(
+ BaseMessages.getString(
+ PKG,
+ "MaskFields.Log.CatalogFields",
+ catalogName,
+ ref.describe(),
+ Integer.toString(matches.size()),
+ Integer.toString(fields.size())));
+ return matches;
+ }
+ }
+
+ /** The tag or term of a rule with its variables resolved, or null when one
isn't set. */
+ static String resolvedTag(IVariables variables, MaskTagRule rule) {
+ String label = StringUtils.trimToNull(variables.resolve(rule.getTag()));
+ return label == null || label.contains("${") ? null : label;
+ }
+
+ /** The answer, or an exception saying why there is none. */
+ private <T> T known(CatalogAnswer<T> answer, DatasetRef ref, String
catalogName)
+ throws HopException {
+ return switch (answer) {
+ case CatalogAnswer.Known<T> known -> known.value();
+ case CatalogAnswer.NotFound<T> notFound ->
+ throw new HopException(
+ BaseMessages.getString(
+ PKG, "MaskFields.Error.CatalogNotFound", catalogName,
ref.describe()));
+ case CatalogAnswer.Unknown<T> unknown ->
+ throw new HopException(
+ BaseMessages.getString(
+ PKG,
+ "MaskFields.Error.CatalogUnknown",
+ catalogName,
+ ref.describe(),
+ unknown.reason().name(),
+ StringUtils.defaultString(unknown.detail())));
+ };
+ }
+}
diff --git
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFields.java
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFields.java
index 860a78a16f..bd04625920 100644
---
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFields.java
+++
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFields.java
@@ -18,9 +18,12 @@
package org.apache.hop.pipeline.transforms.maskfields;
import java.util.ArrayList;
+import java.util.HashSet;
import java.util.LinkedHashMap;
import java.util.List;
+import java.util.Locale;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.commons.lang3.StringUtils;
import org.apache.hop.core.database.DatabaseMeta;
@@ -61,8 +64,7 @@ public class MaskFields extends BaseTransform<MaskFieldsMeta,
MaskFieldsData> {
data.lease = lease;
try {
List<Binding> bindings = new ArrayList<>();
- Map<String, MaskingPattern> patterns = new LinkedHashMap<>();
- String transformName = getTransformMeta() == null ? "" :
getTransformMeta().getName();
+ data.patterns = new LinkedHashMap<>();
if (meta.getFields() != null) {
for (MaskField field : meta.getFields()) {
if (field == null
@@ -70,27 +72,18 @@ public class MaskFields extends
BaseTransform<MaskFieldsMeta, MaskFieldsData> {
|| StringUtils.isEmpty(field.getPatternName())) {
continue;
}
- MaskingPattern pattern = patterns.get(field.getPatternName());
- if (pattern == null) {
- pattern = loadRequired(field.getPatternName());
- patterns.put(field.getPatternName(), pattern);
- }
- long start = parseStart(pattern);
- // Check the hash secret before the mapping table is opened.
- MaskingKey key = MaskingKey.forPattern(pattern, this);
- IMaskingStore store = storeFor(pattern, lease);
- bindings.add(
- new Binding(
- field.getFieldName(),
- pattern,
- resolve(pattern.getPrefix()),
- resolve(pattern.getSuffix()),
- start,
- store,
- sharedSequence(lease, transformName, field, pattern, start),
- key));
+ bindings.add(bind(field));
}
}
+ data.catalogFields =
+ meta.activeTagRules().isEmpty()
+ ? Map.of()
+ : new CatalogFieldResolver(this, getLogChannel(), meta,
getMetadataProvider())
+ .resolve();
+ for (CatalogFieldResolver.Match match : data.catalogFields.values()) {
+ // A missing pattern fails now, not at the first row
+ pattern(match.patternName());
+ }
data.engine = new MaskingEngine(bindings);
IPipelineEngine<PipelineMeta> pipeline = getPipeline();
if (pipeline != null) {
@@ -114,6 +107,7 @@ public class MaskFields extends
BaseTransform<MaskFieldsMeta, MaskFieldsData> {
}
if (data.outputRowMeta == null) {
data.outputRowMeta = getInputRowMeta().clone();
+ bindCatalogFields(data.outputRowMeta);
validateTypes(data.outputRowMeta);
}
try {
@@ -161,6 +155,72 @@ public class MaskFields extends
BaseTransform<MaskFieldsMeta, MaskFieldsData> {
super.dispose();
}
+ /** One field and its pattern, ready to mask. */
+ private Binding bind(MaskField field) throws HopException {
+ MaskingPattern pattern = pattern(field.getPatternName());
+ long start = parseStart(pattern);
+ // Check the hash secret before the mapping table is opened.
+ MaskingKey key = MaskingKey.forPattern(pattern, this);
+ IMaskingStore store = storeFor(pattern, data.lease);
+ String transformName = getTransformMeta() == null ? "" :
getTransformMeta().getName();
+ return new Binding(
+ field.getFieldName(),
+ pattern,
+ resolve(pattern.getPrefix()),
+ resolve(pattern.getSuffix()),
+ start,
+ store,
+ sharedSequence(data.lease, transformName, field, pattern, start),
+ key);
+ }
+
+ private MaskingPattern pattern(String name) throws HopException {
+ MaskingPattern pattern = data.patterns.get(name);
+ if (pattern == null) {
+ pattern = loadRequired(name);
+ data.patterns.put(name, pattern);
+ }
+ return pattern;
+ }
+
+ /**
+ * Masks the incoming fields the data catalog labels with a tag or term of
this transform. Done at
+ * the first row, from the fields that really arrive: a field that only
shows up at run time is
+ * masked too. A field with a row of its own keeps the pattern of that row.
+ */
+ private void bindCatalogFields(IRowMeta rowMeta) throws HopException {
+ if (data.catalogFields.isEmpty()) {
+ return;
+ }
+ Set<String> explicit = new HashSet<>();
+ for (Binding binding : data.engine.getBindings()) {
+ explicit.add(binding.fieldName.toLowerCase(Locale.ROOT));
+ }
+ Set<String> arrived = new HashSet<>();
+ for (IValueMeta valueMeta : rowMeta.getValueMetaList()) {
+ String key = valueMeta.getName().toLowerCase(Locale.ROOT);
+ arrived.add(key);
+ CatalogFieldResolver.Match match = data.catalogFields.get(key);
+ if (match == null || explicit.contains(key)) {
+ continue;
+ }
+ data.engine.getBindings().add(bind(new MaskField(valueMeta.getName(),
match.patternName())));
+ logBasic(
+ BaseMessages.getString(
+ PKG,
+ "MaskFields.Log.CatalogField",
+ valueMeta.getName(),
+ match.patternName(),
+ match.label()));
+ }
+ for (CatalogFieldResolver.Match match : data.catalogFields.values()) {
+ if (!arrived.contains(match.fieldName().toLowerCase(Locale.ROOT))) {
+ logDetailed(
+ BaseMessages.getString(PKG, "MaskFields.Log.CatalogFieldAbsent",
match.fieldName()));
+ }
+ }
+ }
+
private MaskingPattern loadRequired(String name) throws HopException {
MaskingPattern pattern;
try {
diff --git
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsData.java
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsData.java
index af25d52002..d753b1c2bc 100644
---
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsData.java
+++
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsData.java
@@ -17,6 +17,8 @@
package org.apache.hop.pipeline.transforms.maskfields;
+import java.util.LinkedHashMap;
+import java.util.Map;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.pipeline.transform.BaseTransformData;
import org.apache.hop.pipeline.transform.ITransformData;
@@ -26,4 +28,12 @@ public class MaskFieldsData extends BaseTransformData
implements ITransformData
public IRowMeta outputRowMeta;
public MaskingEngine engine;
public MaskingRuntime.Lease lease;
+
+ /** The patterns of this transform, by name. */
+ public Map<String, MaskingPattern> patterns = new LinkedHashMap<>();
+
+ /**
+ * The fields the data catalog labels with a tag or term of this transform,
by lower-case name.
+ */
+ public Map<String, CatalogFieldResolver.Match> catalogFields = Map.of();
}
diff --git
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsDialog.java
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsDialog.java
index aeb336f464..c922f45c10 100644
---
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsDialog.java
+++
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsDialog.java
@@ -228,12 +228,14 @@ public class MaskFieldsDialog extends BaseTransformDialog
{
}
private void installPatternComboSupplier() {
- TableView table = fieldsTable();
- if (table == null || table.getColumns().length <= 1) {
- return;
+ for (String id : new String[] {MaskFieldsMeta.WIDGET_FIELDS,
MaskFieldsMeta.WIDGET_TAG_RULES}) {
+ Control control = widgets.getWidgetsMap().get(id);
+ if (control instanceof TableView table && table.getColumns().length > 1)
{
+ table.getColumns()[1].setComboValueSupplier(
+ () ->
+ input.patternNames(LogChannel.UI,
hopGuiMetadataProvider()).toArray(new String[0]));
+ }
}
- table.getColumns()[1].setComboValueSupplier(
- () -> input.patternNames(LogChannel.UI,
hopGuiMetadataProvider()).toArray(new String[0]));
}
private TableView fieldsTable() {
diff --git
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMeta.java
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMeta.java
index 3432f4e1de..eafba8dc5e 100644
---
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMeta.java
+++
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMeta.java
@@ -38,8 +38,12 @@ import org.apache.hop.core.logging.ILogChannel;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.datacatalog.CatalogDataset;
+import org.apache.hop.datacatalog.CatalogDatasetType;
+import org.apache.hop.datacatalog.DataCatalog;
import org.apache.hop.i18n.BaseMessages;
import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.metadata.api.HopMetadataPropertyType;
import org.apache.hop.metadata.api.IHopMetadataProvider;
import org.apache.hop.metadata.api.IHopMetadataSerializer;
import org.apache.hop.pipeline.PipelineMeta;
@@ -66,6 +70,8 @@ public class MaskFieldsMeta extends
BaseTransformMeta<MaskFields, MaskFieldsData
public static final String GUI_PLUGIN_ELEMENT_PARENT_ID =
"MaskFields.Dialog";
public static final String WIDGET_EDIT_RULE = "editRule";
public static final String WIDGET_FIELDS = "fields";
+ public static final String WIDGET_TAG_RULES = "tagRules";
+ private static final String GROUP_CATALOG = "i18n::MaskFields.Group.Catalog";
/** Column index of the masking rule in the fields table. 0 is the row
number. */
public static final int RULE_COLUMN = 2;
@@ -100,13 +106,127 @@ public class MaskFieldsMeta extends
BaseTransformMeta<MaskFields, MaskFieldsData
tableRows = 8)
private List<MaskField> fields = new ArrayList<>();
+ /** The data catalog that says which fields carry a tag or term. Empty: no
tag rules. */
+ @GuiWidgetElement(
+ id = "catalogName",
+ order = "1000",
+ type = GuiElementType.METADATA,
+ metadata = DataCatalog.class,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_CATALOG,
+ groupOrder = "20",
+ label = "i18n::MaskFields.Catalog.Label",
+ toolTip = "i18n::MaskFields.Catalog.Tooltip")
+ @HopMetadataProperty(
+ key = "data_catalog",
+ hopMetadataPropertyType = HopMetadataPropertyType.DATA_CATALOG,
+ injectionKeyDescription = "MaskFields.Injection.Catalog")
+ private String catalogName;
+
+ @GuiWidgetElement(
+ id = "datasetType",
+ order = "1100",
+ type = GuiElementType.COMBO,
+ variables = false,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_CATALOG,
+ groupOrder = "20",
+ label = "i18n::MaskFields.DatasetType.Label",
+ toolTip = "i18n::MaskFields.DatasetType.Tooltip")
+ @HopMetadataProperty(key = "dataset_type")
+ private CatalogDatasetType datasetType;
+
+ @GuiWidgetElement(
+ id = "datasetConnection",
+ order = "1200",
+ type = GuiElementType.METADATA,
+ metadata = DatabaseMeta.class,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_CATALOG,
+ groupOrder = "20",
+ label = "i18n::MaskFields.DatasetConnection.Label",
+ toolTip = "i18n::MaskFields.DatasetConnection.Tooltip")
+ @HopMetadataProperty(
+ key = "dataset_connection",
+ hopMetadataPropertyType = HopMetadataPropertyType.RDBMS_CONNECTION)
+ private String datasetConnection;
+
+ @GuiWidgetElement(
+ id = "datasetSchema",
+ order = "1300",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_CATALOG,
+ groupOrder = "20",
+ label = "i18n::MaskFields.DatasetSchema.Label",
+ toolTip = "i18n::MaskFields.DatasetSchema.Tooltip")
+ @HopMetadataProperty(
+ key = "dataset_schema",
+ hopMetadataPropertyType = HopMetadataPropertyType.RDBMS_SCHEMA)
+ private String datasetSchema;
+
+ @GuiWidgetElement(
+ id = "datasetName",
+ order = "1400",
+ type = GuiElementType.TEXT,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_CATALOG,
+ groupOrder = "20",
+ label = "i18n::MaskFields.DatasetName.Label",
+ toolTip = "i18n::MaskFields.DatasetName.Tooltip")
+ @HopMetadataProperty(key = "dataset_name")
+ private String datasetName;
+
+ /** Masks every incoming field the catalog labels with one of these tags or
terms. */
+ @HopMetadataProperty(key = "tag_rule", groupKey = "tag_rules")
+ @GuiWidgetElement(
+ id = WIDGET_TAG_RULES,
+ order = "1500",
+ type = GuiElementType.TABLE,
+ parentId = GUI_PLUGIN_ELEMENT_PARENT_ID,
+ groupType = GuiWidgetGroupType.BOXES,
+ group = GROUP_CATALOG,
+ groupOrder = "20",
+ toolTip = "i18n::MaskFields.TagRules.Tooltip",
+ tableRows = 4)
+ private List<MaskTagRule> tagRules = new ArrayList<>();
+
public MaskFieldsMeta() {
this.fields = new ArrayList<>();
+ this.tagRules = new ArrayList<>();
+ this.datasetType = CatalogDatasetType.TABLE;
}
@Override
public void setDefault() {
fields = new ArrayList<>();
+ tagRules = new ArrayList<>();
+ datasetType = CatalogDatasetType.TABLE;
+ }
+
+ /** The tag rules with a tag and a pattern. */
+ public List<MaskTagRule> activeTagRules() {
+ List<MaskTagRule> active = new ArrayList<>();
+ if (tagRules != null) {
+ for (MaskTagRule rule : tagRules) {
+ if (rule != null
+ && StringUtils.isNotBlank(rule.getTag())
+ && StringUtils.isNotBlank(rule.getPatternName())) {
+ active.add(rule);
+ }
+ }
+ }
+ return active;
+ }
+
+ /** The dataset to read the tags and terms of. */
+ public CatalogDataset catalogDataset() {
+ return new CatalogDataset(datasetType, datasetConnection, datasetSchema,
datasetName);
}
/** Pattern names for the grid combo. */
@@ -153,6 +273,22 @@ public class MaskFieldsMeta extends
BaseTransformMeta<MaskFields, MaskFieldsData
if (input == null || input.length == 0) {
error(remarks, transformMeta, "MaskFields.Check.NoInput");
}
+ if (!activeTagRules().isEmpty()) {
+ if (StringUtils.isBlank(catalogName)) {
+ error(remarks, transformMeta,
"MaskFields.Check.TagRulesWithoutCatalog");
+ }
+ if (StringUtils.isBlank(datasetName)) {
+ error(remarks, transformMeta,
"MaskFields.Check.TagRulesWithoutDataset");
+ }
+ for (MaskTagRule rule : activeTagRules()) {
+ if (CatalogFieldResolver.resolvedTag(variables, rule) == null) {
+ error(remarks, transformMeta, "MaskFields.Check.UnresolvedTag",
rule.getTag());
+ }
+ if (loadQuietly(metadataProvider, rule.getPatternName()) == null) {
+ error(remarks, transformMeta, "MaskFields.Check.MissingPattern",
rule.getPatternName());
+ }
+ }
+ }
Set<String> seen = new HashSet<>();
if (fields != null) {
diff --git
a/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskTagRule.java
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskTagRule.java
new file mode 100644
index 0000000000..d1e8de1d91
--- /dev/null
+++
b/plugins/transforms/maskfields/src/main/java/org/apache/hop/pipeline/transforms/maskfields/MaskTagRule.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.maskfields;
+
+import lombok.Getter;
+import lombok.Setter;
+import org.apache.hop.core.gui.plugin.GuiTableColumn;
+import org.apache.hop.core.gui.plugin.GuiTableColumnType;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+
+/**
+ * A tag or glossary term in a data catalog, and the masking pattern for every
field the catalog
+ * labels with it.
+ */
+@Getter
+@Setter
+public class MaskTagRule {
+
+ // Injection keys of their own: "pattern" is the injection key of the Fields
table
+ @HopMetadataProperty(
+ key = "tag",
+ injectionKey = "tag_rule_tag",
+ injectionKeyDescription = "MaskFields.Injection.Tag")
+ @GuiTableColumn(
+ id = "tag",
+ order = "10",
+ type = GuiTableColumnType.TEXT,
+ label = "i18n::MaskFields.Column.Tag.Label",
+ variables = true,
+ width = 200)
+ private String tag = "";
+
+ @HopMetadataProperty(
+ key = "pattern",
+ injectionKey = "tag_rule_pattern",
+ injectionKeyDescription = "MaskFields.Injection.TagPattern")
+ @GuiTableColumn(
+ id = "patternName",
+ order = "20",
+ type = GuiTableColumnType.COMBO,
+ comboValuesMethod = "patternNames",
+ label = "i18n::MaskFields.Column.Pattern.Label",
+ variables = false,
+ width = 240)
+ private String patternName = "";
+
+ public MaskTagRule() {}
+
+ public MaskTagRule(String tag, String patternName) {
+ this.tag = tag;
+ this.patternName = patternName;
+ }
+}
diff --git
a/plugins/transforms/maskfields/src/main/resources/org/apache/hop/pipeline/transforms/maskfields/messages/messages_en_US.properties
b/plugins/transforms/maskfields/src/main/resources/org/apache/hop/pipeline/transforms/maskfields/messages/messages_en_US.properties
index 9872821700..19eb44b7c4 100644
---
a/plugins/transforms/maskfields/src/main/resources/org/apache/hop/pipeline/transforms/maskfields/messages/messages_en_US.properties
+++
b/plugins/transforms/maskfields/src/main/resources/org/apache/hop/pipeline/transforms/maskfields/messages/messages_en_US.properties
@@ -105,3 +105,30 @@ MaskingPattern.Schema.Label=Schema
MaskingPattern.Schema.Tooltip=Schema of the mapping table. Leave this empty to
use the connection default.
MaskingPattern.Table.Label=Table
MaskingPattern.Table.Tooltip=Mapping table. A second table with the suffix
_seq stores the next sequence number. Both are created when they are missing. A
new mapping table holds replacements of up to 255 characters.
+MaskFields.Group.Catalog=Data catalog
+MaskFields.Catalog.Label=Data catalog
+MaskFields.Catalog.Tooltip=The data catalog, such as DataHub, that says which
fields carry the tags and terms below. Leave it empty not to use one.
+MaskFields.DatasetType.Label=Dataset type
+MaskFields.DatasetType.Tooltip=How the dataset is named: a table of a
relational connection, a file, or the id of the dataset in the catalog
+MaskFields.DatasetConnection.Label=Dataset connection
+MaskFields.DatasetConnection.Tooltip=The relational connection of a table
+MaskFields.DatasetSchema.Label=Dataset schema
+MaskFields.DatasetSchema.Tooltip=The schema of a table. Leave it empty for the
preferred schema of the connection.
+MaskFields.DatasetName.Label=Dataset
+MaskFields.DatasetName.Tooltip=The table, the file name, or the id of the
dataset in the catalog whose tags and terms say which fields to mask
+MaskFields.TagRules.Tooltip=Every incoming field that the catalog labels with
the tag or glossary term on a row is masked with that row''s pattern. The first
row that matches wins, and a field in the Fields table keeps its own pattern.
+MaskFields.Column.Tag.Label=Tag or term
+MaskFields.Injection.Catalog=Data catalog
+MaskFields.Injection.Tag=Tag or glossary term
+MaskFields.Injection.TagPattern=Masking pattern of the tag or term
+MaskFields.Check.TagRulesWithoutCatalog=The tag and term rules need a data
catalog
+MaskFields.Check.TagRulesWithoutDataset=The tag and term rules need a dataset
to read the tags and terms of
+MaskFields.Check.UnresolvedTag=Tag or term ''{0}'' holds a variable that is
not set
+MaskFields.Error.UnresolvedTag=Tag or term ''{0}'' holds a variable that is
not set: the fields to mask are unknown, so the transform doesn''t start
+MaskFields.Error.MissingCatalog=Data catalog ''{0}'' can''t be found
+MaskFields.Error.NoSchemaSupport=Data catalog ''{0}'' can''t tell the tags and
terms of a dataset''s fields
+MaskFields.Error.CatalogNotFound=Data catalog ''{0}'' has no dataset {1}, or
no fields for it: the fields to mask are unknown, so the transform doesn''t
start
+MaskFields.Error.CatalogUnknown=Data catalog ''{0}'' could not say which
fields of {1} to mask ({2}: {3}), so the transform doesn''t start
+MaskFields.Log.CatalogFields=Data catalog ''{0}'' labels {2} of the {3} fields
of {1} with a tag or term to mask
+MaskFields.Log.CatalogField=Field {0} is masked with pattern ''{1}'': the data
catalog labels it ''{2}''
+MaskFields.Log.CatalogFieldAbsent=Field {0} is labelled in the data catalog,
but doesn''t reach this transform
diff --git
a/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMetaTest.java
b/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMetaTest.java
index 130046b72e..c7df6a983c 100644
---
a/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMetaTest.java
+++
b/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsMetaTest.java
@@ -28,15 +28,24 @@ import java.util.ArrayList;
import java.util.List;
import org.apache.hop.core.CheckResult;
import org.apache.hop.core.ICheckResult;
+import org.apache.hop.core.RowMetaAndData;
+import org.apache.hop.core.injection.bean.BeanInjectionInfo;
+import org.apache.hop.core.injection.bean.BeanInjector;
+import org.apache.hop.core.logging.HopLogStore;
+import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.RowMeta;
+import org.apache.hop.core.row.RowMetaBuilder;
import org.apache.hop.core.row.value.ValueMetaDate;
import org.apache.hop.core.row.value.ValueMetaInteger;
import org.apache.hop.core.row.value.ValueMetaString;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.datacatalog.CatalogDatasetType;
import org.apache.hop.metadata.serializer.json.JsonMetadataParser;
import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
+import org.apache.hop.metadata.serializer.xml.XmlMetadataUtil;
import org.apache.hop.pipeline.transform.TransformMeta;
+import org.apache.hop.pipeline.transform.TransformSerializationTestUtil;
import org.json.simple.JSONObject;
import org.junit.jupiter.api.Test;
import org.w3c.dom.Node;
@@ -174,6 +183,75 @@ class MaskFieldsMetaTest {
}
}
+ @Test
+ void theDataCatalogOptionsSurviveASaveAndLoad() throws Exception {
+ MaskFieldsMeta meta =
+ TransformSerializationTestUtil.testSerialization(
+ "/mask-fields-catalog-transform.xml", MaskFieldsMeta.class);
+
+ assertEquals("prod-datahub", meta.getCatalogName());
+ assertEquals(CatalogDatasetType.TABLE, meta.getDatasetType());
+ assertEquals("warehouse", meta.getDatasetConnection());
+ assertEquals("crm", meta.getDatasetSchema());
+ assertEquals("customers", meta.getDatasetName());
+ assertEquals(2, meta.activeTagRules().size());
+ assertEquals("Classification.Personal",
meta.getTagRules().get(1).getTag());
+ assertEquals("Hidden", meta.getTagRules().get(1).getPatternName());
+ assertEquals(1, meta.getFields().size());
+ }
+
+ @Test
+ void aTransformSavedBeforeTheCatalogOptionsHasNoTagRules() throws Exception {
+ Node node =
+ XmlHandler.getSubNode(
+ XmlHandler.loadXmlString(
+ "<transform><fields><field><name>name</name><pattern>First
name</pattern>"
+ + "</field></fields></transform>"),
+ "transform");
+
+ MaskFieldsMeta meta =
+ XmlMetadataUtil.deSerializeFromXml(
+ node, MaskFieldsMeta.class, new MemoryMetadataProvider());
+
+ assertTrue(meta.activeTagRules().isEmpty());
+ assertNull(meta.getCatalogName());
+ assertEquals(1, meta.getFields().size());
+ }
+
+ @Test
+ void injectedFieldsKeepTheirPatternBesideTheTagRules() throws Exception {
+ HopLogStore.init();
+ BeanInjectionInfo<MaskFieldsMeta> info = new
BeanInjectionInfo<>(MaskFieldsMeta.class);
+ BeanInjector<MaskFieldsMeta> injector = new BeanInjector<>(info, new
MemoryMetadataProvider());
+ IRowMeta rowMeta =
+ new RowMetaBuilder()
+ .addString("field")
+ .addString("pattern")
+ .addString("tag")
+ .addString("tag pattern")
+ .build();
+ List<RowMetaAndData> rows =
+ List.of(new RowMetaAndData(rowMeta, "email", "Masked email", "pii",
"Hidden"));
+
+ MaskFieldsMeta meta = new MaskFieldsMeta();
+ // The injection keys of Mask fields before the tag rules
+ injector.setProperty(meta, "name", rows, "field");
+ injector.setProperty(meta, "pattern", rows, "pattern");
+
+ assertEquals(1, meta.getFields().size());
+ assertEquals("email", meta.getFields().get(0).getFieldName());
+ assertEquals("Masked email", meta.getFields().get(0).getPatternName());
+ assertTrue(meta.getTagRules().isEmpty());
+
+ injector.setProperty(meta, "tag_rule_tag", rows, "tag");
+ injector.setProperty(meta, "tag_rule_pattern", rows, "tag pattern");
+
+ assertEquals("Masked email", meta.getFields().get(0).getPatternName());
+ assertEquals(1, meta.getTagRules().size());
+ assertEquals("pii", meta.getTagRules().get(0).getTag());
+ assertEquals("Hidden", meta.getTagRules().get(0).getPatternName());
+ }
+
private static MaskingPattern synthetic(String name, MaskingToken token) {
MaskingPattern pattern = new MaskingPattern();
pattern.setName(name);
diff --git
a/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsTest.java
b/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsTest.java
index 2ad8f79698..6515a03587 100644
---
a/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsTest.java
+++
b/plugins/transforms/maskfields/src/test/java/org/apache/hop/pipeline/transforms/maskfields/MaskFieldsTest.java
@@ -25,7 +25,11 @@ import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;
import java.util.Date;
+import java.util.List;
+import java.util.Set;
import java.util.concurrent.TimeUnit;
+import lombok.Getter;
+import lombok.Setter;
import org.apache.hop.core.BlockingRowSet;
import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.logging.ILoggingObject;
@@ -33,7 +37,17 @@ import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.row.value.ValueMetaDate;
import org.apache.hop.core.row.value.ValueMetaString;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.datacatalog.CatalogAnswer;
+import org.apache.hop.datacatalog.CatalogDatasetType;
+import org.apache.hop.datacatalog.CatalogEntityId;
+import org.apache.hop.datacatalog.DataCatalog;
+import org.apache.hop.datacatalog.DatasetRef;
+import org.apache.hop.datacatalog.IDataCatalog;
+import org.apache.hop.datacatalog.ISchemaCapability;
+import org.apache.hop.datacatalog.ISchemaCapability.CatalogField;
import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import org.apache.hop.metadata.api.IHopMetadataProvider;
import org.apache.hop.metadata.serializer.memory.MemoryMetadataProvider;
import org.apache.hop.pipeline.transforms.mock.TransformMockHelper;
import org.junit.jupiter.api.AfterEach;
@@ -191,4 +205,165 @@ class MaskFieldsTest {
rowSet.setRowMeta(rowMeta);
return rowSet;
}
+
+ /** A data catalog that answers with the schema each test sets. */
+ @Getter
+ @Setter
+ public static class SchemaCatalog implements IDataCatalog, ISchemaCapability
{
+ static CatalogAnswer<CatalogEntityId> resolved;
+ static CatalogAnswer<List<CatalogField>> schema;
+
+ private String pluginId = "schema";
+ private String pluginName = "Schema catalog";
+
+ @Override
+ public SchemaCatalog clone() {
+ return new SchemaCatalog();
+ }
+
+ @Override
+ public void initialize(IVariables variables, IHopMetadataProvider
metadataProvider) {
+ // Nothing to prepare
+ }
+
+ @Override
+ public CatalogAnswer<String> ping() {
+ return CatalogAnswer.known("schema");
+ }
+
+ @Override
+ public CatalogAnswer<CatalogEntityId> resolve(DatasetRef dataset) {
+ return resolved;
+ }
+
+ @Override
+ public CatalogAnswer<List<CatalogField>> schema(CatalogEntityId id) {
+ return schema;
+ }
+
+ @Override
+ public void close() {
+ // Nothing to release
+ }
+ }
+
+ private static CatalogField field(String name, Set<String> tags, Set<String>
terms) {
+ return new CatalogField(name, null, "STRING", null, tags, terms);
+ }
+
+ /** Patterns, a catalog whose customers table tags email PII and terms name
Personal. */
+ private MemoryMetadataProvider catalogProvider() throws Exception {
+ SchemaCatalog.resolved = CatalogAnswer.known(new
CatalogEntityId("urn:customers"));
+ SchemaCatalog.schema =
+ CatalogAnswer.known(
+ List.of(
+ field("email", Set.of("PII", "urn:li:tag:PII"), Set.of()),
+ field("name", Set.of(), Set.of("Personal")),
+ field("city", Set.of(), Set.of())));
+ MemoryMetadataProvider provider = new MemoryMetadataProvider();
+ MaskingPattern email = new MaskingPattern();
+ email.setName("Masked email");
+ email.setValueSource(MaskingValueSource.SYNTHETIC);
+ email.setToken(MaskingToken.SEQUENCE);
+ email.setStorage(MaskingStorage.NONE);
+ email.setPrefix("email-");
+ email.setSequenceStart("1");
+ provider.getSerializer(MaskingPattern.class).save(email);
+ MaskingPattern hidden = new MaskingPattern();
+ hidden.setName("Hidden");
+ hidden.setValueSource(MaskingValueSource.SET_NULL);
+ provider.getSerializer(MaskingPattern.class).save(hidden);
+ provider
+ .getSerializer(DataCatalog.class)
+ .save(new DataCatalog("catalog", null, new SchemaCatalog()));
+ return provider;
+ }
+
+ private static MaskFieldsMeta catalogMeta() {
+ MaskFieldsMeta meta = new MaskFieldsMeta();
+ meta.setCatalogName("catalog");
+ meta.setDatasetType(CatalogDatasetType.CATALOG_ID);
+ meta.setDatasetName("urn:customers");
+ meta.getTagRules().add(new MaskTagRule("pii", "Masked email"));
+ meta.getTagRules().add(new MaskTagRule("Personal", "Hidden"));
+ return meta;
+ }
+
+ private Object[] maskOneRow(MaskFieldsMeta meta, MemoryMetadataProvider
provider)
+ throws Exception {
+ MaskFields transform = createTransform(meta);
+ transform.setMetadataProvider(provider);
+ assertTrue(transform.init());
+ RowMeta rowMeta = new RowMeta();
+ rowMeta.addValueMeta(new ValueMetaString("EMAIL"));
+ rowMeta.addValueMeta(new ValueMetaString("name"));
+ rowMeta.addValueMeta(new ValueMetaString("city"));
+ BlockingRowSet output = output(rowMeta);
+ transform.addRowSetToInputRowSets(
+ input("people", rowMeta, new Object[] {"[email protected]", "Ann",
"Gent"}));
+ transform.addRowSetToOutputRowSets(output);
+ assertTrue(transform.processRow());
+ Object[] row = output.getRowWait(1, TimeUnit.SECONDS);
+ transform.dispose();
+ return row;
+ }
+
+ @Test
+ void aFieldTheCatalogLabelsIsMaskedWithThePatternOfItsTagOrTerm() throws
Exception {
+ Object[] row = maskOneRow(catalogMeta(), catalogProvider());
+
+ // EMAIL matches email whatever the case; city carries no tag and is left
alone
+ assertArrayEquals(new Object[] {"email-1", null, "Gent"}, row);
+ }
+
+ @Test
+ void aFieldWithARowOfItsOwnKeepsThatPattern() throws Exception {
+ MaskFieldsMeta meta = catalogMeta();
+ meta.getFields().add(new MaskField("name", "Masked email"));
+
+ Object[] row = maskOneRow(meta, catalogProvider());
+
+ // name keeps its own pattern instead of the Hidden of its term; each
field counts on its own
+ assertArrayEquals(new Object[] {"email-1", "email-1", "Gent"}, row);
+ }
+
+ @Test
+ void aCatalogThatCannotAnswerStopsTheTransform() throws Exception {
+ MemoryMetadataProvider provider = catalogProvider();
+ SchemaCatalog.schema =
CatalogAnswer.unknown(CatalogAnswer.Reason.UNREACHABLE, "refused");
+ MaskFields transform = createTransform(catalogMeta());
+ transform.setMetadataProvider(provider);
+
+ assertFalse(transform.init());
+ }
+
+ @Test
+ void aDatasetTheCatalogDoesNotHaveStopsTheTransform() throws Exception {
+ MemoryMetadataProvider provider = catalogProvider();
+ SchemaCatalog.resolved = CatalogAnswer.notFound();
+ MaskFields transform = createTransform(catalogMeta());
+ transform.setMetadataProvider(provider);
+
+ assertFalse(transform.init());
+ }
+
+ @Test
+ void aTagWhoseVariableIsNotSetStopsTheTransform() throws Exception {
+ MaskFieldsMeta meta = catalogMeta();
+ meta.getTagRules().add(new MaskTagRule("${PII_TAG}", "Hidden"));
+ MaskFields transform = createTransform(meta);
+ transform.setMetadataProvider(catalogProvider());
+
+ assertFalse(transform.init());
+ }
+
+ @Test
+ void tagRulesWithoutACatalogStopTheTransform() throws Exception {
+ MaskFieldsMeta meta = catalogMeta();
+ meta.setCatalogName("");
+ MaskFields transform = createTransform(meta);
+ transform.setMetadataProvider(catalogProvider());
+
+ assertFalse(transform.init());
+ }
}
diff --git
a/plugins/transforms/maskfields/src/test/resources/mask-fields-catalog-transform.xml
b/plugins/transforms/maskfields/src/test/resources/mask-fields-catalog-transform.xml
new file mode 100644
index 0000000000..64b7148232
--- /dev/null
+++
b/plugins/transforms/maskfields/src/test/resources/mask-fields-catalog-transform.xml
@@ -0,0 +1,41 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+<transform>
+ <fields>
+ <field>
+ <name>name</name>
+ <pattern>First name</pattern>
+ </field>
+ </fields>
+ <data_catalog>prod-datahub</data_catalog>
+ <dataset_type>TABLE</dataset_type>
+ <dataset_connection>warehouse</dataset_connection>
+ <dataset_schema>crm</dataset_schema>
+ <dataset_name>customers</dataset_name>
+ <tag_rules>
+ <tag_rule>
+ <tag>PII</tag>
+ <pattern>Email</pattern>
+ </tag_rule>
+ <tag_rule>
+ <tag>Classification.Personal</tag>
+ <pattern>Hidden</pattern>
+ </tag_rule>
+ </tag_rules>
+</transform>