dmainou opened a new issue, #7571:
URL: https://github.com/apache/hop/issues/7571
### What would you like to happen?
## Summary
Enhance the **Data Validator** transform to support loading validation rules
from a **Static Schema Definition** (Metadata Perspective) using a JSON
document.
Instead of manually configuring every validation rule inside the transform,
users would be able to reference a JSON schema stored in the Metadata
Perspective. The transform would interpret the schema and create the validation
rules at runtime.
This would allow validation logic to be externalised from the pipeline while
remaining version-controlled, reusable and understandable by non-Hop users.
---
## Problem
The current Data Validator requires all validation rules to be manually
configured within the transform.
This becomes difficult when:
- Validation rules are generated externally.
- Multiple test suites require different validation sets.
- The same validation logic is reused across many pipelines.
- Governance teams need to review or approve validation rules.
- Validation rules evolve independently of pipeline logic.
Changing validation rules currently requires editing the pipeline itself.
---
## Proposed Solution
Add support for selecting a **Static Schema Definition** from the Metadata
Perspective.
The selected schema would contain the validator definitions in JSON.
The Data Validator would:
1. Load the JSON schema.
2. Convert it internally into the existing validator definitions.
3. Execute validations exactly as it does today.
This preserves full backwards compatibility while allowing validations to be
managed independently of the pipeline.
---
## Example
### Current XML representation
```xml
<validator_field>
<allowed_value>
<value>AAA</value>
<value>ABC</value>
</allowed_value>
<conversion_mask>ddmm</conversion_mask>
<data_type>String</data_type>
<data_type_verified>Y</data_type_verified>
<decimal_symbol>.</decimal_symbol>
<end_string>@#$</end_string>
<end_string_not_allowed>df</end_string_not_allowed>
<error_code>Error1</error_code>
<error_description>Error 1</error_description>
<grouping_symbol>,</grouping_symbol>
<is_sourcing_values>Y</is_sourcing_values>
<max_length>5</max_length>
<max_value>234</max_value>
<min_length>234</min_length>
<min_value>234</min_value>
<name>String1</name>
<null_allowed>Y</null_allowed>
<only_null_allowed>Y</only_null_allowed>
<only_numeric_allowed>Y</only_numeric_allowed>
<regular_expression>.*</regular_expression>
<regular_expression_not_allowed>.*</regular_expression_not_allowed>
<sourcing_field>field</sourcing_field>
<sourcing_transform>src</sourcing_transform>
<start_string>34</start_string>
<start_string_not_allowed>QW</start_string_not_allowed>
<validation_name>Sample1</validation_name>
</validator_field>
<validator_field>
<allowed_value>
<value>df</value>
<value>df</value>
</allowed_value>
<conversion_mask>234rt</conversion_mask>
<data_type>ert</data_type>
<data_type_verified>Y</data_type_verified>
<decimal_symbol>234</decimal_symbol>
<end_string>2wer</end_string>
<end_string_not_allowed>wer</end_string_not_allowed>
<error_code>Error2</error_code>
<error_description>Error2</error_description>
<grouping_symbol>34rt</grouping_symbol>
<is_sourcing_values>Y</is_sourcing_values>
<max_length>234r</max_length>
<max_value>we</max_value>
<min_length>wer</min_length>
<min_value>wer</min_value>
<name>String2</name>
<null_allowed>Y</null_allowed>
<only_null_allowed>Y</only_null_allowed>
<only_numeric_allowed>Y</only_numeric_allowed>
<regular_expression>we</regular_expression>
<regular_expression_not_allowed>we</regular_expression_not_allowed>
<sourcing_field>dfv</sourcing_field>
<start_string>we</start_string>
<start_string_not_allowed>er</start_string_not_allowed>
<validation_name>Sample2</validation_name>
</validator_field>
```
### Equivalent JSON
```json
[
{
"allowedValue": [
"AAA",
"ABC"
],
"conversionMask": "ddmm",
"dataType": "String",
"dataTypeVerified": true,
"decimalSymbol": ".",
"endString": "@#$",
"endStringNotAllowed": "df",
"errorCode": "Error1",
"errorDescription": "Error 1",
"groupingSymbol": ",",
"isSourcingValues": true,
"maxLength": 5,
"maxValue": "234",
"minLength": 234,
"minValue": "234",
"name": "String1",
"nullAllowed": true,
"onlyNullAllowed": true,
"onlyNumericAllowed": true,
"regularExpression": ".*",
"regularExpressionNotAllowed": ".*",
"sourcingField": "field",
"sourcingTransform": "src",
"startString": "34",
"startStringNotAllowed": "QW",
"validationName": "Sample1"
},
{
"allowedValue": [
"df",
"df"
],
"conversionMask": "234rt",
"dataType": "ert",
"dataTypeVerified": true,
"decimalSymbol": "234",
"endString": "2wer",
"endStringNotAllowed": "wer",
"errorCode": "Error2",
"errorDescription": "Error2",
"groupingSymbol": "34rt",
"isSourcingValues": true,
"maxLength": "234r",
"maxValue": "we",
"minLength": "wer",
"minValue": "wer",
"name": "String2",
"nullAllowed": true,
"onlyNullAllowed": true,
"onlyNumericAllowed": true,
"regularExpression": "we",
"regularExpressionNotAllowed": "we",
"sourcingField": "dfv",
"startString": "we",
"startStringNotAllowed": "er",
"validationName": "Sample2"
}
]
```
---
## Benefits
### Reusable validation
The same validation schema can be used across many pipelines without
duplication.
### Easier testing
Entire validation suites can be swapped simply by selecting a different
Static Schema Definition.
For example:
- Development validations
- Integration validations
- Production validations
- Client-specific validations
No pipeline changes required.
### Reusable validation pipelines
A generic validation pipeline could be built where only three metadata
objects change:
- Input schema
- Validation schema
- Output schema
This enables highly reusable, metadata-driven pipelines.
### Better governance
JSON is significantly easier for business users, governance teams and data
stewards to review than transform XML.
Validation rules can be:
- Peer reviewed
- Version controlled
- Automatically generated
- Validated using JSON Schema
- Managed outside Hop
### Easier upgrades
Validation logic can evolve independently from pipeline logic.
Changing validation rules would not require opening, modifying or
redeploying pipelines.
### Better interoperability
External applications could generate validation definitions
programmatically, allowing Hop to consume validation rules from governance
platforms, metadata repositories or code generators.
This also opens the possibility of generating validation schemas from other
metadata sources or enterprise data catalogues.
---
## Backwards Compatibility
This feature could be completely optional.
Existing Data Validator transforms would continue to work exactly as they do
today.
Users could choose between:
- Embedded validator definitions (current behaviour)
- Static Schema Definition (JSON)
<img width="1139" height="953" alt="Image"
src="https://github.com/user-attachments/assets/211b1fde-88b4-4927-a8f6-0deb31b2ecc1";
/>
<img width="1446" height="864" alt="Image"
src="https://github.com/user-attachments/assets/2025dfd8-667c-413f-a291-e930b00046c8";
/>
<img width="863" height="980" alt="Image"
src="https://github.com/user-attachments/assets/6fd0d370-ad35-4de7-92f9-8e7894549271";
/>
### Issue Priority
Priority: 2
### Issue Component
Component: Hop Gui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
