CarlosJuncher03 opened a new issue, #8626: URL: https://github.com/apache/hop/issues/8626
### What would you like to happen? I would like to propose a discussion about adding data quality alerts to Apache Hop Execution Information. Hop already stores valuable execution details such as status, logs, metrics, and profiled data. The idea would be to allow users to evaluate that information against user-defined rules and generate persistent alerts that can also be displayed in the Hop GUI. For example, a pipeline could define checks such as: - Null CPF values must not be greater than zero - Null name values must be below 1% - Rows sent to the “Error Map” hop must be below 500 - Rows loaded to the target must represent at least 95% of input rows - Pipeline duration must be below 30 minutes At the end of an execution, Hop could evaluate these rules and save the results together with the Execution Information. Conceptual example: Execution: person_load #12345 Status: Completed with alerts Rule: Null CPF values Actual value: 84 Threshold: 0 Severity: Failure Rule: Rows sent to Error Map Actual value: 2,184 Threshold: 500 Severity: Warning In addition to being linked to an execution, each alert could be stored in a dedicated structure in the configured Execution Information Location. This would make it possible to query and monitor alerts across multiple pipelines and workflows. The Hop GUI could expose these alerts through the existing notifications/problems area and provide a simple lifecycle: Open → Under investigation → Resolved / Ignored When opening an alert, users could see: - Related pipeline, workflow, and execution - Related transform or hop - Configured rule - Expected and actual values - Execution logs and metrics - Analysis comments - History of similar occurrences It could also be useful to define an action for each rule: - Record only - Raise an alert - Mark execution as warning - Mark execution as failed - Stop execution - Route rows to quarantine The intention is not to assume that every metric deviation is an error. For example, rejected rows can be expected in some data flows. Therefore, rules should explicitly identify which transform, hop, field, or execution metric is being monitored. Points for community discussion: Should this be part of the standard Execution Information model, or should it start as a plugin? What would be the best metadata model for data quality rules? Should alerts be stored in the same Execution Information backend or in a dedicated table/index? How could this integrate with existing data profiling, metrics, lineage, and logging capabilities? Which actions should be supported without making pipeline execution too complex? How can this be implemented with minimal performance impact for high-volume pipelines? I believe this could help evolve Hop from only recording execution details to also identifying, tracking, and auditing data quality deviations in a structured way. ### Issue Priority Priority: 3 ### Issue Component Component: Hop Gui, Component: Infrastructure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
