akshayrai opened a new pull request #5398:
URL: https://github.com/apache/incubator-pinot/pull/5398
**Overall Updates:**
* Added ability to configure Data Quality (DQ) alerts from Yaml. See a
sample below.
* DQ pipeline leverages & follows the existing detection merging logic
pattern.
* Refactored code to modularize & reuse the detection translators and extend
them to data quality translators.
* Updated the DATA_QUALITY Task and Pipeline Runner.
* Added following metrics to track - # DQ Tasks, # DQ Success Tasks & #DQ
Fail Tasks
Doc Link: To Be Updated
**Tests:**
* Added a bunch of unit testing around translating DQ yaml to internal json
representation
* Updated a bunch of unit testing to validate the Data SLA checker.
* Tested the Data Quality/sla pipeline by deploying it locally.
**Example:**
```
...
rules:
- quality:
- type: DATA_SLA
name: slaRule1
params:
sla: 1_DAYS
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
