zixi0825 opened a new issue #4283:
URL: https://github.com/apache/dolphinscheduler/issues/4283
# 1 Summary
Data quality inspection is an important part of the data processing process.
After data synchronization and data processing, it is usually necessary to
check the accuracy of the data, such as comparing the difference in the number
of data between the source table and the target table, or checking according to
a certain rule that calculate a certain column and compare the standard value
and the calculated value to judge. At present, there is no such type of data
quality check in the task type of DS, so it is necessary to add a new data
quality task type so that the data quality check task can be directly added
when defining the workflow, so that the entire data processing process is more
complete.
# 2 Requirements Analysis
For data quality inspection tasks, the core functions are rule management,
specific task execution, and execution result alarms. To achieve a lightweight
data quality, the following functions must be met:
## 2.1 Rule Manager
### 2.1.1 RuleType
* SingleTableRule
* MultiTableRule
### 2.1.2 Rule Implementation
* InnerRule
* NullCheck
* RowCountCheck
* AverageCheck
* TimelinessCheck
* DuplicateCheck
* AccuracyCheck
* others
* CustomRule
### 2.1.3 Rule Definition and Parser
#### 2.1.3.1 Rule Definition
The complete rules should include connector information, executed SQL
statements, the type of comparison value, the type of inspection, etc., that
is, the parameters needed to define a data quality task can be obtained through
the rules
#### 2.1.3.2 Rule Parser
The main responsibility of rule parser is to obtain an parameter that
conforms to the execution of the data quality task by parsing the parameter
value input by the user and the rule definition.
## 2.2 Task Execution Mode
Based on the existing task execution method of DolphinScheduler, a more
appropriate way is to use Spark as the execution engine for data quality tasks,
pass specific execution SQL to the Spark job to run through configuration, and
write the execution results to the specified storage engine
### 2.1.2 Alert
Each rule configure alertrules, when the check result is abnormal, an
alertoccurs. Use DS's alert module for alarm
# 3 Summary Design
## 3.1 Rule Manager Design
### 3.1.1 Rule Component Design
#### 3.1.1.1 Single Rule
* RuleDefinition
* RuleType
* rule_type
* RuleName
* rule_name
* RuleInputEntry
* DefaultInputEntry
* connector_type，eg. JDBC,HIVE
* datasource_id
* table
* filed
* filter
* StatisticsInputEntry
* statistics_name
* RuleMidExecuteSQLDefinition
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* StatisticsExecuteSQLDefinition
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* ComparsionExecuteSQLDefinition
* FixedValue
* InputEntry
* ComparsionRuleTilte：comparsion_title
* ComparsionName：comparsion_name
* ComparsionValue：comparsion_value
* CalculateValue
* InputEntry
* ComparsionRuleTitle：comparsion_title
* ComparsionName：comparsion_name
* ComparsionValue：comparsion_value
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* CheckInputEntry
* CheckType：fixed/percentage， ${check_type}
* Threshold：1000/30%，${threshold}
* Operator：=，<，>，>=，<= ${operator}
* ResultOutputSQL
* according the comparsion type
* Fiexed
* get the ${comparsion_value} as result field value
* Calculate
* get the comparsion execute sql value as result field value
#### 3.1.1.2 MultiTableAccuracyRule
* RuleDefinition
* RuleType
* rule_type
* RuleName
* rule_name
* RuleInputEntry
* DefaultInputEntry
* src_connector_type
* src_datasource_id
* src_table
* src_filter
* target_connector_type
* target_datasource_id
* target_table
* target_filter
* mapping_columns
* on_clause
* StatisticsInputEntry
* statistics_name
* RuleMidExecuteSQLDefinition
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* StatisticsExecuteSQLDefinition
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* ComparsionExecuteSQLDefinition
* FixedValue
* InputEntry
* ComparsionRuleTilte：comparsion_title
* ComparsionName：comparsion_name
* ComparsionValue：comparsion_value
* CalculateValue
* InputEntry
* ComparsionRuleTitle：comparsion_title
* ComparsionName：comparsion_name
* ComparsionValue：comparsion_value
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* CheckInputEntry
* CheckType：fixed/percentage， ${check_type}
* Threshold：1000/30%，${threshold}
* Operator：=，<，>，>=，<= ${operator}
* ResultOutputSQL
* according the comparsion type
* Fiexed
* get the ${comparsion_value} as result field value
* Calculate
* get the comparsion execute sql value as result field value
#### 3.1.1.3 ,MultiTableValueComparsionRule
* RuleDefinition
* RuleType
* rule_type
* RuleName
* rule_name
* RuleInputEntry
* DefaultInputEntry
* src_connector_type
* src_datasource_id
* statistics_name
* statistics_execute_sql
* target_connector_type
* target_datasource_id
* comparsion_name
* comparsion_execute_sql
* CheckInputEntry
* CheckType：fixed/percentage， ${check_type}
* Threshold：1000/30%，${threshold}
* Operator：=，<，>，>=，<= ${operator}
* ResultOutputSQL
* select ${statistics_name} as statistics_value,${comparsion_name} as
coparsion_value from ${statistics_execute_sql} full join
${comparsion_execute_sql}
### 3.1.2 Custom Rule
* Support single table for custom rule calculation for single column
* Support cross-table value calculation comparison for single-column
## 3.2 Task Execute Process Design
### 3.2.1 Execution Engine
* Spark2.0+
### 3.2.2 Task Execution Process
## 3.3 Task Manager Design
Data quality tasks do not support separate definition and scheduled
scheduling, which can be defined and scheduled in the workflow
## 3.4 Data Quality Task Definition UI Design
### 3.4.1 UI Generation Method
The data quality task definition UI interface will automatically generated
by the front-end component according to a JSON string.
### 3.4.2 Task Define UI Prototype Diagram
### 3.4.3 Custom Rule UI Prototype Diagram
# 4 Detail Design
# 4.1 Database Design
### 4.1.1 RuleInfo
| column | type | comment |
| :---------- | :----- | :---------------------------------- |
| id | int | id |
| name | string | rule name |
| type | int | rule type：single-table/multi-table |
| rule_json | text | rule definition |
| create_time | date | create time |
| update_time | date | update time |
### 4.1.2 CheckResultInfo
| column | type | comment |
| :--------------- | :----- | :------------------------------------ |
| id | int | id |
| task_id | long | Task ID |
| task_instance_id | long | TaskInstance ID |
| rule_type | int | rule type |
| statistics_value | double | statistics value |
| comparsion_value | double | comparsion value |
| check_type | int | check type，fixed value or percentage |
| threshold | double | threshold |
| operator | int | operator：>，<，=，>=，<= |
| create_time | date | create time |
| update_time | date | update time |
### 4.1.3 CheckResultStatisticsInfo
# 4.2 Class Design
### 4.2.1 Rule Design
#### 4.2.1.1Rule Related Model
* RuleDefinition
* RuleInputEntry
* ExecuteSqlDefinition
* InputType
* FormType
* OptionSourceType
* ValueType
* RuleType
* ConnectorType
* ComparsionValueType
* CheckType
* FixedComparsionValueParameter
* CalculateComparsionValueParameter
* ConnectorDefinition
#### 4.2.1.2 RuleParser
* Different types of rules have default input items, you can add exclusive
input items on this basis
* After selecting the rule, it will read the rule input items in the content
of the rule, construct a json string conforming to the form-create
specification, and return it to the front end to generate the corresponding UI
interface
* After filling in the rule parameters, after submission, the configured
parameters will be constructed into a map and stored in the task parameters
* After the task is asssigned, the parameters are analyzed, and the
parameters required for the task are constructed to execute the spark job
1）Connector Parameter Parser
To get the information of datasource including url, database, table,
username, password according the datasource_id and constructed information of
connector
2）Replace the placeholders in executeSQL to construct an executeSQL list
3）Construct writer configuration, including construct writer connector
configuration and saveSQL
* The pseudo code for constructing save sql in Writer is as follows：
```plain
if(comparsionType == FIXED){
map.put("${comparsion_name}","fixed_value")
sql = "select ${comparsion_name} as comparsion_value from
${staticsTableName}
} else {
sql = "select ${comparsion_name} as comparsion_value from
${statics_table_Name} full join ${comparsion_table_Name}
}
resultSQL = sql.replacePlaceholder(map)
```
Finally, it will be constructed into the json string parameter and passed to
the Spark application
### 4.2.2 Task Design
#### 4.2.2.1 DolphinScheduler Task Design
* DataQualityParameter
* DataQualityTask
* The main responsibility is to execute spark job
#### 4.2.2.2 Spark Data Quality Task Design
1）The data quality task is actually a Spark task. The main responsibilities
of this task are as follows:
* Parse the parameters and obtain the parameters needed to construct Reader,
Executor and Writer
* Construct corresponding types of Reader, Executor and Writer according to
the parameters
* The main responsibility of Reader is to read the data of the specified
data source and create a temporary table for subsequent use
* The main responsibility of Executor is to run the intermediate step SQL
statement, the statistical step SQL statement and the comparison value
calculation SQL statement
* The main responsibility of Writer is to write the calculation results of
data quality tasks to the corresponding storage engine. Currently, it only
supports writing back to the ds database
2）The execute mode has the follow options
* Package a DQApplicaiton.jar and upload it to hdfs, and load the default
upload address when running
* This method is relatively more saving storage space and reducing jar
upload time
* Put the packaged DQApplication.jar into the lib directory, and load the
jar package when running
* This method is more insensitive to user deployment
# 5 Todo List
* DataQualityTask UI Component
* DataQuailtyTask Component
* DataQualityParameter
* DataQualityTask
* Rule Component
* RuleModel
* RuleManager
* RuleConverter
* Spark DataQuality Application
# 6 related issue and pr
[issue: DataQuality
Application](https://github.com/apache/incubator-dolphinscheduler/issues/4829)
[pr: DataQuality Common
Entity](https://github.com/apache/incubator-dolphinscheduler/pull/4883)
---
# 1 摘要
数据质量检查是数据处理流程中比较重要的环节，在数据同步和数据处理后通常是需要检查数据的准确性，例如比较源表和目标表之间的数据条数差，或者根据某个规则对某一列进行计算，将标准值和计算值进行比较判断。目前在
DS 的任务类型没有数据质量检查这样的类型，所以需要新增数据质量任务类型，以便于在定义工作流的时候可以直接添加数据质量检查任务，让整个数据处理流程更加的完整。
# 2 需求分析
对于数据质量检查任务来说，核心的功能就是规则管理、具体的任务执行以及执行结果告警，实现一个轻量级的数据质量需要满足以下功能：
## 2.1 规则管理
### 2.1.1 规则类型
* 单表规则
* 跨表规则
### 2.1.2 规则实现方式
* 内置规则
* 空值检查
* 表行数检查
* 均值检查
* 及时性检查
* 重复性检查
* 准确性检查
* 等等
* 自定义规则
### 2.1.3 规则的定义和解析
#### 2.1.3.1 规则定义
完整的规则应该包括 connector 信息、执行的 sql 语句、比较值的类型，检查的类型等，即通过规则可以获取定义一个数据质量任务所需要的参数
#### 2.1.3.2 规则解析
规则解析主要职责是通过解析用户输入的参数值和规则定义得到一个符合数据质量任务运行的输入参数
## 2.2 任务的执行方式
基于 DolphinScheduler 现有的任务执行方式，比较合适的方式就是使用 Spark 作为数据质量任务的执行引擎，通过配置的方式将具体的执行
SQL 传入 Spark 作业来运行，并将执行的结果写到指定的存储引擎中
## 2.3 检查结果告警
每个规则都会配置告警规则，当检查结果为异常的话，则会进行告警。使用 DolphinScheduler 的告警模块进行告警
# 3 概要设计
## 3.1 规则管理设计
### 3.1.1 规则组成设计
#### 3.1.1.1 单表规则
* 规则的定义
* 规则的类型
* rule_type
* 规则的名称
* rule_name
* 规则输入项
* 常规输入项
* connector_type，例如 JDBC，HIVE
* datasource_id
* table
* filed
* filter
* 统计指标输入项
* statistics_name
* 规则中间运行语句
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* 规则统计运行语句
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* 比较值相关输入项
* 固定型
* 输入项
* 比较值规则标题：comparsion_title
* 比较值名：comparsion_name
* 比较值：comparsion_value
* 计算型
* 输入项
* 比较值规则标题：comparsion_title
* 比较值名：comparsion_name
* 比较值：comparsion_value
* 计算过程语句
* ExecuteSQL & TableAlias
* 校验规则输入项
* 检查方式：固定条数/百分比， ${check_type}
* 阈值：1000/30%，${threshold}
* 比较符：等于，小于，大于，大于等于，小于等于 ${operator}
* 结果输出运行语句（即将数据输出到某个存储引擎中，系统会自动生成）
* 根据比较值类型进行生成
* 固定型
* 直接读取参数值 comparsion_value 作为字段值
* 计算型
* 将统计运行语句的表和比较值计算语句进行 join 以后获取两个统计指标的值作为字段值进行插入
#### 3.1.1.2 跨表准确性规则
* 规则的定义
* 规则的类型
* rule_type
* 规则的名称
* rule_name
* 规则输入项
* 常规输入项
* src_connector_type
* src_datasource_id
* src_table
* src_filter
* target_connector_type
* target_datasource_id
* target_table
* target_filter
* mapping_columns
* on_clause
* 统计指标输入项
* statistics_name
* 规则中间运行语句
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* 规则统计运行语句
* ExecuteSQLDefinition
* ExecuteSQL & TableAlias
* 比较值相关输入项
* 固定型
* 输入项
* 比较值规则标题：comparsion_title
* 比较值名：comparsion_name
* 比较值：comparsion_value
* 计算型
* 输入项
* 比较值规则标题：comparsion_title
* 比较值名：comparsion_name
* 比较值：comparsion_value
* 计算过程语句
* ExecuteSQL & TableAlias
* 校验规则输入项
* 检查方式：固定条数/百分比， ${check_type}
* 阈值：1000/30%，${threshold}
* 比较符：等于，小于，大于，大于等于，小于等于 ${operator}
* 结果输出运行语句（即将数据输出到某个存储引擎中，系统会自动生成）
* 根据比较值类型进行生成
* 固定型
* 直接读取参数值作为字段值
* 计算型
* 将统计运行语句的表和比较值计算语句进行 join 以后获取两个统计指标的值作为字段值进行插入
#### 3.1.1.3 跨表值比对规则
* 规则的定义
* 规则的类型
* rule_type
* 规则的名称
* rule_name
* 规则输入项
* 常规输入项
* src_connector_type
* src_datasource_id
* statistics_name
* statistics_execute_sql
* target_connector_type
* target_datasource_id
* comparsion_name
* comparsion_execute_sql
* 校验规则输入项
* 检查方式：固定条数/百分比， ${check_type}
* 阈值：1000/30%，${threshold}
* 比较符：等于，小于，大于，大于等于，小于等于 ${operator}
* 结果输出运行语句（即将数据输出到某个存储引擎中，系统会自动生成）
select ${statistics_name} as statistics_value,${comparsion_name} as
coparsion_value from ${statistics_execute_sql} full join
${comparsion_execute_sql}
### 3.1.2 自定义规则
* 支持单表的对单列进行自定义规则计算
* 支持跨表对单列的值计算对比
## 3.2 任务执行流程设计
### 3.2.1 执行引擎
* Spark 计算引擎，2.0 以上
### 3.2.2 任务执行流程
## 3.3 任务管理设计
数据质量任务不支持单独定义和定时调度，可以通过在工作流中定义和定时调度
## 3.4 数据质量任务定义 UI 设计
### 3.4.1 UI 页面生成方式
数据质量任务定义 UI 界面会根据不同规则的参数生成 JSON 串由前端组件自动生成
### 3.4.2 任务定义 UI 示意图
### 3.4.3 自定义规则界面 UI 示意图
# 4 详细设计
# 4.1 数据库设计
### 4.1.1 规则表
| 字段 | 类型 | 解释 |
| :---------- | :----- | :-------------------------- |
| id | int | id |
| name | string | 规则名称 |
| type | int | 规则类型：单表规则/跨表规则 |
| rule_json | text | 规则定义 |
| create_time | date | 创建时间 |
| update_time | date | 更新时间 |
### 4.1.2 检查结果表
| 字段 | 类型 | 解释
|
| :--------------- | :----- |
:--------------------------------------------------- |
| id | int | id
|
| task_id | long | 任务 ID
|
| task_instance_id | long | 任务实例 ID
|
| rule_type | int | 规则类型
|
| statistics_value | double | 计算的指标值
|
| comparsion_value | double | 比对的指标值
|
| check_type | int | 检测类型，数值比较或者百分比 |
| threshold | double | 阈值
|
| operator | int | 操作符：大于，小于，等于，不等于，大于等于，小于等于 |
| create_time | date | 创建时间
|
| update_time | date | 更新时间
|
### 4.1.3 检查结果统计表
# 4.2 类设计
### 4.2.1 规则相关
#### 4.2.1.1 规则实体
* RuleDefinition 规则定义
* RuleInputEntry 输入项定义，每一个 RuleInputEntry 都有默认值，可以不修改
* ExecuteSqlDefinition 执行 SQL 的定义
* InputType 输入项的类型
* FormType前端表单控间类型
* OptionSourceType 前端表单控间所需 Options 来源
* ValueType 输入项的值的类型
* RuleType 规则类型
* ConnectorType 数据源类型
* ComparsionValueType 比较值类型
* CheckType 检验类型
* FixedComparsionValueParameter 固定数值比较值参数
* CalculateComparsionValueParameter 计算类型比较值参数
* ConnectorDefinition
#### 4.2.1.2 规则解析
1） 规则使用的流程分析
* 对于不同类型的规则都会有预先设计好的输入项，可以在此基础上添加专属输入项
* 选择规则以后，会读取规则的内容中的规则输入项，构造一个符合 form-create 规范的 json 字符串，返回给前端生成对应的 UI 界面
* 填写完成规则参数以后，提交以后会将配置好的的参数构造成一个 map 存储在任务参数中
* 下发任务后进行参数的解析，构造成任务运行所需要的参数执行 spark 作业
2）规则解析具体内容
* connector 数据解析
根据 datasource_id 拿到相关的数据源信息，包括 url，database，table，username，password，构造
connector 配置
* 对 executeSQL 中的占位符进行替换，构造 executeSQL 列表
* 构造 Writer 配置，包括构造 Writer 的 Connector 配置以及 saveSQL
* writer 中构造 saveSQL 的伪代码如下：
```plain
if(comparsionType == FIXED){
map.put("${comparsion_name}","fixed_value")
sql = "select ${comparsion_name} as comparsion_value from
${staticsTableName}
} else {
sql = "select ${comparsion_name} as comparsion_value from
${statics_table_Name} full join ${comparsion_table_Name}
}
resultSQL = sql.replacePlaceholder(map)
```
3）最终会构造成json 格式 的参数传给 Spark 应用
### 4.2.2 任务相关
#### 4.2.2.1 DolphinScheduler 任务设计
* DataQualityParameter
* DataQualityTask
* 主要职责就是运行一个 Spark 作业
#### 4.2.2.2 Spark 数据质量任务设计
1）数据质量任务实际上是一个 Spark 任务，这个任务的主要责任是如下：
* 解析参数，获取构造 Reader，Executor 和 Writer 所需要的参数
* 根据参数构造对应类型的 Reader、Executor 和 Writer
* Reader 的主要职责是读取指定数据源的数据，并创建临时表供后续使用
* Executor 的主要职责是运行中间步骤 SQL 语句、统计步骤 SQL 语句和比较值计算 SQL 语句
* Writer 的主要职责是将数据质量任务的计算结果写到相应的存储引擎中，目前只支持写回 ds 的数据库中
2）运行方式可如下：
* 打包一个 DQApplicaiton.jar 上传至 hdfs 上，运行的时候加载默认的上传地址
* 这种方式相对来说更节省存储空间和减少 jar 上传的时间
* 将打包好的 DQApplication.jar 放至 lib 目录下，运行的时候加载该 jar 包
* 这种方式对于用户的部署更加无感
# 5 Todo List
* DataQualityTask UI 相关开发
* DataQuailtyTask 相关开发
* DataQualityParameter设计
* DataQualityTask 设计
* Rule 相关开发
* RuleModel相关设计
* RuleManager设计开发
* RuleConverter
* Spark DataQuality 相关开发
# 6 相关 issue 和 pr
[issue: DataQuality
Application](https://github.com/apache/incubator-dolphinscheduler/issues/4829)
[pr: DataQuality Common
Entity](https://github.com/apache/incubator-dolphinscheduler/pull/4883)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
