ic4y commented on issue #3431:
URL:
https://github.com/apache/incubator-seatunnel/issues/3431#issuecomment-1328614465
For ST-Metrics, we can start with JobMetrics, because JobMetrics can only be
implemented through customization in each engine, and it is more convenient to
unify.
The format for JobMetrics like
```
jobID:123
timestamp：1669606042
PipelineMap：{
PipelineLocation : {
TaskGroupLocation :{
TaskLocation : {
TaskInfo: source or sink …
MetricsName: value,
}
…
}
…
}
…
}
…
```
Seatunnel-Api:
Define Metrics name and Type. Unified indicator names and types
Metrics collection location：
1. ST-Engine collects Metrics such as RowCount in LifeCycle
2. For spark and flink, consider collecting in the translation layer
ST-Engine JobMetrics related interfaces：
1. Provide the latest JobMetrics via jobId
2. Provide all metrics data of the job from the beginning to the present
(there will be pressure on the data volume of long-running jobs)
Some problems：
Under different engines, ST-Engine can store JobMetrics for a period of
time, while spark flink can only obtain them instantly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
