GitHub user luoluoyuyu edited a discussion: Add Prometheus Endpoint to the core
and extensions modules of streampipes.
# Purpose
To support prometheus in monitoring streampipes.
# Solution
## Using 'spring-boot-starter-actuator'
Adds monitoring metrics to a springboot application using
'spring-boot-starter-actuator', 'spring-boot-starter-actuator' provides
information about the memory usage of the springboot application, the number of
threads, and information related to CPU, etc.
## Adding Custom Metrics
Add custom metrics on top of the metrics provided by
'spring-boot-starter-actuator', add custom metrics about prometheus in
streampipes and put this related code into the streampipes-commons module.
# Specific implementation
## Integrate actuator
Add actuator library and actuator configuration to StreamPipesServiceBase
module to integrate actuator with core and extensions modules.
```
private void runApplication(Class<?> serviceClass,
Integer port) {
SpringApplication app = new SpringApplication(serviceClass);
app.setDefaultProperties(getProperties(port));
app.run();
}
private Properties getProperties(Integer port) {
Properties properties = new Properties();
properties.setProperty("management.endpoints.web.exposure.include",
"health,prometheus");
properties.setProperty("management.metrics.tags.application",
"${spring.application.name}");
properties.setProperty("server.port", port.toString());
return properties;
}
```
## Custom Indicators
1、Implement a global registry
```
public class StreamPipesCollectorRegistry {
private static final CollectorRegistry collectorRegistry=new
CollectorRegistry(true);
public static CollectorRegistry getCollectorRegistry(){
return collectorRegistry;
}
}
```
2, in straempipes to create a Prometheus module, in the module to define
indicators such as pipline statistics class.
```
public class PipelinesStats {
private int allPipelines;
private int runningPipelines;
private int stoppedPipelines;
private int healthyPipelines;
private int failedPipelines;
private int attentionRequiredPipelines;
}
```
3, create a Prometheus indicator class
```
public class PipelinesStatsMetrics {
public static final Gauge allPipelinesGauge = StreamPipesCollectorRegistry
.registerGauge(
"all_pipelines",
"Total number of
pipelines");
//....
}
```
4、Data Buried Points
Buried in the execution method
```
public class PipelineHealthCheck implements Runnable {
private static final PipelinesStats pipelinesStats = new PipelinesStats();
public void checkAndRestorePipelineElements() {
//......
pipelinesStats.clear();
pipelinesStats.setAllPipelines(allPipelines.size());
pipelinesStats.setRunningPipelines(runningPipelines.size());
pipelinesStats.setStoppedPipelines(pipelinesStats.getAllPipelines() -
pipelinesStats.getRunningPipelines());
if (runningPipelines.size() > 0) {
//..
runningPipelines.forEach(pipeline -> {
//.....
if (shouldUpdatePipeline.get()) {
if (failedInstances.size() > 0) {
//...
pipelinesStats.FailedIncrease();
} else if (recoveredInstances.size() > 0) {
//...
pipelinesStats.attentionRequiredIncrease();
}
//...
}
});
int healthNum=pipelinesStats.getRunningPipelines() -
pipelinesStats.getFailedPipelines() -
pipelinesStats.getAttentionRequiredPipelines();
pipelinesStats.setHealthyPipelines(healthNum);
pipelinesStats.metrics();
}
}
}
```
5、Register
In the core module to register a custom CollectorRegistry, so that the
CollectorRegistry can be assembled into the PrometheusMeterRegistry.
```
@Configuration
public class StreamPipesPrometheus {
@Bean
public CollectorRegistry collectorRegistry() {
return StreamPipesCollectorRegistry.getCollectorRegistry();
}
}
```
GitHub link: https://github.com/apache/streampipes/discussions/1925
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
