Cyanty opened a new issue, #3557:
URL: https://github.com/apache/hertzbeat/issues/3557
### Feature Request
Warehouse module has currently successfully implemented the batch insert
logic for storing historical data using `Victoria Metrics`.
For a batch write function, a common approach is to use `Scheduler + Buffer
+ Batch Writer` for processing.
In warehouse module, this is respectively represented by `HashedWheelTimer +
BlockingQueue + sendVictoriaMetrics`.
Among them, in the `doSaveData()` function, the logic for writing multiple
or a single set of indicator data is implemented.
And hzb supports multiple types of TSDB, all of which have the requirement
for batch insert. The current batch insert processing is very suitable for
abstracting this feature, making it a common method that can be applied to all
data storage sources.
This can be regarded as the following modification in the configuration file:
```yaml
warehouse:
store:
# Batch insert parameters need to be configured separately for each data
source (currently only support victoria-metrics)
victoria-metrics:
...
insert:
buffer-size: 100
flush-interval: 3
# Extract batch-insert parameters into a unified configuration, which will
take effect for all data sources
warehouse:
batch-insert:
enabled: false
buffer-size: 100
flush-interval: 3
```
In terms of code implementation, the batch insert processing can be
reflected in the `AbstractHistoryDataStorage` abstract class, using the
template method pattern to standardize the writing process:
```java
public abstract class AbstractHistoryDataStorage implements
HistoryDataReader, HistoryDataWriter, DisposableBean {
// Provide a function for writing data sources, implemented separately
in the subclass
abstract void doSaveData(List<? extends MetricsContent> batch);
public void sendMetrics(List contentList) {
// Some scheduling and caching-related processing
...
// Insert data
doSaveData(contentList);
}
}
public class VictoriaMetricsDataStorage extends AbstractHistoryDataStorage {
@Override
public void saveData(CollectRep.MetricsData metricsData) {
// MetricsData conversion processing
...
metricsData => contentList
// saveData() function is only responsible for the conversion
process
// insert-data process is determined by the parent class
template method
sendMetrics(contentList);
}
}
```
It has been observed that the current methods for writing to the data source
include JDBC / HTTP / Client. If appropriate, these can be modified one by one.
What are everyone's thoughts on this?
### Is your feature request related to a problem? Please describe
_No response_
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
notifications-unsubscr...@hertzbeat.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscr...@hertzbeat.apache.org
For additional commands, e-mail: notifications-h...@hertzbeat.apache.org
