azhsmesos opened a new issue, #5418:
URL: https://github.com/apache/rocketmq/issues/5418
The current topic compaction data and commitLog are kept on the broker, and
topic compression scans commitLog for the latest offset.
I have two questions now
1. If a large number of topics are compressed, a large number of disk I/OS
may be generated, reducing service availability.
2.commitLog stores old data and compressed files store the latest compressed
data. In this case, if the compressed file is similar to the Stock ticker
theme, you need to query the old data. Since rocketmq can only hold data for
about 3 days, this is not sufficient in many scenarios. Do we need to implement
hierarchical storage to offload old data to an external cheap storage medium
such as gfs, hdfs, s3, etc. The broker only holds the previous ordinary
messages and compresses them, while the old data in the compressed scenario is
kept in external storage.
For example Just a simple demonstration interface through which to offload
data
```java
interface Offloader {
void offload(Message msg);
CompletableFuture<Message> readOffloaded(long messageID);
void deleteOffloaded(long messageID);
}
```
This reduces broker pressure, and the latest messages that are read
frequently can be stored in commitlogs, while messages that do not require read
frequent but important can be stored in external storage, improving broker
availability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
