GitHub user l-7-l added a comment to the discussion: Exploring Delayed Message Support in Iggy: Use Cases & Implementation Ideas
I found an article about introducing [qmq](https://github.com/qunarcorp/qmq) delayed messages: QMQ offers delayed/scheduled messaging for any time within a configurable period (up to two years by default). You can specify the delivery time of a message at any point within this future timeframe. I placed QMQ last because I consider its design for delayed messages to be the most rational among open-source message queues (MQ). The core design, simply put, is multi-level timing wheel + lazy loading + dedicated disk storage for delayed messages. QMQ implements delayed/scheduled messages using a two-layer hash wheel: Disk Layer: Uses hourly intervals as the unit (default; adjustable via configuration). Each interval generates a log file (schedule log). Since QMQ supports delayed messages up to two years (configurable), this results in a maximum of 2 * 366 * 24 = 17,568 files (fewer files are needed if the maximum delay time is shorter). Memory Layer: When a message's delivery time approaches, the indices for messages within that hour (containing the offset and size within the schedule log) are loaded from disk into a hash wheel in memory. This in-memory hash wheel uses 500ms intervals. <img width="600" height="330" alt="image" src="https://github.com/user-attachments/assets/41f78e59-6440-4567-b3c2-ac4b82dfd1be"; /> Summary of Key Design Highlights: 1. The Timing Wheel algorithm is ideal for delayed/scheduled messaging. It eliminates the need for sorting delayed messages, offering O(1) time complexity for both insertion and deletion operations. 2. The multi-level timing wheel design enables support for delayed messages with extremely large time spans. 3. Lazy loading ensures only messages nearing their delivery time reside in memory. Messages scheduled further in the future are stored on disk, making the design memory efficient. 4. Storing delayed messages separately in schedule log files prevents them from interfering with the space reclamation of normal messages. GitHub link: https://github.com/apache/iggy/discussions/2031#discussioncomment-13877635 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
