GitHub user torwig added a comment to the discussion: Talks about the design of consumer group in stream
@Yangsx-1 vector/unordered_map/other_container is just a C++ representation. I'm not sure you will be operating with exactly those containers. I think the more important thing is the choice of how to store consumer groups, consumers and related data in RocksDB (since we will not keep all the data in memory like Redis; we will read/write a sufficient amount of data when executing commands). Since each consumer group (CG) has its metadata/state and even every consumer has its metadata/state and most commands that operate on CG are changing the state of CG, I tend to store the CG's state separately from the stream's metadata. The `XINFO STREAM` command has a response that includes the number of CGs. For this case, we can have the counter of the stream's CGs as a field of the stream's metadata and update it on creating/destroying a CG (luckily, these commands will be relatively rare). The PEL (pending entry list) of each CG should be stored in a way that gives us the ability to quickly: - remove entry from the PEL on `XACK` - change the owner of the entry - track each entry's last delivery time and the number of deliveries. One question is whether we need to lock the stream when executing write operations on one of its CGs? I also was wondering whether we need a separate column family for CGs? And how to delete CGs, consumers and related data on the stream deletion? GitHub link: https://github.com/apache/kvrocks/discussions/1654#discussioncomment-6680530 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
