I would like to see more explicit discussion of topic retention and share 
groups. There are a few options here from simple to more sophisticated. There 
are also topic-level and share-group level options.

The simple thing would be to ensure that the SPSO of each share group is 
bounded by the Log Start Offset (LSO) of each partition which itself is managed 
by the retention policy. This is a topic-level control which applies to all 
share-groups. I would say that this shared retention is the largest drawback of 
modeling queues on shared logs and this is worth noting.

More sophisticated approaches can be to allow the LSO to advance not (only) by 
retention policy but by the advancement of the lowest SPSO. This can keep the 
amount of data lower by garbage collecting messages that have been acknowledged 
by all share groups. Some people may like that behaviour on those topics where 
share groups are the only consumption model and no replay is needed.

There are per-share-group possibilities such as share-group TTLs where messages 
can be archived on a per share group basis.

Thanks
Jack

Reply via email to