coderzc commented on code in PR #21541: URL: https://github.com/apache/pulsar/pull/21541#discussion_r1390606449
########## pip/pip-318.md: ########## @@ -0,0 +1,49 @@ +# PIP-318: Don't retain null-key messages during topic compaction + +# Background knowledge + +Apache Pulsar is supported [Topic Compaction](https://pulsar.apache.org/docs/en/concepts-topic-compaction/) which is a key-based data retention mechanism. + +# Motivation + +Currently, we retain all null-key messages during topic compaction, which I don't think is necessary because when you use topic compaction, it means that you want to retain the value according to the key, so retaining null-key messages is meaningless. + +Additionally, retaining all null-key messages will double the storage cost, and we'll never be able to clean them up since the compacted topic has not supported the retention policy yet. + +In summary, I don't think we should retain null-key messages during topic compaction. + +# Goals + +# High Level Design + +In order to avoid introducing break changes to release version, we need to add a configuration to control whether to retain null-key messages during topic compaction. +If the configuration is true, we will retain null-key messages during topic compaction, otherwise, we will not retain null-key messages during topic compaction. + +## Public-facing Changes + +### Configuration + +Add config to broker.conf +```properties +compactionRemainNullKey=false Review Comment: `topicCompactionRemainNullKey` seems to be better since it needs to cherry-pick to the old branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
