[kafka] branch trunk updated: KAFKA-14393; Default metadata retition by bytes (#12884)

[jsancio]

This is an automated email from the ASF dual-hosted git repository.

jsancio pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/trunk by this push:
     new 2ab1e6cb571 KAFKA-14393; Default metadata retition by bytes (#12884)
2ab1e6cb571 is described below

commit 2ab1e6cb571fc4900809243f221a296dcda0cf15
Author: José Armando García Sancio <jsan...@users.noreply.github.com>
AuthorDate: Mon Nov 28 10:39:57 2022 -0800

    KAFKA-14393; Default metadata retition by bytes (#12884)
    
    Now that Kafka is generating a metadata snapshot every hour and
    the default metadata retention is to delete snapshots after 7 days,
    every cluster metadata partition will have 168 (1 snapshot per hour
    * 24 hours per day * 7 days) snapshots. If we assume that in most
    cases the size of the snapshot is determined by the number of
    partitions in a cluster, a cluster with 100K partitions will have a
    snapshot size of roughly 10MB (100 bytes per partition * 100k
    partitions). For this kind of clusters the cluster metadata partition
    will always consume around 1.7GB.
    
    KIP-876 changed the default value for metadata.max.retention.bytes
    to 100MB. This should limit the size of the cluster metadata
    partition for large clusters but keep 7 days worth of snapshots for
    small clusters.
    
    Reviewers: Jason Gustafson <ja...@confluent.io>
---
 core/src/main/scala/kafka/server/KafkaConfig.scala | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/kafka/server/KafkaConfig.scala 
b/core/src/main/scala/kafka/server/KafkaConfig.scala
index 98953e338de..e2c8db8dbb7 100755
--- a/core/src/main/scala/kafka/server/KafkaConfig.scala
+++ b/core/src/main/scala/kafka/server/KafkaConfig.scala
@@ -83,6 +83,7 @@ object Defaults {
   val MetadataSnapshotMaxNewRecordBytes = 20 * 1024 * 1024
   val MetadataSnapshotMaxIntervalMs = TimeUnit.HOURS.toMillis(1);
   val MetadataMaxIdleIntervalMs = 500
+  val MetadataMaxRetentionBytes = 100 * 1024 * 1024
 
   /** KRaft mode configs */
   val EmptyNodeId: Int = -1
@@ -1185,7 +1186,7 @@ object KafkaConfig {
       .define(MetadataLogSegmentBytesProp, INT, Defaults.LogSegmentBytes, 
atLeast(Records.LOG_OVERHEAD), HIGH, MetadataLogSegmentBytesDoc)
       .defineInternal(MetadataLogSegmentMinBytesProp, INT, 8 * 1024 * 1024, 
atLeast(Records.LOG_OVERHEAD), HIGH, MetadataLogSegmentMinBytesDoc)
       .define(MetadataLogSegmentMillisProp, LONG, Defaults.LogRollHours * 60 * 
60 * 1000L, null, HIGH, MetadataLogSegmentMillisDoc)
-      .define(MetadataMaxRetentionBytesProp, LONG, Defaults.LogRetentionBytes, 
null, HIGH, MetadataMaxRetentionBytesDoc)
+      .define(MetadataMaxRetentionBytesProp, LONG, 
Defaults.MetadataMaxRetentionBytes, null, HIGH, MetadataMaxRetentionBytesDoc)
       .define(MetadataMaxRetentionMillisProp, LONG, Defaults.LogRetentionHours 
* 60 * 60 * 1000L, null, HIGH, MetadataMaxRetentionMillisDoc)
       .define(MetadataMaxIdleIntervalMsProp, INT, 
Defaults.MetadataMaxIdleIntervalMs, atLeast(0), LOW, 
MetadataMaxIdleIntervalMsDoc)