[
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christo Lolov updated KAFKA-17584:
----------------------------------
Description:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the value for log.retention.ms. This
poses a durability issue if users have set their retention by using
log.retention.hours or log.retention.minutes. In other words, if a user has set
log.retention.hours=-1 (infinite retention) and they dynamically change
`message.max.bytes` their retention will immediately change back to the default
of 604800000 ms (7 days) and data before this will be scheduled for deletion
immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
{code:java}
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512{code}
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
{code:java}
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609{code}
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
was:
Updating certain dynamic configurations (for example `message.max.bytes`)
causes retention based on time to reset to the value for log.retention.ms. This
poses a durability issue if users have set their retention by using
log.retention.hours or log.retention.minutes. In other words, if a user has set
log.retention.hours=-1 (infinite retention) and they dynamically change
`message.max.bytes` their retention will immediately change back to the default
of 604800000 ms (7 days) and data before this will be scheduled for deletion
immediately.
Steps to reproduce:
# Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
server.properties
# Start a single ZK or KRaft instance + a single Kafka instance
# Create a topic using
```
bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
--replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
segment.bytes=512
```
# Create a few segments with the console producer
# Observe that they are deleted after 1 minute
# Use the following command
```
bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
--entity-default --alter --add-config message.max.bytes=1048609
```
(the value of `message.max.bytes` is irrelevant)
# Create a few more segments with the console producer
# Observe that segments are no longer deleted after 1 minute
> Changing dynamic configurations resets retention
> ------------------------------------------------
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
> Issue Type: Bug
> Reporter: Christo Lolov
> Assignee: Christo Lolov
> Priority: Major
>
> Updating certain dynamic configurations (for example `message.max.bytes`)
> causes retention based on time to reset to the value for log.retention.ms.
> This poses a durability issue if users have set their retention by using
> log.retention.hours or log.retention.minutes. In other words, if a user has
> set log.retention.hours=-1 (infinite retention) and they dynamically change
> `message.max.bytes` their retention will immediately change back to the
> default of 604800000 ms (7 days) and data before this will be scheduled for
> deletion immediately.
> Steps to reproduce:
> # Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to
> server.properties
> # Start a single ZK or KRaft instance + a single Kafka instance
> # Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config
> segment.bytes=512{code}
> # Create a few segments with the console producer
> # Observe that they are deleted after 1 minute
> # Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
> # Create a few more segments with the console producer
> # Observe that segments are no longer deleted after 1 minute
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
