Hey Luke, Thanks for the feedback — I really appreciate you taking the time to walk through that scenario. Data durability is absolutely a top priority, and I'm glad we're digging into this.
I'd like to offer a slightly different perspective on the risk you described, though. The scenario you outlined — where a user sets retention.ms to 14 days on a topic that has remote-storage-enabled data spanning 30 days — is actually a risk that exists *today*, independent of this KIP. As soon as a user enables tiered storage for a topic, they already bear the responsibility of configuring retention.ms correctly to avoid unintended data deletion. This KIP doesn't introduce that risk; it inherits it from the original tiered storage design. If we believe this risk is too significant to ask operators to manage during a migration window, then I think we'd also need to acknowledge that it's too significant to let operators manage in normal production usage right now — because the exposure is the same. In both cases, a misconfigured retention.ms on a tiered-storage-enabled topic can lead to data loss. That said, I do want to highlight what this KIP is trying to accomplish: once the migration is complete, this risk goes away. The new design eliminates the need for operators to carefully coordinate retention.ms with their remote storage retention, because compaction can apply to new data going forward. So in a sense, the migration path is the bridge that gets users *out of* this risky situation, not into it. Even in the scenario where we simply respect the retention.ms that the user has already configured — without requiring any additional manual checks — the end state is still strictly better than what users have today, because new messages will benefit from compaction. I completely agree that we should make the migration as safe and smooth as possible, and I'll take a closer look at KIP-866 and KIP-848 for inspiration on how they handled their migration paths. If there are reasonable guardrails we can add (such as automatic configuration of the retention.ms based on current longest remote-storage-enabled topic's retention time), I'm very open to that. I just want to make sure we're framing the risk accurately — as a pre-existing concern rather than one introduced by this proposal. Happy to discuss further. Thanks again for the thoughtful review! Thanks, Lijun Tong Luke Chen <[email protected]> 于2026年8月11日周二 01:01写道： > Hi Lijun, > > Thanks for your reply. > I’ll send a detailed reply later when available, but I wanted to address a > core concern first. > > About the migration strategy, we can't rely on documentation to prevent > potential data loss. We have to protect users from operational mistakes > because data durability is why users trust Kafka in the first place. > > In the current design of the migration path, for example, we want users to > set retention.ms higher than the retention period of the longest > remote-storage-enabled topic. That means the operator needs to check it > before migrating to version 1. It is easy for a 10 topic cluster. But for a > hundreds of topic cluster, it'll be difficult and error-prone. Even if they > complete the check and set to 14 days retention time, some users could > still change the retention period for a remote-storage-enabled topic to 30 > days later. Then 14 days later, the retention deletion kicks off, it will > cause data loss for the topics, right? Also, if the remote-storage-enabled > topic is set as retention size, not retention time, what can they do? > > This is just an example to highlight the flaw in the current design. We > need to rethink this path so the system can protect unexpected issues > happening rather than leaving it as a manual effort for operators. > > You can check other KIPs to see how they have considered to achieve the > migration. The best example is KIP-866[1] : ZK migrating to KRaft. The > other good example is KIP-848[2]. > > I really appreciate the thought and effort put into this proposal! Please > let me know if you have any questions, and we can discuss further. > > > [1]: > > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/225153708/KIP-866+ZooKeeper+to+KRaft+Migration > [2] > > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/217387038/KIP-848+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848%3ATheNextGenerationoftheConsumerRebalanceProtocol-SupportingOnlineConsumerGroupUpgrade > > > > Thank you, > Luke > > > > On Mon, Aug 10, 2026 at 12:49 PM Lijun Tong <[email protected]> > wrote: > > > Hey Luke, > > > > Thanks for your response, I have answered your questions below. > > > > LC1. In scenario 1, we have 2 records with different keys (one with > :UPDATE > > suffix), how could we achieve the final result after log compaction: > > Retains only latest → abc123:topicA:0:1000:3 = FINISHED (T1 compacted > away) > > > > Current log compaction only compares the record key and keeps the latest > > value for the same key. So in this case, shouldn't we keep these 2 > records > > after log compaction? > > > > *Re LC1:* This is correct. My apologies for the inconsistency in the > > proposal. I have updated the proposal to fix this issue. > > > > LC2. What does "Audit topic" mean here? > > > > *Re LC2:* Please disregard the reference to the audit topic. I thought I > > had already refactored the proposal to remove it. > > > > For some background, my original intention was to retain the keyless > topic > > as an audit topic. However, after discussing this with others earlier in > > this email thread, we concluded that an audit topic is unnecessary, so I > > removed this concept from the design. > > > > LC3. If we already store the endOffset and brokerLeaderEpoch in the > record > > key, why should we still need them stored in the record value? > > > > *Re LC3:* This is a good question. The reason is related to how > > RemoteLogManager currently interacts with RemoteMetadataManager. > > > > With the current implementation, RemoteLogManager only has access to the > > metadata value; it does not have the record key available at that point. > > TopicBasedRemoteLogMetadataManager is only one implementation for the > > RemoteLogMetadataManager, the key needs to be constructed from the > metadata > > value rather than the other way around. > > > > For example, see: > > > > > https://github.com/apache/kafka/blob/4.3.1/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L1053 > > > > LC4. About the migration: > > > > LC4.1. For "upgrade-to-v1" and "upgrade-to-v2", what's the difference > with: > > bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature > > remote.log.storage.version=1 > > > > *Re LC4.1:* I actually changed the feature name from > > remote.log.storage.version to remote.log.metadata.version, since I think > > the latter better reflects what is being versioned. > > > > If the user runs: > > > > kafka-remote-log-metadata-migration.sh --bootstrap-server localhost:9092 > > --upgrade-to-v1 > > > > the metadata version will be upgraded from 0 to 1. > > > > If the user runs the script with --upgrade-to-v2, the metadata version > will > > be upgraded from 1 to 2. > > > > The migration script is different from directly changing the feature > > version with kafka-features.sh because the migration also performs the > > necessary data migration and validation steps. > > > > LC4.2. If the user runs the kafka-features.sh command directly to upgrade > > from 0 → 2, what will happen? > > > > *Re LC4.2:* Currently it will complain about the version jump, but it > will > > still proceed with the change, this means we lost all the keyless > messages > > and data loss. > > > > I’m considering adding a safeguard to prevent users from upgrading > directly > > from version 0 to version 2 through kafka-features.sh, since some > migration > > steps are required between these versions should be performed by the > > migration script. Furthermore, I have documented the migration process in > > the feature doc > > < > > > https://github.com/apache/kafka/blob/1ccb860cf83f8d1ba53f3b0706b18f3ca71e5b34/docs/operations/remote-log-metadata-topic-compaction.md > > > > > and clearly directed users to use the migration tool for this upgrade. > > > > I’m open to suggestions on how we can best ensure that users follow the > > migration path. > > > > LC4.3. What will happen if the setting of retention.ms is too short > before > > the existing consumers complete the fetch? > > > > *Re LC4.3:* With the current design, it is already the cluster operator's > > responsibility to configure retention.ms appropriately. If retention.ms > is > > too short and the metadata for an existing remote log segment is deleted > > before it is consumed, that metadata is lost permanently, which can > result > > in data loss. This requirement exists independently of this migration and > > the proposed feature. And this biggest motivation for this proposal is to > > lift this burden from the kafka operator and after the migration, there > is > > no such risk. > > > > The migration does not introduce a new retention requirement. As long as > > the operator follows the existing requirement and sets retention.ms > > appropriately, the migration is safe. > > > > Once the migration is complete, the keyed metadata log no longer needs to > > be protected by this additional retention consideration. > > > > LC4.4. If the user sets 14 days retention.ms and upgrades to v1, and > some > > more metadata is added. If one month later they are still on v1, will the > > keyed metadata log be deleted unexpectedly due to log retention? > > > > *Re LC4.4:* The keyed metadata log can be truncated according to the > > configured retention policy while the cluster remains on v1. I don't > think > > this should be considered unexpected, since it is the expected behavior > of > > the retention policy. > > > > I also don't think this creates a correctness issue, provided that > > retention.ms should be configured to be at least as long as the > retention > > period of the longest remote-storage-enabled topic. Under that condition, > > even if some older keyed metadata records are removed, they will no > longer > > be needed by the remote storage system. > > > > > > LC4.6. "Log cleaner won't compact yet (blocked by min.compaction.lag.ms > )", > > what will happen if a user or operator changes the config to a shorter > > value? > > > > *Re LC4.6:* The migration script will set min.compaction.lag.ms to the > > same > > value as retention.ms. Therefore, if the operator changes the > > configuration > > before running the migration, the migration script will overwrite that > > value. > > > > However, if the operator changes min.compaction.lag.ms after the > migration > > has started and sets it to a value that is shorter than the required > > period, metadata associated with active remote log segments could be > > compacted away prematurely. I will clarify this requirement in the KIP. > > > > LC4.7. In "upgrade-to-v2", what does "Tool displays retention reminder > and > > checks if enough time has passed" mean? > > > > *Re LC4.7:* When migrating from remote.log.metadata.version=1 to 2, the > > migration script will scan the topic and verify whether all existing > > messages have keys. > > > > If it finds keyless messages, it will record the latest timestamp among > > those messages. Based on that timestamp and the topic's retention.ms, > the > > tool can calculate an estimated time when the last keyless message should > > have been removed by retention. > > > > The tool will display this information to the user. The user can then > rerun > > the migration after that estimated time and proceed to v2 once the > keyless > > messages have expired. > > > > LC4.8. In "upgrade-to-v2", "Scans topic for null-key messages", does that > > mean we'll launch a consumer and read the whole log of the > > __remote_log_metadata topic? If each broker already loads all logs in > some > > __remote_log_metadata partitions, can't we do the scan during loading and > > directly return the result to the admin client? > > > > *Re LC4.8:* This is true: each broker already loads all records from its > > assigned partitions of __remote_log_metadata during startup. > > > > However, the validation required for the migration—checking whether > records > > are keyed and, for keyless records, determining their timestamps—is > > unrelated to the existing loading logic. > > > > I would prefer not to introduce migration-specific logic into the > existing > > metadata loading path and then remove or maintain that logic after the > > migration is complete. Keeping the migration scan separate makes the > > existing runtime path simpler and avoids introducing additional > complexity > > into the first version of this feature. > > > > LC4.9. Could you show the command result of > > kafka-remote-log-metadata-migration.sh in the KIP? > > > > *Re LC4.9:* Updated. Thanks for the suggestion! > > > > LC4.10. Also, could you show the --help result of > > kafka-remote-log-metadata-migration.sh? > > > > *Re LC4.10:* Updated. Thanks for the suggestion! > > > > LC4.11. Have you considered how a Kubernetes operator can automate the > > migration steps? > > > > *Re LC4.11:* I’ll need some time to investigate this and understand the > > implications for Kubernetes operators. I’ll follow up once I’ve had a > > chance to look into it in more detail. > > > > LC4.12. I think the migration plan is the most critical part of this KIP. > > Users cannot accept any metadata deleted before it is handled completely. > > We have to make sure all the edge cases are covered before moving this > > forward. > > > > *Re LC4.12:* I agree that the migration plan is one of the most critical > > parts of this KIP. > > > > With the current migration design, the key requirement is that > > retention.ms > > must not be shorter than the retention period of the longest > > remote-storage-enabled topic. As long as this requirement is satisfied, > the > > migration does not introduce any additional data-loss scenario beyond the > > requirements of the existing remote storage implementation. > > > > I will make this requirement more explicit in the migration section and > > clarify the relevant edge cases. > > > > Let me know if you have more questions, thanks! > > > > > > Best, > > > > Lijun Tong > > >
