Hi Lijun, Thanks for your reply. I’ll send a detailed reply later when available, but I wanted to address a core concern first.
About the migration strategy, we can't rely on documentation to prevent potential data loss. We have to protect users from operational mistakes because data durability is why users trust Kafka in the first place. In the current design of the migration path, for example, we want users to set retention.ms higher than the retention period of the longest remote-storage-enabled topic. That means the operator needs to check it before migrating to version 1. It is easy for a 10 topic cluster. But for a hundreds of topic cluster, it'll be difficult and error-prone. Even if they complete the check and set to 14 days retention time, some users could still change the retention period for a remote-storage-enabled topic to 30 days later. Then 14 days later, the retention deletion kicks off, it will cause data loss for the topics, right? Also, if the remote-storage-enabled topic is set as retention size, not retention time, what can they do? This is just an example to highlight the flaw in the current design. We need to rethink this path so the system can protect unexpected issues happening rather than leaving it as a manual effort for operators. You can check other KIPs to see how they have considered to achieve the migration. The best example is KIP-866[1] : ZK migrating to KRaft. The other good example is KIP-848[2]. I really appreciate the thought and effort put into this proposal! Please let me know if you have any questions, and we can discuss further. [1]: https://cwiki.apache.org/confluence/spaces/KAFKA/pages/225153708/KIP-866+ZooKeeper+to+KRaft+Migration [2] https://cwiki.apache.org/confluence/spaces/KAFKA/pages/217387038/KIP-848+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848%3ATheNextGenerationoftheConsumerRebalanceProtocol-SupportingOnlineConsumerGroupUpgrade Thank you, Luke On Mon, Aug 10, 2026 at 12:49 PM Lijun Tong <[email protected]> wrote: > Hey Luke, > > Thanks for your response, I have answered your questions below. > > LC1. In scenario 1, we have 2 records with different keys (one with :UPDATE > suffix), how could we achieve the final result after log compaction: > Retains only latest → abc123:topicA:0:1000:3 = FINISHED (T1 compacted away) > > Current log compaction only compares the record key and keeps the latest > value for the same key. So in this case, shouldn't we keep these 2 records > after log compaction? > > *Re LC1:* This is correct. My apologies for the inconsistency in the > proposal. I have updated the proposal to fix this issue. > > LC2. What does "Audit topic" mean here? > > *Re LC2:* Please disregard the reference to the audit topic. I thought I > had already refactored the proposal to remove it. > > For some background, my original intention was to retain the keyless topic > as an audit topic. However, after discussing this with others earlier in > this email thread, we concluded that an audit topic is unnecessary, so I > removed this concept from the design. > > LC3. If we already store the endOffset and brokerLeaderEpoch in the record > key, why should we still need them stored in the record value? > > *Re LC3:* This is a good question. The reason is related to how > RemoteLogManager currently interacts with RemoteMetadataManager. > > With the current implementation, RemoteLogManager only has access to the > metadata value; it does not have the record key available at that point. > TopicBasedRemoteLogMetadataManager is only one implementation for the > RemoteLogMetadataManager, the key needs to be constructed from the metadata > value rather than the other way around. > > For example, see: > > https://github.com/apache/kafka/blob/4.3.1/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L1053 > > LC4. About the migration: > > LC4.1. For "upgrade-to-v1" and "upgrade-to-v2", what's the difference with: > bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature > remote.log.storage.version=1 > > *Re LC4.1:* I actually changed the feature name from > remote.log.storage.version to remote.log.metadata.version, since I think > the latter better reflects what is being versioned. > > If the user runs: > > kafka-remote-log-metadata-migration.sh --bootstrap-server localhost:9092 > --upgrade-to-v1 > > the metadata version will be upgraded from 0 to 1. > > If the user runs the script with --upgrade-to-v2, the metadata version will > be upgraded from 1 to 2. > > The migration script is different from directly changing the feature > version with kafka-features.sh because the migration also performs the > necessary data migration and validation steps. > > LC4.2. If the user runs the kafka-features.sh command directly to upgrade > from 0 → 2, what will happen? > > *Re LC4.2:* Currently it will complain about the version jump, but it will > still proceed with the change, this means we lost all the keyless messages > and data loss. > > I’m considering adding a safeguard to prevent users from upgrading directly > from version 0 to version 2 through kafka-features.sh, since some migration > steps are required between these versions should be performed by the > migration script. Furthermore, I have documented the migration process in > the feature doc > < > https://github.com/apache/kafka/blob/1ccb860cf83f8d1ba53f3b0706b18f3ca71e5b34/docs/operations/remote-log-metadata-topic-compaction.md > > > and clearly directed users to use the migration tool for this upgrade. > > I’m open to suggestions on how we can best ensure that users follow the > migration path. > > LC4.3. What will happen if the setting of retention.ms is too short before > the existing consumers complete the fetch? > > *Re LC4.3:* With the current design, it is already the cluster operator's > responsibility to configure retention.ms appropriately. If retention.ms is > too short and the metadata for an existing remote log segment is deleted > before it is consumed, that metadata is lost permanently, which can result > in data loss. This requirement exists independently of this migration and > the proposed feature. And this biggest motivation for this proposal is to > lift this burden from the kafka operator and after the migration, there is > no such risk. > > The migration does not introduce a new retention requirement. As long as > the operator follows the existing requirement and sets retention.ms > appropriately, the migration is safe. > > Once the migration is complete, the keyed metadata log no longer needs to > be protected by this additional retention consideration. > > LC4.4. If the user sets 14 days retention.ms and upgrades to v1, and some > more metadata is added. If one month later they are still on v1, will the > keyed metadata log be deleted unexpectedly due to log retention? > > *Re LC4.4:* The keyed metadata log can be truncated according to the > configured retention policy while the cluster remains on v1. I don't think > this should be considered unexpected, since it is the expected behavior of > the retention policy. > > I also don't think this creates a correctness issue, provided that > retention.ms should be configured to be at least as long as the retention > period of the longest remote-storage-enabled topic. Under that condition, > even if some older keyed metadata records are removed, they will no longer > be needed by the remote storage system. > > > LC4.6. "Log cleaner won't compact yet (blocked by min.compaction.lag.ms)", > what will happen if a user or operator changes the config to a shorter > value? > > *Re LC4.6:* The migration script will set min.compaction.lag.ms to the > same > value as retention.ms. Therefore, if the operator changes the > configuration > before running the migration, the migration script will overwrite that > value. > > However, if the operator changes min.compaction.lag.ms after the migration > has started and sets it to a value that is shorter than the required > period, metadata associated with active remote log segments could be > compacted away prematurely. I will clarify this requirement in the KIP. > > LC4.7. In "upgrade-to-v2", what does "Tool displays retention reminder and > checks if enough time has passed" mean? > > *Re LC4.7:* When migrating from remote.log.metadata.version=1 to 2, the > migration script will scan the topic and verify whether all existing > messages have keys. > > If it finds keyless messages, it will record the latest timestamp among > those messages. Based on that timestamp and the topic's retention.ms, the > tool can calculate an estimated time when the last keyless message should > have been removed by retention. > > The tool will display this information to the user. The user can then rerun > the migration after that estimated time and proceed to v2 once the keyless > messages have expired. > > LC4.8. In "upgrade-to-v2", "Scans topic for null-key messages", does that > mean we'll launch a consumer and read the whole log of the > __remote_log_metadata topic? If each broker already loads all logs in some > __remote_log_metadata partitions, can't we do the scan during loading and > directly return the result to the admin client? > > *Re LC4.8:* This is true: each broker already loads all records from its > assigned partitions of __remote_log_metadata during startup. > > However, the validation required for the migration—checking whether records > are keyed and, for keyless records, determining their timestamps—is > unrelated to the existing loading logic. > > I would prefer not to introduce migration-specific logic into the existing > metadata loading path and then remove or maintain that logic after the > migration is complete. Keeping the migration scan separate makes the > existing runtime path simpler and avoids introducing additional complexity > into the first version of this feature. > > LC4.9. Could you show the command result of > kafka-remote-log-metadata-migration.sh in the KIP? > > *Re LC4.9:* Updated. Thanks for the suggestion! > > LC4.10. Also, could you show the --help result of > kafka-remote-log-metadata-migration.sh? > > *Re LC4.10:* Updated. Thanks for the suggestion! > > LC4.11. Have you considered how a Kubernetes operator can automate the > migration steps? > > *Re LC4.11:* I’ll need some time to investigate this and understand the > implications for Kubernetes operators. I’ll follow up once I’ve had a > chance to look into it in more detail. > > LC4.12. I think the migration plan is the most critical part of this KIP. > Users cannot accept any metadata deleted before it is handled completely. > We have to make sure all the edge cases are covered before moving this > forward. > > *Re LC4.12:* I agree that the migration plan is one of the most critical > parts of this KIP. > > With the current migration design, the key requirement is that > retention.ms > must not be shorter than the retention period of the longest > remote-storage-enabled topic. As long as this requirement is satisfied, the > migration does not introduce any additional data-loss scenario beyond the > requirements of the existing remote storage implementation. > > I will make this requirement more explicit in the migration section and > clarify the relevant edge cases. > > Let me know if you have more questions, thanks! > > > Best, > > Lijun Tong >
