Hi Luke, I understand your point, but the only plausible solution that completely eliminates the risk would be a one-size-fits-all mechanism that safely calculates retention.ms automatically for every cluster. In reality, it is impossible for the system to do a better job of determining whether a piece of metadata is still needed than the operators themselves, who have far more context and information about their workloads and topic configurations.
I don't think this is a problem that can be solved at the system level, as it ultimately relies on the operator's operational knowledge. In fact, if there is such good dynamic mechanism to calculate an appropriate retention.ms, then maybe that’s the ideal solution for limiting the size of the metadata topic, instead of the compacting solution itself. But if you have any good idea on that, I would love to hear your thoughts. Best, Lijun Tong On Tue, Aug 18, 2026 at 11:42 PM Luke Chen <[email protected]> wrote: > Hi Lijun, > > Sorry, I think more about this, and read your reply again, I think I > misunderstood it. > > > If the user is fine with -1 as the retention.ms, which means they're not > troubled by this metadata increasing issue yet, they don't need to do the > migration. > > I don't think this is accepted. Firstly, no one is OK with the unbounded > size topic. They use -1 just because it is the default value and the disk > size is still enough so far. So, everyone needs a migration plan. > > Therefore, the goal is: how we can make the existing retention.ms=-1 users > successfully migrate to the compacted topic without data loss. > > Look forward to your good solution. :) > > Luke > > > On Wed, Aug 19, 2026 at 11:28 AM Luke Chen <[email protected]> wrote: > > > Hi Lijun, > > > > I like the updated solution. > > 1. When upgrading to the new version containing this feature, the ` > > remote.log.metadata.topic.retention.ms` is still -1 (or any > > existing value). The only change is that the new metadata records will be > > written with a key. > > 2. Then we need the detailed design to this: "we can directly do some > > message format check and decide whether it's ok to make the topic > > compacted". So users don't need to worry about the data loss during the > > migration process. > > > > Look forward to the detailed design. > > > > Thank you, > > Luke > > > > > > On Wed, Aug 19, 2026 at 8:49 AM Lijun Tong <[email protected]> > > wrote: > > > >> Hey Luke, > >> > >> Thanks for your response! > >> > >> > "and once the old keyless records age out, the transition to > >> > compaction-based cleanup is complete." > >> > > >> > How will you achieve this? Could you elaborate more on this? > >> > >> > >> I do want to point out that even without this KIP's change, currently > user > >> needs to set their __remote_log_metadata topic's retention time so their > >> remote metadata will not increase permanently, this is recorded in the > >> tiered storage KIP[1] like I excerpted below: > >> > >> *remote.log.metadata.topic.retention.ms > >> <http://remote.log.metadata.topic.retention.ms>* > >> > >> Retention of the topic in milli seconds. > >> > >> Default: -1, that means unlimited. > >> > >> Users can configure this value based on their usecases. To avoid any > data > >> loss, this value should be more than the maximum retention period of any > >> topic enabled with tiered storage in the cluster. > >> > >> So the old keyless records will age out naturally even we don't ask the > >> user to configure retention.ms in the migration path. > >> > >> If the user is fine with -1 as the retention.ms, which means they're > not > >> troubled by this metadata increasing issue yet, they don't need to do > the > >> migration. > >> > >> If we decide to remove the retention.ms from the migration path, I > don't > >> envision we will need a 3 steps migration anymore, we can directly do > some > >> message format check and decide whether it's ok to make the topic > >> compacted, the metadata remote.log.metadata.version will only have 2 > >> value, > >> which is 0 and 1. I think this change will answer most of the edge case > >> related questions you have listed above. > >> > >> > >> Let me know if you have other questions, thanks! > >> > >> [1] > >> > >> > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/97554472/KIP-405+Kafka+Tiered+Storage#KIP405%3AKafkaTieredStorage-Configs > >> > >> Best, > >> Lijun Tong > >> > >> > >> Luke Chen <[email protected]> 于2026年8月18日周二 02:02写道： > >> > >> > Hi Lijun, > >> > > >> > Thanks for the reply. > >> > > >> > > "and once the old keyless records age out, the transition to > >> > compaction-based cleanup is complete." > >> > > >> > How will you achieve this? Could you elaborate more on this? > >> > > >> > I want to make sure our migration strategy covers every edge case to > >> > completely eliminate the risk of data loss, > >> > > >> > I think what I expect is our migration strategy covers every edge case > >> to > >> > completely eliminate the risk of data loss (i.e. unable to fetch from > >> the > >> > remote storage). So maybe you can think more about these questions > >> first: > >> > LC4.2: How can we avoid data loss when users directly move the version > >> from > >> > 0 -> 2? > >> > LC4.3: How can we avoid data loss when a short retention.ms is set > for > >> the > >> > internal topic? If it's the solution above, it needs more detailed > >> > information. > >> > LC4.4: How can we avoid data loss when users stay > >> > in remote.log.metadata.version 1 for a long time? > >> > LC4.6: How can we avoid data loss if the configs are altered > >> > after remote.log.metadata.version moves to 1? > >> > LC4.13. How can we avoid data loss if users are currently using > >> > retention.bytes for tiered storage topics when migration? > >> > > >> > > >> > Thank you, > >> > Luke > >> > > >> > > >> > On Wed, Aug 12, 2026 at 1:11 PM Lijun Tong <[email protected]> > >> > wrote: > >> > > >> > > Hi Luke, > >> > > > >> > > Thanks for the thoughtful feedback — I appreciate you pushing on > this. > >> > > > >> > > > >> > > You raise a fair point about the distinction between user topics and > >> > > internal control-plane topics like __remote_log_metadata. I agree > >> that we > >> > > should treat internal topics with extra care, and that relying on > >> users > >> > to > >> > > manually coordinate retention.ms with their longest > >> > remote-storage-enabled > >> > > topic introduces a fragile safeguard. > >> > > > >> > > > >> > > So here's an alternative approach: what if we simply don't ask the > >> user > >> > to > >> > > explicitly set retention.ms on __remote_log_metadata at all during > >> > > migration? Instead, we can just use whatever retention value is > >> already > >> > > configured on the topic. The migration would still complete > naturally > >> > once > >> > > all the legacy keyless messages have been deleted by the existing > >> > retention > >> > > policy — no manual user intervention required. > >> > > > >> > > This way, we avoid introducing any new risk of data loss during > >> > migration, > >> > > since we're not asking users to change anything on the control-plane > >> > topic. > >> > > The migration simply works with the retention behavior that's > already > >> in > >> > > effect, and once the old keyless records age out, the transition to > >> > > compaction-based cleanup is complete. > >> > > > >> > > > >> > > What do you think? Would this address your concern? > >> > > > >> > > Best, Lijun > >> > > > >> > > Luke Chen <[email protected]> 于2026年8月11日周二 20:25写道： > >> > > > >> > > > Hi Lijun, > >> > > > > >> > > > Thanks for the response. > >> > > > > >> > > > > The scenario you outlined — where a user sets retention.ms to > 14 > >> > days > >> > > > on a > >> > > > topic that has remote-storage-enabled data spanning 30 days — is > >> > > actually a > >> > > > risk that exists *today*, independent of this KIP. As soon as a > user > >> > > > enables tiered storage for a topic, they already bear the > >> > responsibility > >> > > of > >> > > > configuring retention.ms correctly to avoid unintended data > >> deletion. > >> > > This > >> > > > KIP doesn't introduce that risk; it inherits it from the original > >> > tiered > >> > > > storage design. > >> > > > > >> > > > I know what you mean. I think there is a key difference between > >> setting > >> > > > retention.ms on a user topic vs. the internal > __remote_log_metadata > >> > > topic: > >> > > > the __remote_log_metadata is shared and cluster-wide control plane > >> > topic > >> > > > (that's why we call it internal topic), which serves for all > tiered > >> > > storage > >> > > > topics. Its records are system pointers/indexes/metadata, not > >> business > >> > > > data. I think you know the risk, so instead of asking users to > set a > >> > > > retention time they like as a normal user topic, you ask users to > >> set > >> > > > retention.ms higher than the retention period of the longest > >> > > > remote-storage-enabled topic. But then, we lose any other > safeguard > >> > when > >> > > > users extend a user topic retention time from 14 days to 30 days, > >> which > >> > > > causes data loss (unable to fetch from the remote storage). This > is > >> > why I > >> > > > think we should improve it. > >> > > > > >> > > > Again, we're handling the control-plane topic, which should treat > it > >> > more > >> > > > carefully than user topics. > >> > > > > >> > > > > That said, I do want to highlight what this KIP is trying to > >> > > accomplish: > >> > > > once the migration is complete, this risk goes away. The new > design > >> > > > eliminates the need for operators to carefully coordinate > >> retention.ms > >> > > > with > >> > > > their remote storage retention, because compaction can apply to > new > >> > data > >> > > > going forward. So in a sense, the migration path is the bridge > that > >> > gets > >> > > > users *out of* this risky situation, not into it. > >> > > > > >> > > > I know and thanks for the proposal for the migration path. I can > >> see we > >> > > > want to help users on the migration path, but we still need to > >> > eliminate > >> > > > the risks of data loss during migration. > >> > > > > >> > > > > >> > > > Thank you, > >> > > > Luke > >> > > > > >> > > > On Wed, Aug 12, 2026 at 12:15 AM Lijun Tong < > >> [email protected]> > >> > > > wrote: > >> > > > > >> > > > > Hey Luke, > >> > > > > > >> > > > > Thanks for the feedback — I really appreciate you taking the > time > >> to > >> > > walk > >> > > > > through that scenario. Data durability is absolutely a top > >> priority, > >> > > and > >> > > > > I'm glad we're digging into this. > >> > > > > > >> > > > > I'd like to offer a slightly different perspective on the risk > you > >> > > > > described, though. > >> > > > > > >> > > > > The scenario you outlined — where a user sets retention.ms to > 14 > >> > days > >> > > > on a > >> > > > > topic that has remote-storage-enabled data spanning 30 days — is > >> > > > actually a > >> > > > > risk that exists *today*, independent of this KIP. As soon as a > >> user > >> > > > > enables tiered storage for a topic, they already bear the > >> > > responsibility > >> > > > of > >> > > > > configuring retention.ms correctly to avoid unintended data > >> > deletion. > >> > > > This > >> > > > > KIP doesn't introduce that risk; it inherits it from the > original > >> > > tiered > >> > > > > storage design. > >> > > > > > >> > > > > If we believe this risk is too significant to ask operators to > >> manage > >> > > > > during a migration window, then I think we'd also need to > >> acknowledge > >> > > > that > >> > > > > it's too significant to let operators manage in normal > production > >> > usage > >> > > > > right now — because the exposure is the same. In both cases, a > >> > > > > misconfigured retention.ms on a tiered-storage-enabled topic > can > >> > lead > >> > > to > >> > > > > data loss. > >> > > > > > >> > > > > That said, I do want to highlight what this KIP is trying to > >> > > accomplish: > >> > > > > once the migration is complete, this risk goes away. The new > >> design > >> > > > > eliminates the need for operators to carefully coordinate > >> > retention.ms > >> > > > > with > >> > > > > their remote storage retention, because compaction can apply to > >> new > >> > > data > >> > > > > going forward. So in a sense, the migration path is the bridge > >> that > >> > > gets > >> > > > > users *out of* this risky situation, not into it. > >> > > > > > >> > > > > Even in the scenario where we simply respect the retention.ms > >> that > >> > the > >> > > > > user > >> > > > > has already configured — without requiring any additional manual > >> > > checks — > >> > > > > the end state is still strictly better than what users have > today, > >> > > > because > >> > > > > new messages will benefit from compaction. > >> > > > > > >> > > > > I completely agree that we should make the migration as safe and > >> > smooth > >> > > > as > >> > > > > possible, and I'll take a closer look at KIP-866 and KIP-848 for > >> > > > > inspiration on how they handled their migration paths. If there > >> are > >> > > > > reasonable guardrails we can add (such as automatic > configuration > >> of > >> > > the > >> > > > > retention.ms based on current longest remote-storage-enabled > >> topic's > >> > > > > retention time), I'm very open to that. I just want to make sure > >> > we're > >> > > > > framing the risk accurately — as a pre-existing concern rather > >> than > >> > one > >> > > > > introduced by this proposal. > >> > > > > > >> > > > > Happy to discuss further. Thanks again for the thoughtful > review! > >> > > > > > >> > > > > Thanks, > >> > > > > > >> > > > > Lijun Tong > >> > > > > > >> > > > > Luke Chen <[email protected]> 于2026年8月11日周二 01:01写道： > >> > > > > > >> > > > > > Hi Lijun, > >> > > > > > > >> > > > > > Thanks for your reply. > >> > > > > > I’ll send a detailed reply later when available, but I wanted > to > >> > > > address > >> > > > > a > >> > > > > > core concern first. > >> > > > > > > >> > > > > > About the migration strategy, we can't rely on documentation > to > >> > > prevent > >> > > > > > potential data loss. We have to protect users from operational > >> > > mistakes > >> > > > > > because data durability is why users trust Kafka in the first > >> > place. > >> > > > > > > >> > > > > > In the current design of the migration path, for example, we > >> want > >> > > users > >> > > > > to > >> > > > > > set retention.ms higher than the retention period of the > >> longest > >> > > > > > remote-storage-enabled topic. That means the operator needs to > >> > check > >> > > it > >> > > > > > before migrating to version 1. It is easy for a 10 topic > >> cluster. > >> > But > >> > > > > for a > >> > > > > > hundreds of topic cluster, it'll be difficult and error-prone. > >> Even > >> > > if > >> > > > > they > >> > > > > > complete the check and set to 14 days retention time, some > users > >> > > could > >> > > > > > still change the retention period for a remote-storage-enabled > >> > topic > >> > > to > >> > > > > 30 > >> > > > > > days later. Then 14 days later, the retention deletion kicks > >> off, > >> > it > >> > > > will > >> > > > > > cause data loss for the topics, right? Also, if the > >> > > > > remote-storage-enabled > >> > > > > > topic is set as retention size, not retention time, what can > >> they > >> > do? > >> > > > > > > >> > > > > > This is just an example to highlight the flaw in the current > >> > design. > >> > > We > >> > > > > > need to rethink this path so the system can protect unexpected > >> > issues > >> > > > > > happening rather than leaving it as a manual effort for > >> operators. > >> > > > > > > >> > > > > > You can check other KIPs to see how they have considered to > >> achieve > >> > > the > >> > > > > > migration. The best example is KIP-866[1] : ZK migrating to > >> KRaft. > >> > > The > >> > > > > > other good example is KIP-848[2]. > >> > > > > > > >> > > > > > I really appreciate the thought and effort put into this > >> proposal! > >> > > > Please > >> > > > > > let me know if you have any questions, and we can discuss > >> further. > >> > > > > > > >> > > > > > > >> > > > > > [1]: > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/225153708/KIP-866+ZooKeeper+to+KRaft+Migration > >> > > > > > [2] > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/spaces/KAFKA/pages/217387038/KIP-848+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848%3ATheNextGenerationoftheConsumerRebalanceProtocol-SupportingOnlineConsumerGroupUpgrade > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > Thank you, > >> > > > > > Luke > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > On Mon, Aug 10, 2026 at 12:49 PM Lijun Tong < > >> > [email protected] > >> > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Hey Luke, > >> > > > > > > > >> > > > > > > Thanks for your response, I have answered your questions > >> below. > >> > > > > > > > >> > > > > > > LC1. In scenario 1, we have 2 records with different keys > (one > >> > with > >> > > > > > :UPDATE > >> > > > > > > suffix), how could we achieve the final result after log > >> > > compaction: > >> > > > > > > Retains only latest → abc123:topicA:0:1000:3 = FINISHED (T1 > >> > > compacted > >> > > > > > away) > >> > > > > > > > >> > > > > > > Current log compaction only compares the record key and > keeps > >> the > >> > > > > latest > >> > > > > > > value for the same key. So in this case, shouldn't we keep > >> these > >> > 2 > >> > > > > > records > >> > > > > > > after log compaction? > >> > > > > > > > >> > > > > > > *Re LC1:* This is correct. My apologies for the > inconsistency > >> in > >> > > the > >> > > > > > > proposal. I have updated the proposal to fix this issue. > >> > > > > > > > >> > > > > > > LC2. What does "Audit topic" mean here? > >> > > > > > > > >> > > > > > > *Re LC2:* Please disregard the reference to the audit > topic. I > >> > > > thought > >> > > > > I > >> > > > > > > had already refactored the proposal to remove it. > >> > > > > > > > >> > > > > > > For some background, my original intention was to retain the > >> > > keyless > >> > > > > > topic > >> > > > > > > as an audit topic. However, after discussing this with > others > >> > > earlier > >> > > > > in > >> > > > > > > this email thread, we concluded that an audit topic is > >> > unnecessary, > >> > > > so > >> > > > > I > >> > > > > > > removed this concept from the design. > >> > > > > > > > >> > > > > > > LC3. If we already store the endOffset and brokerLeaderEpoch > >> in > >> > the > >> > > > > > record > >> > > > > > > key, why should we still need them stored in the record > value? > >> > > > > > > > >> > > > > > > *Re LC3:* This is a good question. The reason is related to > >> how > >> > > > > > > RemoteLogManager currently interacts with > >> RemoteMetadataManager. > >> > > > > > > > >> > > > > > > With the current implementation, RemoteLogManager only has > >> access > >> > > to > >> > > > > the > >> > > > > > > metadata value; it does not have the record key available at > >> that > >> > > > > point. > >> > > > > > > TopicBasedRemoteLogMetadataManager is only one > implementation > >> > for > >> > > > the > >> > > > > > > RemoteLogMetadataManager, the key needs to be constructed > from > >> > the > >> > > > > > metadata > >> > > > > > > value rather than the other way around. > >> > > > > > > > >> > > > > > > For example, see: > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/kafka/blob/4.3.1/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L1053 > >> > > > > > > > >> > > > > > > LC4. About the migration: > >> > > > > > > > >> > > > > > > LC4.1. For "upgrade-to-v1" and "upgrade-to-v2", what's the > >> > > difference > >> > > > > > with: > >> > > > > > > bin/kafka-features.sh --bootstrap-server localhost:9092 > >> upgrade > >> > > > > --feature > >> > > > > > > remote.log.storage.version=1 > >> > > > > > > > >> > > > > > > *Re LC4.1:* I actually changed the feature name from > >> > > > > > > remote.log.storage.version to remote.log.metadata.version, > >> since > >> > I > >> > > > > think > >> > > > > > > the latter better reflects what is being versioned. > >> > > > > > > > >> > > > > > > If the user runs: > >> > > > > > > > >> > > > > > > kafka-remote-log-metadata-migration.sh --bootstrap-server > >> > > > > localhost:9092 > >> > > > > > > --upgrade-to-v1 > >> > > > > > > > >> > > > > > > the metadata version will be upgraded from 0 to 1. > >> > > > > > > > >> > > > > > > If the user runs the script with --upgrade-to-v2, the > metadata > >> > > > version > >> > > > > > will > >> > > > > > > be upgraded from 1 to 2. > >> > > > > > > > >> > > > > > > The migration script is different from directly changing the > >> > > feature > >> > > > > > > version with kafka-features.sh because the migration also > >> > performs > >> > > > the > >> > > > > > > necessary data migration and validation steps. > >> > > > > > > > >> > > > > > > LC4.2. If the user runs the kafka-features.sh command > >> directly to > >> > > > > upgrade > >> > > > > > > from 0 → 2, what will happen? > >> > > > > > > > >> > > > > > > *Re LC4.2:* Currently it will complain about the version > jump, > >> > but > >> > > it > >> > > > > > will > >> > > > > > > still proceed with the change, this means we lost all the > >> keyless > >> > > > > > messages > >> > > > > > > and data loss. > >> > > > > > > > >> > > > > > > I’m considering adding a safeguard to prevent users from > >> > upgrading > >> > > > > > directly > >> > > > > > > from version 0 to version 2 through kafka-features.sh, since > >> some > >> > > > > > migration > >> > > > > > > steps are required between these versions should be > performed > >> by > >> > > the > >> > > > > > > migration script. Furthermore, I have documented the > migration > >> > > > process > >> > > > > in > >> > > > > > > the feature doc > >> > > > > > > < > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/apache/kafka/blob/1ccb860cf83f8d1ba53f3b0706b18f3ca71e5b34/docs/operations/remote-log-metadata-topic-compaction.md > >> > > > > > > > > >> > > > > > > and clearly directed users to use the migration tool for > this > >> > > > upgrade. > >> > > > > > > > >> > > > > > > I’m open to suggestions on how we can best ensure that users > >> > follow > >> > > > the > >> > > > > > > migration path. > >> > > > > > > > >> > > > > > > LC4.3. What will happen if the setting of retention.ms is > too > >> > > short > >> > > > > > before > >> > > > > > > the existing consumers complete the fetch? > >> > > > > > > > >> > > > > > > *Re LC4.3:* With the current design, it is already the > cluster > >> > > > > operator's > >> > > > > > > responsibility to configure retention.ms appropriately. If > >> > > > > retention.ms > >> > > > > > is > >> > > > > > > too short and the metadata for an existing remote log > segment > >> is > >> > > > > deleted > >> > > > > > > before it is consumed, that metadata is lost permanently, > >> which > >> > can > >> > > > > > result > >> > > > > > > in data loss. This requirement exists independently of this > >> > > migration > >> > > > > and > >> > > > > > > the proposed feature. And this biggest motivation for this > >> > proposal > >> > > > is > >> > > > > to > >> > > > > > > lift this burden from the kafka operator and after the > >> migration, > >> > > > there > >> > > > > > is > >> > > > > > > no such risk. > >> > > > > > > > >> > > > > > > The migration does not introduce a new retention > requirement. > >> As > >> > > long > >> > > > > as > >> > > > > > > the operator follows the existing requirement and sets > >> > > retention.ms > >> > > > > > > appropriately, the migration is safe. > >> > > > > > > > >> > > > > > > Once the migration is complete, the keyed metadata log no > >> longer > >> > > > needs > >> > > > > to > >> > > > > > > be protected by this additional retention consideration. > >> > > > > > > > >> > > > > > > LC4.4. If the user sets 14 days retention.ms and upgrades > to > >> v1, > >> > > and > >> > > > > > some > >> > > > > > > more metadata is added. If one month later they are still on > >> v1, > >> > > will > >> > > > > the > >> > > > > > > keyed metadata log be deleted unexpectedly due to log > >> retention? > >> > > > > > > > >> > > > > > > *Re LC4.4:* The keyed metadata log can be truncated > according > >> to > >> > > the > >> > > > > > > configured retention policy while the cluster remains on > v1. I > >> > > don't > >> > > > > > think > >> > > > > > > this should be considered unexpected, since it is the > expected > >> > > > behavior > >> > > > > > of > >> > > > > > > the retention policy. > >> > > > > > > > >> > > > > > > I also don't think this creates a correctness issue, > provided > >> > that > >> > > > > > > retention.ms should be configured to be at least as long as > >> the > >> > > > > > retention > >> > > > > > > period of the longest remote-storage-enabled topic. Under > that > >> > > > > condition, > >> > > > > > > even if some older keyed metadata records are removed, they > >> will > >> > no > >> > > > > > longer > >> > > > > > > be needed by the remote storage system. > >> > > > > > > > >> > > > > > > > >> > > > > > > LC4.6. "Log cleaner won't compact yet (blocked by > >> > > > > min.compaction.lag.ms > >> > > > > > )", > >> > > > > > > what will happen if a user or operator changes the config > to a > >> > > > shorter > >> > > > > > > value? > >> > > > > > > > >> > > > > > > *Re LC4.6:* The migration script will set > >> min.compaction.lag.ms > >> > to > >> > > > the > >> > > > > > > same > >> > > > > > > value as retention.ms. Therefore, if the operator changes > the > >> > > > > > > configuration > >> > > > > > > before running the migration, the migration script will > >> overwrite > >> > > > that > >> > > > > > > value. > >> > > > > > > > >> > > > > > > However, if the operator changes min.compaction.lag.ms > after > >> the > >> > > > > > migration > >> > > > > > > has started and sets it to a value that is shorter than the > >> > > required > >> > > > > > > period, metadata associated with active remote log segments > >> could > >> > > be > >> > > > > > > compacted away prematurely. I will clarify this requirement > in > >> > the > >> > > > KIP. > >> > > > > > > > >> > > > > > > LC4.7. In "upgrade-to-v2", what does "Tool displays > retention > >> > > > reminder > >> > > > > > and > >> > > > > > > checks if enough time has passed" mean? > >> > > > > > > > >> > > > > > > *Re LC4.7:* When migrating from > remote.log.metadata.version=1 > >> to > >> > 2, > >> > > > the > >> > > > > > > migration script will scan the topic and verify whether all > >> > > existing > >> > > > > > > messages have keys. > >> > > > > > > > >> > > > > > > If it finds keyless messages, it will record the latest > >> timestamp > >> > > > among > >> > > > > > > those messages. Based on that timestamp and the topic's > >> > > retention.ms > >> > > > , > >> > > > > > the > >> > > > > > > tool can calculate an estimated time when the last keyless > >> > message > >> > > > > should > >> > > > > > > have been removed by retention. > >> > > > > > > > >> > > > > > > The tool will display this information to the user. The user > >> can > >> > > then > >> > > > > > rerun > >> > > > > > > the migration after that estimated time and proceed to v2 > once > >> > the > >> > > > > > keyless > >> > > > > > > messages have expired. > >> > > > > > > > >> > > > > > > LC4.8. In "upgrade-to-v2", "Scans topic for null-key > >> messages", > >> > > does > >> > > > > that > >> > > > > > > mean we'll launch a consumer and read the whole log of the > >> > > > > > > __remote_log_metadata topic? If each broker already loads > all > >> > logs > >> > > in > >> > > > > > some > >> > > > > > > __remote_log_metadata partitions, can't we do the scan > during > >> > > loading > >> > > > > and > >> > > > > > > directly return the result to the admin client? > >> > > > > > > > >> > > > > > > *Re LC4.8:* This is true: each broker already loads all > >> records > >> > > from > >> > > > > its > >> > > > > > > assigned partitions of __remote_log_metadata during startup. > >> > > > > > > > >> > > > > > > However, the validation required for the migration—checking > >> > whether > >> > > > > > records > >> > > > > > > are keyed and, for keyless records, determining their > >> > timestamps—is > >> > > > > > > unrelated to the existing loading logic. > >> > > > > > > > >> > > > > > > I would prefer not to introduce migration-specific logic > into > >> the > >> > > > > > existing > >> > > > > > > metadata loading path and then remove or maintain that logic > >> > after > >> > > > the > >> > > > > > > migration is complete. Keeping the migration scan separate > >> makes > >> > > the > >> > > > > > > existing runtime path simpler and avoids introducing > >> additional > >> > > > > > complexity > >> > > > > > > into the first version of this feature. > >> > > > > > > > >> > > > > > > LC4.9. Could you show the command result of > >> > > > > > > kafka-remote-log-metadata-migration.sh in the KIP? > >> > > > > > > > >> > > > > > > *Re LC4.9:* Updated. Thanks for the suggestion! > >> > > > > > > > >> > > > > > > LC4.10. Also, could you show the --help result of > >> > > > > > > kafka-remote-log-metadata-migration.sh? > >> > > > > > > > >> > > > > > > *Re LC4.10:* Updated. Thanks for the suggestion! > >> > > > > > > > >> > > > > > > LC4.11. Have you considered how a Kubernetes operator can > >> > automate > >> > > > the > >> > > > > > > migration steps? > >> > > > > > > > >> > > > > > > *Re LC4.11:* I’ll need some time to investigate this and > >> > understand > >> > > > the > >> > > > > > > implications for Kubernetes operators. I’ll follow up once > >> I’ve > >> > > had a > >> > > > > > > chance to look into it in more detail. > >> > > > > > > > >> > > > > > > LC4.12. I think the migration plan is the most critical part > >> of > >> > > this > >> > > > > KIP. > >> > > > > > > Users cannot accept any metadata deleted before it is > handled > >> > > > > completely. > >> > > > > > > We have to make sure all the edge cases are covered before > >> moving > >> > > > this > >> > > > > > > forward. > >> > > > > > > > >> > > > > > > *Re LC4.12:* I agree that the migration plan is one of the > >> most > >> > > > > critical > >> > > > > > > parts of this KIP. > >> > > > > > > > >> > > > > > > With the current migration design, the key requirement is > that > >> > > > > > > retention.ms > >> > > > > > > must not be shorter than the retention period of the longest > >> > > > > > > remote-storage-enabled topic. As long as this requirement is > >> > > > satisfied, > >> > > > > > the > >> > > > > > > migration does not introduce any additional data-loss > scenario > >> > > beyond > >> > > > > the > >> > > > > > > requirements of the existing remote storage implementation. > >> > > > > > > > >> > > > > > > I will make this requirement more explicit in the migration > >> > section > >> > > > and > >> > > > > > > clarify the relevant edge cases. > >> > > > > > > > >> > > > > > > Let me know if you have more questions, thanks! > >> > > > > > > > >> > > > > > > > >> > > > > > > Best, > >> > > > > > > > >> > > > > > > Lijun Tong > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >
