Hey Kamal, Thanks for the quick response.
One concern is that upload failures don't happen exclusively while the broker is online. If a broker goes down unexpectedly during an upload attempt, how would we later identify that failed attempt and publish a tombstone for the corresponding segmentId? >From an operational perspective, the current key design also provides very little context. It doesn't tell users which broker (or leader epoch) attempted to upload which log segment, making it difficult to reason about the behavior of tiered storage. In practice, it turns the remote metadata into something of a black box. I believe the original key design, TopicIdPartition:TopicName:Partition:EndOffset:BrokerLeaderEpoch, is much more self-descriptive. It clearly identifies the log segment being tracked and provides enough context for users to understand and debug the behavior of the remote metadata manager. Best, Lijun Tong Kamal Chandraprakash <[email protected]> 于2026年7月9日周四 10:00写道： > Hi Lijun, > > Thanks for the follow up! > > > Since a new segmentId is > generated every time a broker retries uploading the same log segment, it > doesn't provide a persistent identity for that log segment. > > RemoteLogManager tries to delete the remote log segment when it fails to > upload [1]. > Only when encountering RetriableRemoteStorageException, it does not invoke > the `deleteRemoteLogSegment` call [2]. > If we extend the behaviour to issue delete remote log segments > (delete_seg_started and del_seg_finished events) in case of any upload > failures. > Does it simplify the overall approach? > > [1]: > > https://sourcegraph.com/r/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java?L1117 > [2]: > > https://sourcegraph.com/r/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java?L1113 > > > Thanks, > Kamal > > On Thu, Jul 9, 2026 at 9:32 PM Lijun Tong <[email protected]> wrote: > > > Hey Kamal, > > > > > > I reviewed your implementation of keyed events. I noticed that the key is > > designed as topicId + partition + segmentId + stateId. > > > > I don't think we've fully discussed the tradeoffs or the potential > > concurrency implications of this key design yet. > > > > My main concern is that segmentId doesn't seem stable enough to represent > > the lifecycle of a log segment in remote storage. Since a new segmentId > is > > generated every time a broker retries uploading the same log segment, it > > doesn't provide a persistent identity for that log segment. > > > > Additionally, there isn't an explicit way to determine which log segment > a > > given segmentId corresponds to. That raises the question of how we would > > publish tombstone messages for upload attempts that failed and were later > > retried. > > > > With this key design, it seems like concurrency control and cleanup logic > > could become fairly complicated. In contrast, with the current key > design, > > concurrency control is naturally straightforward, and the correctness of > > the cleanup logic is much easier to reason about and verify. > > > > > > What do you think? > > > > > > Best, > > > > Lijun Tong > > > > Kamal Chandraprakash <[email protected]> 于2026年7月7日周二 > > 02:42写道： > > > > > Hi Lijun, > > > > > > I made a first pass on the https://github.com/apache/kafka/pull/22459 > > PR. > > > Most of the changes in the schema layer may not be required. And, we > > should > > > be able to achieve the outcome with minimal changes. > > > > > > I've opened a draft proposal to consider: > > > https://github.com/apache/kafka/pull/22774. Could you take a look? > > > > > > Thanks, > > > Kamal > > > > > > On Wed, Jun 3, 2026 at 12:20 PM Lijun Tong <[email protected]> > > > wrote: > > > > > > > Hey team, > > > > > > > > I have finished the code change for this KIP, and PR is here > > > > https://github.com/apache/kafka/pull/22459. There are 2 commits in > > this > > > > PR, > > > > the first commit includes all the source code change, the second > commit > > > > includes all the test changes. > > > > I have adopted some advices from this email chain and have also > updated > > > the > > > > KIP and code accordingly. > > > > > > > > Would love to hear your thoughts. > > > > > > > > Thanks, > > > > Lijun Tong > > > > > > > > jian fu <[email protected]> 于2026年4月11日周六 18:38写道： > > > > > > > > > Hi Kamal and Lijun Tong > > > > > > > > > > Maybe another approach (Not good solution) is when we sure about > the > > > work > > > > > is complete. we send all the keys' to null to delete all the data. > > > > > It mean the keys are: > > > > > clusterid:topic:partition:offset: COPY_SEGMENT_FINISHED > > > > > clusterid:topic:partition:offset: DELETE_SEGMENT_STARTED > > > > > clusterid:topic:partition:offset: DELETE_SEGMENT_FINISHED > > > > > ---------------- > > > > > > > > > > This solution won't change the old payload in the message. Thus the > > > > > solution is somehow strange. > > > > > Or carry all complete message so that keep the latest maybe can > work. > > > > > > > > > > BTW: There are four type message need to take care: > > > > > RemoteLogSegmentMetadataRecord > > > > > RemoteLogSegmentMetadataUpdateRecord > > > > > RemotePartitionDeleteMetadataRecord > > > > > RemoteLogSegmentMetadataSnapshotRecord > > > > > > > > > > Regards > > > > > Jian > > > > > > > > > > > > > > > Lijun Tong <[email protected]> 于2026年4月7日周二 08:35写道： > > > > > > > > > > > Hi Kamal, > > > > > > > > > > > > I now see what you mean regarding the DELETE_SEGMENT_STARTED > > > handling. > > > > > > > > > > > > I was thinking that the broker's logic that currently ignores > log > > > > > segment > > > > > > messages when COPY_SEGMENT_STARTED doesn't exist might need to be > > > > updated > > > > > > for the > > > > > > new message format. Since the new message contains useful > > > > information > > > > > > like endOffset, topicID, and partition, it could provide > sufficient > > > > > context > > > > > > to help > > > > > > with the retry of deletion operations. This suggests the broker > > > might > > > > > not > > > > > > need to ignore these messages solely because of the absence of > > > > > > COPY_SEGMENT_STARTED. > > > > > > > > > > > > With the new key-based format, we would have enough metadata in > > the > > > > > > DELETE_SEGMENT_STARTED event itself to: > > > > > > - Identify the remote segment location > > > > > > - Track and retry the deletion process > > > > > > - Prevent orphaned segments in remote storage > > > > > > > > > > > > I believe this means that with the new keyed message format, we > > > could > > > > > > avoid orphaned remote segments that might occur when > > > > COPY_SEGMENT_STARTED > > > > > > is cleaned > > > > > > by the retention policy. The DELETE_SEGMENT_STARTED event would > > > carry > > > > > all > > > > > > the necessary information to complete the deletion, regardless of > > > > whether > > > > > > the > > > > > > base metadata still exists. > > > > > > > > > > > > For the old format messages, the same mechanism (ignoring when > > > > > > COPY_SEGMENT_STARTED is missing) could still apply, as those > > messages > > > > > lack > > > > > > the necessary > > > > > > information for independent processing. > > > > > > > > > > > > I'd appreciate your thoughts on this approach. > > > > > > > > > > > > Best, > > > > > > Lijun Tong > > > > > > > > > > > > > > > > > > Kamal Chandraprakash <[email protected]> > 于2026年4月5日周日 > > > > > > 17:37写道： > > > > > > > > > > > > > Hi Lijun, > > > > > > > > > > > > > > Yes, we can get on a call to close out on this. My concern is > > that > > > if > > > > > the > > > > > > > same key is maintained for a given segment metadata records, > > > > > > > then the newer messages (COPY_FINSIHED, DELETE_STARTED) might > > > > > > > override/compact the previous COPY_STARTED events. > > > > > > > This is not about the old / new format of messages. Assume that > > all > > > > the > > > > > > > messages in the topic are in the new format and contain keys. > > > > > > > > > > > > > > From > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.confluent.io/kafka/design/log_compaction.html#topic-compaction > > > > > > > : > > > > > > > > > > > > > > Topic compaction is a mechanism that allows you to retain the > > > latest > > > > > > value > > > > > > > for each message key in a topic, while discarding older values. > > It > > > > > > > guarantees that the latest value for each message key is always > > > > > retained > > > > > > > within the log of data contained in that topic, making it ideal > > for > > > > use > > > > > > > cases such as restoring state after system failure or reloading > > > > caches > > > > > > > after application restarts. > > > > > > > > > > > > > > Thanks, > > > > > > > Kamal > > > > > > > > > > > > > > On Sun, Apr 5, 2026 at 11:37 PM Lijun Tong < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Kamal, > > > > > > > > > > > > > > > > Thanks for raising this. > > > > > > > > > > > > > > > > Currently, only the existing version of > > > > > > > > RemoteLogSegmentMetadataUpdateRecord > > > > > > > > does not include those fields. We rely on the time-based > > > retention > > > > > > policy > > > > > > > > for cleanup, and this does not impact the ability to rebuild > > the > > > > > > > > RemoteLogMetadataCache. > > > > > > > > > > > > > > > > The cache reconstruction should still work correctly because > it > > > > > depends > > > > > > > on > > > > > > > > the value, and we have not removed any fields from the value. > > > > > > > > > > > > > > > > Regarding the scenario where there are 10 remote segments and > > the > > > > > > > > __remote_log_metadata topic contains only > COPY_SEGMENT_FINISHED > > > > > events: > > > > > > > the > > > > > > > > COPY_SEGMENT_STARTED events will not be compacted in this > case, > > > > since > > > > > > > > messages with a null key are not subject to compaction. > > > > > > > > > > > > > > > > Once older-format messages are cleaned up by the time-based > > > > retention > > > > > > > > policy and compaction is enabled, records with the same key > > will > > > be > > > > > > > > compacted asynchronously and correctly. Given this, I don’t > > > believe > > > > > we > > > > > > > need > > > > > > > > to introduce a separate key for COPY_SEGMENT_STARTED events. > > > > > > > > > > > > > > > > Happy to jump on a call if it’s easier to discuss further. > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Lijun Tong > > > > > > > > > > > > > > > > Lijun Tong <[email protected]> 于2026年4月5日周日 10:56写道： > > > > > > > > > > > > > > > > > Hey Kamal, > > > > > > > > > > > > > > > > > > I am not very clear on what's the question you mentioned > > > above, I > > > > > am > > > > > > > > > happy to jump to a call to discuss further, and I lived in > > PST > > > > time > > > > > > > zone. > > > > > > > > > Maybe we can meet online through google meet? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Lijun Tong > > > > > > > > > > > > > > > > > > Kamal Chandraprakash <[email protected]> > > > > 于2026年4月1日周三 > > > > > > > > > 02:38写道： > > > > > > > > > > > > > > > > > >> Hi Lijun, > > > > > > > > >> > > > > > > > > >> Thanks for the update! I'm still not clear on this. > > > > > > > > >> > > > > > > > > >> The RemoteLogSegmentMetadataUpdateRecord does not contain > > the > > > > > below > > > > > > > > fields > > > > > > > > >> compared to RemoteLogSegmentMetadataRecord: > > > > > > > > >> > > > > > > > > >> - startOffset > > > > > > > > >> - endOffset (will be added as a tagged field) > > > > > > > > >> - MaxTimestampMs > > > > > > > > >> - SegmentLeaderEpochs > > > > > > > > >> - SegmentSizeInBytes and > > > > > > > > >> - TxnIndexEmpty > > > > > > > > >> > > > > > > > > >> When a broker gets restarted, will it be able to rebuild > > > > > > > > >> the RemoteLogMetadataCache? Assume that there are 10 > remote > > > > > > > > >> segments and the __remote_log_metadata topic contains only > > the > > > > > > > > >> COPY_SEGMENT_FINISHED events; the COPY_SEGMENT_STARTED > event > > > > > > > > >> gets compacted as the key is the same. > > > > > > > > >> > > > > > > > > >> Do we need a separate key for the COPY_SEGMENT_STARTED > event > > > and > > > > > > > another > > > > > > > > >> key for the remaining states? > > > > > > > > >> > > > > > > > > >> Current key format: > > > TopicIdPartition:EndOffset:BrokerLeaderEpoch > > > > > > > > >> Proposed key format: > > > > > > TopicIdPartition:EndOffset:BrokerLeaderEpoch:x/y > > > > > > > > >> where > > > > > > > > >> x denotes a identifier for COPY_SEGMENT_STARTED and y > denote > > > for > > > > > all > > > > > > > the > > > > > > > > >> other events. > > > > > > > > >> > > > > > > > > >> Thanks, > > > > > > > > >> Kamal > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> On Tue, Mar 31, 2026 at 8:23 AM Lijun Tong < > > > > > [email protected] > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > Hi Kamal, > > > > > > > > >> > > > > > > > > > >> > The scenario you described only happened with the old > > > version > > > > > > > > >> > RemoteLogSegmentUpdateMetadata message, since the > > endOffset > > > > will > > > > > > be > > > > > > > > >> added > > > > > > > > >> > in the new RemoteLogSegmentUpdateMetadata schema. For > the > > > > > existing > > > > > > > > >> > RemoteLogSegmentUpdateMetadata messages, we rely on the > > time > > > > > based > > > > > > > > >> > retention policy to clean up. Does that make sense? > > > > > > > > >> > > > > > > > > > >> > Best, > > > > > > > > >> > Lijun Tong > > > > > > > > >> > > > > > > > > > >> > Kamal Chandraprakash <[email protected]> > > > > > > 于2026年3月30日周一 > > > > > > > > >> > 18:14写道： > > > > > > > > >> > > > > > > > > > >> > > Hi Lijun, > > > > > > > > >> > > > > > > > > > > >> > > RemoteLogSegmentUpdateMetadata event does not have all > > the > > > > > > > > >> > > fields/attributes similar to RemoteLogSegmentMetadata > > > event. > > > > > > > > >> > > > > > > > > > > >> > > Assume that after compaction, for a segment, we have > > only > > > > > > > > >> > > COPY_SEGMENT_FINISHED records. How do you plan to > > retrieve > > > > the > > > > > > > other > > > > > > > > >> > fields > > > > > > > > >> > > after broker restart? > > > > > > > > >> > > > > > > > > > > >> > > Thanks, > > > > > > > > >> > > Kamal > > > > > > > > >> > > > > > > > > > > >> > > On Mon, Mar 30, 2026, 23:22 Lijun Tong < > > > > > [email protected] > > > > > > > > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > > >> > > > Hi Kamal, > > > > > > > > >> > > > > > > > > > > > >> > > > Thanks for taking another look at the KIP. > > > > > > > > >> > > > 1. I have removed the left-over line about using > > another > > > > new > > > > > > > topic > > > > > > > > >> from > > > > > > > > >> > > the > > > > > > > > >> > > > KIP. > > > > > > > > >> > > > 2. > > > > > > > > >> > > > > > > > > > > > >> > > > > 2. Assume that the topic is enabled with > compaction > > > and > > > > > only > > > > > > > one > > > > > > > > >> > event > > > > > > > > >> > > is > > > > > > > > >> > > > > maintained per segment. If there is a transient > > error > > > in > > > > > the > > > > > > > > >> remote > > > > > > > > >> > log > > > > > > > > >> > > > > deletion, > > > > > > > > >> > > > > then the COPY_SEGMENT started / finished > events > > > > might > > > > > be > > > > > > > > >> > compacted > > > > > > > > >> > > by > > > > > > > > >> > > > > the DELETE_SEGMENT_STARTED events. If the broker > is > > > > > > restarted > > > > > > > > >> during > > > > > > > > >> > > > > this time, will there be dangling remote log > > > > segments? > > > > > > > > >> Currently, > > > > > > > > >> > > > > during restart, the broker discards the events if > it > > > > does > > > > > > not > > > > > > > > see > > > > > > > > >> the > > > > > > > > >> > > > > COPY_SEGMENT_STARTED events. > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > I am glad you asked this question, I didn't mention > > this > > > > > part > > > > > > in > > > > > > > > my > > > > > > > > >> > > current > > > > > > > > >> > > > design to avoid distractions from the main design, > > but I > > > > > plan > > > > > > to > > > > > > > > add > > > > > > > > >> > > > another background thread to clean up all the stale > > > > messages > > > > > > by > > > > > > > > >> > comparing > > > > > > > > >> > > > the message's endOffset with the topic partition's > log > > > > start > > > > > > > > >> offset. I > > > > > > > > >> > > > believe this would help remove all the dangling > > > messages. > > > > > > > > >> > > > > > > > > > > > >> > > > Thanks, > > > > > > > > >> > > > Lijun TOng > > > > > > > > >> > > > > > > > > > > > >> > > > Kamal Chandraprakash < > [email protected]> > > > > > > > > 于2026年3月29日周日 > > > > > > > > >> > > > 22:48写道： > > > > > > > > >> > > > > > > > > > > > >> > > > > Hi Lijun, > > > > > > > > >> > > > > > > > > > > > > >> > > > > Sorry for the late reply. Went over the KIP again. > > > > Overall > > > > > > > LGTM. > > > > > > > > >> Few > > > > > > > > >> > > > > points: > > > > > > > > >> > > > > > > > > > > > > >> > > > > > This KIP aims to solve this issue through > > > introducing > > > > > > > another > > > > > > > > >> > > compacted > > > > > > > > >> > > > > topic for the brokers to bootstrap the state from > > > > > > > > >> > > > > > > > > > > > > >> > > > > 1. Shall we update the motivation section to > mention > > > > that > > > > > > > > another > > > > > > > > >> > topic > > > > > > > > >> > > > is > > > > > > > > >> > > > > not introduced? > > > > > > > > >> > > > > 2. Assume that the topic is enabled with > compaction > > > and > > > > > only > > > > > > > one > > > > > > > > >> > event > > > > > > > > >> > > is > > > > > > > > >> > > > > maintained per segment. If there is a transient > > error > > > in > > > > > the > > > > > > > > >> remote > > > > > > > > >> > log > > > > > > > > >> > > > > deletion, > > > > > > > > >> > > > > then the COPY_SEGMENT started / finished > events > > > > might > > > > > be > > > > > > > > >> > compacted > > > > > > > > >> > > by > > > > > > > > >> > > > > the DELETE_SEGMENT_STARTED events. If the broker > is > > > > > > restarted > > > > > > > > >> during > > > > > > > > >> > > > > this time, will there be dangling remote log > > > > segments? > > > > > > > > >> Currently, > > > > > > > > >> > > > > during restart, the broker discards the events if > it > > > > does > > > > > > not > > > > > > > > see > > > > > > > > >> the > > > > > > > > >> > > > > COPY_SEGMENT_STARTED events. > > > > > > > > >> > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > >> > > > > Kamal > > > > > > > > >> > > > > > > > > > > > > >> > > > > On Thu, Mar 26, 2026 at 5:08 AM Lijun Tong < > > > > > > > > >> [email protected]> > > > > > > > > >> > > > > wrote: > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Hi, > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > I have started a Vote thread for this KIP, > > > considering > > > > > all > > > > > > > > >> > questions > > > > > > > > >> > > > > raised > > > > > > > > >> > > > > > so far have been answered. I am happy to > continue > > > the > > > > > > > > >> discussion if > > > > > > > > >> > > > > needed, > > > > > > > > >> > > > > > otherwise, this is a friendly reminder on the > vote > > > for > > > > > > this > > > > > > > > KIP. > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > >> > > > > > Lijun Tong > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > Lijun Tong <[email protected]> > > 于2026年1月19日周一 > > > > > > 17:59写道： > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Hey Kamal, > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Thanks for raising these questions. Here are > my > > > > > > responses > > > > > > > to > > > > > > > > >> your > > > > > > > > >> > > > > > > questions: > > > > > > > > >> > > > > > > Q1 and Q2: > > > > > > > > >> > > > > > > I think both questions boil down to how to > > release > > > > > this > > > > > > > new > > > > > > > > >> > > feature, > > > > > > > > >> > > > > both > > > > > > > > >> > > > > > > questions are valid concerns. The solution I > > have > > > in > > > > > > mind > > > > > > > is > > > > > > > > >> this > > > > > > > > >> > > > > feature > > > > > > > > >> > > > > > > is *gated by the metadata version*. The new > > > > tombstone > > > > > > > > >> semantics > > > > > > > > >> > and > > > > > > > > >> > > > the > > > > > > > > >> > > > > > > additional fields (for example in > > > > > > > > >> RemoteLogSegmentUpdateRecord) > > > > > > > > >> > are > > > > > > > > >> > > > > only > > > > > > > > >> > > > > > > enabled once the cluster metadata version is > > > > upgraded > > > > > to > > > > > > > the > > > > > > > > >> > > version > > > > > > > > >> > > > > that > > > > > > > > >> > > > > > > introduces this feature. As long as the > cluster > > > > > metadata > > > > > > > > >> version > > > > > > > > >> > is > > > > > > > > >> > > > not > > > > > > > > >> > > > > > > bumped, the system will not produce tombstone > > > > records. > > > > > > > > >> Therefore, > > > > > > > > >> > > > > during > > > > > > > > >> > > > > > > rolling upgrades (mixed 4.2/4.3 brokers), the > > > > feature > > > > > > > > remains > > > > > > > > >> > > > > effectively > > > > > > > > >> > > > > > > disabled. Tombstones will only start being > > > produced > > > > > > after > > > > > > > > the > > > > > > > > >> > > > metadata > > > > > > > > >> > > > > > > version is upgraded, at which point all > brokers > > > are > > > > > > > already > > > > > > > > >> > > required > > > > > > > > >> > > > to > > > > > > > > >> > > > > > > support the new behavior. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Since Kafka does not support metadata version > > > > > downgrades > > > > > > > at > > > > > > > > >> the > > > > > > > > >> > > > moment, > > > > > > > > >> > > > > > > once a metadata version that supports this > > feature > > > > is > > > > > > > > >> enabled, it > > > > > > > > >> > > > > cannot > > > > > > > > >> > > > > > be > > > > > > > > >> > > > > > > downgraded to a version that does not support > > it. > > > I > > > > > will > > > > > > > add > > > > > > > > >> > these > > > > > > > > >> > > > > > details > > > > > > > > >> > > > > > > to the KIP later. > > > > > > > > >> > > > > > > Q3. This is an *editing mistake* in the KIP. > > > Thanks > > > > > for > > > > > > > > >> pointing > > > > > > > > >> > it > > > > > > > > >> > > > > out — > > > > > > > > >> > > > > > > the enum value has already been corrected in > the > > > > > latest > > > > > > > > >> revision > > > > > > > > >> > to > > > > > > > > >> > > > > > remove > > > > > > > > >> > > > > > > the unused placeholder and keep the state > values > > > > > > > contiguous > > > > > > > > >> and > > > > > > > > >> > > > > > consistent. > > > > > > > > >> > > > > > > Q4. I don't foresee the quota mechanism will > > > > interfere > > > > > > > with > > > > > > > > >> the > > > > > > > > >> > > state > > > > > > > > >> > > > > > > transition in any way so far, let me know if > any > > > > > concern > > > > > > > > hits > > > > > > > > >> > you. > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > >> > > > > > > Lijun > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > Kamal Chandraprakash < > > > > [email protected]> > > > > > > > > >> > > 于2026年1月18日周日 > > > > > > > > >> > > > > > > 00:40写道： > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > >> Hi Lijun, > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> Thanks for updating the KIP! > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> The updated migration plan looks clean to me. > > Few > > > > > > > > questions: > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> 1. The ConsumerTask in 4.2 Kafka build does > not > > > > > handle > > > > > > > the > > > > > > > > >> > > tombstone > > > > > > > > >> > > > > > >> records. Should the tombstone records be sent > > > only > > > > > when > > > > > > > all > > > > > > > > >> the > > > > > > > > >> > > > > brokers > > > > > > > > >> > > > > > >> are > > > > > > > > >> > > > > > >> upgraded to 4.3 version? > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> 2. Once all the brokers are upgraded and the > > > > > > > > >> > __remote_log_metadata > > > > > > > > >> > > > > topic > > > > > > > > >> > > > > > >> cleanup policy changed to compact. Then, > > > > downgrading > > > > > > the > > > > > > > > >> brokers > > > > > > > > >> > > is > > > > > > > > >> > > > > not > > > > > > > > >> > > > > > >> allowed as the records without key will throw > > an > > > > > error > > > > > > > > while > > > > > > > > >> > > > producing > > > > > > > > >> > > > > > the > > > > > > > > >> > > > > > >> compacted topic. Shall we mention this in the > > > > > > > compatibility > > > > > > > > >> > > section? > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> 3. In the RemoteLogSegmentState Enum, why is > > the > > > > > value > > > > > > 1 > > > > > > > > >> marked > > > > > > > > >> > as > > > > > > > > >> > > > > > unused? > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> 4. Regarding the key > > > > > > > > >> > > (TopicIdPartition:EndOffset:BrokerLeaderEpoch), > > > > > > > > >> > > > > we > > > > > > > > >> > > > > > >> may > > > > > > > > >> > > > > > >> have to check for scenarios where there is > > > segment > > > > > lag > > > > > > > due > > > > > > > > to > > > > > > > > >> > > remote > > > > > > > > >> > > > > log > > > > > > > > >> > > > > > >> write quota. Will the state transition work > > > > > correctly? > > > > > > > Will > > > > > > > > >> come > > > > > > > > >> > > > back > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > >> this later. > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> Thanks, > > > > > > > > >> > > > > > >> Kamal > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> On Fri, Jan 16, 2026 at 4:50 AM jian fu < > > > > > > > > >> [email protected]> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > >> > Hi Lijun and Kamal > > > > > > > > >> > > > > > >> > I also think we don't need to keep > > > delJIanpolicy > > > > in > > > > > > > final > > > > > > > > >> > > > config，if > > > > > > > > >> > > > > > >> so，we > > > > > > > > >> > > > > > >> > should always keep remembering all of our > > topic > > > > > > > retention > > > > > > > > >> time > > > > > > > > >> > > > must > > > > > > > > >> > > > > > less > > > > > > > > >> > > > > > >> > than the value，right？It is one protect with > > > risk > > > > > > > > involved. > > > > > > > > >> > > > > > >> > Regards > > > > > > > > >> > > > > > >> > JIan > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > Lijun Tong <[email protected] > > > >于2026年1月16日 > > > > > > > > 周五06:45写道： > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > Hey Kamal, > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > Some additional points about the Q4, > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > The user can decide when to change > their > > > > > internal > > > > > > > > topic > > > > > > > > >> > > > cleanup > > > > > > > > >> > > > > > >> policy > > > > > > > > >> > > > > > >> > to > > > > > > > > >> > > > > > >> > > > compact. If someone retains > > > > > > > > >> > > > > > >> > > > the data in the remote storage for 3 > > > months, > > > > > then > > > > > > > > they > > > > > > > > >> can > > > > > > > > >> > > > > migrate > > > > > > > > >> > > > > > >> to > > > > > > > > >> > > > > > >> > the > > > > > > > > >> > > > > > >> > > > compacted topic after 3 months > > > > > > > > >> > > > > > >> > > > post rolling out this change. And, > update > > > > their > > > > > > > > cleanup > > > > > > > > >> > > policy > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > >> > > [compact, > > > > > > > > >> > > > > > >> > > > delete]. > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > I don't think it's a good idea to keep > > delete > > > > in > > > > > > the > > > > > > > > >> final > > > > > > > > >> > > > cleanup > > > > > > > > >> > > > > > >> policy > > > > > > > > >> > > > > > >> > > for the topic `__remote_log_metadata`, as > > > this > > > > > > still > > > > > > > > >> > requires > > > > > > > > >> > > > the > > > > > > > > >> > > > > > >> user to > > > > > > > > >> > > > > > >> > > keep track of the max retention hours of > > > topics > > > > > > that > > > > > > > > have > > > > > > > > >> > > remote > > > > > > > > >> > > > > > >> storage > > > > > > > > >> > > > > > >> > > enabled, and it's operational burden. > It's > > > also > > > > > > hard > > > > > > > to > > > > > > > > >> > reason > > > > > > > > >> > > > > about > > > > > > > > >> > > > > > >> what > > > > > > > > >> > > > > > >> > > will happen if the user configures the > > wrong > > > > > > > > >> retention.ms. > > > > > > > > >> > I > > > > > > > > >> > > > hope > > > > > > > > >> > > > > > >> this > > > > > > > > >> > > > > > >> > > makes sense. > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > Thanks, > > > > > > > > >> > > > > > >> > > Lijun Tong > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > Lijun Tong <[email protected]> > > > > > 于2026年1月15日周四 > > > > > > > > >> 11:43写道： > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > Hey Kamal, > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > Thanks for your reply! I am glad we are > > on > > > > the > > > > > > same > > > > > > > > >> page > > > > > > > > >> > > with > > > > > > > > >> > > > > > making > > > > > > > > >> > > > > > >> > the > > > > > > > > >> > > > > > >> > > > __remote_log_metadata topic compacted > > > > optional > > > > > > for > > > > > > > > the > > > > > > > > >> > user > > > > > > > > >> > > > > now, I > > > > > > > > >> > > > > > >> will > > > > > > > > >> > > > > > >> > > > update the KIP with this change. > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > For the Q2: > > > > > > > > >> > > > > > >> > > > With the key designed as > > > > > > > > >> > > > > > >> > TopicId:Partition:EndOffset:BrokerLeaderEpoch, > > > > > > > > >> > > > > > >> > > > even the same broker retries the upload > > > > > multiple > > > > > > > > times > > > > > > > > >> for > > > > > > > > >> > > the > > > > > > > > >> > > > > > same > > > > > > > > >> > > > > > >> log > > > > > > > > >> > > > > > >> > > > segment, the latest retry attempt with > > the > > > > > latest > > > > > > > > >> segment > > > > > > > > >> > > UUID > > > > > > > > >> > > > > > will > > > > > > > > >> > > > > > >> > > > overwrite the previous attempts' value > > > since > > > > > they > > > > > > > > share > > > > > > > > >> > the > > > > > > > > >> > > > same > > > > > > > > >> > > > > > >> key, > > > > > > > > >> > > > > > >> > so > > > > > > > > >> > > > > > >> > > we > > > > > > > > >> > > > > > >> > > > don't need to explicitly track the > failed > > > > > upload > > > > > > > > >> metadata, > > > > > > > > >> > > > > because > > > > > > > > >> > > > > > >> it's > > > > > > > > >> > > > > > >> > > > gone already by the later attempt. > That's > > > my > > > > > > > > >> understanding > > > > > > > > >> > > > about > > > > > > > > >> > > > > > the > > > > > > > > >> > > > > > >> > > > RLMCopyTask, correct me if I am wrong. > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > Thanks, > > > > > > > > >> > > > > > >> > > > Lijun Tong > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > Kamal Chandraprakash < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > >> > > > > > 于2026年1月14日周三 > > > > > > > > >> > > > > > >> > > > 21:18写道： > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > >> Hi Lijun, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> Thanks for the reply! > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> Q1: Sounds good. Could you clarify it > in > > > the > > > > > KIP > > > > > > > > that > > > > > > > > >> the > > > > > > > > >> > > > same > > > > > > > > >> > > > > > >> > > partitioner > > > > > > > > >> > > > > > >> > > >> will be used? > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> Q2: With > > > > > > > > TopicId:Partition:EndOffset:BrokerLeaderEpoch > > > > > > > > >> > key, > > > > > > > > >> > > > if > > > > > > > > >> > > > > > the > > > > > > > > >> > > > > > >> > same > > > > > > > > >> > > > > > >> > > >> broker retries the upload due to > > > > intermittent > > > > > > > > >> > > > > > >> > > >> issues in object storage (or) RLMM. > > Then, > > > > > those > > > > > > > > failed > > > > > > > > >> > > upload > > > > > > > > >> > > > > > >> metadata > > > > > > > > >> > > > > > >> > > >> also > > > > > > > > >> > > > > > >> > > >> need to be cleared. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> Q3: We may have to skip the null value > > > > records > > > > > > in > > > > > > > > the > > > > > > > > >> > > > > > ConsumerTask. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> Q4a: The idea is to keep the cleanup > > > policy > > > > as > > > > > > > > >> "delete" > > > > > > > > >> > and > > > > > > > > >> > > > > also > > > > > > > > >> > > > > > >> send > > > > > > > > >> > > > > > >> > > the > > > > > > > > >> > > > > > >> > > >> tombstone markers > > > > > > > > >> > > > > > >> > > >> to the existing > `__remote_log_metadata` > > > > topic. > > > > > > > And, > > > > > > > > >> > handle > > > > > > > > >> > > > the > > > > > > > > >> > > > > > >> > tombstone > > > > > > > > >> > > > > > >> > > >> records in the ConsumerTask. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> The user can decide when to change > their > > > > > > internal > > > > > > > > >> topic > > > > > > > > >> > > > cleanup > > > > > > > > >> > > > > > >> policy > > > > > > > > >> > > > > > >> > > to > > > > > > > > >> > > > > > >> > > >> compact. If someone retains > > > > > > > > >> > > > > > >> > > >> the data in the remote storage for 3 > > > months, > > > > > > then > > > > > > > > they > > > > > > > > >> > can > > > > > > > > >> > > > > > migrate > > > > > > > > >> > > > > > >> to > > > > > > > > >> > > > > > >> > > the > > > > > > > > >> > > > > > >> > > >> compacted topic after 3 months > > > > > > > > >> > > > > > >> > > >> post rolling out this change. And, > > update > > > > > their > > > > > > > > >> cleanup > > > > > > > > >> > > > policy > > > > > > > > >> > > > > to > > > > > > > > >> > > > > > >> > > >> [compact, > > > > > > > > >> > > > > > >> > > >> delete]. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> Thanks, > > > > > > > > >> > > > > > >> > > >> Kamal > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> On Thu, Jan 15, 2026 at 4:12 AM Lijun > > > Tong < > > > > > > > > >> > > > > > >> [email protected]> > > > > > > > > >> > > > > > >> > > >> wrote: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > >> > Hey Jian, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > Thanks for your time to review this > > > KIP. I > > > > > > > > >> appreciate > > > > > > > > >> > > that > > > > > > > > >> > > > > you > > > > > > > > >> > > > > > >> > > propose a > > > > > > > > >> > > > > > >> > > >> > simpler migration solution to > onboard > > > the > > > > > new > > > > > > > > >> feature. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > There are 2 points that I think can > be > > > > > further > > > > > > > > >> refined > > > > > > > > >> > > on: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > 1). make the topic compacted > optional, > > > > > > although > > > > > > > > the > > > > > > > > >> new > > > > > > > > >> > > > > feature > > > > > > > > >> > > > > > >> will > > > > > > > > >> > > > > > >> > > >> > continue to emit tombstone message > for > > > > those > > > > > > > > expired > > > > > > > > >> > log > > > > > > > > >> > > > > > segments > > > > > > > > >> > > > > > >> > even > > > > > > > > >> > > > > > >> > > >> when > > > > > > > > >> > > > > > >> > > >> > the topic is still on time-based > > > retention > > > > > > mode, > > > > > > > > so > > > > > > > > >> > once > > > > > > > > >> > > > user > > > > > > > > >> > > > > > >> > switched > > > > > > > > >> > > > > > >> > > >> to > > > > > > > > >> > > > > > >> > > >> > using the compacted topic, those > > expired > > > > > > > messages > > > > > > > > >> can > > > > > > > > >> > > still > > > > > > > > >> > > > > be > > > > > > > > >> > > > > > >> > deleted > > > > > > > > >> > > > > > >> > > >> > despite the topic is not retention > > based > > > > > > > anymore. > > > > > > > > >> > > > > > >> > > >> > 2). we need to expose some flag to > the > > > > user > > > > > to > > > > > > > > >> indicate > > > > > > > > >> > > > > whether > > > > > > > > >> > > > > > >> the > > > > > > > > >> > > > > > >> > > >> topic > > > > > > > > >> > > > > > >> > > >> > can be flipped to compacted by > > checking > > > > > > whether > > > > > > > > all > > > > > > > > >> the > > > > > > > > >> > > old > > > > > > > > >> > > > > > >> format > > > > > > > > >> > > > > > >> > > >> > keyed-less message has expired, and > > > allow > > > > > user > > > > > > > to > > > > > > > > >> > choose > > > > > > > > >> > > to > > > > > > > > >> > > > > > flip > > > > > > > > >> > > > > > >> to > > > > > > > > >> > > > > > >> > > >> > compacted only when the flag is > true. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > Thanks for sharing your idea. I will > > > > update > > > > > > the > > > > > > > > KIP > > > > > > > > >> > later > > > > > > > > >> > > > > with > > > > > > > > >> > > > > > >> this > > > > > > > > >> > > > > > >> > > new > > > > > > > > >> > > > > > >> > > >> > idea. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > Best, > > > > > > > > >> > > > > > >> > > >> > Lijun Tong > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > jian fu <[email protected]> > > > > > 于2026年1月12日周一 > > > > > > > > >> 04:55写道： > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > Hi Lijun Tong: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > Thanks for your KIP which raise > this > > > > > > critical > > > > > > > > >> issue. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > what about just keep one topic > > instead > > > > of > > > > > > > > involve > > > > > > > > >> > > another > > > > > > > > >> > > > > > >> topic. > > > > > > > > >> > > > > > >> > > >> > > for existed topic data's > migration. > > > > maybe > > > > > we > > > > > > > can > > > > > > > > >> use > > > > > > > > >> > > this > > > > > > > > >> > > > > way > > > > > > > > >> > > > > > >> to > > > > > > > > >> > > > > > >> > > solve > > > > > > > > >> > > > > > >> > > >> > the > > > > > > > > >> > > > > > >> > > >> > > issue: > > > > > > > > >> > > > > > >> > > >> > > (1) set the retention date > all > of > > > > topic > > > > > > > which > > > > > > > > >> > enable > > > > > > > > >> > > > > remote > > > > > > > > >> > > > > > >> > > >> storage's > > > > > > > > >> > > > > > >> > > >> > > retention time > > > > > > > > >> > > > > > >> > > >> > > (2) deploy new kafka version with > > > > feature: > > > > > > > > which > > > > > > > > >> > send > > > > > > > > >> > > > the > > > > > > > > >> > > > > > >> message > > > > > > > > >> > > > > > >> > > >> with > > > > > > > > >> > > > > > >> > > >> > key > > > > > > > > >> > > > > > >> > > >> > > (3) wait all the message expired > and > > > new > > > > > > > message > > > > > > > > >> with > > > > > > > > >> > > key > > > > > > > > >> > > > > > >> coming > > > > > > > > >> > > > > > >> > to > > > > > > > > >> > > > > > >> > > >> the > > > > > > > > >> > > > > > >> > > >> > > topic > > > > > > > > >> > > > > > >> > > >> > > (4) convert the topic to compact > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > I don't test it. Just propose this > > > > > solution > > > > > > > > >> according > > > > > > > > >> > > to > > > > > > > > >> > > > > code > > > > > > > > >> > > > > > >> > review > > > > > > > > >> > > > > > >> > > >> > > result. just for your reference. > > > > > > > > >> > > > > > >> > > >> > > The steps maybe a little complex. > > but > > > it > > > > > can > > > > > > > > >> avoiding > > > > > > > > >> > > add > > > > > > > > >> > > > > new > > > > > > > > >> > > > > > >> > topic. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > Regards > > > > > > > > >> > > > > > >> > > >> > > Jian > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > Lijun Tong < > [email protected] > > > > > > > > > > > 于2026年1月8日周四 > > > > > > > > >> > > > 09:17写道： > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > Hey Kamal, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Thanks for your time for the > > review. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Here is my response to your > > > questions: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Q1 At this point, I don’t see a > > need > > > > to > > > > > > > change > > > > > > > > >> > > > > > >> > > >> > > > > RemoteLogMetadataTopicPartitioner > > > for > > > > > this > > > > > > > > >> design. > > > > > > > > >> > > > > Nothing > > > > > > > > >> > > > > > in > > > > > > > > >> > > > > > >> > the > > > > > > > > >> > > > > > >> > > >> > current > > > > > > > > >> > > > > > >> > > >> > > > approach appears to require a > > > > > partitioner > > > > > > > > >> change, > > > > > > > > >> > but > > > > > > > > >> > > > I’m > > > > > > > > >> > > > > > >> open > > > > > > > > >> > > > > > >> > to > > > > > > > > >> > > > > > >> > > >> > > > revisiting if a concrete need > > > arises. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Q2 I have some reservations > about > > > > using > > > > > > > > >> > > SegmentId:State > > > > > > > > >> > > > > as > > > > > > > > >> > > > > > >> the > > > > > > > > >> > > > > > >> > > key. > > > > > > > > >> > > > > > >> > > >> A > > > > > > > > >> > > > > > >> > > >> > > > practical challenge we see today > > is > > > > that > > > > > > the > > > > > > > > >> same > > > > > > > > >> > > > logical > > > > > > > > >> > > > > > >> > segment > > > > > > > > >> > > > > > >> > > >> can > > > > > > > > >> > > > > > >> > > >> > be > > > > > > > > >> > > > > > >> > > >> > > > retried multiple times with > > > different > > > > > > > > SegmentIds > > > > > > > > >> > > across > > > > > > > > >> > > > > > >> brokers. > > > > > > > > >> > > > > > >> > > If > > > > > > > > >> > > > > > >> > > >> the > > > > > > > > >> > > > > > >> > > >> > > key > > > > > > > > >> > > > > > >> > > >> > > > is SegmentId-based, it becomes > > > harder > > > > to > > > > > > > > >> discover > > > > > > > > >> > and > > > > > > > > >> > > > > > >> tombstone > > > > > > > > >> > > > > > >> > > all > > > > > > > > >> > > > > > >> > > >> > > related > > > > > > > > >> > > > > > >> > > >> > > > attempts when the segment > > eventually > > > > > > > expires. > > > > > > > > >> The > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > TopicId:Partition:EndOffset:BrokerLeaderEpoch > > > > > > > > >> key > > > > > > > > >> > is > > > > > > > > >> > > > > > >> > deterministic > > > > > > > > >> > > > > > >> > > >> for > > > > > > > > >> > > > > > >> > > >> > a > > > > > > > > >> > > > > > >> > > >> > > > logical segment attempt and > helps > > > > group > > > > > > > > retries > > > > > > > > >> by > > > > > > > > >> > > > epoch, > > > > > > > > >> > > > > > >> which > > > > > > > > >> > > > > > >> > > >> > > simplifies > > > > > > > > >> > > > > > >> > > >> > > > cleanup and reasoning about > state. > > > I’d > > > > > > love > > > > > > > to > > > > > > > > >> > > > understand > > > > > > > > >> > > > > > the > > > > > > > > >> > > > > > >> > > >> benefits > > > > > > > > >> > > > > > >> > > >> > > > you’re seeing with > SegmentId:State > > > > > > compared > > > > > > > to > > > > > > > > >> the > > > > > > > > >> > > > > > >> > > >> offset/epoch-based > > > > > > > > >> > > > > > >> > > >> > key > > > > > > > > >> > > > > > >> > > >> > > > so we can weigh the trade-offs. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > On partitioning: with this > > proposal, > > > > all > > > > > > > > states > > > > > > > > >> > for a > > > > > > > > >> > > > > given > > > > > > > > >> > > > > > >> user > > > > > > > > >> > > > > > >> > > >> > > > topic-partition still map to the > > > same > > > > > > > metadata > > > > > > > > >> > > > partition. > > > > > > > > >> > > > > > >> That > > > > > > > > >> > > > > > >> > > >> remains > > > > > > > > >> > > > > > >> > > >> > > true > > > > > > > > >> > > > > > >> > > >> > > > for the existing > > > __remote_log_metadata > > > > > > > > >> (unchanged > > > > > > > > >> > > > > > >> partitioner) > > > > > > > > >> > > > > > >> > and > > > > > > > > >> > > > > > >> > > >> for > > > > > > > > >> > > > > > >> > > >> > > the > > > > > > > > >> > > > > > >> > > >> > > > new > > __remote_log_metadata_compacted, > > > > > > > > preserving > > > > > > > > >> the > > > > > > > > >> > > > > > >> properties > > > > > > > > >> > > > > > >> > > >> > > > RemoteMetadataCache relies on. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Q3 It should be fine for > > > ConsumerTask > > > > to > > > > > > > > ignore > > > > > > > > >> > > > tombstone > > > > > > > > >> > > > > > >> > records > > > > > > > > >> > > > > > >> > > >> (null > > > > > > > > >> > > > > > >> > > >> > > > values) and no-op. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Q4 Although TBRLMM is a sample > > RLMM > > > > > > > > >> implementation, > > > > > > > > >> > > > it’s > > > > > > > > >> > > > > > >> > currently > > > > > > > > >> > > > > > >> > > >> the > > > > > > > > >> > > > > > >> > > >> > > only > > > > > > > > >> > > > > > >> > > >> > > > OSS option and is widely used. > The > > > new > > > > > > > > >> > > > > > >> > > >> __remote_log_metadata_compacted > > > > > > > > >> > > > > > >> > > >> > > > topic offers clear operational > > > > benefits > > > > > in > > > > > > > > that > > > > > > > > >> > > > context. > > > > > > > > >> > > > > We > > > > > > > > >> > > > > > >> can > > > > > > > > >> > > > > > >> > > also > > > > > > > > >> > > > > > >> > > >> > > > provide a configuration to let > > users > > > > > > choose > > > > > > > > >> whether > > > > > > > > >> > > > they > > > > > > > > >> > > > > > >> want to > > > > > > > > >> > > > > > >> > > >> keep > > > > > > > > >> > > > > > >> > > >> > the > > > > > > > > >> > > > > > >> > > >> > > > audit topic > > (__remote_log_metadata) > > > in > > > > > > their > > > > > > > > >> > cluster. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Q4a Enabling compaction on > > > > > > > > __remote_log_metadata > > > > > > > > >> > > alone > > > > > > > > >> > > > > may > > > > > > > > >> > > > > > >> not > > > > > > > > >> > > > > > >> > > fully > > > > > > > > >> > > > > > >> > > >> > > > address the unbounded growth, > > since > > > we > > > > > > also > > > > > > > > >> need to > > > > > > > > >> > > > emit > > > > > > > > >> > > > > > >> > > tombstones > > > > > > > > >> > > > > > >> > > >> for > > > > > > > > >> > > > > > >> > > >> > > > expired keys to delete them. > > > Deferring > > > > > > > > >> compaction > > > > > > > > >> > and > > > > > > > > >> > > > > > >> > tombstoning > > > > > > > > >> > > > > > >> > > to > > > > > > > > >> > > > > > >> > > >> > user > > > > > > > > >> > > > > > >> > > >> > > > configuration could make the > code > > > flow > > > > > > > > >> complicated, > > > > > > > > >> > > > also > > > > > > > > >> > > > > > add > > > > > > > > >> > > > > > >> > > >> > operational > > > > > > > > >> > > > > > >> > > >> > > > complexity and make outcomes > less > > > > > > > predictable. > > > > > > > > >> The > > > > > > > > >> > > > > proposal > > > > > > > > >> > > > > > >> aims > > > > > > > > >> > > > > > >> > > to > > > > > > > > >> > > > > > >> > > >> > > provide > > > > > > > > >> > > > > > >> > > >> > > > a consistent experience by > > defining > > > > > > > > >> deterministic > > > > > > > > >> > > keys > > > > > > > > >> > > > > and > > > > > > > > >> > > > > > >> > > emitting > > > > > > > > >> > > > > > >> > > >> > > > tombstones as part of the > broker’s > > > > > > > > >> > responsibilities, > > > > > > > > >> > > > > while > > > > > > > > >> > > > > > >> still > > > > > > > > >> > > > > > >> > > >> > allowing > > > > > > > > >> > > > > > >> > > >> > > > users to opt out of the audit > > topic > > > if > > > > > > they > > > > > > > > >> prefer. > > > > > > > > >> > > > But I > > > > > > > > >> > > > > > am > > > > > > > > >> > > > > > >> > open > > > > > > > > >> > > > > > >> > > to > > > > > > > > >> > > > > > >> > > >> > more > > > > > > > > >> > > > > > >> > > >> > > > discussion if there is any > > concrete > > > > > need I > > > > > > > > don't > > > > > > > > >> > > > foresee. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Thanks, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Lijun Tong > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > Kamal Chandraprakash < > > > > > > > > >> > [email protected] > > > > > > > > >> > > > > > > > > > > > >> > > > > > >> > > 于2026年1月6日周二 > > > > > > > > >> > > > > > >> > > >> > > > 01:01写道： > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > Hi Lijun, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > Thanks for the KIP! Went over > > the > > > > > first > > > > > > > > pass. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > Few Questions: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > 1. Are we going to maintain > the > > > same > > > > > > > > >> > > > > > >> > > >> > RemoteLogMetadataTopicPartitioner > > > > > > > > >> > > > > > >> > > >> > > > > < > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogMetadataTopicPartitioner.java > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > for both the topics? It is not > > > clear > > > > > in > > > > > > > the > > > > > > > > >> KIP, > > > > > > > > >> > > > could > > > > > > > > >> > > > > > you > > > > > > > > >> > > > > > >> > > clarify > > > > > > > > >> > > > > > >> > > >> > it? > > > > > > > > >> > > > > > >> > > >> > > > > 2. Can the key be changed to > > > > > > > SegmentId:State > > > > > > > > >> > > instead > > > > > > > > >> > > > of > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > TopicId:Partition:EndOffset:BrokerLeaderEpoch > > > > > > > > >> if > > > > > > > > >> > > the > > > > > > > > >> > > > > same > > > > > > > > >> > > > > > >> > > >> partitioner > > > > > > > > >> > > > > > >> > > >> > > is > > > > > > > > >> > > > > > >> > > >> > > > > used? It is good to maintain > all > > > the > > > > > > > segment > > > > > > > > >> > states > > > > > > > > >> > > > > for a > > > > > > > > >> > > > > > >> > > >> > > > > user-topic-partition in the > same > > > > > > metadata > > > > > > > > >> > > partition. > > > > > > > > >> > > > > > >> > > >> > > > > 3. Should we have to handle > the > > > > > records > > > > > > > with > > > > > > > > >> null > > > > > > > > >> > > > value > > > > > > > > >> > > > > > >> > > >> (tombstone) > > > > > > > > >> > > > > > >> > > >> > in > > > > > > > > >> > > > > > >> > > >> > > > the > > > > > > > > >> > > > > > >> > > >> > > > > ConsumerTask > > > > > > > > >> > > > > > >> > > >> > > > > < > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/ConsumerTask.java?L166 > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > ? > > > > > > > > >> > > > > > >> > > >> > > > > 4. TBRLMM > > > > > > > > >> > > > > > >> > > >> > > > > < > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://sourcegraph.com/github.com/apache/kafka/-/blob/storage/src/main/java/org/apache/kafka/server/log/remote/metadata/storage/TopicBasedRemoteLogMetadataManager.java > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > is a sample plugin > > implementation > > > of > > > > > > RLMM. > > > > > > > > Not > > > > > > > > >> > sure > > > > > > > > >> > > > > > whether > > > > > > > > >> > > > > > >> > the > > > > > > > > >> > > > > > >> > > >> > > community > > > > > > > > >> > > > > > >> > > >> > > > > will agree to add one more > > > internal > > > > > > topic > > > > > > > > for > > > > > > > > >> > this > > > > > > > > >> > > > > plugin > > > > > > > > >> > > > > > >> > impl. > > > > > > > > >> > > > > > >> > > >> > > > > 4a. Can we modify the new > > messages > > > > to > > > > > > the > > > > > > > > >> > > > > > >> > __remote_log_metadata > > > > > > > > >> > > > > > >> > > >> topic > > > > > > > > >> > > > > > >> > > >> > > to > > > > > > > > >> > > > > > >> > > >> > > > > contain the key and leave it > to > > > the > > > > > user > > > > > > > to > > > > > > > > >> > enable > > > > > > > > >> > > > > > >> compaction > > > > > > > > >> > > > > > >> > > for > > > > > > > > >> > > > > > >> > > >> > this > > > > > > > > >> > > > > > >> > > >> > > > > topic if they need? > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > Thanks, > > > > > > > > >> > > > > > >> > > >> > > > > Kamal > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > On Tue, Jan 6, 2026 at 7:35 AM > > > Lijun > > > > > > Tong > > > > > > > < > > > > > > > > >> > > > > > >> > > >> [email protected]> > > > > > > > > >> > > > > > >> > > >> > > > wrote: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Hey Henry, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Thank you for your time and > > > > > response! > > > > > > I > > > > > > > > >> really > > > > > > > > >> > > like > > > > > > > > >> > > > > > your > > > > > > > > >> > > > > > >> > > >> KIP-1248 > > > > > > > > >> > > > > > >> > > >> > > about > > > > > > > > >> > > > > > >> > > >> > > > > > offloading the consumption > of > > > > remote > > > > > > log > > > > > > > > >> away > > > > > > > > >> > > from > > > > > > > > >> > > > > the > > > > > > > > >> > > > > > >> > broker, > > > > > > > > >> > > > > > >> > > >> and > > > > > > > > >> > > > > > >> > > >> > I > > > > > > > > >> > > > > > >> > > >> > > > > think > > > > > > > > >> > > > > > >> > > >> > > > > > with that change, the topic > > that > > > > > > enables > > > > > > > > the > > > > > > > > >> > > tiered > > > > > > > > >> > > > > > >> storage > > > > > > > > >> > > > > > >> > > can > > > > > > > > >> > > > > > >> > > >> > also > > > > > > > > >> > > > > > >> > > >> > > > have > > > > > > > > >> > > > > > >> > > >> > > > > > longer retention > > configurations > > > > and > > > > > > > would > > > > > > > > >> > benefit > > > > > > > > >> > > > > from > > > > > > > > >> > > > > > >> this > > > > > > > > >> > > > > > >> > > KIP > > > > > > > > >> > > > > > >> > > >> > too. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Some suggestions: In your > > > example > > > > > > > > >> scenarios, it > > > > > > > > >> > > > would > > > > > > > > >> > > > > > >> also > > > > > > > > >> > > > > > >> > be > > > > > > > > >> > > > > > >> > > >> good > > > > > > > > >> > > > > > >> > > >> > to > > > > > > > > >> > > > > > >> > > >> > > > add > > > > > > > > >> > > > > > >> > > >> > > > > > > an example of remote log > > > segment > > > > > > > > deletion > > > > > > > > >> > > > triggered > > > > > > > > >> > > > > > by > > > > > > > > >> > > > > > >> > > >> retention > > > > > > > > >> > > > > > >> > > >> > > > policy > > > > > > > > >> > > > > > >> > > >> > > > > > > which will trigger > > generation > > > of > > > > > > > > tombstone > > > > > > > > >> > > event > > > > > > > > >> > > > > into > > > > > > > > >> > > > > > >> > > metadata > > > > > > > > >> > > > > > >> > > >> > > topic > > > > > > > > >> > > > > > >> > > >> > > > > and > > > > > > > > >> > > > > > >> > > >> > > > > > > trigger log > > > compaction/deletion > > > > 24 > > > > > > > hour > > > > > > > > >> > later, > > > > > > > > >> > > I > > > > > > > > >> > > > > > think > > > > > > > > >> > > > > > >> > this > > > > > > > > >> > > > > > >> > > is > > > > > > > > >> > > > > > >> > > >> > the > > > > > > > > >> > > > > > >> > > >> > > > key > > > > > > > > >> > > > > > >> > > >> > > > > > > event to cap the metadata > > > topic > > > > > > size. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Regarding to this > suggestion, > > I > > > am > > > > > not > > > > > > > > sure > > > > > > > > >> > > whether > > > > > > > > >> > > > > > >> > Scenario 4 > > > > > > > > >> > > > > > >> > > >> > > > > > < > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=406618613#KIP1266:BoundingTheNumberOfRemoteLogMetadataMessagesviaCompactedTopic-Scenario4:SegmentDeletion > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > has > > > > > > > > >> > > > > > >> > > >> > > > > > covered it. I can add more > > rows > > > in > > > > > the > > > > > > > > >> Timeline > > > > > > > > >> > > > Table > > > > > > > > >> > > > > > >> like > > > > > > > > >> > > > > > >> > > >> > T5+24hour > > > > > > > > >> > > > > > >> > > >> > > to > > > > > > > > >> > > > > > >> > > >> > > > > > indicate the messages are > gone > > > by > > > > > then > > > > > > > to > > > > > > > > >> > > > explicitly > > > > > > > > >> > > > > > show > > > > > > > > >> > > > > > >> > that > > > > > > > > >> > > > > > >> > > >> > > messages > > > > > > > > >> > > > > > >> > > >> > > > > are > > > > > > > > >> > > > > > >> > > >> > > > > > deleted, thus the number of > > > > messages > > > > > > are > > > > > > > > >> capped > > > > > > > > >> > > in > > > > > > > > >> > > > > the > > > > > > > > >> > > > > > >> > topic. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Regarding whether the topic > > > > > > > > >> > __remote_log_metadata > > > > > > > > >> > > > is > > > > > > > > >> > > > > > >> still > > > > > > > > >> > > > > > >> > > >> > > necessary, I > > > > > > > > >> > > > > > >> > > >> > > > > am > > > > > > > > >> > > > > > >> > > >> > > > > > inclined to continue to have > > > this > > > > > > topic > > > > > > > at > > > > > > > > >> > least > > > > > > > > >> > > > for > > > > > > > > >> > > > > > >> > debugging > > > > > > > > >> > > > > > >> > > >> > > purposes > > > > > > > > >> > > > > > >> > > >> > > > > so > > > > > > > > >> > > > > > >> > > >> > > > > > we can build confidence > about > > > the > > > > > > > > compacted > > > > > > > > >> > topic > > > > > > > > >> > > > > > >> change, we > > > > > > > > >> > > > > > >> > > can > > > > > > > > >> > > > > > >> > > >> > > > > > always choose to remove this > > > topic > > > > > in > > > > > > > the > > > > > > > > >> > future > > > > > > > > >> > > > once > > > > > > > > >> > > > > > we > > > > > > > > >> > > > > > >> all > > > > > > > > >> > > > > > >> > > >> agree > > > > > > > > >> > > > > > >> > > >> > it > > > > > > > > >> > > > > > >> > > >> > > > > > provides limited value for > the > > > > > users. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Thanks, > > > > > > > > >> > > > > > >> > > >> > > > > > Lijun Tong > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > Henry Haiying Cai via dev < > > > > > > > > >> > [email protected]> > > > > > > > > >> > > > > > >> > 于2026年1月5日周一 > > > > > > > > >> > > > > > >> > > >> > > 16:19写道： > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Lijun, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Thanks for the proposal > and > > I > > > > > liked > > > > > > > your > > > > > > > > >> idea > > > > > > > > >> > > of > > > > > > > > >> > > > > > using > > > > > > > > >> > > > > > >> a > > > > > > > > >> > > > > > >> > > >> > compacted > > > > > > > > >> > > > > > >> > > >> > > > > topic > > > > > > > > >> > > > > > >> > > >> > > > > > > for tiered storage > metadata > > > > topic. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > In our setup, we have set > a > > > > > shorter > > > > > > > > >> retention > > > > > > > > >> > > (3 > > > > > > > > >> > > > > > days) > > > > > > > > >> > > > > > >> for > > > > > > > > >> > > > > > >> > > the > > > > > > > > >> > > > > > >> > > >> > > tiered > > > > > > > > >> > > > > > >> > > >> > > > > > > storage metadata topic to > > > > control > > > > > > the > > > > > > > > size > > > > > > > > >> > > > growth. > > > > > > > > >> > > > > > We > > > > > > > > >> > > > > > >> can > > > > > > > > >> > > > > > >> > > do > > > > > > > > >> > > > > > >> > > >> > that > > > > > > > > >> > > > > > >> > > >> > > > > since > > > > > > > > >> > > > > > >> > > >> > > > > > we > > > > > > > > >> > > > > > >> > > >> > > > > > > control all topic's > > retention > > > > > policy > > > > > > > in > > > > > > > > >> our > > > > > > > > >> > > > > clusters > > > > > > > > >> > > > > > >> and > > > > > > > > >> > > > > > >> > we > > > > > > > > >> > > > > > >> > > >> set a > > > > > > > > >> > > > > > >> > > >> > > > > uniform > > > > > > > > >> > > > > > >> > > >> > > > > > > retention.policy for all > our > > > > > tiered > > > > > > > > >> storage > > > > > > > > >> > > > topics. > > > > > > > > >> > > > > > I > > > > > > > > >> > > > > > >> can > > > > > > > > >> > > > > > >> > > see > > > > > > > > >> > > > > > >> > > >> > > other > > > > > > > > >> > > > > > >> > > >> > > > > > > users/companies will not > be > > > able > > > > > to > > > > > > > > >> enforce > > > > > > > > >> > > that > > > > > > > > >> > > > > > >> retention > > > > > > > > >> > > > > > >> > > >> policy > > > > > > > > >> > > > > > >> > > >> > > to > > > > > > > > >> > > > > > >> > > >> > > > > all > > > > > > > > >> > > > > > >> > > >> > > > > > > tiered storage topics. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Some suggestions: In your > > > > example > > > > > > > > >> scenarios, > > > > > > > > >> > it > > > > > > > > >> > > > > would > > > > > > > > >> > > > > > >> also > > > > > > > > >> > > > > > >> > > be > > > > > > > > >> > > > > > >> > > >> > good > > > > > > > > >> > > > > > >> > > >> > > to > > > > > > > > >> > > > > > >> > > >> > > > > add > > > > > > > > >> > > > > > >> > > >> > > > > > > an example of remote log > > > segment > > > > > > > > deletion > > > > > > > > >> > > > triggered > > > > > > > > >> > > > > > by > > > > > > > > >> > > > > > >> > > >> retention > > > > > > > > >> > > > > > >> > > >> > > > policy > > > > > > > > >> > > > > > >> > > >> > > > > > > which will trigger > > generation > > > of > > > > > > > > tombstone > > > > > > > > >> > > event > > > > > > > > >> > > > > into > > > > > > > > >> > > > > > >> > > metadata > > > > > > > > >> > > > > > >> > > >> > > topic > > > > > > > > >> > > > > > >> > > >> > > > > and > > > > > > > > >> > > > > > >> > > >> > > > > > > trigger log > > > compaction/deletion > > > > 24 > > > > > > > hour > > > > > > > > >> > later, > > > > > > > > >> > > I > > > > > > > > >> > > > > > think > > > > > > > > >> > > > > > >> > this > > > > > > > > >> > > > > > >> > > is > > > > > > > > >> > > > > > >> > > >> > the > > > > > > > > >> > > > > > >> > > >> > > > key > > > > > > > > >> > > > > > >> > > >> > > > > > > event to cap the metadata > > > topic > > > > > > size. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > For the original unbounded > > > > > > > > >> > remote_log_metadata > > > > > > > > >> > > > > topic, > > > > > > > > >> > > > > > >> I am > > > > > > > > >> > > > > > >> > > not > > > > > > > > >> > > > > > >> > > >> > sure > > > > > > > > >> > > > > > >> > > >> > > > > > > whether we still need it > or > > > not. > > > > > If > > > > > > > it > > > > > > > > is > > > > > > > > >> > left > > > > > > > > >> > > > > only > > > > > > > > >> > > > > > >> for > > > > > > > > >> > > > > > >> > > audit > > > > > > > > >> > > > > > >> > > >> > > trail > > > > > > > > >> > > > > > >> > > >> > > > > > > purpose, people can set > up a > > > > data > > > > > > > > >> ingestion > > > > > > > > >> > > > > pipeline > > > > > > > > >> > > > > > to > > > > > > > > >> > > > > > >> > > ingest > > > > > > > > >> > > > > > >> > > >> > the > > > > > > > > >> > > > > > >> > > >> > > > > > content > > > > > > > > >> > > > > > >> > > >> > > > > > > of metadata topic into a > > > > separate > > > > > > > > storage > > > > > > > > >> > > > location. > > > > > > > > >> > > > > > I > > > > > > > > >> > > > > > >> > think > > > > > > > > >> > > > > > >> > > >> we > > > > > > > > >> > > > > > >> > > >> > can > > > > > > > > >> > > > > > >> > > >> > > > > have > > > > > > > > >> > > > > > >> > > >> > > > > > a > > > > > > > > >> > > > > > >> > > >> > > > > > > flag to have only one > > metadata > > > > > topic > > > > > > > > (the > > > > > > > > >> > > > compacted > > > > > > > > >> > > > > > >> > > version). > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > On Monday, January 5, 2026 > > at > > > > > > 01:22:42 > > > > > > > > PM > > > > > > > > >> > PST, > > > > > > > > >> > > > > Lijun > > > > > > > > >> > > > > > >> Tong > > > > > > > > >> > > > > > >> > < > > > > > > > > >> > > > > > >> > > >> > > > > > > [email protected]> > > > wrote: > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Hello Kafka Community, > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > I would like to start a > > > > discussion > > > > > > on > > > > > > > > >> > KIP-1266, > > > > > > > > >> > > > > which > > > > > > > > >> > > > > > >> > > >> proposes to > > > > > > > > >> > > > > > >> > > >> > > add > > > > > > > > >> > > > > > >> > > >> > > > > > > another new compacted > remote > > > log > > > > > > > > metadata > > > > > > > > >> > topic > > > > > > > > >> > > > for > > > > > > > > >> > > > > > the > > > > > > > > >> > > > > > >> > > tiered > > > > > > > > >> > > > > > >> > > >> > > > storage, > > > > > > > > >> > > > > > >> > > >> > > > > > to > > > > > > > > >> > > > > > >> > > >> > > > > > > limit the number of > messages > > > > that > > > > > > need > > > > > > > > to > > > > > > > > >> be > > > > > > > > >> > > > > iterated > > > > > > > > >> > > > > > >> to > > > > > > > > >> > > > > > >> > > build > > > > > > > > >> > > > > > >> > > >> > the > > > > > > > > >> > > > > > >> > > >> > > > > remote > > > > > > > > >> > > > > > >> > > >> > > > > > > metadata state. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > KIP link: KIP-1266 > Bounding > > > The > > > > > > Number > > > > > > > > Of > > > > > > > > >> > > > > > >> > RemoteLogMetadata > > > > > > > > >> > > > > > >> > > >> > > Messages > > > > > > > > >> > > > > > >> > > >> > > > > via > > > > > > > > >> > > > > > >> > > >> > > > > > > Compacted > RemoteLogMetadata > > > > Topic > > > > > > > > >> > > > > > >> > > >> > > > > > > < > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1266%3A+Bounding+The+Number+Of+RemoteLogMetadata+Messages+via+Compacted+Topic > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Background: > > > > > > > > >> > > > > > >> > > >> > > > > > > The current Tiered Storage > > > > > > > > implementation > > > > > > > > >> > uses > > > > > > > > >> > > a > > > > > > > > >> > > > > > >> > > >> > > > __remote_log_metadata > > > > > > > > >> > > > > > >> > > >> > > > > > > topic with infinite > > retention > > > > and > > > > > > > > >> > delete-based > > > > > > > > >> > > > > > cleanup > > > > > > > > >> > > > > > >> > > policy, > > > > > > > > >> > > > > > >> > > >> > > > causing > > > > > > > > >> > > > > > >> > > >> > > > > > > unbounded growth, slow > > broker > > > > > > > bootstrap, > > > > > > > > >> no > > > > > > > > >> > > > > mechanism > > > > > > > > >> > > > > > >> to > > > > > > > > >> > > > > > >> > > >> clean up > > > > > > > > >> > > > > > >> > > >> > > > > expired > > > > > > > > >> > > > > > >> > > >> > > > > > > segment metadata, and > > > > inefficient > > > > > > > > >> re-reading > > > > > > > > >> > > from > > > > > > > > >> > > > > > >> offset 0 > > > > > > > > >> > > > > > >> > > >> during > > > > > > > > >> > > > > > >> > > >> > > > > > > leadership changes. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Proposal: > > > > > > > > >> > > > > > >> > > >> > > > > > > A dual-topic approach that > > > > > > introduces > > > > > > > a > > > > > > > > >> new > > > > > > > > >> > > > > > >> > > >> > > > > > > > __remote_log_metadata_compacted > > > > > > > > >> > > > > > >> > > >> > > > > > > topic using log compaction > > > with > > > > > > > > >> deterministic > > > > > > > > >> > > > > > >> offset-based > > > > > > > > >> > > > > > >> > > >> keys, > > > > > > > > >> > > > > > >> > > >> > > > while > > > > > > > > >> > > > > > >> > > >> > > > > > > preserving the existing > > topic > > > > for > > > > > > > audit > > > > > > > > >> > > history; > > > > > > > > >> > > > > this > > > > > > > > >> > > > > > >> > allows > > > > > > > > >> > > > > > >> > > >> > > brokers > > > > > > > > >> > > > > > >> > > >> > > > to > > > > > > > > >> > > > > > >> > > >> > > > > > > build their metadata cache > > > > > > exclusively > > > > > > > > >> from > > > > > > > > >> > the > > > > > > > > >> > > > > > >> compacted > > > > > > > > >> > > > > > >> > > >> topic, > > > > > > > > >> > > > > > >> > > >> > > > > enables > > > > > > > > >> > > > > > >> > > >> > > > > > > cleanup of expired segment > > > > > metadata > > > > > > > > >> through > > > > > > > > >> > > > > > tombstones, > > > > > > > > >> > > > > > >> > and > > > > > > > > >> > > > > > >> > > >> > > includes > > > > > > > > >> > > > > > >> > > >> > > > a > > > > > > > > >> > > > > > >> > > >> > > > > > > migration strategy to > > populate > > > > the > > > > > > new > > > > > > > > >> topic > > > > > > > > >> > > > during > > > > > > > > >> > > > > > >> > > >> > > > upgrade—delivering > > > > > > > > >> > > > > > >> > > >> > > > > > > bounded metadata growth > and > > > > faster > > > > > > > > broker > > > > > > > > >> > > startup > > > > > > > > >> > > > > > while > > > > > > > > >> > > > > > >> > > >> > maintaining > > > > > > > > >> > > > > > >> > > >> > > > > > > backward compatibility. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > More details are in the > > > attached > > > > > KIP > > > > > > > > link. > > > > > > > > >> > > > > > >> > > >> > > > > > > Looking forward to your > > > > thoughts. > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Thank you for your time! > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > Best, > > > > > > > > >> > > > > > >> > > >> > > > > > > Lijun Tong > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > > >> > > > > > >> > > >> > > > > > > > > >> > > > > > >> > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > >> > > > > > >> > > > > > > > > > >> > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
