Hey Luke, Thanks for your response, I have answered your questions below.
LC1. In scenario 1, we have 2 records with different keys (one with :UPDATE suffix), how could we achieve the final result after log compaction: Retains only latest → abc123:topicA:0:1000:3 = FINISHED (T1 compacted away) Current log compaction only compares the record key and keeps the latest value for the same key. So in this case, shouldn't we keep these 2 records after log compaction? *Re LC1:* This is correct. My apologies for the inconsistency in the proposal. I have updated the proposal to fix this issue. LC2. What does "Audit topic" mean here? *Re LC2:* Please disregard the reference to the audit topic. I thought I had already refactored the proposal to remove it. For some background, my original intention was to retain the keyless topic as an audit topic. However, after discussing this with others earlier in this email thread, we concluded that an audit topic is unnecessary, so I removed this concept from the design. LC3. If we already store the endOffset and brokerLeaderEpoch in the record key, why should we still need them stored in the record value? *Re LC3:* This is a good question. The reason is related to how RemoteLogManager currently interacts with RemoteMetadataManager. With the current implementation, RemoteLogManager only has access to the metadata value; it does not have the record key available at that point. TopicBasedRemoteLogMetadataManager is only one implementation for the RemoteLogMetadataManager, the key needs to be constructed from the metadata value rather than the other way around. For example, see: https://github.com/apache/kafka/blob/4.3.1/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java#L1053 LC4. About the migration: LC4.1. For "upgrade-to-v1" and "upgrade-to-v2", what's the difference with: bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature remote.log.storage.version=1 *Re LC4.1:* I actually changed the feature name from remote.log.storage.version to remote.log.metadata.version, since I think the latter better reflects what is being versioned. If the user runs: kafka-remote-log-metadata-migration.sh --bootstrap-server localhost:9092 --upgrade-to-v1 the metadata version will be upgraded from 0 to 1. If the user runs the script with --upgrade-to-v2, the metadata version will be upgraded from 1 to 2. The migration script is different from directly changing the feature version with kafka-features.sh because the migration also performs the necessary data migration and validation steps. LC4.2. If the user runs the kafka-features.sh command directly to upgrade from 0 → 2, what will happen? *Re LC4.2:* Currently it will complain about the version jump, but it will still proceed with the change, this means we lost all the keyless messages and data loss. I’m considering adding a safeguard to prevent users from upgrading directly from version 0 to version 2 through kafka-features.sh, since some migration steps are required between these versions should be performed by the migration script. Furthermore, I have documented the migration process in the feature doc <https://github.com/apache/kafka/blob/1ccb860cf83f8d1ba53f3b0706b18f3ca71e5b34/docs/operations/remote-log-metadata-topic-compaction.md> and clearly directed users to use the migration tool for this upgrade. I’m open to suggestions on how we can best ensure that users follow the migration path. LC4.3. What will happen if the setting of retention.ms is too short before the existing consumers complete the fetch? *Re LC4.3:* With the current design, it is already the cluster operator's responsibility to configure retention.ms appropriately. If retention.ms is too short and the metadata for an existing remote log segment is deleted before it is consumed, that metadata is lost permanently, which can result in data loss. This requirement exists independently of this migration and the proposed feature. And this biggest motivation for this proposal is to lift this burden from the kafka operator and after the migration, there is no such risk. The migration does not introduce a new retention requirement. As long as the operator follows the existing requirement and sets retention.ms appropriately, the migration is safe. Once the migration is complete, the keyed metadata log no longer needs to be protected by this additional retention consideration. LC4.4. If the user sets 14 days retention.ms and upgrades to v1, and some more metadata is added. If one month later they are still on v1, will the keyed metadata log be deleted unexpectedly due to log retention? *Re LC4.4:* The keyed metadata log can be truncated according to the configured retention policy while the cluster remains on v1. I don't think this should be considered unexpected, since it is the expected behavior of the retention policy. I also don't think this creates a correctness issue, provided that retention.ms should be configured to be at least as long as the retention period of the longest remote-storage-enabled topic. Under that condition, even if some older keyed metadata records are removed, they will no longer be needed by the remote storage system. LC4.6. "Log cleaner won't compact yet (blocked by min.compaction.lag.ms)", what will happen if a user or operator changes the config to a shorter value? *Re LC4.6:* The migration script will set min.compaction.lag.ms to the same value as retention.ms. Therefore, if the operator changes the configuration before running the migration, the migration script will overwrite that value. However, if the operator changes min.compaction.lag.ms after the migration has started and sets it to a value that is shorter than the required period, metadata associated with active remote log segments could be compacted away prematurely. I will clarify this requirement in the KIP. LC4.7. In "upgrade-to-v2", what does "Tool displays retention reminder and checks if enough time has passed" mean? *Re LC4.7:* When migrating from remote.log.metadata.version=1 to 2, the migration script will scan the topic and verify whether all existing messages have keys. If it finds keyless messages, it will record the latest timestamp among those messages. Based on that timestamp and the topic's retention.ms, the tool can calculate an estimated time when the last keyless message should have been removed by retention. The tool will display this information to the user. The user can then rerun the migration after that estimated time and proceed to v2 once the keyless messages have expired. LC4.8. In "upgrade-to-v2", "Scans topic for null-key messages", does that mean we'll launch a consumer and read the whole log of the __remote_log_metadata topic? If each broker already loads all logs in some __remote_log_metadata partitions, can't we do the scan during loading and directly return the result to the admin client? *Re LC4.8:* This is true: each broker already loads all records from its assigned partitions of __remote_log_metadata during startup. However, the validation required for the migration—checking whether records are keyed and, for keyless records, determining their timestamps—is unrelated to the existing loading logic. I would prefer not to introduce migration-specific logic into the existing metadata loading path and then remove or maintain that logic after the migration is complete. Keeping the migration scan separate makes the existing runtime path simpler and avoids introducing additional complexity into the first version of this feature. LC4.9. Could you show the command result of kafka-remote-log-metadata-migration.sh in the KIP? *Re LC4.9:* Updated. Thanks for the suggestion! LC4.10. Also, could you show the --help result of kafka-remote-log-metadata-migration.sh? *Re LC4.10:* Updated. Thanks for the suggestion! LC4.11. Have you considered how a Kubernetes operator can automate the migration steps? *Re LC4.11:* I’ll need some time to investigate this and understand the implications for Kubernetes operators. I’ll follow up once I’ve had a chance to look into it in more detail. LC4.12. I think the migration plan is the most critical part of this KIP. Users cannot accept any metadata deleted before it is handled completely. We have to make sure all the edge cases are covered before moving this forward. *Re LC4.12:* I agree that the migration plan is one of the most critical parts of this KIP. With the current migration design, the key requirement is that retention.ms must not be shorter than the retention period of the longest remote-storage-enabled topic. As long as this requirement is satisfied, the migration does not introduce any additional data-loss scenario beyond the requirements of the existing remote storage implementation. I will make this requirement more explicit in the migration section and clarify the relevant edge cases. Let me know if you have more questions, thanks! Best, Lijun Tong
