Hi Ivan, Thanks for the KIP. The motivation of the KIP looks reasonable to me. It requires a way for RSM providers to add custom metadata with the existing RemoteLogSegmentMetadata. I remember we wanted to introduce a very similar change in the earlier proposals called RemoteLogMetadataContext. But we dropped that as we did not feel a strong need at that time and we wanted to revisit it if needed. But I see there is a clear need for this kind of custom metadata to keep with RemoteLogSegmentMetadata.
It is better to introduce a new class for this custom metadata in RemoteLogSegmentMetadata like below for any changes in the future. RemoteLogSegmentMetadata will have this as an optional value. public class RemoteLogSegmentMetadata { ... public static class CustomMetadata { private final byte[] value; ... } ... } This is completely opaque and it is the RSM implementation provider's responsibility in serializing and deserializing the bytes. We can introduce a property to guard the size with a configurable property with a default value to avoid any unwanted large size values. Thanks, Satish. On Tue, 30 May 2023 at 10:59, Ivan Yurchenko <ivan0yurche...@gmail.com> wrote: > > Hi all, > > I want to bring this to a conclusion (positive or negative), so if there > are no more questions in a couple of days, I'll put the KIP to the vote. > > Best, > Ivan > > > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko <ivan0yurche...@gmail.com> > wrote: > > > Hi Alexandre, > > > > > combining custom > > > metadata with rlmMetadata increases coupling between Kafka and the > > > plugin. > > > > This is true. However, (if I understand your concern correctly,) > > rlmMetadata in the current form may be independent from RSM plugins, but > > data they point to are accessible only via the particular plugin (the one > > that wrote the data or a compatible one). It seems, this coupling already > > exists, but it is implicit. To make my point more concrete, imagine an S3 > > RSM which maps RemoteLogSegmentMetadata objects to S3 object keys. This > > mapping logic is a part of the RSM plugin and without it the metadata is > > useless. I think it will not get worse if (to follow the example) the > > plugin makes the said S3 object keys explicit by adding them to the > > metadata. From the high level point of view, moving the custom metadata to > > a separate topic doesn't change the picture: it's still the plugin that > > binds the standard and custom metadata together. > > > > > > > For instance, the custom metadata may need to be modified > > > outside of Kafka, but the rlmMetadata would still be cached on brokers > > > independently of any update of custom metadata. Since both types of > > > metadata are authored by different systems, and are cached in > > > different layers, this may become a problem, or make plugin migration > > > more difficult. What do you think? > > > > This is indeed a problem. I think a solution to this would be to clearly > > state that metadata being modified outside of Kafka is out of scope and > > instruct the plugin authors that custom metadata could be provided only > > reactively from the copyLogSegmentData method and must remain immutable > > after that. Does it make sense? > > > > > > > Yes, you are right that the suggested alternative is to let the plugin > > > store its own metadata separately with a solution chosen by the admin > > > or plugin provider. For instance, it could be using a dedicated topic > > > if chosen to, or relying on an external key-value store. > > > > I see. Yes, this option always exists and doesn't even require a KIP. The > > biggest drawback I see is that a plugin will need to reimplement the > > consumer/producer + caching mechanics that will exist on the broker side > > for the standard remote metadata. I'd like to avoid this and this KIP is > > the best solution I see. > > > > Best, > > Ivan > > > > > > > > On Tue, 18 Apr 2023 at 13:02, Alexandre Dupriez < > > alexandre.dupr...@gmail.com> wrote: > > > >> Hi Ivan, > >> > >> Thanks for the follow-up. > >> > >> Yes, you are right that the suggested alternative is to let the plugin > >> store its own metadata separately with a solution chosen by the admin > >> or plugin provider. For instance, it could be using a dedicated topic > >> if chosen to, or relying on an external key-value store. > >> > >> I agree with you on the existing risks associated with running > >> third-party code inside Apache Kafka. That said, combining custom > >> metadata with rlmMetadata increases coupling between Kafka and the > >> plugin. For instance, the custom metadata may need to be modified > >> outside of Kafka, but the rlmMetadata would still be cached on brokers > >> independently of any update of custom metadata. Since both types of > >> metadata are authored by different systems, and are cached in > >> different layers, this may become a problem, or make plugin migration > >> more difficult. What do you think? > >> > >> I have a vague memory of this being discussed back when the tiered > >> storage KIP was started. Maybe Satish has more background on this. > >> > >> Thanks, > >> Alexandre > >> > >> Le lun. 17 avr. 2023 à 16:50, Ivan Yurchenko > >> <ivan0yurche...@gmail.com> a écrit : > >> > > >> > Hi Alexandre, > >> > > >> > Thank you for your feedback! > >> > > >> > > One question I would have is, what is the benefit of adding these > >> > > custom metadata in the rlmMetadata rather than letting the plugin > >> > > manage access and persistence to them? > >> > > >> > Could you please elaborate? Do I understand correctly that the idea is > >> that > >> > the plugin will have its own storage for those custom metadata, for > >> example > >> > a special topic? > >> > > >> > > It would be possible for a user > >> > > to use custom metadata large enough to adversely impact access to and > >> > > caching of the rlmMetadata by Kafka. > >> > > >> > Since the custom metadata is 100% under control of the RSM plugin, the > >> risk > >> > is as big as the risk of running a third-party code (i.e. the RSM > >> plugin). > >> > The cluster admin must make the decision if they trust it. > >> > To mitigate this risk and put it under control, the RSM plugin > >> > implementations could document what custom metadata they use and > >> estimate > >> > their size. > >> > > >> > Best, > >> > Ivan > >> > > >> > > >> > On Mon, 17 Apr 2023 at 18:14, Alexandre Dupriez < > >> alexandre.dupr...@gmail.com> > >> > wrote: > >> > > >> > > Hi Ivan, > >> > > > >> > > Thank you for the KIP. > >> > > > >> > > I think the KIP clearly explains the need for out-of-band metadata > >> > > authored and used by an implementation of the remote storage manager. > >> > > One question I would have is, what is the benefit of adding these > >> > > custom metadata in the rlmMetadata rather than letting the plugin > >> > > manage access and persistence to them? > >> > > > >> > > Maybe one disadvantage and potential risk with the approach proposed > >> > > in the KIP is that the rlmMetadata is not of a predefined, relatively > >> > > constant size (although corner cases with thousands of leader epochs > >> > > in the leader epoch map are possible). It would be possible for a user > >> > > to use custom metadata large enough to adversely impact access to and > >> > > caching of the rlmMetadata by Kafka. > >> > > > >> > > Thanks, > >> > > Alexandre > >> > > > >> > > Le jeu. 6 avr. 2023 à 16:03, hzh0425 <hzhka...@163.com> a écrit : > >> > > > > >> > > > I think it's a good idea as we may want to store remote segments in > >> > > different buckets > >> > > > > >> > > > > >> > > > > >> > > > | | > >> > > > hzhka...@163.com > >> > > > | > >> > > > | > >> > > > 邮箱:hzhka...@163.com > >> > > > | > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > ---- 回复的原邮件 ---- > >> > > > | 发件人 | Ivan Yurchenko<ivan0yurche...@gmail.com> | > >> > > > | 日期 | 2023年04月06日 22:37 | > >> > > > | 收件人 | dev@kafka.apache.org<dev@kafka.apache.org> | > >> > > > | 抄送至 | | > >> > > > | 主题 | [DISCUSS] KIP-917: Additional custom metadata for remote log > >> > > segment | > >> > > > Hello! > >> > > > > >> > > > I would like to start the discussion thread on KIP-917: Additional > >> custom > >> > > > metadata for remote log segment [1] > >> > > > This KIP is fairly small and proposes to add a new field to the > >> remote > >> > > > segment metadata. > >> > > > > >> > > > Thank you! > >> > > > > >> > > > Best, > >> > > > Ivan > >> > > > > >> > > > [1] > >> > > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment > >> > > > >> > >