Hello all,
Thanks for continuing the discussion! I have a few responses to your points.
Tom: You are correct in that this KIP has not mentioned the
DeleteTopicsRequest. I think that this would be out of scope for now, but
may be something worth adding in the future.
John: We did consider sequence ids, but there are a few reasons to favor
UUIDs. There are several cases where topics from different clusters may
interact now and in the future. For example, Mirror Maker 2 may benefit
from being able to detect when a cluster being mirrored is deleted and
recreated and globally unique identifiers would make resolving issues
easier than sequence IDs which may collide between clusters. KIP-405
(tiered storage) will also benefit from globally unique IDs as shared
buckets may be used between clusters.
Globally unique IDs would also make functionality like moving topics
between disparate clusters easier in the future, simplify any future
implementations of backups and restores, and more. In general, unique IDs
would ensure that the source cluster topics do not conflict with the
destination cluster topics.
If we were to use sequence ids, we would need sufficiently large cluster
ids to be stored with the topic identifiers or we run the risk of
collisions. This will give up any advantage in compactness that sequence
numbers may bring. Given these advantages I think it makes sense to use
UUIDs.
Gokul: This is an interesting idea, but this is a breaking change. Out of
scope for now, but maybe worth discussing in the future.
Hope this explains some of the decisions,
Justine
On Fri, Sep 11, 2020 at 8:27 AM Gokul Ramanan Subramanian <
gokul24...@gmail.com> wrote:
> Hi.
>
> Thanks for the KIP.
>
> Have you thought about whether it makes sense to support authorizing a
> principal for a topic ID rather than a topic name to achieve tighter
> security?
>
> Or is the topic ID fundamentally an internal detail similar to epochs used
> in a bunch of other places in Kafka?
>
> Thanks.
>
> On Fri, Sep 11, 2020 at 4:06 PM John Roesler <vvcep...@apache.org> wrote:
>
> > Hello Justine,
> >
> > Thanks for the KIP!
> >
> > I happen to have been confronted recently with the need to keep track of
> a
> > large number of topics as compactly as possible. I was going to come up
> > with some way to dictionary encode the topic names as integers, but this
> > seems much better!
> >
> > Apologies if this has been raised before, but I’m wondering about the
> > choice of UUID vs sequence number for the ids. Typically, I’ve seen UUIDs
> > in two situations:
> > 1. When processes need to generate non-colliding identifiers without
> > coordination.
> > 2. When the identifier needs to be “universally unique”; I.e., the
> > identifier needs to distinguish the entity from all other entities that
> > could ever exist. This is useful in cases where entities from all kinds
> of
> > systems get mixed together, such as when dumping logs from all processes
> in
> > a company into a common system.
> >
> > Maybe I’m being short-sighted, but it doesn’t seem like either really
> > applies here. It seems like the brokers could and would achieve consensus
> > when creating a topic anyway, which is all that’s required to generate
> > non-colliding sequence ids. For the second, as you mention, we could
> assign
> > a UUID to the cluster as a whole, which would render any resource scoped
> to
> > the broker universally unique as well.
> >
> > The reason I mention this is that, although a UUID is way more compact
> > than topic names, it’s still 16 bytes. In contrast, a 4-byte integer
> > sequence id would give us 4 billion unique topics per cluster, which
> seems
> > like enough ;)
> >
> > Considering the number of different times these topic identifiers are
> sent
> > over the wire or stored in memory, it seems like it might be worth the
> > additional 4x space savings.
> >
> > What do you think about this?
> >
> > Thanks,
> > John
> >
> > On Fri, Sep 11, 2020, at 03:20, Tom Bentley wrote:
> > > Hi Justine,
> > >
> > > This looks like a very welcome improvement. Thanks!
> > >
> > > Maybe I missed it, but the KIP doesn't seem to mention changing
> > > DeleteTopicsRequest to identify the topic using an id. Maybe that's out
> > of
> > > scope, but DeleteTopicsRequest is not listed among the Future Work APIs
> > > either.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > > On Thu, Sep 10, 2020 at 3:59 PM Satish Duggana <
> satish.dugg...@gmail.com
> > >
> > > wrote:
> > >
> > > > Thanks Lucas/Justine for the nice KIP.
> > > >
> > > > It has several benefits which also include simplifying the topic
> > > > deletion process by controller and logs cleanup by brokers in corner
> > > > cases.
> > > >
> > > > Best,
> > > > Satish.
> > > >
> > > > On Wed, Sep 9, 2020 at 10:07 PM Justine Olshan <jols...@confluent.io
> >
> > > > wrote:
> > > > >
> > > > > Hello all, it's been almost a year! I've made some changes to this
> > KIP
> > > > and hope to continue the discussion.
> > > > >
> > > > > One of the main changes I've added is now the metadata response
> will
> > > > include the topic ID (as Colin suggested). Clients can obtain the
> > topicID
> > > > of a given topic through a TopicDescription. The topicId will also be
> > > > included with the UpdateMetadata request.
> > > > >
> > > > > Let me know what you all think.
> > > > > Thank you,
> > > > > Justine
> > > > >
> > > > > On 2019/09/13 16:38:26, "Colin McCabe" <cmcc...@apache.org> wrote:
> > > > > > Hi Lucas,
> > > > > >
> > > > > > Thanks for tackling this. Topic IDs are a great idea, and this
> is
> > a
> > > > really good writeup.
> > > > > >
> > > > > > For /brokers/topics/[topic], the schema version should be bumped
> to
> > > > version 3, rather than 2. KIP-455 bumped the version of this znode
> to
> > 2
> > > > already :)
> > > > > >
> > > > > > Given that we're going to be seeing these things as strings as
> lot
> > (in
> > > > logs, in ZooKeeper, on the command-line, etc.), does it make sense to
> > use
> > > > base64 when converting them to strings?
> > > > > >
> > > > > > Here is an example of the hex representation:
> > > > > > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> > > > > >
> > > > > > And here is an example in base64.
> > > > > > b8tRS7h4TJ2Vt43Dp85v2A
> > > > > >
> > > > > > The base64 version saves 15 letters (to be fair, 4 of those were
> > > > dashes that we could have elided in the hex representation.)
> > > > > >
> > > > > > Another thing to consider is that we should specify that the
> > > > all-zeroes UUID is not a valid topic UUID. We can't use null for
> this
> > > > because we can't pass a null UUID over the RPC protocol (there is no
> > > > special pattern for null, nor do we want to waste space reserving
> such
> > a
> > > > pattern.)
> > > > > >
> > > > > > Maybe I missed it, but did you describe "migration of... existing
> > > > topic[s] without topic IDs" in detail in any section? It seems like
> > when
> > > > the new controller becomes active, it should just generate random
> > UUIDs for
> > > > these, and write the random UUIDs back to ZooKeeper. It would be
> good
> > to
> > > > spell that out. We should make it clear that this happens regardless
> > of
> > > > the inter-broker protocol version (it's a compatible change).
> > > > > >
> > > > > > "LeaderAndIsrRequests including an is_every_partition flag"
> seems a
> > > > bit wordy. Can we just call these "full LeaderAndIsrRequests"? Then
> > the
> > > > RPC field could be named "full". Also, it would probably be better
> > for the
> > > > RPC field to be an enum of { UNSPECIFIED, INCREMENTAL, FULL }, so
> that
> > we
> > > > can cleanly handle old versions (by treating them as UNSPECIFIED)
> > > > > >
> > > > > > In the LeaderAndIsrRequest section, you write "A final deletion
> > event
> > > > will be secheduled for X ms after the LeaderAndIsrRequest was first
> > > > received..." I guess the X was a placeholder that you intended to
> > replace
> > > > before posting? :) In any case, this seems like the kind of thing
> we'd
> > > > want a configuration for. Let's describe that configuration key
> > somewhere
> > > > in this KIP, including what its default value is.
> > > > > >
> > > > > > We should probably also log a bunch of messages at WARN level
> when
> > > > something is scheduled for deletion, as well. (Maybe this was
> > assumed, but
> > > > it would be good to mention it).
> > > > > >
> > > > > > I feel like there are a few sections that should be moved to
> > "rejected
> > > > alternatives." For example, in the DeleteTopics section, since we're
> > not
> > > > going to do option 1 or 2, these should be moved into "rejected
> > > > alternatives," rather than appearing inline. Another case is the
> > "Should
> > > > we remove topic name from the protocol where possible" section. This
> > is
> > > > clearly discussing a design alternative that we're not proposing to
> > > > implement: removing the topic name from those protocols.
> > > > > >
> > > > > > Is it really necessary to have a new /admin/delete_topics_by_id
> > path
> > > > in ZooKeeper? It seems like we don't really need this. Whenever
> > there is
> > > > a new controller, we'll send out full LeaderAndIsrRequests which will
> > > > trigger the stale topics to be cleaned up. The active controller
> will
> > > > also send the full LeaderAndIsrRequest to brokers that are just
> > starting
> > > > up. So we don't really need this kind of two-phase commit (send
> out
> > > > StopReplicasRequest, get ACKs from all nodes, commit by removing
> > > > /admin/delete_topics node) any more.
> > > > > >
> > > > > > You mention that FetchRequest will now include UUID to avoid
> issues
> > > > where requests are made to stale partitions. However, adding a UUID
> to
> > > > MetadataRequest is listed as future work, out of scope for this KIP.
> > How
> > > > will the client learn what the topic UUID is, if the metadata
> response
> > > > doesn't include that information? It seems like adding the UUID to
> > > > MetadataResponse would be an improvement here that might not be too
> > hard to
> > > > make.
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 9, 2019, at 17:48, Ryanne Dolan wrote:
> > > > > > > Lucas, this would be great. I've run into issues with topics
> > being
> > > > > > > resurrected accidentally, since a client cannot easily
> > distinguish
> > > > between
> > > > > > > a deleted topic and a new topic with the same name. I'd need
> the
> > ID
> > > > > > > accessible from the client to solve that issue, but this is a
> > good
> > > > first
> > > > > > > step.
> > > > > > >
> > > > > > > Ryanne
> > > > > > >
> > > > > > > On Wed, Sep 4, 2019 at 1:41 PM Lucas Bradstreet <
> > lu...@confluent.io>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I would like to kick off discussion of KIP-516, an
> > implementation
> > > > of topic
> > > > > > > > IDs for Kafka. Topic IDs aim to solve topic uniqueness
> > problems in
> > > > Kafka,
> > > > > > > > where referring to a topic by name alone is insufficient.
> Such
> > > > cases
> > > > > > > > include when a topic has been deleted and recreated with the
> > same
> > > > name.
> > > > > > > >
> > > > > > > > Unique identifiers will help simplify and improve Kafka's
> topic
> > > > deletion
> > > > > > > > process, as well as prevent cases where brokers may
> incorrectly
> > > > interact
> > > > > > > > with stale versions of topics.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> > > > > > > >
> > > > > > > > Looking forward to your thoughts.
> > > > > > > >
> > > > > > > > Lucas
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
